From what I've come across, it's from a combination of their secrecy (historically to the point of death, read about ~~Hiram Abiff~~ William Morgan who was mobbed to death by Freemasons just North of where I used to live), their links to the upper class, their place in the spiritual sphere (they have Anglican/Templar associations, which is why the pope forbids joining, and these put their links to the British crown into perspective, as well as the fact they have their very own equivalent to the Vatican Secret Archives, which is a common theme, with the more gender-inclusive and Knights-Hospitaller-sprung Sovereign Military Order of Malta being their strictest rivals), their feud with what has come to be known as the LDS church (Joseph Smith was said to have been a Freemason who took off with their secret "ideas" to make the Book of Mormon), the fact they have historically looked down on those who leave or operate from other societies such as the Oddfellows, and some of their practices, such as the fact they used to be unwilling to testify against each other in court (I don't know if this is still true, but to put that into perspective, the United States recently reprimanded Scientology for the same reason), how "expensive" it is to actually be a member, their overlapping with what would today be called Gnosticism (oddly the G symbol does not stand for Gnosticism, though one cannot deny what comes across as some very sectarian observations/tendencies), and how it's 2025 and they still don't allow women to join (they also used to not allow people of color to join either, up until recently, and they still require someone to have a spiritual upbringing), which is why I am not one (I could join the Eastern Star, but it's almost knock-off-esque compared to the actual thing, which actually used to frown upon the Eastern Star as "missing the point", plus they wouldn't take kindly to my upbringing since my details would fall outside their range of knowledge).

In a way, it's comparable to how we might critique a British megachurch, if that megachurch was formatted like a university fraternity club. I had known many Freemasons, which is the norm where I used to live because there is a high enough Masonic presence in the area that they built the streets (arranging the sidewalk in a literal square and compass design), with family members of my friends participating in the group. I have nothing against them on their own, but with their sense of superiority and duty (especially with foreign entities involved) that often gets stereotypically mixed in with their demeanor, they can be as overbearing as sand here (coarse and rough and irritating and getting everywhere), which for a long time has not just led me to speculate myself but also forced my hand in a way. When you combine an obsolete sense of self with extreme exclusivity, well, there you go.