The referer header tells the site which specific users and which specific clicks came from lemmy world. That's flat-out invasive. Revealing the number of users (as Mozilla wants to do) is also invasive even if it doesn't single out the user (of course that's much less direct and people usually tolerate it until they become attuned to the issue).

The thing to ask yourself when site X wants information Y is "what does X want to do with the information?". If the answer can possibly be "something bad", then X should not get the information unless the user opts into sending it. That is even if it's statistical or aggregated information. Being included in the count is like casting a vote for X, which (as we see with Trump getting elected) can have significant effects even with no identification of the individual voters.