how can you make non-arbitrary distinctions between the two and who should have the authority to decide?

The core of the problem, I think, is that there is no non-arbitrary distinction for every case and no one should have absolute authority about these decisions. Cue decentralised communities. The discussions around these topics are messy and exhausting, but I think that’s as good as it gets. Communicating and changing again and again where this arbitrary lines get drawn is what people have to do, even when that doesn’t feel exactly satisfying.

For the suffrage example, if you look up what the counter-arguments where, it’s actually that they believed women weren’t capable of voting because they supposedly didn’t have the time or mental capacity to think about politics. Or that women “don’t want the vote”. I wouldn’t call that feeling threatened in their freedom, they felt threatened in their worldview.

I do think a lot of people do not see this distinction (is my freedom threatened or my worldview?) because they happen to not really fear that their rights might actually be stripped away someday. For someone who struggles to empathise with this fear it might seem overly dramatic how other people react to them just asking questions.