Maybe it's just me, but this feels like a double standard.
Let's say I create a community that allows male and female content. But a lot of users join the community who dislike male content and consistently downvote it as a way to discourage it, then those downvoters are the ones trying to change the community. To me, it is the downvoters who are trying to, in a sense, hijack the community and make it something it wasn't intended to be.
I'd break out the SPF 1000 and get to work. It might take a while.