For me, personally, it’s more about the focus of votes vs actual discussion. I’m worried it would turn the tides and make people much more focused on the votes than actual discussion. It might make it echo chamber-ish.

I recently disabled showing votes on my side (through Voyager app). Even if I get downvoted a hundred times over, if I just get a respectful discussion with links to (trustable) sources. I’m all alright with that.