If you really don’t want to explain why you are down-voting, I really don’t think people should be down-voting.

there are many times that you can down-vote without a requirement of explaining. Sometimes your point has already been made by another person, other times it's just a really bad take or the person is so dead-set that honestly you couldn't change the persons mind even if you explained it. Sometimes the comment is just hostile to the current situation or the OP, sometimes the comment is just super off-topic. Some situations allow for down-votes without explaining it.

I personally down-vote for off-topic and harassing posts as it helps the system sort what is considered helpful to the discussion. I would refuse to down-vote for harassing and off-topic if this system is in place, as it creates an attack vector for the person to come after me, a situation that would require either blocking them or bugging a mod for, which is something that personally I just don't want to deal with in my life so I would simply just not participate in the vote.

Not sure where you are pulling the “new users get filtered out as untrustworthy”,

The type of system proposed inherently causes it as a side effect. When you have a system that is crowdsourced from the popular opinion, you create an echo chamber that only shows content from sources that have been deemed as appropriate, as such not only do you lose the arguing side, you also lose content from people who are not established/just starting out as they are not profiled as that side. as for examples? Two examples of sites that use that style system include Stackoverflow, which uses a rep system to decide how much access you can get into, and some of the larger reddit sites which went off the karma system to even allow posting in them. There are also other examples in reddit, but the karma block system was the most predominant (followed by sub rule filters which filtered out based off bias).

I do believe that a karma system is best type of system however I believe that the metric should be hidden from sight. This will allow for helpful comments to rise to the top, but will remove the hard focus "score" ideology that everyone has. In this system you wouldn't know if you were down-voted in the first place, which means you wouldn't be aware of someone maliciously down-voting you, and it would also do what you want where it would force someone if they had a super big issue with what was posted to actually comment on it. That being said, this system can not exist in a federated environment so therefore the next best thing is either anonymous (to all but mods/admins due to moderation and federation control reasons) or just not having the system as a whole.