You may think this means that only 5% of people, 1 in 20, will be able to have kids. But actually, it takes two, and if there's no way to predict who can and cannot, it becomes closer to 1 in 400. The 1/20 chance of the male partner being fertile is multiplied by the 1/20 chance of the female partner being fertile.

It's difficult to prove someone infertile, but if someone can conceive a child, that proves their fertility. This is of limited utility in the case of women, but I suppose a man of proven fertility could make a living as a stud, attempting to knock up eligible women. So once you've identified a population of fertile males, you knock that half of the equation back down to a 1/1 and the women can go back to the much better 1/20 odds. Of course, you'd need to re-identify potential studs over time from the newer generations as older ones die off. And unless each woman is having twenty daughters you're still suffering from rapid population decline and the attendant societal collapse.

So, in summary: It'd be real bad, and even if it didn't kill off humanity in a generation it probably would knock us back to the bronze age within a century. And even if we somehow manage to dodge that it will still change human society permanently in unpredictable ways.