The problem is it really depends who you ask. On road.cc, a fairly reputable bike magazine/website, you can find this: link

1. Do you need to change your helmet after a certain amount of time, even if you haven't had a crash?

Yes, but that's not necessarily because the expanded polystyrene foam degrades over time, as April Beard, Bontrager's (link is external) product manager for helmets, explains: "It really depends on how good of care you take of the helmet. There is no evidence that the EPS liner will deteriorate from age. Still, there are things such as solvents, chemicals and environmental exposure that can degrade the performance of the helmet."

Paul Caswell, the senior brand manager for Giro and Bell helmets' distributor ZyroFisher agrees: "We normally recommend a new lid every three years depending on usage as even with no impacts there is a constant knocking and pressing of the EPS as the helmet is stored, dropped, placed on hard surfaces etc. Due to tiny impact after tiny impact over time, the EPS will gradually lose its volume making it less able to deal with the energy in the unfortunate event of an impact. Of course, the more the helmet is used, the more it will deteriorate, so one rule for all does not work here, but three years is a good guide."

Helmets.org, a non-profit consumer-funded program providing information about bicycle helmets, largely backs up the manufacturer's stance. They cite data from an MEA Forensic study in 2015 that found that the foam liners of used but not crashed helmets retained their performance over many years, with some of the helmets tested being 26 years old.

When you actually look on helmets.org I am not sure what part of the manufacturer’s stance they imply that this program backs. link

Personally, I like to replace it every 4 years because I use mine a lot and I would rather fall for a selling trick than risk a higher chance of brain injury. In any case, a system like MIPS or wavecel would, in my opinion, justify buying a new helmet anyway if you don’t have that on your old one.

At a guess, it could be because the materials just naturally degrade over time. Any foam will start disintegrating, sun exposure could weaken the plastic, and the straps will fray.

I ride a motorcycle as my commuting vehicle and I find my helmet needs replacing every 5 years max, as the inner foam lining has compressed to point where it is no longer snug to my face and therefore unsafe.
Push bike helmets would be made of similar materials, just with less coverage.
So as @wander1236@sh.itjust.works says, it would be materials degradation.

I'm on a long-term hiatus from motorcycles at the moment after getting sideswiped by an SUV a few years ago. I was traveling at about 40MPH and had a "high-side" get-off, landed on my right side and face, then had my 300lb motorcycle strike me as I was sliding.

Sounds likely to result in serious injury that requires physical therapy to relearn walking and possibly some reconstructive surgery, right? Well, thanks modern protective gear, I still have a face, and the only injuries that I had were a minor and rather rare internal injury akin to a bruise that required no medical intervention, a sprained ankle, and an abrasion about the size of a nickel on my arm that was treated with some ointment and a bandaid (at a criminally inflated price).

Modern helmets and protective gear are incredible. I provided my experience in the hope that it gives you more hope. They make the stuff for skateboarding and bicycling the same way, for the most part (heavy, kevlar mesh is a bit impractical and unnecessary at the lower speeds). It can keep you safe, as long as you use the safety gear between your ears.