Spores are everywhere like you say and you only really see a tiny percentage of mycelium. Fungi kinda is everywhere already, but where it can grow well is much more limited.

Fungi can be remarkably picky about its growing conditions to thrive, otherwise, it's growth will be remarkably slow. However, if you put a tablespoon of dirt under the microscope, there could be dozens of mycelial strands in it trying to survive. They can all survive, to a degree, but there are a couple of issues preventing dominance.

If it can find a place to settle in and grow, chances are that many other spores may be trying to take hold as well. Fungi is insanely competitive and is constantly fighting for space. Fast growing fungi is what we normally see take over food sources and it's usually a type of trichoderma. Trichoderma will literally choke out other fungal growths simply due to its rapid development. If an existing colony is weakened for one reason or another and it gets a trich infection, it's game over.

For commercial mycelium development, (button mushrooms, oysters, etc.) growing conditions are generally perfect and the substrate used is tailored specifically per species. (It's mostly sanitized poo or specific types of wood.) Temperatures need to be adjusted for each growth phase as fungi can be very sensitive to that. Some strains of shiitake are rumored to require a physical shock to fruit. (Like, the substrate bag needs to be physically smacked hard. It's an odd characteristic.)

To sum all of this up, it usually comes down to competition. Where there isn't fungi, there is bacteria. Plants even have chemical defenses to both. Small critters and insects may eat all three of those things.

Next time you look at your garden, just remember you are looking at an actual battleground for millions of critters of all shapes and sizes.