We have dual redundant links via separate physical hardware from our side to our Tier 1 ISP. We unexpectedly discovered their equipment is a single point failure.

Our Tier 1 ISP has just confirmed hardware failure of the upstream router (after 16 hours!) and will provision new replacement device once it has been shipped to site.

I wonder why the OSM team isn't calling the ISP out by name. Not having on-site spares at a Tier 1 is just, insane.

TL;DR

The OSM team is very small and constrained heavily by financials. They had a single ISP which acted as a single point of failure. They are moving to a new ISP but they expect the outage to last until Wednesday

Updates: It should be operational as of tonight. OSM are switching to their new ISP.

https://community.openstreetmap.org/t/openstreetmap-org-currently-offline-operations-team-are-working-to-restore-15-december-2024-updated/122930/53

You heard it here first…

We should have all service back by the European evening tomorrow (Tuesday). We are switching to the new ISP.

(Minor matter of me having to get new ISP setup working on our routers with some virtual networking required for simultaneous upstream ISPs, and some special VRRP setup required for the new ISP redundancy setup)

• Firefishy, OpenStreetMap Community