The more time goes by, the more convinced I become that centralizing power never solves the problem it promises to solve. It just relocates the problem to one spot and raises the stakes. This week proved it not once but twice. Microsoft routed Outlook, Teams, Copilot, and OneDrive through the same shared plumbing, and one botched network change took all of them down at once. The FAA ran every en route center in the country over one telecom backbone, and one overloaded batch of routers grounded the entire nation for more than four hours. In both cases, the people in charge sold centralization as efficiency. In both cases, efficiency meant that when something broke, everything broke.
The same logic applies well beyond servers and radar screens. Every time Washington pulls another decision away from states, towns, and individuals, it builds another single point of failure, and the people who run it never have to live with the outage. A bad call in one state hurts one state, and the other 49 can watch, learn, and do better. A bad call in Washington hits all 50 at once, with no neighbor left to pick up the slack. The Founders understood that instinctively. They split power between the branches and between the states because they knew no one, however smart or well-meaning, gets it right every time. They designed for failure. Today’s planners, in Redmond and in Washington, keep designing for perfection, and assuming it, and then act shocked when perfection doesn’t show up.
This is pretty much a follow-on to my piece on the Microsoft 365 meltdown. If you've not seen it, you may want to start there. I see Rick Moran has also leaned into the FAA aspect of this, and Matt Margolis did, as well. My thrust, however, both yesterday and today, is the issue, the frailty, of centralized anything.
Yesterday, while I was writing about the Microsoft meltdown, I made a brief mention of the FAA and its announcement that it would finally phase floppy disks out of air traffic control by 2028. I never dreamed that less than 24 hours after I filed that piece, the FAA would ground every departing flight in the country over what it called "communications issues." One gets the impression that our ATC is run by stone knives and bear skins.
Here it is, gang: Two of the biggest systems Americans depend on, one private and one government, fell over in the same week, and for the same reason. Somebody changed a shared network, and everything hanging off that network went down with it. That isn't bad luck; that's the bill for centralization coming due.
Funny, isn't it, how the FAA systems failed so quickly after I mentioned its push to centralize things. Oh, I'm not saying there's a connection between my mentioning the obvious and further breakdowns happening, so don't get it twisted. But if little old me could see the problem even before it happened, why didn't they?
Related: Microsoft 365: a Living Example of the Centralization Two Step
What happened Friday
At 3:54 p.m. Eastern, the FAA ordered a nationwide ground stop after a communications problem hit every Air Route Traffic Control Center in the country. It was the first nationwide halt since January 2023 and only the second since 9/11. Departures didn't resume until about 8:15 p.m. By then, airlines had canceled more than 4,700 flights and delayed 10,000, with about 700 Saturday cancellations already posted.
Transportation Secretary Sean Duffy blamed switches and routers that overloaded and shut down once they started spitting out inconsistent data. He called the whole setup "an incredibly old air-traffic control system" running on what he termed "analog telecom."
Then came the more interesting admission. Officials conceded that new technology added to the system overloaded older components, which caused "intermittent" radar and communication gaps. Homeland Security Secretary Markwayne Mullin said investigators are looking into incompatibilities between the old gear and the new equipment going in.
Translation: The upgrade broke the thing it was supposed to fix.
Because the failure hit every center at once, the usual safety valve disappeared. When one facility goes dark, its neighbors normally pick up the slack. On Friday, there were no neighbors left to call.
On paper, the FAA's control structure looks spread out: 21 en route centers, scores of approach facilities, and hundreds of towers. Underneath all of them runs one backbone, the FAA Telecommunications Infrastructure network. Contractor L3Harris operates it and calls it "the communications backbone serving air traffic facilities." On top of it all sits the Air Traffic Control System Command Center in Warrenton, Va., which balances traffic for the entire country and sends its stop orders to every tower at once.
Citing people briefed on the matter, The Wall Street Journal reported that L3Harris oversaw the affected network, and CBS sources tied the outage to a network update and reboot. Nobody has confirmed either report, and the FAA hasn't assigned blame. Still, look at the shape of the thing: one change, one network, every center.
We've seen this movie before, just on smaller screens:
- Sept. 21: The FAA had already moved Newark's airspace to Philadelphia. That day, Philadelphia's primary circuit failed, and construction crews severed the backup fiber. Ground stops rolled out from Philadelphia and Newark to JFK, LaGuardia, and Westchester.
- March 13: An overheated circuit board at Potomac TRACON, which shares a site with the national Command Center in Warrenton, grounded every major airport around Washington plus Richmond.
- Jan. 11, 2023: Contract workers deleted files while syncing the NOTAM system's primary and backup databases and stopped the whole country for 90 minutes. The backup went down with the primary because somebody had tied them together.
Microsoft runs the same script
Microsoft's outage began around 1 p.m. Eastern on Nov. 12 and knocked out Outlook, Teams, Copilot, and OneDrive worldwide. About a day later, Microsoft apologized. It blamed a problem introduced during an earlier network capacity upgrade that surfaced after a recent network change and called it "a technical issue not related to cyber or AI / agents." The company promises a full post-incident report.
By the way, there doesn't appear to be a common cause between these two failures, past a failure of design involving the concept of centralization and lack of redundancy.
Redmond has played this tune before. In July, it blamed a networking change and rolled it back. In January, it pointed to heavy load during a maintenance window plus a third-party networking provider's performance problems. The details change, but the pattern doesn't. Every time, one change to shared plumbing takes out a dozen services that sell themselves as separate products. As Jimmy Doohan (Scotty from Star Trek) once said, "The more complicated the plumbing, the easier it is to stop up the drain".
As I alluded to the other day, FAA Administrator Bryan Bedford made the case for ditching floppy disks by describing each facility as its own "silo" that can't talk to the others. The new system, he says, will let controllers share information instantly.
Fine. But a silo has one great virtue: When it burns, the one next door doesn't. I spent enough years in network administration to know that every link you add between two boxes is also a path for failure to travel. Anyone who has dealt with, say, a virus in a corporate environment knows this as a matter of instinct. The FAA's new backbone aims to tie together roughly 4,500 facilities, and the administration's overhaul calls for six new control centers. The net effect is to trade many small, local failures for fewer, bigger ones.
The Government Accountability Office saw this coming. In its September 14 report, the watchdog warned that "the FAA is buying delivery speed at the risk of cost uncertainty and potential disruptions." It found 11,389 project schedules that nobody had stitched into one master plan and a telecom price tag that jumped from $4.75 billion to $5.91 billion. Friday's meltdown looks a lot like the "potential disruptions" the GAO had in mind. Government efficiency is, of course, a massive oxymoron.
Indeed, I'm not saying any of this is an argument for keeping floppy disks and paper strips. The old system has to go. But Duffy now wants $50 billion, and the White House asked for another $30.1 billion for aviation infrastructure just hours before the ground stop. More money poured into a more centralized network without truly independent backups just buys a bigger, more expensive, and possibly deadly single point of failure.
Conservatives have understood this principle in politics for a long time. Concentrate everything in one place, and one mistake there becomes everybody's problem. The Founders spread power around for the same reason good engineers build redundancy. Friday afternoon, about 11,000 flights carrying more than 1.5 million passengers were still waiting to take off when the system went down. Redundancy isn't waste. It's what keeps one router from stranding a nation.
