Find the Right GRC Solution

Search and compare GRC technology built for the problems you’re trying to solve

Three Outages, Three Failure Modes, One Week

Three Outages, Three Failure Modes, One Week

By
Key Takeaways
  • Microsoft Traces Outage to a July Update: Its preliminary report says a defect in a Windows update to Microsoft 365 networking gateway servers went undetected for more than three months before combining with a Teams traffic failover and reduced server capacity to disrupt services for nearly 24 hours.
  • Automated Recovery Spread the Failure: After more than 80% of gateway servers stopped responding, automated traffic management shifted requests onto the remaining servers until they overloaded, and regional containment safeguards did not stop the outage from spreading globally.
  • FAA Findings Point to Upgrade Work: Preliminary findings tie Friday's nationwide ground stop to excess data from a router in El Dorado, Ark., and a radar site in Amarillo, Texas, during contractor work to replace copper lines with fiber. Officials say there was no total loss of communications or radar, and the cause remains under investigation.
  • Bank Disruptions Remain Unexplained: Chase attributed Monday's problems to a technical issue and said customers kept access to their funds. No bank has disclosed a common cause or linked the disruptions to the Microsoft or FAA outages.
  • Outages Do Not Automatically Trigger Reporting: Under DORA and the U.S. 36-hour notification rule, reporting obligations depend on classification criteria and materiality thresholds, not on whether an attacker was involved.
Deep Dive

Microsoft said a defect introduced by a Windows update to its own servers in July went undetected for more than three months before it set off an outage that disrupted Microsoft 365 services worldwide for nearly 24 hours last week. The company's preliminary post-incident report, dated Nov. 15 and summarized by Wccftech and Petri, traces the Nov. 12 failure to that defect, combined with a routine shift of Teams traffic and server capacity already reduced by an unrelated problem.

The report came during a week of high-profile disruptions with no established connection to one another. On Friday, the Federal Aviation Administration halted departures nationwide for more than four hours after what officials, citing preliminary findings, described as a router flooding its network during work to replace aging copper telecommunications lines. On Monday, customers of JPMorgan Chase, Bank of America, Wells Fargo and U.S. Bank reported trouble logging in, checking balances and moving money, and Chase attributed its problems to a technical issue. None of the organizations has reported evidence of malicious activity, and none has linked its disruption to the others.

Microsoft

The outage began at 16:20 UTC on Nov. 12, according to the report. Exchange Online was hit hardest, leaving users unable to reliably reach their mailboxes or send and receive email, and features in Teams, Copilot, SharePoint and OneDrive that depend on Exchange Online failed with it. Microsoft tracked the event as incidents EX1550543 and MO1550821. The report puts the end of customer impact at 16:10 UTC on Nov. 13, although Microsoft continued posting service-health updates into Nov. 14.

The report identifies three contributing factors. The first was a defect in a Windows update deployed in July to the networking gateway servers that carry Microsoft 365 traffic. Under certain workloads, an internal Windows process encountered lock contention and consumed CPU and memory until the servers stopped responding. Testing did not catch the problem because it ran on less powerful hardware with fewer CPU cores. Microsoft says the defect affects only that gateway infrastructure, not Windows deployments generally.

The second factor was a shift in traffic. About 15 hours before the outage, Teams traffic was automatically moved from an alternative gateway to the primary gateway that also serves Exchange Online. Microsoft describes such failovers as routine, but this one added load. The third was capacity. Roughly 14% of the gateway's servers were already offline because of an unrelated issue the day before.

Within two hours, more than 80% of the gateway servers had stopped responding. Automated traffic management redirected requests to the servers still running, which then became overloaded themselves, and regional containment safeguards did not keep the disruption from spreading globally. Because the defective change had been in place for months, rolling it back did not resolve the problem. Engineers instead deployed a fix that returned the affected connection-handling code to its earlier working state.

Microsoft says it has since added safeguards on how traffic is distributed, expanded monitoring and increased server capacity, and that email queued during the outage has been delivered. Early on Nov. 13, the company said in a status update that it was "not aware of any malicious activity that is causing this incident," according to a compilation of its statements. Its final post-incident report is due Nov. 20.

The FAA Ground Stop

Air traffic controllers began seeing inconsistencies in radar data and in communications with aircraft at 3:36 p.m. Eastern time on Friday, Transportation Secretary Sean Duffy said, and the FAA ordered a nationwide ground stop for departing flights at 3:54 p.m., NBC News reported. Aircraft already in the air were allowed to land. Duffy said airspace reopened at 8:15 p.m. Al Jazeera described the halt as the longest U.S. ground stop in 25 years and, citing Flightradar24, reported more than 4,700 cancellations and 10,000 delays on Friday.

The FAA attributed the stop to a telecommunications outage affecting its air route traffic control centers, which handle flights traveling across the country. Officials have described network instability rather than a blackout. Duffy said switches and routers were overloaded with data and went down, but that there was no total loss of communications or radar. Officials later acknowledged that new technology added to the system had overloaded older components, causing intermittent gaps in communications and radar data, according to Al Jazeera.

The work under way at the time was part of the agency's modernization program. Duffy said a contractor "was doing routine maintenance to switch telecommunication lines from old copper wire" to fiber at at least one remote FAA location. The contractor, L3Harris, is the FAA's longtime telecommunications provider. During the conversion, a router began sending more than 1,000 connection attempts every four to five seconds across the FAA network, Scripps News reported.

On Sunday, the agency said preliminary findings pointed to a router at a radio facility in El Dorado, Ark., and a radar site in Amarillo, Texas, as sources of the excess data, The Hill reported. Duffy told ABC's "This Week" that the Arkansas router was at an L3Harris lab in Florida. Investigators have not said whether human error, an installation mistake or faulty equipment was responsible. "We see zero evidence of anything nefarious and zero evidence of anything AI-related," FAA Administrator Bryan Bedford said. L3Harris said it would take part in the Transportation Department's review and that, with its telecommunications partners and the FAA, it has completed more than 70% of the planned copper-to-fiber conversion.

The disruption differs from Microsoft's in where it began. Microsoft's outage originated in a commercial cloud platform shared by customers around the world. On the preliminary account, the FAA's originated in upgrade work on the agency's own network, carried out by its contractor. It also adds to warnings that predate it. A Government Accountability Office report in September cautioned about aging copper lines, slow modernization and poorly scheduled changes that overwhelmed the system, according to Al Jazeera. Duffy has called for $50 billion in air traffic control upgrades.

Online Banking

Bank customers began reporting problems Monday morning. By 1:50 p.m. Eastern time, Downdetector, which tracks user-submitted outage reports, showed more than 4,600 reports for Chase, more than 1,900 for Bank of America, more than 900 for U.S. Bank and about 800 for Wells Fargo, according to Yahoo Finance.

"Due to a technical issue, some transactions are taking longer than usual to complete but customers still have access to their accounts and funds," Chase said in a statement. A person familiar with the matter told NBC News the issue was internal to Chase, and NBC New York, citing a source not authorized to speak publicly, reported that it was a technical bug rather than a cyber issue. The bank said the disruption spread from its website and app to some branches and to outside apps that connect to Chase accounts, Fortune reported.

Wells Fargo said it was working to resolve a technical issue, and Zelle said its payments network was not down but that problems at financial institutions could affect its services, ABC7 reported. As of Monday afternoon, no bank had tied the problems to the Microsoft outage, and it was not clear whether the disruptions at different banks were connected, Fast Company reported. Downdetector counts reflect what users report, not confirmed system failures, and some complaints about one institution may stem from problems at another or at a shared payment service.

The Regulatory Picture

Microsoft is among 19 technology companies that the European Supervisory Authorities designated in November 2025 as critical ICT third-party providers under the Digital Operational Resilience Act, placing them under direct EU-level oversight. That oversight is aimed at the providers. It does not shift responsibility away from the banks, insurers and investment firms that rely on them, which remain fully accountable under DORA for their own ICT risk management and third-party risk. An outage at a provider is not automatically a reportable event for those firms. Whether a disruption amounts to a major ICT-related incident for a given firm depends on DORA's classification criteria, which take into account factors such as the number of clients affected, how long services were down and how critical those services are.

In the United States, a 2021 rule issued jointly by the Office of the Comptroller of the Currency, the Federal Reserve and the Federal Deposit Insurance Corp. requires a banking organization to notify its primary federal regulator as soon as possible, and no later than 36 hours, after it determines that a "notification incident" has occurred. The threshold has two parts. A computer-security incident is an occurrence that results in actual harm to the confidentiality, integrity or availability of an information system or the information it holds, a definition that does not require an attacker. It rises to a notification incident only if it has materially disrupted or degraded, or is reasonably likely to materially disrupt or degrade, a bank's ability to serve a material portion of its customers, business lines whose failure would cause material loss, or operations whose failure would threaten U.S. financial stability. Each bank makes that determination itself, and none of the banks affected Monday has said publicly whether it notified regulators.

The rule imposes a separate obligation on bank service providers. A provider must notify each affected banking organization customer as soon as possible after determining that a computer-security incident has materially disrupted or degraded, or is reasonably likely to materially disrupt or degrade, covered services for four or more hours.

Where the Investigations Stand

Microsoft's final post-incident report is due Nov. 20. The Transportation Department's review of the ground stop is continuing, and investigators have yet to say whether human error, an installation mistake or faulty equipment caused the router to flood the network. Duffy has said he expects normal operations by the Thanksgiving travel peak, which begins in less than two weeks.

As of Monday afternoon, Chase had not explained the technical issue behind its disruption, Wells Fargo had described its problem only as a technical issue, and Bank of America and U.S. Bank had not publicly given a cause. Officials have also not addressed a question raised by Ron Bartsch, an aviation law professor, about why redundant backup systems did not take over immediately when the FAA network began to fail.

 The GRC Report is your premier destination for the latest in governance, risk, and compliance news. As your reliable source for comprehensive coverage, we ensure you stay informed and ready to navigate the dynamic landscape of GRC. Beyond being a news source, the GRC Report represents a thriving community of professionals who, like you, are dedicated to GRC excellence. Explore our insightful articles and breaking news, and actively participate in the conversation to enhance your GRC journey.

Oops! Something went wrong