AdvertiseDonate
Headlines Times · 3597 East Monarch Sky Lane, Suite 240, Meridian, ID 83646

How a Dormant Bug Triggered Microsoft's Massive Outlook Outage

A latent defect in Microsoft 365 networking gateway servers, combined with a Teams failover and reduced capacity, knocked out Outlook email for millions of users for nearly 24 hours last week.

By Bianca Shevchenko ·

Illustration for: How a Dormant Bug Triggered Microsoft's Massive Outlook Outage
Illustration generated for this article, not a news photograph.

What began as a normal Thursday morning on Microsoft's campus last week quickly turned into one of the worst outages in the company's history. Within about an hour of the workday starting, Outlook email became unavailable to employees, businesses, and numerous governments, and it stayed down for nearly 24 hours.

The failure began in Switzerland, according to a source familiar with the incident, when Exchange Online servers powering Outlook started going offline. The disruption spread to Mexico and then worldwide as Microsoft's failover systems attempted to shift email traffic elsewhere. Within a couple of hours, more than 80 percent of Exchange servers were unresponsive. Microsoft classified the incident as "sev0" internally, its most severe category, and hundreds of engineers were mobilized to find the cause.

In the early hours, engineers pored over crash reports, trace logs, and kernel dumps. Some servers had slipped into maintenance mode and stopped responding under unusually high CPU loads. Attempts to restart them failed because the machines were almost immediately flooded by waiting traffic. Because Exchange servers boot over a network using the Preboot Execution Environment, the surge of rebooting machines overwhelmed the monitoring systems, creating a reboot loop in which servers repeatedly tried and failed to come back online.

As speculation grew outside the company about a possible cyberattack or sloppy AI-generated code, engineers investigated potential attack vectors and consulted DDoS specialists. Crash reports and logs quickly ruled out an attack. After rollbacks failed, the team examined the networking stack and found an unusual bug that had been sitting unnoticed on Microsoft 365 networking gateway servers since July.

"A latent defect caused servers to become overwhelmed and unresponsive under certain load conditions," wrote Charles Lamanna, head of Microsoft's Copilot, Agents, and Platform team, in a post on X. The company's preliminary post-incident report describes a flaw affecting select Windows servers "due to lock contention inside an internal Windows process." The bug does not affect regular versions of Windows, since it stemmed from a fix built specifically for the gateway servers. Lamanna acknowledged testing gaps, writing that "the contention issue is worsened on machines with more cores and newer hardware; whereas our load testing tended to focus on weaker hardware with fewer cores."

Three factors combined to awaken the dormant defect. Fifteen hours before the outage, Microsoft Teams failed over from an alternative networking gateway to the main gateway supporting Exchange Online, which Lamanna said "increased the load on the networking gateway and changed its traffic patterns." At the same time, roughly 14 percent of server capacity for that gateway was out of service because of an unrelated issue from the previous day. Together, the bug, the failover, and the reduced capacity triggered the collapse.

Restoring service proved difficult. "Recovery took longer than usual because as servers returned to service, they were immediately overwhelmed by waiting traffic and became unresponsive again," Lamanna wrote, adding that automated recovery processes were not designed for such a broadly unresponsive state, and that the latent bug meant normal rollback tools, flighting, and feature flags could not quickly revert the change. A running bridge call for engineers at one point hit its maximum of 1,000 participants, and overlapping interventions from multiple teams at times complicated coordination. It took nearly 24 hours to stabilize the Exchange servers.

Microsoft plans to publish a final post-incident review later this week. The episode makes clear the company will need to invest more heavily in its testing processes so that a similar sequence of events cannot knock out Outlook for millions of people again.

More News · National news · Report a business · Got a Story?

Get Headlines Times in your inboxReporting on the stories people talk about. Free, a few times a week, unsubscribe any time.

Related stories

Get Headlines Times in your inboxReporting on the stories people talk about. Free, a few times a week, unsubscribe any time.