MGB
MGB // Cyber Blog
← All posts

Office 3654 outage

For about 24 hours on 12 and 13 November, most people could not get to their Microsoft 365 mailbox, and mail was delayed. It wasn't an attack, so strictly it isn't a cyber story. But availability is one third of security, and this one is worth understanding.

TL;DR
  • What happened: a networking gateway that sits in front of Exchange Online and other Microsoft 365 services was overwhelmed worldwide, starting around 16:20 GMT on Thursday 12 November. Customers were no longer actively affected by 16:10 GMT on Friday 13 November.
  • Why: Microsoft's preliminary report names three things that combined: a latent bug, extra load from a Teams failover, and some capacity already out of service.
  • Was it an attack? Microsoft's report doesn't say so. It describes a software defect and a capacity cascade.
  • What it means for you: assume your email can disappear for a day. Plan how you talk to staff, how you receive security alerts, and how you avoid making risky changes in a rush.
~24hStart to recovery
3Contributing causes
14%Capacity already out
80%+Servers unresponsive at worst

What people saw

Outlook and other mail clients could not connect to Exchange Online for most users, and email delivery was delayed for many. Microsoft's report says parts of Copilot, Teams, SharePoint, OneDrive, Defender and Purview that depend on Exchange Online were affected as well, though most of those stayed online. How bad it was depended on the region, how you connect, and the time of day.

What went wrong

Microsoft says a global outage of the networking gateway in front of many Microsoft 365 services came from three factors lining up:

  1. A latent bug. A lock contention problem in an internal Windows process could use up server CPU and memory under certain loads. It came from a fix made for the gateway servers and had been in production since July without causing trouble.
  2. A load surge. About fifteen hours earlier, Teams traffic automatically failed over onto the gateway that also serves Exchange Online. More load and different traffic patterns set the scene.
  3. Reduced capacity. Nearly 14% of server capacity was already out of service because of an unrelated issue the day before. On its own, that left enough headroom.

Once servers started to fail, traffic rerouted from them landed on the remaining servers and pushed those over too. At the lowest point more than 80% of servers were unresponsive. The boundaries that should have kept the problem inside one region did not hold, so it spread. BleepingComputer's account of the report adds that Microsoft's load testing missed the defect because the test labs used less powerful hardware than production.

Why it took so long to recover

Restarting servers did not simply fix it. Microsoft says restored servers were immediately overwhelmed again by the backlog of traffic, and the usual rollback and feature-flag controls could not quickly undo the change. Servers that had recovered went unresponsive again late on Thursday.

Timeline (all times GMT)

  • Thu 12 Nov, 01:07: automated failover moves Teams traffic onto the gateway shared with Exchange Online.
  • 16:03: monitoring detects rising connectivity problems.
  • 16:20: a high-priority alert on Exchange Online impact leads to a Service Health notice.
  • 19:00: Microsoft pauses all ongoing and planned deployments and configuration changes.
  • 23:55: recovered servers become unresponsive again under traffic.
  • Fri 13 Nov, 07:20: a mitigation reverting the defective change is applied to part of the capacity, then widened from 08:00, starting with regions in peak business hours.
  • 11:35: Europe returns to healthy levels.
  • 14:17: global webmail availability is back to expected levels.
  • 16:10: customers are no longer actively affected, and monitoring and backlog recovery continue.

In the UK, GMT is also local time at this time of year, so for most of us it hit in the late afternoon and was fixed by the next afternoon.

What Microsoft has done since

  • Returned the connection-handling code path to its previously known good state.
  • Extended the email delivery window so delayed messages were processed rather than bounced as undeliverable.
  • Added capacity, plus monitoring for traffic-redirection patterns and connection-handling health.
  • Put guardrails on traffic distribution to stop a failure cascading in the same way.

This is a preliminary report. A final report with the full list of repair items was due on 20 November, so details may change.

Is it a cyber issue?

Not on the evidence so far. Microsoft's report points to a bug, a traffic shift and lost capacity, with no mention of an attacker. I'd treat any claim of an attack as unproven unless Microsoft or a government agency says so. It still belongs here because confidentiality, integrity and availability are the three things security protects, and email being gone for a day is an availability failure.

What to take from it (my view)

These are my own suggestions, not Microsoft's.

  • Have a way to reach people that doesn't run on Microsoft 365. A phone tree, a text group or a status page hosted elsewhere. If email and Teams are both down, you still need to tell staff what is going on.
  • Give security alerts a second route. If your alerting arrives by email, an email outage can hide a real incident. Make sure the important alerts also reach a phone or a separate system.
  • Don't loosen security to get things working. Outages are when people disable policies or add exceptions in a hurry and forget to put them back. Keep change control, even in a rush.
  • Expect opportunists. Phishing about "restoring your mailbox" or "re-verifying your account" turns up fast after a big outage. Warn staff in your first message.
  • Know your own tolerance. Ask what a day without email costs, which processes depend on it (invoice approvals, supplier contact) and what the manual fallback is.
  • Use the official status channels. The Microsoft 365 admin centre Service Health page, not social media, is the place for incident IDs and updates. This one was tracked as EX1550543 and MO1550821.

The bottom line

Nobody broke in. A hidden bug met a busy network with less spare capacity than usual, and the safeguards that should have contained it didn't. The lesson for the rest of us isn't to distrust the cloud, but to plan for the day it isn't there.