That big Microsoft 365, Teams, and Outlook outage? Here’s what went wrong

Microsoft says an replace on a router was behind an enormous multi-hour outage affecting the Microsoft Wide Area Network (WAN) that made Azure, Microsoft 365 apps, and Power Platform inaccessible to prospects throughout the globe final week. 

The multi-hour outage final Wednesday impacted Microsoft Teams, Exchange Online, Outlook, SharePoint Online, OneDrive for Business, Microsoft Graph, PowerBi, M365 Admin Portal, Microsoft Intune, Microsoft Defender for Cloud Apps, and Microsoft Defender for Identity.  

Prior to the outage, Microsoft had warned prospects {that a} deliberate replace would possibly trigger latency or timeouts from 07:05 UTC on Wednesday when prospects tried to connect with Azure assets in Public Azure areas, Microsoft 365, and Power BI. But as staff in Europe began the day, the replace prompted greater than latency points and began impacting community gadgets throughout the Microsoft WAN, which dropped connections between companies in information facilities in addition to connections on ExpressRoute, Microsoft’s non-public community for purchasers to switch information between information facilities. 

Microsoft says in its preliminary post-incident evaluate that almost all areas and companies had recovered by 09:00 UTC on Wednesday, however they weren’t absolutely recovered till 12:43 UTC on 25 January. The outage additionally affected Azure Government cloud companies that had been depending on Azure public cloud, in response to Microsoft.

Also: The greatest cloud storage companies: Are free ones price it?

“We determined that a change made to the Microsoft Wide Area Network (WAN) impacted connectivity between clients on the internet to Azure, connectivity across regions, as well as cross-premises connectivity via ExpressRoute,” Microsoft says in its report first noticed by Bleeping Computer.

“As part of a planned change to update the IP address on a WAN router, a command given to the router caused it to send messages to all other routers in the WAN, which resulted in all of them recomputing their adjacency and forwarding tables. During this re-computation process, the routers were unable to correctly forward packets traversing them. The command that caused the issue has different behaviors on different network devices, and the command had not been vetted using our full qualification process on the router on which it was executed.”

Microsoft’s monitoring techniques detected area title service (DNS) and WAN points at 07:12 UTC. After reviewing latest modifications, whereas computerized restoration was taking place at 08:20 UTC, engineers found the “problematic command” behind the problems.

Also: Technology spending will rise subsequent yr. And this previous favorite remains to be a high precedence

“Due to the WAN impact, our automated systems for maintaining the health of the WAN were paused, including the systems for identifying and removing unhealthy devices, and the traffic engineering system for optimizing the flow of data across the network,” Microsoft mentioned. 

“Due to the pause in these systems, some paths in the network experienced increased packet loss from 09:35 UTC until those systems were manually restarted, restoring the WAN to optimal operating conditions. This recovery was completed at 12:43 UTC.”

Microsoft says it has now “blocked highly impactful commands from getting executed on the devices” to mitigate future occurrences. It’s additionally now requiring all command execution on the networks gadgets to observe secure change pointers. 

Microsoft plans to publish a remaining post-incident report inside the subsequent two weeks.