Office 365 Monitoring: Teams Outage (April 27, 2021)
On April 27, 2021, at ~6:30 am UTC, Microsoft reported an issue with Microsoft Teams impacting...
On Monday, April 24, 2023, at approximately 7:17 AM ET, Microsoft sent out its first of many tweets for the day @MSFT365status regarding an issue in which users were unable to use certain search functions in various Microsoft 365 services. With this first tweet, Microsoft disclosed no information as to what geographic regions were impacted.
For system admins and IT professionals with access to the Microsoft 365 Admin Center, the service incident number to reference was MO545600.
We're investigating an issue where users may be unable to use the search functionality in multiple Microsoft 365 services. Further information can be found under MO545600 within the Microsoft 365 admin center.— Microsoft 365 Status (@MSFT365Status) April 24, 2023
Prior to today, Microsoft's last reported service incident was on April 20, 2023, and this prior incident involved access issues for Microsoft 365 online apps and the Microsoft Teams Admin Center.
There were many responses from the community throughout the day, many of them voicing frustration as to issues with Microsoft services yet again. Other community responses confirmed issues with Office 365 search functions and some questioned whether this issue was related to an issue in which some users were seeing error screens when starting Microsoft Teams.
At 9:45 AM ET, Microsoft sent out a second message, announcing that a "fix" had been developed internally but not yet deployed.
We've developed a fix to address the impact to the search feature, which we're currently validating internally to ensure it is efficient before deploying. Additionally, we're exploring if an ECS configuration change may provide quicker relief.— Microsoft 365 Status (@MSFT365Status) April 24, 2023
Approximately 5 hours after first reporting the issue on Twitter, Microsoft's next tweet provided only status quo information, that they were still only testing internally a possible fix.
We're performing the final stages of our internal testing of the fix, which we expect to complete within the next 60 minutes. If successful, it will be applied across the affected environments to mitigate impact. Additional details can be found under MO545600 in the admin center.— Microsoft 365 Status (@MSFT365Status) April 24, 2023
Two hours later, after having promised a 60 minute completion time in their previous message, Microsoft was still status quo with the issue: a possible "fix" was still being tested internally.
We're conducting our final testing of the fix, and additional time is needed for validation. We've observed positive results from an interim change, and we're evaluating further updates to improve the experience. Additional info is available under MO545600 in the admin center.— Microsoft 365 Status (@MSFT365Status) April 24, 2023
For their next public message, Microsoft was again making a "60-minute to complete" promise to the community, despite testing of the fix still being conducted on their end.
Our validations are progressing successfully, and we anticipate to complete within 60 minutes. Tests for a secondary change have shown positive results, and we're applying that change to a broader scope. Further details can be found in the admin center under MO545600.— Microsoft 365 Status (@MSFT365Status) April 24, 2023
Some 12 hours after their first tweet regarding MO545600, the issue was still not resolved. Microsoft had at this time confirmed that a temporary fix deployment was underway, and that impacted users should see some sort of service restoration. Fix deployments, as well as ongoing validation that the fix was indeed working, would continue throughout the evening for Microsoft.
Additionally, Microsoft made it clear that they were at the same time still working on validating a primary and long-term fix for this issue.
We've begun deploying our secondary fix to a broader scope after further validation and monitoring. Users should begin to experience relief as the deployment progresses. In parallel, we're continuing to validate our longer-term primary fix. Refer to MO545600 in the admin center.— Microsoft 365 Status (@MSFT365Status) April 24, 2023
We’re continuing our deployment of various changes and monitoring their progress. Some users' functionality may be restored as changes reach them. Concurrently, our validation continues for our long-term fix. Refer to MO545600 via the Microsoft 365 admin center for more detail.— Microsoft 365 Status (@MSFT365Status) April 25, 2023
We're heavily focused on validating the efficacy of our changes in restoring search availability. In parallel, we're still working on our longer-term fix. Refer to MO545600 via the Microsoft 365 admin center for more detail.— Microsoft 365 Status (@MSFT365Status) April 25, 2023
As Monday turned into Tuesday, Microsoft had still not fully resolved the matter. Their next tweets on April 25 were somewhat hopeful in that Microsoft was at least confirming that search functions had returned for some users. But, a full restoration was still not confirmed.
We've confirmed that search availability has been returned back to the service for some tenants. We are continuing implement our changes to fully restore availability. Refer to MO545600 in the Microsoft 365 admin center for more info.— Microsoft 365 Status (@MSFT365Status) April 25, 2023
We believe that the search functionality is now working for the majority of users. We're continuing to implement further mitigations to fully restore search functionality. For more details, please look for MO545600 in the Microsoft 365 admin center.— Microsoft 365 Status (@MSFT365Status) April 25, 2023
It should be noted that while service incident MO545600 was still not fully resolved, Microsoft during this time reported and resolved a separate service incident EX546390 involving Exchange Online connectivity issues for users in North America.
Finally, at approximately 6:30 PM ET, Microsoft sent out their final tweet regarding MO545600 declaring the issue completely resolved and all users would no longer experience this search functionality issue in Micrsosoft 365.
We identified and mitigated an issue within the search service that was preventing users from access to search functionality within Microsoft 365 platforms, and users should no longer see this issue. Please refer to MO545600 in the Microsoft 365 admin center for more details.— Microsoft 365 Status (@MSFT365Status) April 25, 2023
In a cloud-world, outages are bound to happen. While Microsoft is responsible for restoring service during outages, IT needs to take ownership of their environment and user experience. It is crucial to have greater visibility into business impacts during a service outage the moment it happens.
ENow’s Microsoft 365 Monitoring and Reporting solution enables IT Pros to pinpoint the exact services affected and the root cause of the issues an organization is experiencing during a service outage by providing: