2017 Office 365 Outages Recap: Ducks vs Horses - Office 365 Monitoring
Reddit, a popular online discussion site, has a running joke: would rather fight a horse-sized duck...
Jaap Wesselius
25 years ago, I was working for Microsoft. And one morning we received a message with the subject line “Why am I on this mailing list? Please remove me from it.” This was followed by another email saying “Me Too!”, and then another “Me Too!”, and another, and another. These were followed by multiple emails with subject line “People stop with the reply alls”. Before we knew it, our Exchange environment was brought to a standstill because of the enormous amount of email sent via this Distribution Group.
Nobody was able to send email anymore. And it took a couple of days before the Microsoft internal IT staff had fixed this.
The Distribution Groups that were used were called “Bedlam DLx” (there were several Distribution Groups) and a couple of months later there were even T-shirts in Microsoft with “I survived Bedlam” printed on it 😊
The underlying problem here is twofold:
- Why use Distribution Groups with tons of users (more that 10.000) and why is a member of this Distribution Group allowed to send email to this Distribution Group?
- Ignorant (or dumb!) people that use the Reply All button when they should not have. This one is hard to solve, and as my good friend Ed Crowley always says, “"There are seldom GOOD technological solutions to BEHAVIORAL problems" although the “Ignore Conversation” in Outlook should work for these people.
In short, it took Microsoft more than 20 years to get a solution to this problem. When there’s risk of such a mail storm Exchange Online will block this email thread and generate the following NDR:
So when does Exchange online do this?
- When people are replying to the same email thread.- When there are 10 replies to all recipients.
- When it is sent to more than 5000 recipients.
- When this happens in less than an hour.
When this block is triggered, email is throttled for four hours, after the last message is sent.
A couple of remarks regarding the reply all mailstorm protection:
- It is currently being rolled-out in Exchange Online and will take a couple of weeks to complete.- It is only available in Exchange Online, I haven’t heard anything about Exchange 2016 or Exchange 2019. Looking at the lack of new features in the past Cumulative Updates for both versions of Exchange I doubt it will be backported to Exchange on-premises.
- Exchange admins have no tools to control this behavior.
- You don’t have to enable this feature, it will be available for all tenants. However, it can only be used in larger organization with over 5,000 mailboxes.
- You cannot test this (unless you want to send test messages to over 5,000 mailboxes and wait for people to hit the reply all button).
Summary
The reply all mail storm prevention feature is a nice addition for Exchange Online and it will prevent a possible mail storm when people are using the reply all button too much. However, there’s no technical solution for a behavior problem. Better is to control the use of Distribution Groups to prevent people to use the Reply All button, and to prevent them from being able to send an email to these Distribution Groups at all.
Monitor Your Hybrid - Office 365 Environment with ENow
ENow’s Office 365 Monitoring solution is like your own personal outage detector that pertains solely to you environment. ENow’s solution monitors all crucial components including your hybrid servers, the network, and Office 365 from a single pane of glass. Knowing immediately when a problem happens, where the fault lies, and why the issue has occurred, ensures that any outages are detected and solved as quickly as possible.
