Thoughts on Microsoft's latest news on Office, Office 365 and Exchange
Over the past two weeks, Microsoft has made a range of announcements around updates and new...
Any seasoned Exchange administrator has at one time needed to deal with a massive storm of “reply all” emails circulating the organization. It’s a chaotic situation, and not much fun to deal with, although you may enjoy telling the story to friends for years afterward.If you haven’t had the pleasure of a “reply all” storm in your career so far, here’s how it happens. First, someone in the company sends an email to a very large audience, either by adding too many people to the “To” field of the email or by sending to a very large distribution list. Next, someone who receives the email decides to “reply all.” A common reply is “Why am I on this list?” or “Please remove me from this list,” usually followed by more people clicking “reply all” and writing “Please remove me as well!” Things can quickly escalate with people sending abusive emails telling others to “stop replying all.”
The sudden surge in email traffic has the potential to bring the whole system down, as most email systems are sized to handle normal traffic loads with the occasional spike, not prolonged periods of excess traffic. Transport servers can only process messages so fast, and mailbox databases can overwhelm disk storage with too many IO operations.
No company is immune to the human errors that lead to this type of incident, including international news agency Reuters who reported to have experienced an email storm in late August after an employee named Vince emailed 33,000 people at once. Around 700 people replied to all in the first four hours, which adds up to over 20 million emails in a four-hour period, far more than an organization that size would be used to handling. And that’s before any read receipts or out of office messages are taken into consideration.
As an Exchange administrator, there are two things you need to know about managing “reply all” storms:
The tips I will share with you in this article apply to Exchange 2010 or later, as well as Exchange Online (Office 365), although the administrative interface may vary slightly between them.
If an unauthorized user tries to send to the distribution group, they will receive a non-delivery report, and the distribution group members are spared yet another unwanted email.
If the emails are going to a distribution group, the first step you can take is to block the distribution list from receiving emails from your users.
Note that when you set this it will stop any replies from going to the distribution group again, but other individual recipients will still receive the reply. On the one hand, this is OK because you’ve stopped further proliferation of the “reply all” messages to the entire group. On the other hand, some of the individuals the email is addressed to may still be annoyed by the continued messages. We can address that problem with a transport rule.
Transport rules in Exchange provide many powerful capabilities, one of which is to drop or reject messages that match some specific criteria. Before we continue, let me caution you that transport rules can be very destructive if not carefully configured. More than once I have witnessed a careless transport rule rejecting thousands of messages that it was not intended to reject.
A transport rule that matches the subject line of the email that is causing the “reply all” storm can be added to silently drop (i.e., no NDR is sent back to the sender) any message with that subject line. If you want to be safe, you can redirect a copy of the message to a temporary mailbox as well. This provides you the opportunity to recover any email that should not have been blocked, as well as identify any individuals who continue to reply to the emails. Such individuals may enjoy a re-education about corporate email policies.
If the risk of blocking legitimate emails is a concern, then instead of blocking the message, you can use a similar transport rule to hold it for approval instead. There is a risk that the person moderating the message will simply approve it without considering the impact, so make sure the person is well aware of what you’re trying to achieve here.
It is better to be proactive and try to prevent these incidents, and Exchange provides several methods that you can apply to your environment.
Applying restrictions to your largest distribution groups before an incident occurs is a wise move, but first you need to find your large distribution groups. You could probably guess many of them in your environment – the “All Staff” group is a prime candidate. However, due to nesting of groups within other groups, some of your largest distribution groups won’t be so easy to find without using a PowerShell script such as Get-DGMemberCounts.ps1.
Group size is not the only factor here. You likely also want to restrict who can send to sensitive groups such as the board of directors or all department heads. That will require a little more detective work on your part. I recommend grabbing a copy of the org chart for the company and looking at the group membership of the people at the top of the chart.
Once you’ve identified the high-risk groups, simply apply restrictions as demonstrated earlier, or consider using moderated transport, which is discussed next.
In some cases, blocking emails to larger distribution groups is not desirable. Consider a situation where a company has a social club distribution group that members are allowed to use within certain guidelines. In this situation, we can use moderated transport, allowing one or more trusted people to review email messages sent to the group and approve/decline them.
As with the earlier use of moderation, the approvers must be able to recognize misuse of distribution groups and not simply approve every message that is held for moderation.
Exchange includes a number of pre-configured Mail Tips that will display messages to Outlook and OWA users when they are composing email messages. For example, if a user is sending to a very large distribution group, then a mail tip appears to warn them. However, it does not prevent them from sending to it. Let’s face it - most people will not see it or will ignore it if they do see it.
But Mail Tips can also be manually configured to alert senders to restrictions on distribution groups before they send their message. For example, a distribution group that has been configured for moderated transport can have a mail tip added to advise the sender that their email may be delayed due to the approval process.
If you’re planning to use moderated transport on your larger distribution groups, then a mail tip to make it clear to end users what is happening is a good idea, as this will avoid support calls from users wondering why their emails have not yet been delivered.
One of the weaknesses of the controls we can apply to distribution groups is that the sender can simply expand the distribution group in Outlook and send the email message to all of the members as individual recipients. To avoid this, we can use dynamic distribution groups, which can’t be expanded in Outlook.
We can also apply other controls mentioned earlier, such as delivery restrictions and moderated transport, to dynamic distribution groups for a multi-layered approach.
Dynamic distribution groups aren’t always a feasible solution, though. One reason is that dynamic distribution groups cannot also be used as security groups, so duplication of groups becomes an administrative burden. In this case, the risk of distribution group expansion in Outlook can be mitigated by limiting the maximum number of recipients per message to a low, but reasonable number.
An unexpanded distribution group counts as one recipient, so you can still send emails to larger numbers of recipients using distribution groups even when the max recipients is set to a low number.
As you can see, there are a variety of controls that Exchange administrators can apply in their environments to prevent message storms due to misuse of distribution groups and the “reply all” button in Outlook. There is not a one-size-fits-all approach here; each type of control has its strengths and weaknesses, and you’ll need to assess each one against your organization’s needs. But with a little proactive effort, you can avoid a disaster.