The Curious Logic Behind Some EOP Routing Decisions
Recently, one of my customers reached out to me stating they were having trouble delivering emails...
Patching Exchange servers. Three words a lot of us wish would just go away forever. Well, the good news is, in some capacity patching old Microsoft Exchange servers will become a thing of the past. We’ve heard about this tirelessly for years in the tech community, from Microsoft themselves to Microsoft MVP’s and thought leaders, but it’s not just ‘talk’ anymore – it’s rapidly turning into reality.
On July 26, 2023, Exchange Online will start throttling and blocking inbound traffic from obsolete Exchange 2007 servers, a specific subset that connect to Exchange Online over an inbound connector type of OnPremises. From there, this enforcement system incrementally adds other Exchange Server versions to the fold, culminating with all versions of Exchange Server as well as all inbound email for Exchange Online. Side note: Monday, April 11, 2023 is end of support for Exchange 2013.
So there you have it - there's no way around it. You absolutely MUST patch any management servers in use, or better yet – migrate to the latest version of Exchange or Exchange Online already. Because we all know that at some point, eventually you’ll have to migrate to the cloud, and patching old Exchange servers is just a temporary band-aid and not a long-term solution.
In the first blog of a two-part series, we add some color to help frame the situation at hand, emphasizing the potential business ramifications that will result from inaction, and the criticality of taking immediate action. In part two we'll provide a helpful solution with a step-by-step tutorial and some best practices around different Microsoft Exchange mailbox migration scenarios.
I’m sure few of you need a reminder about the HAFNIUM attack that transpired in March 2021, where thousands of Exchange servers were attacked worldwide, afterwards sending many IT pros in a tailspin to quickly establish stricter IT security protocols. Others found themselves having to rebuild completely from scratch, either from the loss of data or from the identification of breached servers.
While it seems like around that time, most admins saw the warning signs, the writing on the wall, and had their systems patched. Unfortunately there are still outliers who continue to ignore all warning signs and ignore their Exchange servers. To this day, Exchange servers across the globe are still being exploited using the same exact attack vectors and exploits from the March ‘21 HAFNIUM attack - an attack not to be taken lightly.
To help put the magnitude of an attack like HAFNIUM in perspective - when an Exchange server is breached and attacked, this allows remote access to the server where the attacker can then elevate privileges and perform lateral movement, with the end-goal of extorting information from the company in demand for ransom. To make matters even worse, often during the wake of a breach other hacking groups will attempt to capitalize on the situation, leveraging tools made available on the dark web from the exploit, in order to carry out their own adjacent attacks to steal information, deliver Ransomware and malware, and at the end of the day, probably demand ransom as well.
Recently, a colleague of mine was telling me about a situation where a customer was running Exchange 2010 RTM, internet-facing, and a recent Windows update caused the server to crash. There are numerous red flags with that situation, but to begin with, an immediate big red flag around the fact that they were running an outdated and unsupported Exchange Server Version that was no longer maintained by Microsoft.
Slightly alarmed, I strongly advised that my colleague upgrade his client to the latest supported version of Exchange, and quickly. His response back to me essentially said 'the crash was just a small hurdle, a minor setback' and that 'the server works just fine' (even though it had, in fact, just crashed), and that it was 'highly unlikely the company was going to spend thousands of dollars on new hardware, Office licenses to support the new version, etc.' And I’m sure that many other admins who still haven’t patched or migrated or upgraded are probably singing the same exact tune. Unfortunately, it’s just a matter of time before my colleague’s server, and those of many others, are compromised. Unless of course they’ve already been compromised and just don’t know it yet.
The main takeaway here is - it doesn’t matter how long you procrastinate – eventually you will have to make some kind of a move, whether it’s on your timeline and your terms, or because you waited until the last minute and the clock ran out. Either way, if you don't upgrade or migrate, you won’t be able to send email to Exchange online if your Exchange servers are old and outdated.
The same principles that apply to unsupported servers, also apply to unpatched Microsoft Exchange servers running on unsupported software. That’s because in that scenario, the extreme amount of vulnerability one is exposed to is comparable because none of the updates that address multiple CVE exploits have been applied. To be clear, it doesn’t necessarily mean you have an old version of Exchange - but you might as well have an older version, because you haven't applied any updates. For example, you could be running Exchange Server 2019 RTM with no Security (SU) Updates or Cumulative Updates (CU).
Microsoft does regularly release updates to address known vulnerabilities, as well as fixes for issues brought to light by Microsoft MVP’s, Technology Adoption Program (TAP) customers and nuances that are continuously brought up and discussed in forums. Whenever new, known vulnerabilities are identified, and different patches for Exchange are released from Microsoft, I cannot stress to you enough how important it is to stay current with these updates. This holds true for management servers as well - you must patch them. And with the ease of access to a multitude of cloud resources, customers can now quickly spin up a lab in Azure and test patches prior to production, shutting the lab down afterward to save on costs.
Again, if you do not stay up-to-date with upgrading and patching your Microsoft Exchange servers, you will eventually be blocked and be completely unable to send email via Exchange.
Now that you have a better idea of where Microsoft is ultimately going with all this, it would be wise to start getting your change controls in place, or move forward with those pesky migrations that keep getting put on the back-burner. To reiterate, if you find yourself in this particular scenario, I strongly advise you immediately put a plan in motion, because the inability to send email will cripple any business, but especially if customers you are trying to reach are on Exchange Online and blocked from ever receiving your emails again.
If you are unsure how exactly to go about upgrading or migrating your Exchange mailboxes, stay tuned for a second blog where we'll provide a tutorial and some best practices for migrating Microsoft Exchange mailboxes.
Do you have numerous Exchange servers that need to be patched? Understanding the version and patch you are currently running enables you to access the security risk in your environment and ensure the patch was successfully installed. The Exchange version report simply returns back the information needed to understand what version your servers are running and if the security patch was successful. (PS - don’t forget to reboot your server after applying the patch!)