Reaction to Microsoft Exchange 0-day attack

On March 6, Microsoft pointed out vulnerabilities in the Microsoft Exchange software. IONOS had already learned of the vulnerability on March 3 and immediately applied updates provided by Microsoft to all Exchange systems it operated itself in order to eliminate the vulnerabilities. IONOS systems were not affected by the attack wave.

In this interview with Lead Exchange Engineer, John Barnes, we try to clarify the most important questions on the matter.

We found out about the vulnerability very early in the morning of March 3rd and we immediately began assessing which servers needed to be patched and splitting up the work among the team to expedite the patching process.

We knew immediately that this update was very important, because it is rare for Microsoft to release patches outside of their standard monthly process. As soon as we got into the details of the vulnerabilities and understood the technical ramifications it became apparent that this patching needed to happen absolutely as quickly as possible.

We build our platforms with a significant amount of redundancy, so in circumstances such as these we are able to patch our platforms during business hours without causing a loss of service to our customers. Where possible, this also allows us to automate the patching process so that a significant proportion of our servers had already installed the patch prior to the start of the day.

The other parts of our platform we began patching in stages, moving workload between servers to avoid service interruption which took most of the day on Wednesday, prioritising those parts of the platform we felt were most vulnerable. We also became vigilant in looking for any suspicious activity on our platforms.

While the patching work was ongoing, we also joined a call from Microsoft regarding the update and coordinated with the IONOS Security Team to ensure information was distributed throughout the wider business. The Microsoft call was particularly enlightening with how much emphasis was being placed on patching right now, and not trying to mitigate the exploit in other ways.

Once the patching process was complete, we turned our primary focus to scanning for Indicators of Compromise. This is a lengthy process due to the size of our platforms, and how seriously we took these vulnerabilities, so we devoted several days to this scanning process.

Patching the vulnerability as quickly as possible should be the number one priority, which can be difficult sometimes with systems that are business critical such as email. In cases such as these, where the vulnerability can be so damaging, it is worth considering whether to cause a service outage in order to deploy the patch faster.

As a hosting provider we are geared up to act quickly and decisively on these 'incidents'. As this is something we do every day, we have the contacts and the information to know about these things as soon as possible - often before public releases. 

Our processes are highly automated and managed, therefore we ensure that we can react quickly and we operate globally scaled platforms, with geo-redundancy, meaning updates can be applied without business disruption.  As a Hoster we have been running global Hosted Exchange platforms since 2010 and have unrivalled experience outside of Microsoft.  We understand that these things happen - and it is how you can react that is most important.

I think the companies most at risk are the smaller companies running Exchange Servers on premise, that don’t necessarily have the resources to keep on top of the quarterly Cumulative Update schedule from Microsoft.

When Microsoft initially released the updates they were only available for the latest two Cumulative Updates on each version of Exchange, which meant that companies that were not up to date would need to install the latest Cumulative Updates before they could patch the vulnerability. This can be a much more involved process than just installing a Security Update for Exchange and will take significantly longer, increasing the company’s exposure to this vulnerability.

For this specific vulnerability: Microsoft recommends this HealthChecker to find the patch level of your servers and determine if you need to update your servers. In particular it may not be apparent through Microsoft Update that you have an outstanding update if your Exchange Servers are not patched to the latest Cumulative Update. The Test-ProxyLogon script is a Microsoft script which will help you identify any Indicators of Compromise.

In general usage I have found the Nessus Vulnerability Scanner to be particularly effective at identifying vulnerabilities and unpatched servers.

This exploit was particularly nasty, an ‘Unauthenticated Remote Code Execution as System’ exploit. It is particularly dangerous in this case, because Microsoft Exchange Servers typically have high levels of Privilege within Active Directory which is the primary Authentication and Authorisation system for Windows systems within a business.

This means that the risks to the business are extremely high, with a significant possibility of stolen/destroyed business data, loss of ability to function etc. It is difficult to understate the possible impact to the business if an attacker was able to successfully exploit this vulnerability.

I think it's always difficult to give a definitive 'all clear' with these sorts of vulnerabilities. We've done all the scans, based on the information currently provided by Microsoft and we've not seen anything of note, but a really good malicious actor can make it very difficult to spot a compromise.

In particular it has now come to light in various news sources that these particular vulnerabilities were first identified in early January and reported responsibly to Microsoft. In late February, more widespread attacks were identified just prior to Microsoft releasing the patches. This means that there is a significant time frame for the initial exploit to occur.

I believe this may make companies reassess whether they wish to continue hosting their email systems on premise and the real cost of maintaining those systems versus hosting these business critical systems with a trusted cloud provider.

Further Information:

https://www.microsoft.com/security/blog/2021/03/02/hafnium-targeting-exchange-servers/ 

https://msrc-blog.microsoft.com/2021/03/02/multiple-security-updates-released-for-exchange-server/ 

https://msrc-blog.microsoft.com/2021/03/05/microsoft-exchange-server-vulnerabilities-mitigations-march-2021/