Recap: Lessons Learned During the Kaseya VSA Supply Chain Attack

On July 2, 2021, as many people in the United States were preparing for the July 4 holiday, a major ransomware attack began to unfold.

The REvil ransomware group carried out a sophisticated supply chain attack against Kaseya’s VSA product. The attack is believed to have affected between 50 and 60 MSPs—and between 1,500 and 2,000 of their customers.

This attack was a prime example of attacker tradecraft becoming more sophisticated. The targets—IT resellers and MSPs—act as a gateway to dozens, hundreds or even thousands of SMBs. Unfortunately, this was one of the most complex and successful ransomware attacks in recent memory. Look no further than the news headlines that surfaced to see how groundbreaking and sophisticated this attack was.

Now that a decryption key is available and we seem to be on the downward slope of the rollercoaster, we have an opportunity to look back and capture some important lessons and learnings that can help this industry try to combat these threats more effectively. 

What Happened

The Huntress team first became aware of the incident after three separate MSP partners reached out to us, noting that they—along with their customers—had been hit with ransomware. These reports reached us within half an hour of each other, which immediately made us question if this was part of a larger attack. As the number of compromised victims continued to grow, it became clear that the only commonality between them was the usage of on-premises Kaseya VSA.

As soon as we had enough information to do so, we began providing real-time updates on our rapid response blog as well as our Reddit post. Now, we want to dive a bit deeper into the incident to learn all we can from it to be better prepared for the next event.

The Attackers' Perspective

This attack marked an exceptional opportunity from the attackers’ perspective. Why target one business when they could directly target an MSP and get dozens—even tens or hundreds—of businesses in one go?

REvil conducted a novel supply chain attack because of the ripple effect it would have. The impact trickled down from the compromised VSA server to all of the affected MSPs’ machines. Then, it impacted all of the downstream SMB endpoints. In one swoop, the attackers hit the mothership and cut across the entire vertical.

Remaining Undetected

There were three main components to the threat actors’ attack chain:

1) Authentication bypass

2) Arbitrary file upload

3) Code execution

The interface to access the Kaseya VSA server is publicly open on the internet, so hackers could find and access those targets with ease. How they performed the initial authentication bypass, however, is something our Security Researchers are still wondering about.

Next Steps

Once the attackers had an authenticated session, they uploaded two files: agent.crt and Screenshot.jpg. We’ve learned that `agent.crt` is the core payload that kicks off ransomware while `Screenshot.jpg` sets the stage by deleting IIS logs and administrative accounts and setting up procedures in a database to push out the ransomware to all endpoints with the agent at a coordinated time.

This attack presented a huge problem, largely because the authentication bypass could completely circumvent multi-factor authentication and gain access to the overarching database to control all client machines. This compromised the remote monitoring and management (RMM) software that a significant number of Kaseya customers used. We’ve heard it be said before that a compromised RMM is the worst-case scenario for MSPs. This was another nightmare.

The Victims

As previously mentioned, the victims included between 50 and 60 MSPs and their customers—however, the impact of this attack stretches far beyond those who were directly impacted. This incident has been an eye-opening experience for all businesses, even those that don’t use the VSA software. They’re left wondering, “Could my own security stack be compromised like this? What about Vendor 'ABC'—could they be next?”

Although the victims who experienced the encryption firsthand felt the brunt of this hack, the greater community has felt the strong ripple effects.

The Significance of This Attack

We say it all the time: threat actors are getting smarter every day. As attacks grow more sophisticated, it becomes more challenging to defend against them. 

The reality is that the attack against Kaseya could have happened to any vendor. If anything, this attack should serve as a wake-up call for organizations. No matter how much or how quickly a business grows, it’s critical that business leaders continue to focus on protecting their core and strengthening their code, policies and procedures. Cybersecurity education and training are a must. One bad breach can make a company crumble.

What Went Well—and What Could've Gone Better

It was incredible to witness the entire MSP community come together during this crisis. We saw this firsthand on our Reddit post as members of the MSP community reached out to lend a helping hand:

If there's a silver lining to this incident, it's how incredible the MSP community is and how we came together to support and collaborate with one another. 💙

There was certainly room for improvement, though. When we consider the VSA vulnerability on its own, the software code base, the notification process and the timeline for remediations and recovery—it leaves something to be desired. We need to hold our vendors accountable for code quality and transparent communication.

The Decryption Key

On July 22, the anticipated news broke: a universal decryption key had surfaced.

The public reception of the news appeared to be mixed. Some have said the key’s arrival is just too little, too late, as they’d already paid the ransom on their own or worked to recover their data manually. Others who hadn’t made much progress recovering their data may benefit from the decryption key to get back up and running.

That leads us to the next question that's consuming many minds: how did they get the decryption key?

Many seemed to question if Kaseya might have paid the ransom; others have posited that the government intervened. Kaseya has noted that they didn’t pay the ransom for the decryption tool, but they haven’t noted where the key came from. All that to say that quite a few details surrounding this event are still undisclosed for now.

The Huntress Team's Involvement

Our team played an active role in this incident. We did what we could to investigate and analyze exactly how this attack was carried out while trying to help businesses that were affected.

Around two hours after the ransomware incidents started, our ThreatOps and Engineering teams were able to review a payload that enabled them to grab a few snippets—just enough to come up with a vaccine that could potentially help the systems we had access to (our partners’).

There were a few concerns about administering the vaccine, however:

• If REvil found out about it, they could have easily changed the name of the file/directory to make the ransomware run as intended.
• The kworking dir was configurable, so if it were named something different, the vaccine wouldn’t work.
• The Huntress agent.exe could have been confused with the REvil agent.exe.

At that point, we had a tough question to ask ourselves: did the potential benefits of the vaccine outweigh the risks? Ultimately for us, they did. 

We often say “our offense is your defense,” but this was a time when our actions spoke louder than those words. So, we pushed out the vaccine. We took a page right out of the hacker handbook in order to protect our partners—and thankfully, we have partners who trust us to make the right decisions to keep them and their customers safe. 😁

Lessons Learned

Thinking back to how this incident unfolded, we’ve unpacked a few lessons worth learning and some tips on moving forward.

• Regularly evaluate your security stack.

What works for your organization today may not stack up against the attacks of tomorrow. You should regularly evaluate the tools you have in your security stack to make sure you’re well-equipped to deal with tomorrow’s attackers.

• Work with your vendors to properly configure your tools.

There’s no out-of-the-box solution that works for every business. Work with each vendor you partner with to make sure your tools are configured the way that works for your business. Further, make sure your vendors stay up-to-date on best practices and the latest security alerts.

• Revisit your incident response plan.

Let's put it this way: in Florida, hurricanes happen. Floridians aren't measured on whether they can prevent a hurricane from happening—that's preposterous! 

Instead, they're measured on how fast they can recover. They keep emergency supplies on hand and have a plan in place for surviving hurricane season.

Cyberattacks are as inevitable as hurricanes. There is not—and will never be—a foolproof way to prevent a cyberattack. Instead, cybersecurity pros should focus their efforts on being prepared for when the worst does happen. That’s where an incident response plan comes in.

Not only can an incident response plan bring you some peace of mind on how to respond to those worst-case scenarios, but it can also help you respond quickly and get you on the path to recovery faster. If you want to test out your current plan, check out our Incident Response Tabletop-in-a-Box.

If you need assistance—even if you’re not a current Huntress partner—please reach out to us at support@huntress.com.