Veeam Backup & Replication CVE-2023-27532 Response

UPDATE 03/13/2023 2252 ET: After taking further inventory of our partner's Veeam service binary details to review the version number, we uncovered many more unpatched and vulnerable hosts. We are sending incident reports for all affected partners and making direct phone calls when Veeam is publicly exposed on the Internet.

On 7 March 2023, Veeam published a knowledge base article outlining CVE-2023-27532, a vulnerability in the Veeam Backup & Replication component that allowed an unauthenticated user to retrieve host credentials stored in the configuration database.

This weakness could ultimately enable an attacker to gain access to hosts and devices managed by the Veeam Backup server. With access to the open TCP port 9401, any individual could obtain credentials and potentially move laterally throughout the network with the newly exposed username and passwords. Any Veeam Backup & Replication version prior to V12  (build 12.0.0.1420 P20230223) and V11a (build 11.0.1.1261 P20230227) is vulnerable.

With a High severity rating and a CVSS score of 7.5, this vulnerability made its rounds across headlines, news outlets, and further scrutiny by security researchers.

If you do not know your Veeam version and build number, the following PowerShell lines will display it.
If the output does not match any of the build numbers mentioned previously, [.underline]you need to patch[.underline].

[.highlight]$CoreDllPath = (Get-ItemProperty -Path "HKLM:\Software\Veeam\Veeam Backup and Replication\" | Select-Object -ExpandProperty CorePath) + "Veeam.Backup.Core.dll"$CoreDll = Get-Item -Path $CoreDllPath$CoreDll.VersionInfo.ProductVersion + " - " + $CoreDll.VersionInfo.Comments[.highlight]

Outreach

The Huntress team proactively identified partners running vulnerable Veeam software versions on servers with a Huntress agent and has sent incident reports with specific hostnames and Huntress agent IDs. If you received a report via email or PSA integration, we urge you to patch and upgrade to the latest available Veeam version as soon as possible.

If you do not have a Huntress agent installed on all hosts and we were unable to detect your Veeam instance, you are of course still urged to patch.

Huntress partners with a publicly exposed Veeam instance and TCP port 9401 open to the Internet have received a direct phone call to urgently patch or block external connections from the firewall. (Frankly, these were single-digit occurrences.)

Out of our 2 million install base, we uncovered north of 7,500 hosts with a vulnerable version of the Veeam Backup & Replication service present.

Investigation

Following the initial advisory of this new Veeam vulnerability, there was not yet a publicly available or commoditized proof-of-concept exploit. At the time of writing, Huntress has not observed any exploitation in-the-wild.

However, on March 9th, Huntress researchers caught wind of a Twitter post from Code White GmbH. 

CVE-2023-27532 in Veeam Backup & Replication is serious, expect exploitation attempts soon. Our teammate @mwulftange was able to develop an exploit just by using the exposed API. pic.twitter.com/86EmYT5zjP

Markus Wulftange, a researcher we have worked with before, had crafted a proof-of-concept exploit to demonstrate both the simplicity of the attack vector as well as the potential impact. Usernames and passwords were easily obtained, decrypted, and this technique merely required access to the exposed API of the Veeam Backup & Replication component (Veeam.Backup.Service.exe).

Seeing the tangible impact, Huntress researchers went to work recreating the proof-of-concept and assessing how this may affect our partners. We reached out to Code White GmbH for further discussion and we are always grateful for their transparency and support in information sharing. We successfully recreated the proof-of-concept and were able to extract login credentials.

Technical Details

Exploitation is achieved by interrogating the local API, served on port 9401 listening on all interfaces (0.0.0.0) by default. 

The implementation of this API is specific to the Veeam Backup & Replication server. After configuring a client to communicate with this API, querying for credentials is trivial. Credentials are initially encrypted, but a separate API call can easily decrypt them into their original plaintext value. There are no child processes created, nor any filesystem or registry artifacts left behind. 

With the default settings of the Veeam Backup & Replication server, no logs related to this attack are left behind. Logs specifically for the API can be configured with the registry changes discussed in the Indicators of Compromise section below.

Our Proof-of-Concept

Huntress researcher Caleb Stewart has recreated the proof-of-concept to demonstrate retrieving credentials from a vulnerable Veeam Backup & Replication server instance. All of our testing for the vulnerability and exploitation was tested against the installation provided by the ISO file VeeamBackupReplication_11.0.1.1261_20211123.iso (SHA256: 0dcebe370ec61bd817ebeadcc827f7aadac3e5011f7a4e5967c50df85380074c)

The following video showcases Caleb’s efforts to leak and decrypt credentials via CVE-2023-27532:

With this proof-of-concept, the Huntress team was able to rapidly detect and safely check for publicly exposed vulnerable instances without any exploitation. This enabled us to make more accurate reporting for partners and determine how urgently they need to react and respond.

Additionally, the Huntress team uncovered other potential API calls that could be weaponized. We were able to further expand our proof-of-concept to achieve arbitrary code execution.

While the unauthenticated credential dump acts as a vector for lateral movement or post-exploitation, the vulnerability in question can also be used for unauthenticated remote code execution - turning the vulnerable Veeam instance itself into a vector for initial access or further compromise.

The patch resolves both these issues of unauthenticated credential dumping and remote code execution.

The following video is a demonstration of Veeam Backup & Replication server exploited to run arbitrary commands:

As a public proof-of-concept has not yet been shared online, and there have not yet been any observed exploitation in the wild, we are withholding specific technical details that might enable adversaries or threat actors to craft these techniques.

Validating the Patch

The official release patch and security update from Veeam successfully remediates CVE-2023-27532 and other unauthenticated attack vectors against the API. 

Veeam Backup & Replication service versions V12.0.0.1420 P20230223 and V11.0.1.1261 P20230227 make changes to the underlying codebase to require an authentication token with all requests to the API. 

This is accomplished with a JSON Web Token (JWT) that is signed with a unique certificate specific to the Veeam Backup & Replication installation. Empty or invalid tokens cause an access exception and are unable to retrieve information or make changes.

Indicators of Compromise

While there are no child processes created following exploitation, logs are generated under C:\ProgramData\Veeam\Backup\Svc.VeeamBackup.log. However, by default, logging is not configured to include API calls.

Without manually changing the log level, the Svc.VeeamBackup.log will not show any entries related to the attack technique.

By default, the Windows registry value HKLM\Software\Veeam\Veeam Backup and Replication\LoggingLevel is set to the DWORD value of 4. To observe API calls, this value must be set to 7.

The following commands ran from an Administrator PowerShell prompt will configure logging to capture API calls:

Without sharing the specific API methods, the pertinent log entries look like the following after our proof-of-concept exploitation:

The official Veeam knowledge base advisory explains these credentials are encrypted, which they are -- the database manager does not return plaintext credentials, but it does return encrypted values and each account UUID. 

Once the account identifiers are obtained, further API calls can be made to decrypt the credentials into a simple Base64 encoding of the original plaintext value.