Beware of Traitorware: Using Splunk for Persistence

In a previous blog post, I covered how Splunk, and by extension, other security tools, can be used for malicious purposes. In that specific example, we looked at a straightforward data exfiltration technique. That blog garnered some attention, but I didn’t feel it exemplified the damage that malicious control of security tools can cause a business.

In this blog, I will detail how the Splunk Universal Forwarder (UF) can be used as traitorware for persistence and remote code execution.

Did you know? Traitorware, as defined by Alberto Rodriguez and Erik Hunstad, is:
1. Software that betrays the trust placed in it to perform malicious actions
2. Trusted software with benign original intent used for malicious actions

For those who may be unfamiliar, Splunk is a robust log collection tool standard in enterprise environments. Certain features and functionalities can be added through technical apps (TAs) and add-ons. These apps can be as straightforward as a configuration file to monitor a single directory or be as complex as a user interface with executable code functionalities.

An example is the Splunk Add-on for Windows, which gives Splunk additional functionality for collecting information from Windows hosts. This is a very common Splunk add-on, with over 415,000 downloads (at the time of writing). This add-on allows the Splunk UF to collect additional log data from the system using PowerShell scripts and custom inputs.

In this scenario, I aim to use the Splunk UF as a Remote Access Trojan (RAT) on the system, allowing the Splunk UF to receive arbitrary PowerShell commands from a remote server. To do this, I’ll use a custom Splunk app that uses the PowerShell input functionality provided by Splunk.

IMPORTANT DISCLAIMER: This POC is for demonstration purposes only and is not a vulnerability, bug, or new exploit within Splunk. These are configurations supported by Splunk and provided in their documentation. The goal is to demonstrate that malicious actors can abuse IT and security tooling and use the core functionality for malicious intent.

Diving In

First, we must create a custom Splunk app to abuse Splunk's PowerShell input functionality. A Splunk app is a bundle of configuration files installed on any Splunk appliance or, in this case, UF. These configuration files can enable new functionalities, provide data inputs, or create new UI elements.

We will create our Splunk app under <Splunk Install>\etc\apps, and it contains the following components: a PowerShell script and an inputs config file used to execute the script.

Here’s the folder structure for our app:

Starting with inputs.conf, we can begin to paint the picture:

Let’s break down each of these settings.

Script

Contains the path to our PowerShell script. [.highlight]$SplunkHome[.highlight] is set during the installation of the UF and is the install directory.

Schedule

The script will run based on this schedule in cron format. This example runs every 10th minute on the hour (12:00, 12:10, 12:20, etc.) This is a built-in persistence mechanism since Splunk will automatically re-run the script. We can also schedule this to only run once, or at system startup, for example.

Disabled, Index, and Sourcetype

These settings aren’t always necessary and aren’t crucial in this scenario. Splunk uses index and sourcetype to know where to route the script’s outputs, but I will not be configuring the UF to ship logs anywhere as part of this POC.

For [.highlight]git.ps1[.highlight], we’ll use [.highlight]Invoke-WebRequest[.highlight] to scrape a webpage's HTML, collect the contents of only the HTML comment tag on the page, and then execute those contents as a command.

For demonstration purposes, [.highlight]10.0.0.235[.highlight] is a webserver I control. Replacing the IP with a domain name will retain the same functionality.

You may notice that we’re using [.highlight][System.Text.Encoding][.highlight] to decode a base64 value stored in [.highlight]$value[.highlight]. The HTML comment tags contain PowerShell commands, and I found that some characters in the PowerShell commands would parse as HTML, which would not return the correct value with [.highlight]Invoke-WebRequest[.highlight] (< and >, for example.) To get around that challenge, I just base64 encoded the contents of the comment tag.

Here’s the HTML of the web page we’re scraping:

Our Base64 encoded command is [.highlight]d2hvYW1pID4gQzpcdXNlcnNcdXNlclxkZXNrdG9wXGhlbGxvLnR4dA==[.highlight] which decodes to [.highlight]whoami > C:\users\user\desktop\hello.txt[.highlight].

After we string everything together, the Splunk UF will pull a command from the web page every 10 minutes past the hour, allowing us to change the webpage to send new code to the victim endpoint.

Here’s a screenshot showing the connection to the web server and the contents of the hello.txt file we created. By default, the Splunk UF runs as [.highlight]NT Authority\System[.highlight] and will execute scripts with the same user context.

Why Does This Work?

The Splunk UF configures the [.highlight]SplunkUniversalForwarder[.highlight] service to run as [.highlight]NT Authority\System[.highlight], and this is because several of the background tasks that Splunk performs for data collection require administrative privileges.

If we pull apart the Splunk Add-on for Windows as mentioned earlier, we’ll find several PowerShell scripts for collecting data such as DNS info, Windows update logs, replication status, etc. This is standard information that most administrators would find helpful. Still, the Universal Forwarder’s script handler doesn’t limit the use of PowerShell or network connections, allowing us to inject just about any command and make outbound connections. While I created a custom Splunk app to demonstrate this, injecting malicious PowerShell code into existing Splunk apps, such as the Add-on for Windows, is possible.

This is not a new technique; security researchers have used Splunk’s UF as an offensive tool for years. Tom Kopchak’s blog on leveraging Splunk as an offensive security tool covers this technique and a similar technique for Linux in much deeper detail.

Mitigation

For Splunk:

Monitor your environment for changes, including new installations of the Splunk UF (especially if you don’t use Splunk,) or Splunk UFs that are restarting unexpectedly or no longer checking in with the deployment server.
Do not run the Splunk UF as root or administrator.

This can be a bit tedious in Windows, as you’ll have to configure and deploy GPOs to manage the Local Security Policies across a large environment. Splunk’s documentation on this process is here.
For Linux, this process is more straightforward. This involves changing some folder permissions and adjusting Splunk’s boot-start configuration. Splunk’s documentation on this process is here.

Limit the outbound traffic allowed by Splunk, such as limiting traffic to Splunk-only ports or disabling internet access.
Use Splunk’s reference architecture as a baseline for a secure deployment.

General mitigation for preventing IT/security tool abuse:

Monitor the internal logs of IT/security tools for suspicious behavior.
Monitor network traffic for anomalous connections, persistence, or unnecessary port exposure.
Network segmentation of security tools and critical infrastructure.
Limit outbound traffic to only necessary ports and protocols.
Use a layered defense strategy for network and host-based detection methods.
Use of SSO and MFA with strong password policies for administrators accessing these tools.