Understanding Evil: How to Reverse Engineer Malware

How do security teams know exactly what malware is doing and how to stop it?

In a recent episode of Tradecraft Tuesday, I was joined by Principal Security Researcher Caleb Stewart to dive into the thought process behind reverse engineering malware and the value it brings.

Understanding how malware functions is critical to developing a stronger security posture. Not only does it aid in improving defense measures by learning how malware behaves, but it can also give valuable intel into how threat actors use malware to operate. A real-world example of this is shown in our recent blog about Qakbot when Huntress discovered how to stop it during its execution.

Let’s dive into the basics behind reverse engineering malware and how you can get started.

The Fundamentals of Malware Reverse Engineering

It’s dangerous in the wild wild web, and detonating malware with no preparation or regard for safety can have disastrous consequences. Instead, you should carefully craft a secure, isolated environment for your malware analysis.

Below are some simple steps you can take to ensure that you are as prepared as possible for your analysis.

1. Prepare your device

Create a Virtual Machine either locally or in the cloud.
Download any tools you deem necessary for the analysis and ensure they are in working order.
Create a snapshot of your clean device with all the tools in place!

2. Acquire the malware

Download the malware directly to your analysis machine. Don’t risk putting known bad software on your normal computer!
If you are acquiring the sample from the internet, consider a VPN or Tor to minimize the personal information you are sending to the internet.
Don’t immediately detonate the malware on your computer! Wait for the last step.

3. Isolate your analysis environment

This can be done a number of ways by either completely disconnecting the device from the internet or restricting network traffic via firewalls (note: some malware can defeat host-based firewalls)
It’s safer to execute the malware without networking; however, it may be necessary to allow for network traffic for complete analysis. If so, be sure to watch the traffic!

Tools of the Trade

There are countless reverse engineering tools available, but here’s a few we recommend:

Disassemblers

The most commonly used disassemblers are Ghidra and IDA Pro. These handy tools read the machine code of a program and attempt to translate the instructions into a higher-level programming language of your choice. The better disassemblers provide features like control flow analysis that will guide you through the steps the malware is taking and prevent you from jumping all around on your own.

Dynamic debuggers

Dynamic debuggers have similar functionality as a developer's integrated development environment. They allow you to execute the program and pause throughout at your will to inspect the state and see variable values or other useful information. Generally, sticking with Ghidra or IDA Pro is a good choice since they have the capability to do both disassembly and dynamic debugging.

Other handy tools

Sometimes you'll run into challenges when attempting to understand what the source code is doing, like when a program is written in Golang, which hampers most disassemblers because it’s not built on top of the C programming language. Fortunately, there are other options for dynamic evaluation of a piece of malware:

Sysinternals Suite has countless Windows monitoring tools, including Procmon, that allow you to see each action a process takes on a machine.
Wireshark is a sniffer that captures network traffic that could help with insights into an attacker's command & control structure.
Automation tools like using CAPE in conjunction with YARA rules to identify and categorize malware automatically.

Challenges in Reverse Engineering

Hackers don’t want to make it easy for us to understand their code. The less investigation, the better for them. You’re likely to find a few techniques used to prevent pesky reverse engineers from discovering the secrets of their source of income, including:

Poly and metamorphic code, which allow the code to change its programming every execution. These changes in the source code make static analysis differ from execution to execution. You’re often best off doing dynamic analysis against these pieces of software because while the source code changes, more often than not, the overall objective and end state of the malware will be the same.
Anti-analysis techniques intend to hamper a reverse engineer's ability to decipher the code. This can range from basic software packing, which readily eliminates most strings and human-readable information in the program, to interesting techniques similar to code golf, where the hacker obfuscates the source code. Imagine looking at a program and every part is just the same letter!
Evasion techniques enable the software to detect the environment it’s executing in. Some malware may recognize that you are in a virtual environment and block execution. Working around this can be frustrating, as often you will need to put the malware on a live box. You should still follow as many safety precautions as possible, but you will lose your ability to easily revert to a snapshot.

Putting It All Together

Reverse engineering isn’t as simple as just throwing tools on a box and detonating your malware. It is a methodical approach.

You need to identify the malware and determine what type of analysis will work best. Then, look for the first thing the malware does when execution starts. You will encounter unexpected challenges along the way, and you will need critical thinking and rapid problem-solving skills to help work your way through the nuances of each sample. Just like all aspects of cybersecurity, mastering reverse engineering requires time and discipline. It can take hundreds of repetitions to learn how to differentiate from what is normal or not, but by utilizing safe practices and having a little patience, you can get there!