A Day in the Life of a Security Researcher

Last week, we hopped behind the virtual shoulders of Dray Agha, a ThreatOps analyst here at Huntress. He detailed the careful choreography that takes place on our global ThreatOps team to make sure that seamless shift changes happen between threat analysts.

This week, we’re pivoting to a different function of our ThreatOps team: our security researchers.

We asked three of our security researchers—Dave Kleinatland, Caleb Stewart and John Hammond*—some questions about their job duties, including which tools they use most frequently and what they particularly enjoy about their respective roles.

Pull up a chair and read on to learn about a day in the life of a security researcher.

*These individuals have been sorted by most to least number of mentions of “needing caffeine” to make it through the day.

Caleb Stewart

Caffeine mentions: 2

Q: What’s the first thing you do when you’re ready to start your day?

I normally grab a beverage with caffeine (either a Monster or a coffee) and look through Slack. We have multiple channels under ThreatOps like threatops-alerts, threatops-center, threatops-research, etc. These could contain recent incident information or just interesting articles and research from colleagues, so I think it’s important to stay up-to-date there.

Q: How have your education/training and past experiences paid off in your day-to-day role?

My professional certifications are mainly focused on offensive operations/tooling and reverse engineering. My undergrad degree is in electrical engineering, and reverse engineering is obviously very applicable in incident response scenarios as they occur. On the other hand, the R&D team has our hands in a lot of projects which are intended to improve the automated detection and response to malicious (or offensive) activity, so the knowledge of offensive tactics and techniques is invaluable when developing tools or features to combat those techniques.

Q: Do you have your own preferred methodology for doing your job?

Being a security researcher is inherently a fairly broad job. I don’t think you could pin down one “methodology.” We get tasked with many different types of projects from improving or adding to the agent to creating internal tools for our analysts to improve their detection and response. 

In a broad sense, I suppose the methodology is just keeping your eyes open for opportunities to improve parts of the company—whether that be writing new code for the agent or portal or developing detectors for new tactics and techniques in ELK, or improving some internal processes. It’s about having an open mind and being willing to consider changing up how we’ve always done something in the spirit of finding a better way.

Q: What’s your favorite part about being a security researcher?

We have a lot of freedom. I think you’ll see while reading the answers from other researchers that we all have a slightly different experience based on our own strengths and weaknesses. The common thread is that we are all experts in some facet of security, and we have the freedom to apply that knowledge in any way we find interesting as long as we are striving to make something better for Huntress, our partners or the community at large.

Q: What are some common challenges that you face while working?

With the large amount of freedom we have in our position, I think the hardest part is maintaining a good work/life balance. This is a common thread outside of just our small group, so we definitely are not alone here, but I think it’s an interesting problem in this case because it’s often self-inflicted. 

The people that end up in a role like our security researchers are often self-motivated people who genuinely love the field. I personally love writing code, and I will very easily work through on the night or even through a weekend on a project for work just because I genuinely enjoy doing it, and that kind of work is also my biggest hobby. But then, I’ll look back and realize I just worked through a whole weekend. I’m not upset about it. I genuinely enjoy it, but it’s something to keep an eye on for sure.

Q: What does collaboration look like with other members of the ThreatOps team (support, analysts, etc.)?

We have a pretty small team of researchers, so we often get on Zoom calls when we have no meetings. We call them our “Zoom jams.” There’s no agenda or schedule or anything. We’re just working while on a call, and we may chat about possible solutions to projects we’re working on or collaborate if there’s a project two or more of us are working on together. Or sometimes, we’ll just hang out and chat about life to maintain “the team” if you will. It’s been a great way to combat the occasional awkwardness of not working directly with your direct peers.

Q: What software and tools do you use in your job?

We do a lot of virtualization under R&D, whether it's for researching a new threat/vulnerability, analyzing a piece of malware or just testing software. I use KVM which is the built-in virtualization solution on Linux for all my virtualization.

I’m also constantly in a code editor. My personal editor of choice is Emacs.

Q: Do you have any unique tricks or strategies that help you as a security researcher?

For me personally, I think not being afraid to just dive into code is very helpful. Many things in the cyber field have poor documentation. Having the ability and confidence to just open someone else's code, read it and understand what is going on is very helpful when it comes to using a community tool or implementing a new feature.

This ties into another really helpful strategy, which is running with ideas as soon as you have them. Most of the time, if an idea comes up for improving or adding on to some piece of Huntress, I’ve found it’s best to just develop some kind of “proof-of-concept” and say “look, it could work like this.” I think this normally side-steps a lot of the common back and forth trying to explain what you mean and the hypotheticals of how it could impact X, Y or Z.

Q: Story time! From start to finish, what does a typical day look like for you?

After grabbing my caffeine starter pack, I normally sit down at my desk to look over meetings I have during the day. Next, I’ll normally take a look at Shortcut tickets to review my assigned projects and then dive into one of them. For me personally, a lot of my projects have a development/programming flair to them. I work a lot in Python for tools, Golang for agent tweaks and occasionally Ruby if I hop into any portal changes. From there, my day is pretty straightforward. I keep my head down as much as possible and work on my assigned projects.

Projects are assigned in a few different ways. The simplest and most common way is during our weekly R&D sync call. If there has been a problem or need identified within the company that we would like implemented by R&D, Jamie [the Director of R&D] will ask for volunteers for the project. Sometimes, there’s only one volunteer and you work on a project alone, while other times multiple R&D members sign up, and you work collaboratively. These projects almost always involve a lot of interaction and cooperation with other teams under Engineering or ThreatOps to make sure what we are building or implementing is right.

Occasionally, the R&D team will be tagged in Slack by analysts if they need more help looking at a particularly complicated piece of malware (or potential malware), and we will divert from what we are doing to help out there. This normally consists of hopping into an investigation and helping put the puzzle pieces together, decide whether a binary is malicious or track a malicious actor's actions throughout a network.

***

Dave Kleinatland

Caffeine mentions: Also 2, but does tea actually count?

Q: What’s the first thing you do when you’re ready to start your day?

Aside from ensuring there’s enough caffeine to get through the day’s tasks? 🤣 

My first actual task most days is to go through my list of pending tasks and prioritize what needs to be done “right now” versus “this week.”

Q: What’s the last thing you do when you’re ready to end your day?

Before I log off for the day, I make sure to update my notes and Shortcut tickets. Since most research tasks have a bit of a story arc and can’t be completed in a single workday, it’s important to update these things to ensure I can smoothly pick back up the next day.

Q: How have your education/training and past experiences paid off in your day-to-day role?

The mix of my previous experiences provides me with insight into the types of problems we’re trying to solve for our partners. I spent a lot of time trying to solve these types of issues on the scale of “my IT department” or “our MSP clients,” but at the vendor scale, I’m able to see how my experience and training apply across a wider scope. 

The most interesting thing to me is the bits of random experience that become useful are not always the expected ones. I never really expected that my years of Microsoft Exchange experience would come in handy here at Huntress, but then the Exchange vulnerabilities happened. That experience allowed me to get our test environments spun up very quickly and begin triaging how we could create detectors and defenses for it. 

Q: Do you have your own preferred methodology for doing your job?

Sometimes I wonder if there is a method to the madness—but at its core, it’s just complex problem-solving. 

A unique aspect is that we’ve got a bit of a mantra here: Solve for the first 80 percent first. If you try to scope something the full 100 percent right out of the gate, you might miss things. But you can solve that first 80 percent pretty quickly and then find the most important bits of that remaining 20 percent and focus there. 

Q: What’s your favorite part about being a security researcher?

I love that I’m always solving a “new” problem with the various teams. One of my roles in research is helping the product team shape new features and solutions for our partners. I really enjoy being able to provide them with my insights as a former target partner. 

Another huge favorite is being able to solve the “random complex problem du jour” (e.g. new vulnerability, new feature bug, etc.) with the rest of the R&D team. We’ve got a unique group of folks who all bring alternative perspectives to the table and we seem to be able to flesh out solutions pretty quickly when we work together. 

Q: What are some common challenges that you face while working?

I’d say there are two main things that I catch myself thinking about often in terms of challenges. 

One is the signal-versus-noise ratio from both our own ThreatOps perspective and from the partner’s perspective. The other comes down to scale. When working in the MSP space, I was hyper-aware of the alert fatigue problem that our partners face every day. Because I’ve lived this experience firsthand, I spend a lot of time trying to ensure our product doesn’t add to all the noise. 

As far as scale, I’ve never had to worry about more than a few thousand endpoints before joining Huntress—but now with 1M+ endpoints, we have to consider the challenges that presents as we try not to inundate ourselves with noise while still maintaining efficacy. 

Q: What does collaboration look like with other members of the ThreatOps team (support, analysts, etc.)?

For cross-department collaboration, I’d say we mostly use Slack as well as Shortcut. In our department, I’d say Slack and Zoom are the most popular. While working remotely certainly has its advantages, it also has some drawbacks of not being able to pile a bunch of minds together in a conference room and work through separate problems together. Collaborative tools are king to keep that vibe going. 

Q: What software and tools do you use in your job?

There are so many tools—probably hundreds. I’d say among the most popular are Amazon Web Services (AWS) and VMware Workstation for virtualization. A good code editor is a must—mostly Vim and VSCode for me. Tools like Docker and Ansible help with automating deployments of test machines and environments. 

As far as some of the security researcher-specific tools I’d say the Sysinternals suite stands out with tools like Autoruns, Procmon, Process Explorer and more. I also keep a good clean snapshot of both a Windows and Linux-based analysis machine with my favorite tools installed so I’ve always got a good starting base for tackling a problem. 

Q: Story time! From start to finish, what does a typical day look like for you?

“Typical” day? 🤣 Jokes aside, most days start off the same and are likely not atypical to how most people start their workday: caffeine (tea for me) and a review of Slack for unread messages. We’re a very Slack-oriented organization which allows for a much more collaborative environment than email. Some of our early-stage or prototype features send alerts and points of interest in Slack, and these get reviewed throughout the day depending on what projects I’m working on. 

Most days consist of a mix of project work and meetings. Typical meetings for me are a mix of our product and engineering teams as we primarily work on researching, shaping and testing new features for both our partners and internal ThreatOps staff. This can be something as simple as providing insight from my past experiences, expertise with the product itself or building out full-scale environments to test in. I also serve as an escalation point for our ThreatOps Center, as do a few of my other R&D colleagues. 

When most folks think of the R&D department, this is usually what pops in their mind first. While part of the escalation is often taking apart a complex malware sample, it’s also an excellent opportunity to mentor the team. I enjoy when I get a chance to teach/mentor/discuss these malware breakdowns with the team as it’s often very engaging. Additionally, these escalations are how we continue to improve our product—by identifying how they can be more easily identified and processed by the team. 

Another typical task I take part in—but thankfully not every day—is our Rapid Response efforts where various members across the entire Huntress team respond to larger-scale incidents. These incidents can be limited to one or more partner assets or larger global-scale incidents like SolarWinds, Kaseya, Microsoft Exchange or log4j. 

Other typical tasks include working on our ThreatOps AWS infrastructure, running through product feature testing for engineering, building concepts for new features like Microsoft 365 breach detection analysis and working on tweaking existing features like Managed Microsoft Defender. A much less exciting but far more common occurrence in the day-to-day is studying and reading lots of documentation for the things we are interfacing with, such as MS Defender, M365 APIs, AWS and more. As we expand Huntress as a platform and integrate with more things (or the same things in a wider scope), I find myself continually soaking up knowledge from wherever I can.

***

John Hammond

Caffeine mentions: 0, but we know he's down for an occasional Monster

Q: What’s the first thing you do when you’re ready to start your day?

The very first thing I do when I start work is review our internal Slack workspace for any urgent incidents, tasking or need for extra help with any analysis and reporting. I lightly review some security news and current events and cruise through emails before getting started on the real work!

Q: What’s the last thing you do when you’re ready to end your day?

Before finishing work, I try to set reminders for unfinished tasks to return to the next day and, if applicable, update team members on any recent progress made during the day.

Q: How have your education/training and past experiences paid off in your day-to-day role?

A lot of the professional training I’ve received through either industry certification efforts or general classes and workshops in cybersecurity are often directly applicable to work, especially when uncovering threat actor tradecraft and tactics for persistence and post-exploitation. Techniques used by real hackers are much easier to recognize and identify when you have a solid understanding of how the adversaries work—that is especially why we say “our offense is your defense”!

Q: Do you have your own preferred methodology for doing your job?

I enjoy syncing with my team members and oftentimes suggesting what we affectionately refer to as a “Zoom jam session,” where we can casually get together without an agenda but chip away at the work we need to get done together. This makes for a great feeling of inclusiveness while we work at a remote company, and we can constantly bounce ideas off each other or ask for help whenever we uncover something we don’t understand or just aren’t as familiar with. When we can screenshare and virtually “look over their shoulder” to better understand what our teammates are working on, we can better help or learn new things ourselves!

Q: What’s your favorite part about being a security researcher?

My favorite part about being a researcher is the flexibility and encouragement to work on cool things that interest me personally, with the greater purpose of improving our own product, service, detection capabilities or the security of our partners as a whole. 

When a new incident occurs, we can reverse engineer the entire attack chain to gain a better understanding. We can then prepare new detection capabilities, train our own team members and even recreate the attack chain if we are interested. Other times, we can build out new prototype functionality for our product, or dig into the latest security news and threats. 

The best part of being a researcher is that it often boils down to learning, tinkering, experimenting and trying to make new creative solutions as a professional occupation. Personally, I absolutely love being able to tell the stories of all the great work that we do for better community learning and greater security education. Helping shine the spotlight on our team for the fantastic research and analysis we bring forth is something I take a lot of pride in.

Q: What are some common challenges that you face while working?

I certainly struggle with the ability to really “sign off” for the day. There is a sort of strange joke in our industry about the idea of “if you do what you love, you never work a day in your life.” While that has some truth to it, it can be a bit of a double-edged sword when it blurs the line of what you do for fun and what you do for work. When you really enjoy what you are doing, the proverbial work day might run over into the late evening, or even the weekend. If I’m on vacation and a rapid response incident pops up, I have a certain fear of missing out (FOMO)!

Q: What does collaboration look like with other members of the ThreatOps team (support, analysts, etc.)?

While I often work with other members of the ThreatOps team in a collaborative Zoom session, we’re also messaging through Slack to send files back and forth—whether it be the next stage of the malware sample one of us has deobfuscated, or some Python scripts and code to automate some task, or access keys to jump into a temporary cloud instance for better testing—we are always sharing. When we are collaborating, it’s very cool to see a synergy of everyone’s different skillsets: we can divvy up different tasks or activities to the folks who might be strongest in one form of analysis, development or research efforts. 

Q: What software and tools do you use in your job?

I am almost always inside of a virtual machine while working. I will be bouncing back and forth between a REMnux install and a FLARE VM install for malware analysis, different renditions of Windows workstations or server operating systems for testing or development environments I have staged with different programming language compilers or interpreters—Golang and appropriate IDEs, Python interpreters, etc. When the team is working on something that requires a temporary collaborative cloud instance, we will spin up a machine with AWS or Digital Ocean. 

Q: Do you have any unique tricks or strategies that help you as a security researcher?

I am a huge fan of using automation to help make things faster and easier, so some of my tricks are trying to script out different tasks, or trying to use some aspect of a programming language to help unravel a problem in front of me. Alongside that, one super helpful strategy is not only knowing your strengths, but also your weaknesses—and subsequently knowing folks both on and off your team who can help. Having a growing network of connections in the cybersecurity industry is absolutely an invaluable thing, and that really helps get other opinions and input on research and development efforts!

Q: Story time! From start to finish, what does a typical day look like for you?

A typical day for me as a researcher starts with the usual review of any goings-on in our internal Slack workspace to see what the team might already be up to if anyone needs a hand with any analysis or projects they are working on. If there isn’t anything immediately needing attention, I will try and spend just a few minutes to look through security news or headlines to see what threats the rest of the industry might be tracking. As silly as it sounds to say a lot of that “cybersecurity news” comes from Twitter, I’ve come to accept that literally scrolling through Twitter for threat intelligence is a part of my job. 😅

Occasionally there have been some crazy “the-sky-is-falling” incidents that I will want early notification for (PrintNightmare and Log4Shell to name a few 😜), and I try to help spin up the team for rapid response. If there isn’t anything that seems like a wild emergency, I will tune into “regular work” mode for the day.

For me, personally, this really varies from the occasional marketing to-dos or PR efforts (building out technical presentations for an upcoming conference or event, speaking in a webinar, cranking out a blog post or chatting with reporters) alongside more traditional ThreatOps work.

When it comes to strictly ThreatOps work, the tasking is very dependent on what the overall company is focusing on. If we have a heavy push to get a new feature or service built out, I might be part of the team doing testing or helping prepare some proof-of-concept functionality. If we have a bit of breathing room between upcoming deadlines or big product initiatives, we can focus on ad-hoc projects that augment and improve what we already offer. This could be all across the board—I’ve spent some time creating detectors for more automated findings of malware and threats, helped script some interaction with our frontend portal for faster intel gathering and more. Most recently, I’ve been working on a project to take better inventory of “known goodware,” or software and applications that come by default in Windows. Projects like these are the most fun tasks, because I can get in the weeds with code and programming and try to think of how to solve a large problem. 

Truthfully, I will spend most of my time during the day chipping away at these projects, working alongside other researchers and potentially lending a hand with whatever they are up to. It is a team, through and through. If there are any ad-hoc requests or new tasking that pops up throughout the day, or meetings to attend to help shape or form our roadmap and process, I pivot to those.

***

Thanks to Dave, Caleb and John for jotting down some thoughtful responses and giving us some insight into what they do every day. We owe you guys a round of caffeine.