The Effects of Opening Move Selection on Investigation Speed

What follows is a shortened version of a longer paper that will be released at a later time. You can also learn more about this research by watching my recent Security Onion Conference 2016 video where I discuss these results and other similar experiments.


The core construct of computer network defense is the investigation. It’s here where human analysts receive anomalous signals and pursue evidence that might prove the existence of an information system breach. While little formalized methodology related to the investigation process exists, research in this area is beginning to emerge.

Existing research in human thought and knowledge acquisition is applicable to information security and computer network defense. Daniel Kahneman provided modern research in support of a dual-process theory of thinking that defines two separate processes governing human thought. The first process, called intuitive or system 1 thinking, is automatic and usually kicks in without directly acknowledging it. You don’t have to think about brushing your teeth or starting your car, you just do it. The second process, called reflective or system 2 thinking, is deliberate and requires attentive focus. You have to think about math problems and your next move when playing checkers. Both of these systems are in play during the investigation process.

In an investigation, analysts use intuitive thought when pursuing data related to an alert. This is most often the case when the analyst makes their opening move. The analyst’s opening move is the first data query they execute after receiving an alert. By virtue of being the first move, the analyst doesn’t apply a lot of explicit reflective thought on the issue and they simply jump to the place they believe will provide the quickest and most definitive answer. It’s assumed that in these cases, the analyst’s first move is probably the data source they perceive as being the most valuable.

The goal of the present research is to determine which common data source analysts were more likely to use as their opening move, and to assess the impact of that first move on the speed of the investigation.


The foundation of this research was a purpose built investigation simulator. The investigation simulator was built to recreate the investigation environment in a tool agnostic manner, such that individual scenarios could be loaded for a sample population and the variables could be tightly controlled.

A pool of security analysts was selected based on their employment history. Every analysts selected was currently or recently in a role were they were responsible for investigating security alerts to determine if a security compromise had occurred. Demographic information was collected, and analysts were placed into three skills groups based on their qualifications and level of experience: novice, intermediate, or expert.


Group A – Exploit Kit Infection

The primary experiment group was asked to connect to the investigation simulator remotely and work through the investigation scenario provided to arrive at a conclusion whether an infection or compromise had successful occurred.

The scenario presented the user with a Suricata IDS alert indicating that an internal host visited an exploit kit landing page.



Figure 1: The Suricata IDS alert that initiated the simulation

The following data sources were provided for investigation purposes:

Data Source Query Format Output Style
Full packet capture (PCAP) Search by IP or port TCPDump
Network flow/session data Search by IP or port SiLK IPFIX
Host file system Search by filename File path location
Windows Logs Option: User authentication/process create logs Windows event log text
Windows Registry Option: Autoruns/System restore/application executions (MUI cache) Registry keys and values
Antivirus Logs Search by IP Generic AV log text
Memory Option: Running process list/Shim cache Volatility
Open Source Intelligence Option: IP or domain reputation/file hash reputation/Google search Text output similar to popular intelligence providers

Table 1: Data source provided for Group A

Subjects were instructed that they should work towards a final disposition of true positive (an infection occurred), or false positive (no infection occurred). Whenever they had enough information to reach a conclusion, they were to indicate their final disposition in the tool, at which point the simulation exited.

The simulator logged every query the analysts made during this experiment, along with a timestamp and the start and end time. This produced a timeline of the analysts enter investigation, which was used to evaluate the research questions.


Group B – PCAP Data Replaced with Bro Data

Based on results achieved with group A, a second non-overlapping sample group of analysts were selected to participate in another experiment. Since group indicated a preference for higher context PCAP data, the second scenario removed the PCAP data option and replaced it with Bro data, another high context data source that is more structured and organized. The complete list of data sources provided for this group were:

Data Source Query Format Output Style
Bro Search by IP Bro
Network flow/session data Search by IP or port SiLK IPFIX
Host file system Search by filename File path location
Windows Logs Option: User authentication/process create logs Windows event log text
Windows Registry Option: Autoruns/System restore/application executions (MUI cache) Registry keys and values
Antivirus Logs Search by IP Generic AV log text
Memory Option: Running process list/Shim cache Volatility
Open Source Intelligence Option: IP or domain reputation/file hash reputation/Google search Text output similar to popular intelligence providers

Table 2: Data source provided for Group B

All experiment procedures and investigation logging measures remained in place, consistent with group A.

Group C – Survey Group

A third semi-overlapping group was selected at random to collect self-reported statistics to assess what opening move analysts self reported they would be more likely to make given a generic investigation scenario.

Using a combination of manually polling analysts and collecting responses from Twitter polling, analysts were asked the following question:

In a normal investigation scenario, what data source would you look at first?

The multiple-choice options presented were:

  1. PCAP
  2. Flow
  3. Open Source Intelligence
  4. Other


The first item evaluated was the distribution of opening moves. Simply put, what data source did analysts look at first?

In Group A, an 72% of analysts chose PCAP as their first move, 16% chose flow data, and the remaining 12% chose OSINT. The observed numbers differ significantly from the numbers analysts reported during information polling. In the Group C polling, 49% of analysts reported PCAP would be their first move, 28% chose flow data, and 23% chose OSINT.


Chart 1: Opening move selection observed for Group A

The mean time to disposition (MTTD) metric was calculated for each first move group by determining the difference between start and end investigation time for each analysts and averaging the results of all analysts within the group together. Analyst’s who chose PCAP had a MTTD of 16 minutes, those who chose flow had a MTTD of 10 minutes, and those who chose OSINT had a MTTD of 9 minutes.


Chart 2: Time to disposition for Group A


In Group B where PCAP data was replaced with Bro data, 46% of analysts chose Bro data as their first move, 29% chose OSINT, and 25% chose flow.


Chart 3: Comparison of group A and B opening moves

Analysts who chose Bro had a MTTD of 10 minutes, while those who chose flow and OSINT and MTTDs of 10 minutes and 11 minutes, respectively.


Chart 4: Comparison of group A and B average time to close


While not entirely conclusive, the data gained from this research does provide several suggestions. First, given an overwhelming 72% of people chose to begin their investigation with PCAP data, it’s clear that analysts prefer a higher context data source when its available, even if other lower context data sources available. In these simulations there were multiple ways to come to the correct conclusion, and PCAP data did not have to be examined at all to reach it.

The data also suggests that an opening move to a high context but relatively unorganized data source can negatively affect the speed an analyst reaches an appropriate conclusion. The MTTD for analysts whose opening move was PCAP in Group A was significantly higher than those who started with lower context data sources flow and OSINT. This is likely because PCAP data contains extraneous data that isn’t beneficial to the investigator, and it takes much longer to visually parse and interpret. Examining the results of the group B experiment further supports this finding. PCAP was replaced with Bro log data, which generally contains most of the same useful information that PCAP provides, but organizes it in a much more intuitive way that makes it easier to sift through. Analysts who chose Bro data for their opening move had a MTTD that was much lower than PCAP and comparable to flow and OSINT data sources.

The comparison between observed and reported opening moves highlights another finding that analysts often don’t understand their own tendencies during an investigation. There was a significant difference between the number of people who reported they would choose to investigate an anomaly with PCAP, and those who actually did. Opening move selection is somewhat situational however, so the present study did not introduce enough unique simulations to truly validate the statistics supporting that finding.

Possible limitations for this study mostly center on a limited number of trials, as only one simulation (albeit modified for one group) was used. More trials would serve to strengthen the findings. In addition, there is some selection bias towards analysts who are more specialized in network forensics than host forensics. This likely accounts for no first moves being to host-based data. Additionally, in the simulations conducted here access to all data sources took an equal amount of time. In a real world scenario, some data sources take longer to access. However, since PCAP and other higher context data sources are usually larger in size on disk, the added time to retrieve this data would only strengthen these findings that PCAP data negatively affects investigation speed.


Overall, this research provides insight into the value of better organized higher context data sources. While PCAP data contains an immense level of context, it is also unorganized, hard to filter and sift through compared to other data types, and has extraneous data not useful for furthering the investigation. To improve investigation efficiency, it may be better to make opening moves that start with lower context data sources so that a smaller net can be cast when it comes time to query higher context sources. Furthermore, when more organized higher context data sources are available, they should be used.

While the present research isn’t fully conclusive due to its sample size and a limited number of simulation trials, it does provide unique insight into the investigation process. The methods and strategy used here should be applicable for additional research to further confirm the things this data suggests.


Interested in learning more about the investigation process, choosing the right data sources to look at, and becoming a better analyst? Sign up here to be the first to hear about my new analyst training course being released later this year. Mailing list subscribers will get the first opportunity to sign up for the exclusive web-based course, and space is limited. Proceeds from this course will go to benefit charity.

Video: Tracking Investigations with Timelines

As humans, we rely on visualizing things to solve problems, even when we don’t realize it. In this video, I want to talk about how you can use timelines to visualize investigations. This is useful for tracking active investigations, retracing your steps and identifying gaps in your analysis, and relaying investigation output to management.

If you like this video, you’ll enjoy the course its a part of that I’m releasing in a few months. You can learn more about the course by signing up for my mailing list.

In this thirty minute video I illustrate the complexity of investigations and describe why visualizations are important. From there, I explain how timelines can fit this gap, and the types of events that are notable for tracking on a timeline. From there, I use VisJS to provide an example of how you can create simple timelines to track your investigations.

I’ve also included the following resources:

  • A sample timeline using VisJS
  • A directory structure and HTML page for managing timelines

You can download these resources here.

Accelerating Experience with Investigation Heuristics

ifthenelseWhy is someone who has been investigating security incidents for ten years so much better than someone who has only been doing it for a year?

That’s a simple question, and the simple answer is experience. As an analyst learns the fundamentals, develops a larger tool chest, and encounters more diverse scenarios they will naturally become better at their craft.

That’s straightforward, but consider these alternate scenarios. There are analysts who have been involved with security investigations for three years who are better than analysts who have been involved for ten years. Why is that? Furthermore, if there are two analysts with the same amount of experience, why would one analyst be better at investigating things than the other?

While we like to measure experience in units of time that is rarely an effective way to relate why an analyst is good at their job. Experience is related to expertise, but they don’t always directly correlate.

Today, I want to focus two elements particularly relevant to how expertise can be quantified between novice and expert analysts. These are rule-based reasoning and investigation heuristics.

Rule-Based Reasoning

I recently conducted a series of case studies where I brought in several security analysts of varying experience levels and asked them to describe a case they had worked. Through a technique known as the stimulated recall interview, I had them describe the process from beginning to end, focusing on why they took certain actions as the investigation progressed.

Once I collected a reasonable sample of these case studies, I reviewed each of them and performed a key phrase mapping exercise. I identified a list of categories based on a dual process theory model and mapped relevant statements made by the analyst to those categories. I was left with a distribution of how many responses existed in each category that I could divide based on various analyst demographics, like experience.

One category where there was a significant difference between the number of responses given be novice and expert analysts was rule-based reasoning. The expert analyst had nearly three times as many instances where rule-based reasoning was responsible for their actions.

Rule-based reasoning can be best thought of as an if-then-else statement. It’s a way that many believe humans store, retrieve, and manipulate knowledge, often leading to an action. Of course, as with several matters of the mind there are other theories too.

Regardless, it should come as no surprise that computers were designed to work using if-then-else statements, because computers are in some ways mankind’s attempt to recreate itself. It represents some of our most fundamental understanding of how we think and process information, and it can be demonstrated in all walks of life. Investigations are no different.

Consider the domain When you see that domain appear in an alert you immediately assume the alert is a false positive. This is because you’ve applied a rule like this:

  • If: Domain belongs to a well-known public company
  • Then: It’s probably not hosting malicious content
  • Else: It might have been victim of a strategic web compromise

Now consider the domain When you see this domain in an alert you immediately assume its evil. This could be the result of a rule like this:

  • If: Domain appears to be mostly random alphanumeric characters
  • Then: It might be generated by a domain name generation algorithm and/or owned by an attacker
  • Else: It could be a coincidence, and should be documented in case I run across it again

These are simple rules that can be articulated easily. Of course, not all rules are that cut and dry.

Even if you don’t realize it, any time you review evidence in an investigation you’re evaluating a set of rules to make decisions. Some of these are very deliberate (reflective thinking) and some of them are very automatic (intuitive thinking). These two types of thinking and how they relate define dual process theory.

With that said, a rule-based system is a simplification of something that is insanely more complex. We aren’t just dealing with a linear approach to information processing, but more likely with the activation of millions of neurons in a semantic network or some other form of connectionist model. That goes well beyond the scope of this article and some levels of the current state of human understanding. Although a simplification, a rule-based system is a reasonable one for how humans might take inputs, compare them against existing knowledge (see: top-down processing), and produce outputs.

Accelerating Experience

Given this perspective on rule-based reasoning, it should come as no surprise that expert analysts have a much larger library of rules than novice analysts. These rules can be gained through experience, but as I stated earlier, experience doesn’t correlate perfectly with expertise. Gaining expertise is more about optimizing the analyst’s ability to build mental rules than arbitrarily waiting for the passage of time.

Certainly experience provides more of an opportunity to learn things, but if we can identify those things then there is little reason they can’t be taught in a more direct manner. Practically, this means that it’s possible to accelerate the rate at which an analyst gains experience by subjecting them to an environment that is more suitable for the development of rules.

That’s one reason we get analysts with the same amount of experience but varying levels of expertise (ignoring natural disposition towards the work). One environment might support the development of rules better than another. Experience is accelerated in these environments.

Investigation Heuristics

A simple way to help analysts develop a bigger library of rules is to write them down. The infosec industry has done a poor job of this, as it’s not something you’ll find publicly available. Some organizations have invested in the creation of investigation playbooks, which are a step in the right direction.

To document investigation-focused mental rules, the same if-then-else framework discussed earlier can be applied. If it ain’t broke, don’t fix it. These are more appropriately called heuristics, which are rules used to make decisions, solve problems, or draw conclusions. Better said, heuristics are mental shortcuts to finding answers to questions.

A more formalized heuristic format looks like this:

Heuristic Name
Input: $evidence_type
     $evidence is/has/contains $observation


Each heuristic is given a name for quick reference. It also includes an input evidence type, because in general any investigative conclusion is drawn from some type of observation or analysis on evidence. In many cases, a heuristic could be relevant for input of multiple types of evidence, or may require multiple types.

From there, the if-then-else statement makes up the meat of the heuristic. Similar to normal if-then-else statements, these scenarios can be made infinitely more complex. Of course, the simpler they can be made the better. Humans are processing these, so they don’t have to be perfect or follow all the same guidelines as though we’d expect a computer to be able to interpret them. Here are a few examples.

Domain Fast Flux Heuristic

Input: Domain

  • If: Domain resolves to a large number of IP addresses with diverse registration ownership or geography in a very short period
  • Then: It is likely that the domain is attacker owned and exhibiting fast-flux characteristics.
  • Else: The domain could be owned by a hosting company.
  • Else: The observation could be a coincidence.


File Type Mismatch Heuristic

Input: File

  • If: A file received in an e-mail is identified as a specific type based on its extension, but static analysis identifies a different file type.
  • Then: It is probable that the file is malicious in nature.
  • Else: The observation could be a coincidence.


Isolated POST Heuristic 

Input: IP, URL

  • If: An external IP sends an HTTP POST to one of your web servers, but doesn’t send any HTTP GET requests during the same period.
  • Then: There is a possibility that the internal host has become infected with a web shell, and the communication represents malicious traffic.
  • Else: This could be normal behavior for the system.


These heuristics all share the fact that they probably aren’t strong enough indicators on their own to warrant detection alerts; at least, not as scale grows beyond the small business. They do make useful investigation heuristics given the appropriate input in another investigation, whether alert-driven or human-driven (as in hunting).

This is a simplified example of a structured heuristic, but there is room to add a lot of interesting metadata to this format. For example, adding reference points to specific techniques used to retrieve evidence. Another example would be adding confidence ratings to the conclusions. This is a great place to make use of words of estimative probability so analysts can approach the heuristic with the appropriate weight and scrutiny.

Ultimately, the format doesn’t matter too much as long as this fits into the investigative workflow seamlessly. If you are embracing the investigation method, this should fit well with the question-hypothesis-answer format. These heuristics serve the role of helping develop questions and hypotheses to existing questions. They can also be used to drive initial observations when the investigation takes the form of hunting.

As a Teaching Tool

In an ideal world, the industry rallies around a format for investigation heuristics that can be explained in both a narrative and programmatic form, a standard is developed, and large common bodies of knowledge could exist that teach people how to investigate things.

In reality, the information security industry isn’t great at standards, so it’s probably a bit of a pipe dream; but it’s okay to have goals. In the interim, just maintaining a simple wiki with these types of investigation shortcuts can provide a tremendous benefit to analysts in your environment attempting to gain expertise. Even in environments where you might be a one-man-army network administrator and security analysts, having the reference available and reviewing it within the context of an active investigation is a helpful. It’s a worthwhile up front time investment.

They goal of this article isn’t to give you a format for creating and storing investigation heuristics. Instead, it’s to introduce rule-based reasoning and how the familiar construct of the if-then-else statement can be used to represent investigation shortcuts. It’s up to you to find the best way you can capture and represent this information for your own development, and the nurturing of analysts on your team.



How Analysts Approach Investigations

A  challenge facing information security is our inability to effectively train new analysts. The majority of security knowledge is tacit. We have plenty of practitioners who are good at catching bad guys, but most of them can’t articulate how they do it. I believe that overcoming this issue requires a focus on fundamental thought processes underlying security investigations, which is the foundation of my doctoral research.

Every major thought-based profession has a core construct through which everything is framed. For doctors, it’s the patient case. From this stems the diagnostic process, testing frameworks, and treatment plans. For lawyers, it’s the legal case. From this stems the discovery exercise, the trial, and sentencing. These core constructs are defined as an entities whose whole is greater than the sum of their parts. Each one is a story all its own.

In information security, our core construct is the investigation case. Everything we do is based on determining if malicious activity has happened, and to what extent. I don’t think many would argue this point, but surprisingly, there is very little formal writing out there about the investigation process itself. Many texts gloss over it and merely consider in the sum of its parts, a basic container for related evidence.

I propose that the investigation is so much more.

The Investigation Method

The investigation is at the heart of information security. It is a living, beating thing through which all of our actions are motivated and framed. It is our lens. To understand the investigation you must understand how humans think.

  1. Perception is not reality. What we perceive as reality and what actually exists are two separate things separated by our ability to interpret sensory input and using higher order reasoning. The process of getting from an initial perception to an accurate depiction of reality is the basis for learning and cognition.
  2. Learning comes from questioning. Straight from the womb, humans learn by questioning their environment, themselves, and their limits. By asking questions and employing various techniques to find answers we learn to move, walk, talk, and think. These techniques range from simple experimentation to complex reasoning, and can be motivated by primal needs like food and water, or higher order needs like achievement or respect.
  3. Our biases are always present. There are countless barriers that limit our ability to get from perception to reality. The most dangerous of these is our own mindset and the biases that are inherent to it. Humans are opinionated, and the same questions that drive us toward the pursuit of reality also drive opinions. When those opinions are educated and conscious they are hypotheses, and when none of those conditions are met they are guesses, and more subject to limiting bias.

If you consider this knowledge of human psychology, it begins to paint a picture of an investigation. Instead of trying to create a framework that dictates how investigations should be done, I wanted to take an approach the uncovers how you approach investigations as a form of learning. After all, that’s basically what an investigation is. It’s all about bridging the gap between perception and reality by learning facts. This yields the following definition and method.

“An investigation is the systematic inquiry and examination of evidence and observations in an effort to gain an accurate perception of whether an incident has occurred, and to what extent.”

The Investigation Process

If this looks familiar to you, that’s because it’s not too different from the scientific method. In a similar manner, the scientific manner wasn’t thought up as some way that scientific discovery should be done; it is an identification how most scientific discovery is done based on how humans learn. Even if scientists don’t intentionally set out to use the scientific method, their subconscious mind is doing it. The scientific method is responsible for the vast majority of scientific discovery. The investigation method is similarly responsible for the discovery of network intruders.

The investigation method contains five parts. I’ll briefly cover them here, although each one is worthy of its own article which will come later.


Every investigation begins with some observation that arouses suspicion. This is often machine generated in the form of an IDS alert, but could also be human driven in the form of an observation made while hunting. It doesn’t have to be an internal observation, and may come from a third-party notification. The tactics of the investigation are often shaped by the source of the initial observation, but the general process remains the same.

  • An observation is usually based on some form of initial evidence.
  • An observation can come from anywhere, but should be supportable. Even hunches or gut feelings are supportable when framed appropriately.
  • The first goal of the investigation is usually to validate or invalidate the initial observation as the premise of the investigation. If that observation isn’t valid, the investigation may not need to progress.


An investigation consists of a series of questions for which the analyst must seek answers. Based on the initial observation, the overarching questions will likely be some version of “Did a breach occur?” or “Is this malicious?” To answer those questions, more questions must be asked. Answers to one question will usually generate more questions. At any given point, an analyst should be able to articulate what question they’re trying to answer.

  • The ability to define good questions increases with experience because expert analysts have a larger pool of heuristics (rules) to draw from.
  • Most questions are centered around uncovering relationships, because ultimately it’s the relationships between devices and users that define an attack or breach.
  • Newer analysts will frequently begin answer seeking activities without clearly identifying the question they are attempting to answer. This can lead to wasted effort, but usually diminishes with experience.


You’re usually already slanted towards a specific answer from the moment you define your question, even if you don’t realize it. Your opinion forms based on your mindset, and is shaped by the entirety of your experience, both personal and professional. This is also where bias lives in the investigation process. The ability to articulate a hypothesis is an ideal way to expose bias so that your assumptions can be challenged if necessary. It also provides a clear path to additional questions that can validate or invalidate your hypothesis. Collectively, this leads to better, stronger conclusions.

  • Most hypothesis generation is passive and occurs subconsciously. A trick to making this an active process is to form an “I believe” statement for a hypothesis in response to each question. I believe ______ because _______.
  • Ideally a hypothesis is an educated guess. If you cannot complete the last half of the because statement, your assumptions may be from a place of bias, inexperience, or an inability to articulate well.
  • Every question should provide opportunity for a hypothesis, even if it’s a null hypothesis stating that a scenario isn’t probable.


The area of investigation most analysts are familiar with is answer seeking. It involves familiar tasks like retrieving, manipulating, and reviewing data. Any time you analytically review data or perform research it’s because you’re seeking an answer to your questions, usually to prove or disprove a hypothesis. Traditionally, newer analysts usually learn answer seeking before anything else which explains why the learning curve is so steep. They are trying to find answers for questions they don’t fully understand.

  • The goal of every answer isn’t to solve the investigation, it’s often to provide an opportunity for more questions. The answers you find will only be as good as the question they’re trying to resolve.
  • While it may seem logical to seek answers that prove a hypothesis, seeking to disprove a hypothesis is usually a much faster route to better questions.
  • Some questions won’t be answerable due to a lack of visibility or not enough data retention. Inability to answer a question is notable, because it might have impact on the investigation later. An unanswered question does not equal an invalid hypothesis.


The conclusion of an investigation is its terminal point. The investigation can terminate as a false positive alert, an acceptable risk, a simple malware infection, or a large breach requiring coordinated incident response. When a terminal disposition has been made, the investigation will contain a series of questions, hypotheses, and answers that uncover a (hopefully) accurate representation of events as they have occurred.

  • The strength of conclusions should always be accurately depicted by using estimative language. Certainties should be cited as such and backed up with evidence. Analytic opinions should be weighted based on their estimated certitude and available evidence.
  • If the steps that led you to a conclusion are considered carefully and documented well throughout the process, it should ease the burden of citing supporting information when documenting conclusions.

Framing an Investigation

Let’s look at example of what an investigation looks like through the lens of the investigation method. In this case, our fictional analyst has received an alert from an intrusion detection system.

Initial Observation: IDS Alert – User account was added to a domain admin group

This alert represents activity that might be legitimate, but could be malicious if it was unauthorized. The first question that generally follows an alert of this nature is whether it is malicious or normal activity.

Question 1: Does this alert represent malicious activity?

If the analyst were in a small organization they might be aware of any changes like this that should be occurring. Our analyst works in a very large enterprise, so it’s entirely possible that someone made this change for a legitimate reason without the analyst knowing. Because of this, the analyst believes its legitimate activity.

Hypothesis 1: I believe this is legitimate activity because this is something that happens frequently within the organization. 

To answer the initial question, the analyst must prove or disprove the hypothesis. To do this, more questions must be asked. There are a number of routes the analyst could go here, but one many analysts would pursue relates to follow-up actions taken by the user account.

Question 2: What actions did the user account take after being added to the admin group?

Based on the earlier hypothesis that this is normal behavior, it’s likely the hypothesis to Q2 will be similar.

Hypothesis 2: I believe the account participated in legitimate admin activity because it supports hypothesis 1. 

Seeking an answer to Q2 should be fairly easily with adequate visibility into your system and network logs. The analyst is able to search through logs fed into his SIEM and determine that the user account in question logged into a workstation, opened Outlook, and mounted several C-level executives mailboxes from the Exchange mail server.

Answer 2: The user account logged into a workstation, opened Outlook, and mounted several C-level executives mailboxes from the Exchange mail server.

The answer to Q2 appears to disprove our hypothesis 2, which in turn disproves hypothesis 1. The activity exhibited by the user account is definitely malicious, and answers our first question.

Answer 1: The actions taken by the user account after being added to the domain admin group are malicious in nature due to unauthorized access to multiple sensitive mailboxes.

At this point, the analyst is confident a breach has occurred, and the investigation can continue with that in mind. This should bring up more questions as the investigation evolves, including:

  • Was the user account an existing user account whose credentials were compromised?
  • Are there any indicators of compromise on the workstation normally used by the user who owns this account?
  • How did the potential attacker gain enough access to be able to promote the compromised account into an admin group?
  • How did the user account gain access to the workstation used to mount the Exchange mailboxes?
  • Is there any malware installed on the workstation the mailboxes were mounted from?
  • Were any other accounts accessed from the system belonging to the owner of the compromised account?

As you can see, what I’ve articulated here is only a fraction of what could be a much larger investigation. The key takeaway is that it provides a very structured, easy to follow timeline of the investigation and how it progressed. This makes it much easier to review the investigation process from beginning to end, and to use this investigation as a teaching tool for novice analysts.


As a Universal Method

The investigation method is a universal construct within information security. While the industry often glamorizes unique subspecialties like hunting and malware analysis, they all fit within the same scope of activities. The method still applies.

For example, consider threat hunting. It follows the same process to bridge the gap from perception to reality. The only difference is that the initial observation is usually human-driven. Instead of receiving an IDS alert or an external notification, the analyst asks broad questions based on their library of experience-derived heuristics. The goal of this questioning is for the answers to generate more questions, or lead to the discovery of evidence that represents malicious activity.

This isn’t to say that subspecialties don’t require unique skill sets. They most certainly do. A hunter is usually someone more experienced because they have a larger library of investigative heuristics to work from, which allows them to be more effective at coming up with questions that can drive the discovery of interesting observations. A novice analyst wouldn’t have nearly as many heuristics to rely on, and their efforts would be less fruitful.

The characteristics of a good analyst will vary based on specialization, but the method is universal.

Why It Matters

The investigation method isn’t provided as a framework. The truth is that this is the method you likely already use to investigate security events, even if you aren’t aware of it. That awareness is key, because it gives practitioners a language to express their knowledge. From this comes more insightful analysis, more clearly identified methods that lead to conclusions, and an ability to teach novice analysts how investigations can be performed through the lens of an expert.

If you walk into a hundred SOCs you will find a hundred ways of documenting investigations. There is no standard, and worse yet, most end up adopting whatever format their tooling provides. What happens is that ticketing systems and wikis end up defining how analysts perform investigations. This is tragic.

If you walk into those same hundreds SOC’s, you’ll also typically only find one way of teaching people how investigations should be done — through on the job observation. While observation-based training is a key component of any training program, an education that is founded entirely on observation is sure to fail. I wouldn’t want a surgeon who skipped medical school and went straight to residency to be operating on me. Sure, they might be able to get the job done, but they’ll be missing the fundamentals that make them flexible and prepared for the inevitable unknown.

This is one significant reason why defenders are so badly outpaced by attackers in information security. Our profession hasn’t gone through its cognitive revolution where we seek to understand how we approach the investigation and it’s components. If we want to get there, understanding human thought and the methods that form the investigation are key. This article seeks to shed light in some of those areas, and certainly the articles to follow will as well.

I’d encourage you to consider the method shown here and think through it as you perform your investigations. What questions are you asking? How are your hypotheses swaying your analysis? How strong are your conclusions? How do you express how you approach investigations? These are all useful questions and are pivotal in your own understanding of the craft, as well as those who will come after you.


Writing for Security: Action Items that Provoke Change

quillMost people don’t realize it, but the success of what you write will probably be measured by how actionable it is. I’ve read hundreds of security assessments and forensic reports that go into a perfect level of detail, only to find that they fall short of delivering what every report needs: something actionable.

Imagine watching a great movie. They’ve done a wonderful job developing complex characters, the plot engagingly builds, and you’re on the edge of your seat the entire time. Right as the climax is happening and the story is coming to it’s pivotal point…the credit start rolling. It’s over. Although you might have enjoyed the couple of hours you invested up unto that point, you’re going to walk away with a bad taste in your mouth because you were robbed of a satisfactory conclusion. We all know movies like this, and usually chalk it up to lazy writing. This is exactly what you’re doing when you write without providing something actionable.

Whether you’re writing a security assessment report or an incident response report, your purpose isn’t merely to inform, it’s also to persuade. It isn’t enough that someone knows there is a vulnerability on their network. They have to be persuaded to implement controls that mitigate the risk of that vulnerability. It doesn’t matter if your forensic report does a good job explaining how an attacker got in. It has to persuade the reader to implement the necessary process changes or install the right tools to prevent it from happening again.

There are plenty of techniques you can use to be persuasive when you write, but before you do that you must identify what you want the reader to do. These are your action items, and the ability to identify them is what makes you an expert. Plenty of people can find vulnerabilities or find evidence of an attacker, but if you can’t identify actions to mitigate the risk associated with those findings then you’re report isn’t useful.

Identifying action items is all about mitigating risk. You should give the reader advice that prevents bad guys from doing some thing, or detects when they do it. You should always do both when possible.

Prevention Action Items

Prevention is as simple as making changes that keep bad things from happening. If you can give your reader steps they can take that prevent an attacker from doing something, that’s usually a win.

In reporting, I like to conceptualize change in terms of how difficult it is to accomplish. After all, it’s a lot harder to persuade someone to make a change if it’s going to be insanely difficult. Part of good writing is being honest with your readers, so it helps if to identify the level of effort required with a requested change. If it’s going to be easy you should make that clear so the reader is compelled to do it quickly. If it’s going to be difficult, be up front about and break it into steps. Your readers will appreciate this and will trust you more.

Changes will typically be categorized in terms of people, process, and technology.

  • People: Changing mindsets, providing training, hiring new staff, replace existing staff.
  • Process: Changing the way human or tech-centric things are done, adding new processes.
  • Technology: Configuration changes, additional software, new technology.

In most case, technology changes will be the easiest and people changes will be the hardest. It’s easy to manipulate systems, but it’s very difficult to acquire new people or change the way existing people think. The latter requires a lot more political and financial capital. This hierarchy of difficulty is how you should approach identifying prevention actions in your report. You should also report them in order of easiest to hardest within each individual finding.

In a lot of cases, some changes might touch all three areas. For example, building a security operations team requires hiring new people, building new processes, and implementing new technology. These massive changes should be saved for last and you should provide plenty of ancillary resources for the reader, as they will often involve topics that need to be covered in much larger depth.

When you are ready to start identifying action items, it’s helpful to ask yourself these three questions, filling in the blanks with the pertinent details from the finding you’re addressing:

  1. Are there any changes that can be made to prevent an attacker from __________?
  2. Is there anything new that can be done to prevent an attacker from __________?
  3. Is there anything that should be stopped in order to prevent an attacker from __________?

Let’s look at some examples of common findings and their action items. Notice that some action items combine categories, and some categories aren’t present.


Security Assessment: Web Server – Utilizing Plaintext Authentication

  • Technology: Change authentication method


Security Assessment: Local Windows Admin Account – No Password Rotation

  • Technology: Purchase password management software
  • Process: Institute manual change process


Incident Report: Attacker Guessed VPN Password

  • Technology: Institute lockout after three failed authentication attempts. Enforce stronger password requirements and more frequent rotation.
  • Technology + Process: Implement two-factor authentication.
  • People: Train users to use passwords that can’t be easily guessed


Incident Report: Workstation Compromised Because User Clinked Phishing Link

  • Technology: Install an e-mail threat protection appliance.
  • Process: Force users to use non-admin accounts and escalate privileges when administrative actions are needed.
  • People: Provide phishing awareness training to users.


Incident Report: Attacker Moved Laterally with Ease Due to Flat Network

  • Technology: Architect network for better segmentation.


These are just examples, but you can see where we started with technology and moved towards people. In most cases, the technology solution is going to be the easiest to implement in terms of labor hours. Of course, this doesn’t mean that a technology solution is always the best, but it is a step in the right direction. You want to give the reader the easy wins so they are more compelled to keep working towards to bigger wins. The first step is the hardest to take.

Detection Action Items

Detective controls are designed to detect when bad things happen. The vast majority of reports you’ve read probably don’t include them, which is a shame. Whenever you are making preventive recommendations, you should also make detective recommendations. There are a few reasons why:

  • Many organizations won’t be able to implement protective changes in a timely manner, or at all due to political or budgeting restraints.
  • Prevention eventually fails, and a key tenant of security is having multiple layers of controls.
  • The findings you’ve identified may have already been exploited, and an ability to retroactively detect this can help uncover a breach.

If you’re a consultant, writing detection action items can be difficult because there are a wide array of detection technologies. It’s hard to tailor detection content exclusively for a single customer without an intimate knowledge of their detection strategy. As a place to start, consider asking your client about their detection strategy and relevant technologies so that you can tailor your recommendations to them. This can be a part of the initial scoping call.

If you’re findings are related to your own network, or you’re writing a blog post, it’s a lot easier provide detection action items based on the precise technologies your using, or at a minimum prevailing open source standards. You can start with these questions:

  1. Are there any network-based indicators that can be used for detection?
  2. Are there any network-based behaviors that can be used for detection?
  3. Are there any host-based indicators that can be used for detection?
  4. Are there any host-based behaviors that can be used for detection?

This isn’t all encompassing, but in a lot of cases you will be able to derive some type of host or network based indicators or behaviors. An indicator can be something simple like a list of MD5’s or domain names, and will usually be representative of known evil. A behavior is usually more complex and will indicate a behavior that is normally legitimate, but could be the results of an attackers actions in some cases. This might be an action like a user account being added to an administrative group, or the use of the command “ping -n 1”. Both normal activities, but something that might not be done too often and worth of investigation in relation to the identified activity as it relates to the attacker or breach you’re describing.

In all of these cases, recommendations towards specific technologies are what will differentiate you. Don’t just give someone a list of domain names, also give them a Snort or Suricata rule that will detect them, including relevant context and information links. Don’t just give someone malware characteristics, give them the YARA rule to search for it. You might think that’s time consuming, and you’re right. Don’t be lazy! If you truly want to promote change and you expect your reader to go the extra mile, you’re writing has to do it as well. Something as small as creating a 10 line bash script to detect something will endear your reader/client to you forever, and will show your hands-on expertise.

More on Writing

If the things you write require the user to take action, you’re going to have to work harder to get them to do it. Just because you’ve written a very clear and informative statement on what a problem is and how to fix it doesn’t necessarily mean someone will take the action you want. The easier you can make this for people, the more they will be likely to actually pursue your recommendations. Going the extra mile in your writing will be rewarded with actions. If you can outline some prevention and detection action items, you’ll be writing content that will get people moving.

If you’re interested in learning more about my personal systems for better technical writing, I’ll be releasing more articles in that area soon, as well as a couple of videos. You can subscribe to the mailing list below to get access to that content first, along with a few exclusives that won’t be on the site.

Sign Up for the Mailing List Here