Category Archives: Investigations

The Effects of Opening Move Selection on Investigation Speed

What follows is a shortened version of a longer paper that will be released at a later time. You can also learn more about this research by watching my recent Security Onion Conference 2016 video where I discuss these results and other similar experiments.


The core construct of computer network defense is the investigation. It’s here where human analysts receive anomalous signals and pursue evidence that might prove the existence of an information system breach. While little formalized methodology related to the investigation process exists, research in this area is beginning to emerge.

Existing research in human thought and knowledge acquisition is applicable to information security and computer network defense. Daniel Kahneman provided modern research in support of a dual-process theory of thinking that defines two separate processes governing human thought. The first process, called intuitive or system 1 thinking, is automatic and usually kicks in without directly acknowledging it. You don’t have to think about brushing your teeth or starting your car, you just do it. The second process, called reflective or system 2 thinking, is deliberate and requires attentive focus. You have to think about math problems and your next move when playing checkers. Both of these systems are in play during the investigation process.

In an investigation, analysts use intuitive thought when pursuing data related to an alert. This is most often the case when the analyst makes their opening move. The analyst’s opening move is the first data query they execute after receiving an alert. By virtue of being the first move, the analyst doesn’t apply a lot of explicit reflective thought on the issue and they simply jump to the place they believe will provide the quickest and most definitive answer. It’s assumed that in these cases, the analyst’s first move is probably the data source they perceive as being the most valuable.

The goal of the present research is to determine which common data source analysts were more likely to use as their opening move, and to assess the impact of that first move on the speed of the investigation.


The foundation of this research was a purpose built investigation simulator. The investigation simulator was built to recreate the investigation environment in a tool agnostic manner, such that individual scenarios could be loaded for a sample population and the variables could be tightly controlled.

A pool of security analysts was selected based on their employment history. Every analysts selected was currently or recently in a role were they were responsible for investigating security alerts to determine if a security compromise had occurred. Demographic information was collected, and analysts were placed into three skills groups based on their qualifications and level of experience: novice, intermediate, or expert.


Group A – Exploit Kit Infection

The primary experiment group was asked to connect to the investigation simulator remotely and work through the investigation scenario provided to arrive at a conclusion whether an infection or compromise had successful occurred.

The scenario presented the user with a Suricata IDS alert indicating that an internal host visited an exploit kit landing page.



Figure 1: The Suricata IDS alert that initiated the simulation

The following data sources were provided for investigation purposes:

Data Source Query Format Output Style
Full packet capture (PCAP) Search by IP or port TCPDump
Network flow/session data Search by IP or port SiLK IPFIX
Host file system Search by filename File path location
Windows Logs Option: User authentication/process create logs Windows event log text
Windows Registry Option: Autoruns/System restore/application executions (MUI cache) Registry keys and values
Antivirus Logs Search by IP Generic AV log text
Memory Option: Running process list/Shim cache Volatility
Open Source Intelligence Option: IP or domain reputation/file hash reputation/Google search Text output similar to popular intelligence providers

Table 1: Data source provided for Group A

Subjects were instructed that they should work towards a final disposition of true positive (an infection occurred), or false positive (no infection occurred). Whenever they had enough information to reach a conclusion, they were to indicate their final disposition in the tool, at which point the simulation exited.

The simulator logged every query the analysts made during this experiment, along with a timestamp and the start and end time. This produced a timeline of the analysts enter investigation, which was used to evaluate the research questions.


Group B – PCAP Data Replaced with Bro Data

Based on results achieved with group A, a second non-overlapping sample group of analysts were selected to participate in another experiment. Since group indicated a preference for higher context PCAP data, the second scenario removed the PCAP data option and replaced it with Bro data, another high context data source that is more structured and organized. The complete list of data sources provided for this group were:

Data Source Query Format Output Style
Bro Search by IP Bro
Network flow/session data Search by IP or port SiLK IPFIX
Host file system Search by filename File path location
Windows Logs Option: User authentication/process create logs Windows event log text
Windows Registry Option: Autoruns/System restore/application executions (MUI cache) Registry keys and values
Antivirus Logs Search by IP Generic AV log text
Memory Option: Running process list/Shim cache Volatility
Open Source Intelligence Option: IP or domain reputation/file hash reputation/Google search Text output similar to popular intelligence providers

Table 2: Data source provided for Group B

All experiment procedures and investigation logging measures remained in place, consistent with group A.

Group C – Survey Group

A third semi-overlapping group was selected at random to collect self-reported statistics to assess what opening move analysts self reported they would be more likely to make given a generic investigation scenario.

Using a combination of manually polling analysts and collecting responses from Twitter polling, analysts were asked the following question:

In a normal investigation scenario, what data source would you look at first?

The multiple-choice options presented were:

  1. PCAP
  2. Flow
  3. Open Source Intelligence
  4. Other


The first item evaluated was the distribution of opening moves. Simply put, what data source did analysts look at first?

In Group A, an 72% of analysts chose PCAP as their first move, 16% chose flow data, and the remaining 12% chose OSINT. The observed numbers differ significantly from the numbers analysts reported during information polling. In the Group C polling, 49% of analysts reported PCAP would be their first move, 28% chose flow data, and 23% chose OSINT.


Chart 1: Opening move selection observed for Group A

The mean time to disposition (MTTD) metric was calculated for each first move group by determining the difference between start and end investigation time for each analysts and averaging the results of all analysts within the group together. Analyst’s who chose PCAP had a MTTD of 16 minutes, those who chose flow had a MTTD of 10 minutes, and those who chose OSINT had a MTTD of 9 minutes.


Chart 2: Time to disposition for Group A


In Group B where PCAP data was replaced with Bro data, 46% of analysts chose Bro data as their first move, 29% chose OSINT, and 25% chose flow.


Chart 3: Comparison of group A and B opening moves

Analysts who chose Bro had a MTTD of 10 minutes, while those who chose flow and OSINT and MTTDs of 10 minutes and 11 minutes, respectively.


Chart 4: Comparison of group A and B average time to close


While not entirely conclusive, the data gained from this research does provide several suggestions. First, given an overwhelming 72% of people chose to begin their investigation with PCAP data, it’s clear that analysts prefer a higher context data source when its available, even if other lower context data sources available. In these simulations there were multiple ways to come to the correct conclusion, and PCAP data did not have to be examined at all to reach it.

The data also suggests that an opening move to a high context but relatively unorganized data source can negatively affect the speed an analyst reaches an appropriate conclusion. The MTTD for analysts whose opening move was PCAP in Group A was significantly higher than those who started with lower context data sources flow and OSINT. This is likely because PCAP data contains extraneous data that isn’t beneficial to the investigator, and it takes much longer to visually parse and interpret. Examining the results of the group B experiment further supports this finding. PCAP was replaced with Bro log data, which generally contains most of the same useful information that PCAP provides, but organizes it in a much more intuitive way that makes it easier to sift through. Analysts who chose Bro data for their opening move had a MTTD that was much lower than PCAP and comparable to flow and OSINT data sources.

The comparison between observed and reported opening moves highlights another finding that analysts often don’t understand their own tendencies during an investigation. There was a significant difference between the number of people who reported they would choose to investigate an anomaly with PCAP, and those who actually did. Opening move selection is somewhat situational however, so the present study did not introduce enough unique simulations to truly validate the statistics supporting that finding.

Possible limitations for this study mostly center on a limited number of trials, as only one simulation (albeit modified for one group) was used. More trials would serve to strengthen the findings. In addition, there is some selection bias towards analysts who are more specialized in network forensics than host forensics. This likely accounts for no first moves being to host-based data. Additionally, in the simulations conducted here access to all data sources took an equal amount of time. In a real world scenario, some data sources take longer to access. However, since PCAP and other higher context data sources are usually larger in size on disk, the added time to retrieve this data would only strengthen these findings that PCAP data negatively affects investigation speed.


Overall, this research provides insight into the value of better organized higher context data sources. While PCAP data contains an immense level of context, it is also unorganized, hard to filter and sift through compared to other data types, and has extraneous data not useful for furthering the investigation. To improve investigation efficiency, it may be better to make opening moves that start with lower context data sources so that a smaller net can be cast when it comes time to query higher context sources. Furthermore, when more organized higher context data sources are available, they should be used.

While the present research isn’t fully conclusive due to its sample size and a limited number of simulation trials, it does provide unique insight into the investigation process. The methods and strategy used here should be applicable for additional research to further confirm the things this data suggests.


Interested in learning more about the investigation process, choosing the right data sources to look at, and becoming a better analyst? Sign up here to be the first to hear about my new analyst training course being released later this year. Mailing list subscribers will get the first opportunity to sign up for the exclusive web-based course, and space is limited. Proceeds from this course will go to benefit charity.

Video: Tracking Investigations with Timelines

As humans, we rely on visualizing things to solve problems, even when we don’t realize it. In this video, I want to talk about how you can use timelines to visualize investigations. This is useful for tracking active investigations, retracing your steps and identifying gaps in your analysis, and relaying investigation output to management.

If you like this video, you’ll enjoy the course its a part of that I’m releasing in a few months. You can learn more about the course by signing up for my mailing list.

In this thirty minute video I illustrate the complexity of investigations and describe why visualizations are important. From there, I explain how timelines can fit this gap, and the types of events that are notable for tracking on a timeline. From there, I use VisJS to provide an example of how you can create simple timelines to track your investigations.

I’ve also included the following resources:

  • A sample timeline using VisJS
  • A directory structure and HTML page for managing timelines

You can download these resources here.

Accelerating Experience with Investigation Heuristics

ifthenelseWhy is someone who has been investigating security incidents for ten years so much better than someone who has only been doing it for a year?

That’s a simple question, and the simple answer is experience. As an analyst learns the fundamentals, develops a larger tool chest, and encounters more diverse scenarios they will naturally become better at their craft.

That’s straightforward, but consider these alternate scenarios. There are analysts who have been involved with security investigations for three years who are better than analysts who have been involved for ten years. Why is that? Furthermore, if there are two analysts with the same amount of experience, why would one analyst be better at investigating things than the other?

While we like to measure experience in units of time that is rarely an effective way to relate why an analyst is good at their job. Experience is related to expertise, but they don’t always directly correlate.

Today, I want to focus two elements particularly relevant to how expertise can be quantified between novice and expert analysts. These are rule-based reasoning and investigation heuristics.

Rule-Based Reasoning

I recently conducted a series of case studies where I brought in several security analysts of varying experience levels and asked them to describe a case they had worked. Through a technique known as the stimulated recall interview, I had them describe the process from beginning to end, focusing on why they took certain actions as the investigation progressed.

Once I collected a reasonable sample of these case studies, I reviewed each of them and performed a key phrase mapping exercise. I identified a list of categories based on a dual process theory model and mapped relevant statements made by the analyst to those categories. I was left with a distribution of how many responses existed in each category that I could divide based on various analyst demographics, like experience.

One category where there was a significant difference between the number of responses given be novice and expert analysts was rule-based reasoning. The expert analyst had nearly three times as many instances where rule-based reasoning was responsible for their actions.

Rule-based reasoning can be best thought of as an if-then-else statement. It’s a way that many believe humans store, retrieve, and manipulate knowledge, often leading to an action. Of course, as with several matters of the mind there are other theories too.

Regardless, it should come as no surprise that computers were designed to work using if-then-else statements, because computers are in some ways mankind’s attempt to recreate itself. It represents some of our most fundamental understanding of how we think and process information, and it can be demonstrated in all walks of life. Investigations are no different.

Consider the domain When you see that domain appear in an alert you immediately assume the alert is a false positive. This is because you’ve applied a rule like this:

  • If: Domain belongs to a well-known public company
  • Then: It’s probably not hosting malicious content
  • Else: It might have been victim of a strategic web compromise

Now consider the domain When you see this domain in an alert you immediately assume its evil. This could be the result of a rule like this:

  • If: Domain appears to be mostly random alphanumeric characters
  • Then: It might be generated by a domain name generation algorithm and/or owned by an attacker
  • Else: It could be a coincidence, and should be documented in case I run across it again

These are simple rules that can be articulated easily. Of course, not all rules are that cut and dry.

Even if you don’t realize it, any time you review evidence in an investigation you’re evaluating a set of rules to make decisions. Some of these are very deliberate (reflective thinking) and some of them are very automatic (intuitive thinking). These two types of thinking and how they relate define dual process theory.

With that said, a rule-based system is a simplification of something that is insanely more complex. We aren’t just dealing with a linear approach to information processing, but more likely with the activation of millions of neurons in a semantic network or some other form of connectionist model. That goes well beyond the scope of this article and some levels of the current state of human understanding. Although a simplification, a rule-based system is a reasonable one for how humans might take inputs, compare them against existing knowledge (see: top-down processing), and produce outputs.

Accelerating Experience

Given this perspective on rule-based reasoning, it should come as no surprise that expert analysts have a much larger library of rules than novice analysts. These rules can be gained through experience, but as I stated earlier, experience doesn’t correlate perfectly with expertise. Gaining expertise is more about optimizing the analyst’s ability to build mental rules than arbitrarily waiting for the passage of time.

Certainly experience provides more of an opportunity to learn things, but if we can identify those things then there is little reason they can’t be taught in a more direct manner. Practically, this means that it’s possible to accelerate the rate at which an analyst gains experience by subjecting them to an environment that is more suitable for the development of rules.

That’s one reason we get analysts with the same amount of experience but varying levels of expertise (ignoring natural disposition towards the work). One environment might support the development of rules better than another. Experience is accelerated in these environments.

Investigation Heuristics

A simple way to help analysts develop a bigger library of rules is to write them down. The infosec industry has done a poor job of this, as it’s not something you’ll find publicly available. Some organizations have invested in the creation of investigation playbooks, which are a step in the right direction.

To document investigation-focused mental rules, the same if-then-else framework discussed earlier can be applied. If it ain’t broke, don’t fix it. These are more appropriately called heuristics, which are rules used to make decisions, solve problems, or draw conclusions. Better said, heuristics are mental shortcuts to finding answers to questions.

A more formalized heuristic format looks like this:

Heuristic Name
Input: $evidence_type
     $evidence is/has/contains $observation


Each heuristic is given a name for quick reference. It also includes an input evidence type, because in general any investigative conclusion is drawn from some type of observation or analysis on evidence. In many cases, a heuristic could be relevant for input of multiple types of evidence, or may require multiple types.

From there, the if-then-else statement makes up the meat of the heuristic. Similar to normal if-then-else statements, these scenarios can be made infinitely more complex. Of course, the simpler they can be made the better. Humans are processing these, so they don’t have to be perfect or follow all the same guidelines as though we’d expect a computer to be able to interpret them. Here are a few examples.

Domain Fast Flux Heuristic

Input: Domain

  • If: Domain resolves to a large number of IP addresses with diverse registration ownership or geography in a very short period
  • Then: It is likely that the domain is attacker owned and exhibiting fast-flux characteristics.
  • Else: The domain could be owned by a hosting company.
  • Else: The observation could be a coincidence.


File Type Mismatch Heuristic

Input: File

  • If: A file received in an e-mail is identified as a specific type based on its extension, but static analysis identifies a different file type.
  • Then: It is probable that the file is malicious in nature.
  • Else: The observation could be a coincidence.


Isolated POST Heuristic 

Input: IP, URL

  • If: An external IP sends an HTTP POST to one of your web servers, but doesn’t send any HTTP GET requests during the same period.
  • Then: There is a possibility that the internal host has become infected with a web shell, and the communication represents malicious traffic.
  • Else: This could be normal behavior for the system.


These heuristics all share the fact that they probably aren’t strong enough indicators on their own to warrant detection alerts; at least, not as scale grows beyond the small business. They do make useful investigation heuristics given the appropriate input in another investigation, whether alert-driven or human-driven (as in hunting).

This is a simplified example of a structured heuristic, but there is room to add a lot of interesting metadata to this format. For example, adding reference points to specific techniques used to retrieve evidence. Another example would be adding confidence ratings to the conclusions. This is a great place to make use of words of estimative probability so analysts can approach the heuristic with the appropriate weight and scrutiny.

Ultimately, the format doesn’t matter too much as long as this fits into the investigative workflow seamlessly. If you are embracing the investigation method, this should fit well with the question-hypothesis-answer format. These heuristics serve the role of helping develop questions and hypotheses to existing questions. They can also be used to drive initial observations when the investigation takes the form of hunting.

As a Teaching Tool

In an ideal world, the industry rallies around a format for investigation heuristics that can be explained in both a narrative and programmatic form, a standard is developed, and large common bodies of knowledge could exist that teach people how to investigate things.

In reality, the information security industry isn’t great at standards, so it’s probably a bit of a pipe dream; but it’s okay to have goals. In the interim, just maintaining a simple wiki with these types of investigation shortcuts can provide a tremendous benefit to analysts in your environment attempting to gain expertise. Even in environments where you might be a one-man-army network administrator and security analysts, having the reference available and reviewing it within the context of an active investigation is a helpful. It’s a worthwhile up front time investment.

They goal of this article isn’t to give you a format for creating and storing investigation heuristics. Instead, it’s to introduce rule-based reasoning and how the familiar construct of the if-then-else statement can be used to represent investigation shortcuts. It’s up to you to find the best way you can capture and represent this information for your own development, and the nurturing of analysts on your team.



How Analysts Approach Investigations

A  challenge facing information security is our inability to effectively train new analysts. The majority of security knowledge is tacit. We have plenty of practitioners who are good at catching bad guys, but most of them can’t articulate how they do it. I believe that overcoming this issue requires a focus on fundamental thought processes underlying security investigations, which is the foundation of my doctoral research.

Every major thought-based profession has a core construct through which everything is framed. For doctors, it’s the patient case. From this stems the diagnostic process, testing frameworks, and treatment plans. For lawyers, it’s the legal case. From this stems the discovery exercise, the trial, and sentencing. These core constructs are defined as an entities whose whole is greater than the sum of their parts. Each one is a story all its own.

In information security, our core construct is the investigation case. Everything we do is based on determining if malicious activity has happened, and to what extent. I don’t think many would argue this point, but surprisingly, there is very little formal writing out there about the investigation process itself. Many texts gloss over it and merely consider in the sum of its parts, a basic container for related evidence.

I propose that the investigation is so much more.

The Investigation Method

The investigation is at the heart of information security. It is a living, beating thing through which all of our actions are motivated and framed. It is our lens. To understand the investigation you must understand how humans think.

  1. Perception is not reality. What we perceive as reality and what actually exists are two separate things separated by our ability to interpret sensory input and using higher order reasoning. The process of getting from an initial perception to an accurate depiction of reality is the basis for learning and cognition.
  2. Learning comes from questioning. Straight from the womb, humans learn by questioning their environment, themselves, and their limits. By asking questions and employing various techniques to find answers we learn to move, walk, talk, and think. These techniques range from simple experimentation to complex reasoning, and can be motivated by primal needs like food and water, or higher order needs like achievement or respect.
  3. Our biases are always present. There are countless barriers that limit our ability to get from perception to reality. The most dangerous of these is our own mindset and the biases that are inherent to it. Humans are opinionated, and the same questions that drive us toward the pursuit of reality also drive opinions. When those opinions are educated and conscious they are hypotheses, and when none of those conditions are met they are guesses, and more subject to limiting bias.

If you consider this knowledge of human psychology, it begins to paint a picture of an investigation. Instead of trying to create a framework that dictates how investigations should be done, I wanted to take an approach the uncovers how you approach investigations as a form of learning. After all, that’s basically what an investigation is. It’s all about bridging the gap between perception and reality by learning facts. This yields the following definition and method.

“An investigation is the systematic inquiry and examination of evidence and observations in an effort to gain an accurate perception of whether an incident has occurred, and to what extent.”

The Investigation Process

If this looks familiar to you, that’s because it’s not too different from the scientific method. In a similar manner, the scientific manner wasn’t thought up as some way that scientific discovery should be done; it is an identification how most scientific discovery is done based on how humans learn. Even if scientists don’t intentionally set out to use the scientific method, their subconscious mind is doing it. The scientific method is responsible for the vast majority of scientific discovery. The investigation method is similarly responsible for the discovery of network intruders.

The investigation method contains five parts. I’ll briefly cover them here, although each one is worthy of its own article which will come later.


Every investigation begins with some observation that arouses suspicion. This is often machine generated in the form of an IDS alert, but could also be human driven in the form of an observation made while hunting. It doesn’t have to be an internal observation, and may come from a third-party notification. The tactics of the investigation are often shaped by the source of the initial observation, but the general process remains the same.

  • An observation is usually based on some form of initial evidence.
  • An observation can come from anywhere, but should be supportable. Even hunches or gut feelings are supportable when framed appropriately.
  • The first goal of the investigation is usually to validate or invalidate the initial observation as the premise of the investigation. If that observation isn’t valid, the investigation may not need to progress.


An investigation consists of a series of questions for which the analyst must seek answers. Based on the initial observation, the overarching questions will likely be some version of “Did a breach occur?” or “Is this malicious?” To answer those questions, more questions must be asked. Answers to one question will usually generate more questions. At any given point, an analyst should be able to articulate what question they’re trying to answer.

  • The ability to define good questions increases with experience because expert analysts have a larger pool of heuristics (rules) to draw from.
  • Most questions are centered around uncovering relationships, because ultimately it’s the relationships between devices and users that define an attack or breach.
  • Newer analysts will frequently begin answer seeking activities without clearly identifying the question they are attempting to answer. This can lead to wasted effort, but usually diminishes with experience.


You’re usually already slanted towards a specific answer from the moment you define your question, even if you don’t realize it. Your opinion forms based on your mindset, and is shaped by the entirety of your experience, both personal and professional. This is also where bias lives in the investigation process. The ability to articulate a hypothesis is an ideal way to expose bias so that your assumptions can be challenged if necessary. It also provides a clear path to additional questions that can validate or invalidate your hypothesis. Collectively, this leads to better, stronger conclusions.

  • Most hypothesis generation is passive and occurs subconsciously. A trick to making this an active process is to form an “I believe” statement for a hypothesis in response to each question. I believe ______ because _______.
  • Ideally a hypothesis is an educated guess. If you cannot complete the last half of the because statement, your assumptions may be from a place of bias, inexperience, or an inability to articulate well.
  • Every question should provide opportunity for a hypothesis, even if it’s a null hypothesis stating that a scenario isn’t probable.


The area of investigation most analysts are familiar with is answer seeking. It involves familiar tasks like retrieving, manipulating, and reviewing data. Any time you analytically review data or perform research it’s because you’re seeking an answer to your questions, usually to prove or disprove a hypothesis. Traditionally, newer analysts usually learn answer seeking before anything else which explains why the learning curve is so steep. They are trying to find answers for questions they don’t fully understand.

  • The goal of every answer isn’t to solve the investigation, it’s often to provide an opportunity for more questions. The answers you find will only be as good as the question they’re trying to resolve.
  • While it may seem logical to seek answers that prove a hypothesis, seeking to disprove a hypothesis is usually a much faster route to better questions.
  • Some questions won’t be answerable due to a lack of visibility or not enough data retention. Inability to answer a question is notable, because it might have impact on the investigation later. An unanswered question does not equal an invalid hypothesis.


The conclusion of an investigation is its terminal point. The investigation can terminate as a false positive alert, an acceptable risk, a simple malware infection, or a large breach requiring coordinated incident response. When a terminal disposition has been made, the investigation will contain a series of questions, hypotheses, and answers that uncover a (hopefully) accurate representation of events as they have occurred.

  • The strength of conclusions should always be accurately depicted by using estimative language. Certainties should be cited as such and backed up with evidence. Analytic opinions should be weighted based on their estimated certitude and available evidence.
  • If the steps that led you to a conclusion are considered carefully and documented well throughout the process, it should ease the burden of citing supporting information when documenting conclusions.

Framing an Investigation

Let’s look at example of what an investigation looks like through the lens of the investigation method. In this case, our fictional analyst has received an alert from an intrusion detection system.

Initial Observation: IDS Alert – User account was added to a domain admin group

This alert represents activity that might be legitimate, but could be malicious if it was unauthorized. The first question that generally follows an alert of this nature is whether it is malicious or normal activity.

Question 1: Does this alert represent malicious activity?

If the analyst were in a small organization they might be aware of any changes like this that should be occurring. Our analyst works in a very large enterprise, so it’s entirely possible that someone made this change for a legitimate reason without the analyst knowing. Because of this, the analyst believes its legitimate activity.

Hypothesis 1: I believe this is legitimate activity because this is something that happens frequently within the organization. 

To answer the initial question, the analyst must prove or disprove the hypothesis. To do this, more questions must be asked. There are a number of routes the analyst could go here, but one many analysts would pursue relates to follow-up actions taken by the user account.

Question 2: What actions did the user account take after being added to the admin group?

Based on the earlier hypothesis that this is normal behavior, it’s likely the hypothesis to Q2 will be similar.

Hypothesis 2: I believe the account participated in legitimate admin activity because it supports hypothesis 1. 

Seeking an answer to Q2 should be fairly easily with adequate visibility into your system and network logs. The analyst is able to search through logs fed into his SIEM and determine that the user account in question logged into a workstation, opened Outlook, and mounted several C-level executives mailboxes from the Exchange mail server.

Answer 2: The user account logged into a workstation, opened Outlook, and mounted several C-level executives mailboxes from the Exchange mail server.

The answer to Q2 appears to disprove our hypothesis 2, which in turn disproves hypothesis 1. The activity exhibited by the user account is definitely malicious, and answers our first question.

Answer 1: The actions taken by the user account after being added to the domain admin group are malicious in nature due to unauthorized access to multiple sensitive mailboxes.

At this point, the analyst is confident a breach has occurred, and the investigation can continue with that in mind. This should bring up more questions as the investigation evolves, including:

  • Was the user account an existing user account whose credentials were compromised?
  • Are there any indicators of compromise on the workstation normally used by the user who owns this account?
  • How did the potential attacker gain enough access to be able to promote the compromised account into an admin group?
  • How did the user account gain access to the workstation used to mount the Exchange mailboxes?
  • Is there any malware installed on the workstation the mailboxes were mounted from?
  • Were any other accounts accessed from the system belonging to the owner of the compromised account?

As you can see, what I’ve articulated here is only a fraction of what could be a much larger investigation. The key takeaway is that it provides a very structured, easy to follow timeline of the investigation and how it progressed. This makes it much easier to review the investigation process from beginning to end, and to use this investigation as a teaching tool for novice analysts.


As a Universal Method

The investigation method is a universal construct within information security. While the industry often glamorizes unique subspecialties like hunting and malware analysis, they all fit within the same scope of activities. The method still applies.

For example, consider threat hunting. It follows the same process to bridge the gap from perception to reality. The only difference is that the initial observation is usually human-driven. Instead of receiving an IDS alert or an external notification, the analyst asks broad questions based on their library of experience-derived heuristics. The goal of this questioning is for the answers to generate more questions, or lead to the discovery of evidence that represents malicious activity.

This isn’t to say that subspecialties don’t require unique skill sets. They most certainly do. A hunter is usually someone more experienced because they have a larger library of investigative heuristics to work from, which allows them to be more effective at coming up with questions that can drive the discovery of interesting observations. A novice analyst wouldn’t have nearly as many heuristics to rely on, and their efforts would be less fruitful.

The characteristics of a good analyst will vary based on specialization, but the method is universal.

Why It Matters

The investigation method isn’t provided as a framework. The truth is that this is the method you likely already use to investigate security events, even if you aren’t aware of it. That awareness is key, because it gives practitioners a language to express their knowledge. From this comes more insightful analysis, more clearly identified methods that lead to conclusions, and an ability to teach novice analysts how investigations can be performed through the lens of an expert.

If you walk into a hundred SOCs you will find a hundred ways of documenting investigations. There is no standard, and worse yet, most end up adopting whatever format their tooling provides. What happens is that ticketing systems and wikis end up defining how analysts perform investigations. This is tragic.

If you walk into those same hundreds SOC’s, you’ll also typically only find one way of teaching people how investigations should be done — through on the job observation. While observation-based training is a key component of any training program, an education that is founded entirely on observation is sure to fail. I wouldn’t want a surgeon who skipped medical school and went straight to residency to be operating on me. Sure, they might be able to get the job done, but they’ll be missing the fundamentals that make them flexible and prepared for the inevitable unknown.

This is one significant reason why defenders are so badly outpaced by attackers in information security. Our profession hasn’t gone through its cognitive revolution where we seek to understand how we approach the investigation and it’s components. If we want to get there, understanding human thought and the methods that form the investigation are key. This article seeks to shed light in some of those areas, and certainly the articles to follow will as well.

I’d encourage you to consider the method shown here and think through it as you perform your investigations. What questions are you asking? How are your hypotheses swaying your analysis? How strong are your conclusions? How do you express how you approach investigations? These are all useful questions and are pivotal in your own understanding of the craft, as well as those who will come after you.


The Role of Curiosity in Security Investigations

curiousgeorgeI’ve written a lot about the metacognitive gap in digital forensics and incident response. As an industry, we aren’t very effective at identifying the skills that make our expert practitioners so good at what they do, and we are even worse at teaching them. While there are a myriad of skills that make someone good at finding evil and eradicating adversaries, what’s the most important? What is the “X Factor” that makes an investigator great?

That’s a highly subjective question and everyone has an opinion on it biased towards his or her own experience. Regardless, I recently posed this question to some of my coworkers and Twitter followers. The most common answer I received was centered on curiosity.

Based on these results, I conducted a semi-formal survey where I asked 12 experienced analysts to rate the importance of possessing a highly curious mind while attempting to solve an investigation.

In the first survey item, I asked respondents to address the statement “A curious mind is important for an investigator to arrive at a state of resolution in an investigation with accurate and thorough results.”

All 12 investigators responded Strongly Agree using a 5-point Likert scale.

In a second question, I asked respondents to address the statement “A curious mind is important for an investigator to arrive at a state of resolution in an investigation in a timely manner.”

Using the same rating sale, 10 investigators responded Strongly Agree and 2 responded Agree.

Finally, I asked respondents to address the statement “A curious mind is a primary factor in determining whether an investigator will be successful in resolving/remediating an investigation.”.

Using the same rating sale, all 12 analysts responded Strongly Agree.

Clearly, expert practitioners believe that a curious mind is important in terms of accuracy, thoroughness, and speed at which an investigation is conducted. While curiosity isn’t the only thing makes an investigator successful in their craft, it certainly warrants attention as a key player. In this post I will talk about curiosity as a trait, how it manifests in the investigative process, how it’s measured, and whether it’s a teachable skill.

What is Curiosity?

There are many definitions of curiosity scattered across psychology research text, but the one I think most accurately depicts the construct from an applied perspective comes from Litman and Spielberger (2003). They state that curiosity can be broadly defined as a desire to acquire new information, knowledge, and sensory experience that motivates exploratory behavior.

Lowenstein (1994) also provides relevant insight by defining curiosity as “the desire to know.” In this sense, he describes that a desire to know more can arise when a person encounters stimuli that are inconsistent with an idea he or she holds. When this is experienced, the person may feel some kind of deprivation that can only be alleviated by resolving this inconsistency and closing the knowledge gap that has been identified. This jives well with the thoughts of other great psychology thinkers like Kant and Freud.

Curiosity takes form early in life when infants start exploring the world around them to test the limitations of their own body. Many developmental psychologists agree that this curiosity and simple but constant experimentation is the foundation of early learning and normal development. As we grow older, our curiosity continues to spark experimentation.

While curiosity has been considered a research-worthy construct from a theoretical perspective, there has been little effort put into pinning down the neural substrates that underlie it. This is unfortunate, but something to look forward to as neurology and brain imaging techniques continue to rapidly advance.

As it relates to computer security investigations, curiosity manifests practically in a number of ways that most of us can easily recognize. A few of those include the following:


Dead End Scenarios

The most dreaded scenario in an investigation occurs when an investigator reaches a point where there are still unanswered questions, but there are no leads left to pursue answers. This is common, especially when things like data retention and availability often limit us. In these scenarios a required data source might not be available, a lead from other evidence could run dry, or the data might not point to an obvious next step.

A limited amount of curiosity can be correlated with an increased number of dead end experiences encountered by an investigator. Without adequate motivation to explore additional avenues for answering questions, the investigation might miss logical paths to the answers they are seeking. They may also fail to adequately ask the appropriate questions.


Hypothesis Generation

The investigative process provides many opportunities for open-ended questions, such as “What is responsible for the network traffic?” or “Why would this internal host talk to that external host?” The process of reasoning through these questions is usually characterized initially by divergent thinking to generate ideas to be explored in route to a possible solution. This manifests as an internal dialog when conducted by a single analyst, but can be expressed verbally when a group is involved.

When presented with an open-ended question, curiosity is responsible for motivating the internal evaluation of hypothetical situations. Without curiosity, an individual won’t conduct mind-wandering exercises and may only consider a small number of potential hypotheses when there is potential for many other valid ones. In this scenario an investigator might not be pursuing the correct answers because they haven’t considered all of the potential questions that should be asked.

Note: It’s probably worth noting here that creativity plays a role in this process too, and is linked to curiosity depending on which model you subscribe to. That, however, is a little beyond the scope of what I want to talk about here.


Data Manipulation

Looking at the same data in different ways can yield interesting results. This can include using sed, grep, and awk to pull specific columns out of a data stream for comparison, using uniq and sort to aggregate field values, or reducing PCAP data into flows for comparison of multiple data streams.

While having the skills to manipulate data is a separate discussion, having the desire to find out if the manipulation of existing data into a new format will yield useful results is a product of curiosity. Investigators who lack curiosity to find out if such an exercise would be fruitful end up in more dead end scenarios and may take longer routes towards resolving investigations.


Pivoting to Tangential Evidence

The goal of collecting and reviewing evidence is to yield answers relevant to the question(s) an investigator has asked. However, it’s common for the review of evidence to introduce tangential questions or spawn completely new investigations. Within an investigation, you might review network connections between a friendly internal host and a potentially hostile external host only to find that other friendly devices have communicated with the hostile device and warrant examination. In another example, while examining web server logs for exploitation of a specific vulnerability, you might find unrelated evidence of successful SQL injection that warrants a completely separate investigation.

Curiosity is a key determinant in whether an investigator chooses to pivot to these tangential data points and pursue their investigation. Without the motivation that curiosity provides, an investigator may neglect to provide more than a cursory glance to these data points, or fail to note them down for later review. This can result in missed intrusions or improper scoping.

Relating Curiosity and Experience

Our cognitive processes don’t operate in a vacuum. Any decision we make is influenced by a symphony of different traits and emotions working in concert together. Some work in perfect harmony while others operate as opposing forces, and curiosity is not exempt. When we talk about curiosities role in an investigation, we also have to talk about experience.

Earlier, I mentioned that curiosity is pivotal to human development, and that our experimentation at an early age is motivated by curiosity to learn more about ourselves and the world around us. This isn’t a characteristic that goes away with time; we just become more aware of it. As we get older, we gain more experience and become more selective of what experiments we conduct. This manifests in many forms of our lives and in every day decisions. For example, a person who has never slept in on a Tuesday might hit the snooze button a few times because curiosity motivates them to explore the benefits and/or consequences of that action.

Experience serves as both a motivating and regulating force for curiosity. In an investigation, I believe this is best illustrated by assessing curiosity and experience as they relate to each other. Consider the following scenarios where we assess the level of curiosity (C) and experience (E) possessed by an individual investigator.

High C / Low E:

With a lot of curiosity but little experience, an investigator is jumpy. This person’s curiosity drives them to dig into everything that seems new, and without experience to regulate it, this persons ends up chasing a lot of ghosts. They will encounter dead end scenarios frequently because they will choose to pursue inconsequential leads within the evidence they are reviewing. They will rarely admit to encountering a dead-end scenario because their lack of experience doesn’t permit them to realize they’ve encountered one. This person will generate many ideas when hypothesis generation is required, but many of those ideas will be unrealistic because of a lack of experience to weed out the less useful ones. They will seek alternate views of data constantly, but will spend a considerable amount of time pursuing alternate views that don’t necessarily help them. Instead of learning to use tools that get them close to the views they want, they’ll spend time attempting to do more manual work to get the data precisely how they desire even if going that extra 20% doesn’t provide a discernable benefit to their investigation. Even though this person will spend a lot of time failing, they will fail fast and gain experience quickly.

Low C / High E

An investigator possessing a lot of experience but little curiosity could be described as apathetic. This doesn’t necessarily mean they aren’t effective at all, but it does make them less likely to investigate tangential leads that might be indicative of a larger compromise scope or a secondary compromise. In many cases, a person in this state may have started with a high degree of curiosity, but it may have waned over time as their experience increased. This can result in the investigator using their experience as a crutch to make up for their lack of curiosity. They won’t encounter too many dead end scenarios because of this, but may be more prone to them in new and unfamiliar situations. This person will manipulate data, but will rely on preexisting tools and scripts to do so when possible. They will carefully evaluate the time/reward benefit of their actions and will trust their gut instinct more than anything else. This person’s success in resolving investigations will be defined by the nature of their experience, because they will be significantly less successful in scenarios that don’t relate to that experience. These individuals won’t be as highly motivated in terms of out-of-the-box thinking and may be limited in hypothesis generation.

High C / High E

Because this person has a high level of curiosity they will be more motivated to investigate tangential leads. Because they also possess a high level of experience, they will be more efficient in choosing which leads they follow because they will have a wealth of knowledge to reflect upon. When encountering a dead-end scenario, this person should be able to move past it quickly, or if they claim they’ve hit a true dead end, it’s more likely to be an accurate representation of the truth. This person will excel in hypothesis generation and will provide valuable input to lesser experienced investigators relating to how their time could be best spent. They will seek to perform data manipulation when possible, but will be adept at realizing when to use already available tools and when to create their own. They will realize when they’ve found a data manipulation solution that is good enough, and won’t let perfect be the enemy of good enough. This presents an ideal scenario where the investigator is highly capable of resolving an investigation and doing so in a timely manner. These individuals are ideal candidates for being senior leaders, because they can often effectively guide less experienced investigators regarding what leads are worth pursuing and what the right questions to ask are. This person is always learning and growing, and may have several side projects designed to make your organization better.

Low C / Low E

This presents an undesirable scenario. Not only does this person not have the experience to know what they are looking at, they don’t have enough curiosity to motivate them to perform the necessary research and experimentation needed to learn more. This will handicap their professional growth and have them getting outpaced by their peers with a similar amount of experience.


If you are an investigator or have spent time around a lot of them then the descriptions you read in each of these scenarios might remind you of someone you know, or even yourself at different points in your career. It’s important to also consider progression, because the level of curiosity and experience of a person changes throughout their career. In these scenarios, a person always starts with no experience but their level of curiosity may affect how quickly that experience is gained.


High Curiosity – Sustained


In this ideal scenario, an investigator learns very quickly, and the rate at which they learn also grows. As they realize there is more to learn, they begin to consume more information in more efficient ways.


High Curiosity – Waning


While many start very curious, some experience a waning level of curiosity as their experience grows. When this happens, these investigators will rely more on their experience and their rate of learning will slow.


Low Curiosity – Sustained


An investigator with a sustained level of low curiosity will continually learn, but at a very slow rate through their career. Peers with a similar number of years experience will outpace them quickly.


Low Curiosity – Growing


If an investigator is able to develop an increased level of curiosity over time, their rate of learning will increase. This can result in dramatic mid to late career growth.


Each of these scenarios represents a bit of an extreme case. In truth, the progression of an investigators career is affected by many other factors, and curiosity can often take a back seat to other prevailing forces. Most of us who have served in an investigative capacity also know that curiosity often comes in peaks and valleys as new ideas or technologies are discovered. For instance, new tools like Bro have sparked renewed interest for many in the field of network forensics, while the maturity of memory analysis tools like Volatility have sparked curiosity for many in host-based forensics. A new job or changes in someone’s personal life can also positively or negatively affect curiosity.

Recognizing and Assessing Curiosity

We’ve established that curiosity is a desirable trait, and we’ve reviewed examples of what an investigator possessing varying degrees of curiosity and experience might look like. It’s only logical to consider whether curiosity is a testable characteristic. Several researchers have tackled this problem, and as a result there are different tests that can be used to measure varying degrees of this construct.

Available tests include, but are not limited to, the State-Trait Personality Inventory (Spielberger et al, 1980), the Academic Curiosity Scale (Vidler & Rawan, 1974), and the Melbourne Curiosity Inventory (Naylor, 1981). All of these tests are simple self-reported pencil and paper inventories designed to ask somewhat abstract questions in order to assess different facets of curiosity. Some use likert scales to evaluate whether statements describe them, where as others use agreement/disagreement choices in response to whether specific activities sound interesting. These tests all use different models for curiosity, spanning three, four, and five-factor models. They also all require someone with an understanding of administering personality tests to deliver and interpret the results.

A paper published by Reio, et al (2016) completed a factor analysis study of eleven different test designed to measure facets of curiosity. Their findings confirmed research done other psychologists that supports a three-factor model for curiosity delineated by cognitive, physical thrill seeking, and social thrill seeking components. Of course, the former of those is most interesting in our pursuits.

Psychometrics and personality testing is a very unique field of research. While many tests exist that can measure curiosity to some degree, their delivery, administration, and interpretation isn’t entirely feasible by those outside of the field. Simply choosing which test to administer requires a detailed understanding of test reliability and validity beyond what would be expected in a normal SOC. Of course, there is room for more investigation and research here that might yield simplified versions of these personality inventories that are approachable by technical leaders. This is yet another gap that can be bridged where psychology and information security intersect.

Teaching and Promoting Curiosity

Many believe that there is an aspect of curiosity that is a product of nature, and one that is a product of nurture. That is to say some people are born innately with a higher level of curiosity than others. The nature/nurture debate is one of the most prolific arguments in human history, and it goes well beyond the scope of my this article. However, I think we can stipulate that all humans are born with an innate ability to be curious.

If we know curiosity is important, that humans are born with a capacity for it, and we have models that can assess it, the practical question is whether we can teach it. As the field of cognitive psychology has grown, academics have sought to increase the practical application of research in this manner, incorporating the last hundred years of research on reasoning, memory, learning, and other relevant topics.

Nichols (1963) provides useful insight about scenarios that can inhibit and foster curiosity. He identifies three themes.


Theme 1: Temperance

A state of temperance is a state of moderation or restraint. While we usually think that it’s in our best interest to absorb all the information we can in an investigation, this can actually serve to limit curiosity. In short, a hungry man is much more curious than a well-fed one.

I think Nichols says it best, “Intemperance in a work situation is largely a condition we bring upon ourselves by limiting our mental exercise to a narrow span of interest. This is most commonly manifested in an over-attention paid to the details of what we are doing. Once our mind becomes satiated by an abundance of minor facts, we cannot, merely by definition, provide it with new and fresh ideas that will allow us to expand our intellectual perception. Our capacity to do so is restricted by our inability to cram more into a mind that is already overburdened by minutiae. Instead, if we recognize that our responsibility is to focus on the vital few points rather than the trivial many, we will have released ourselves so that we may—as the juggler does—examine these areas from several vantage points and mentally manipulate them in a way that will be both more productive and give greater self-satisfaction (Nichols, 1963, p.4). “


Theme 2: Success and Failure

When know this from basic principles of conditioning that humans will use avoidance techniques to prevent experiencing a stimulus that is perceived as negative. Because of this, an investigator who repeatedly attempts to perform the same activity and fails will be dissuaded from pursuing that activity. As we’ve established curiosity as a motivation to fill knowledge gaps, it’s clear to see the correlation between repeated failure and decreased curiosity.

For example, an investigator who has little scripting ability might decide that they would like to write a script to output the contents packet capture file and print all of the DNS queries and responses. If they attempt this multiple times and fail, they will eventually just move on to other methods of inquiry. At this point they are much less likely to pursue this same task again, and worse, are much less likely to attempt to solve similar problems using scripting techniques.


Theme 3: Culture

Whenever someone is surrounded by a group of others without any sense of curiosity, it’s likely that their level of curiosity will slow or cease growing at all. Fortunately, the opposite of the previous case is also true, as Nichols noted, “Just as association with a group methodically killing curiosity soon serves to stifle that precious commodity within us, becoming part of a group concerned with intellectual growth stimulates our personal curiosity and growth. This does not mean that each of us must throw over our present job, don a white lab coat, and head for the research and development department. It does mean that we should be discriminating in our choice of attitudinal surroundings both on and off the job. Specifically, it requires that we surround ourselves with doers, with competition that will give us incentive to exercise the creative abilities that grow out of intellectual curiosity. We all have the opportunity to find and benefit from an environment that stimulates our curiosity if we only seek it (Nichols, 1963, p.4).”


I’ve written extensively about creating a culture of learning, but there is something to be said for creating a culture of curiosity as a part of that. In a more recent study (Koranda & Sheehan, 2014), a group of researchers concerned with organizational psychology in the advertising field built upon that practical implications of Nichols’ work and designed a course with the goal of promoting curiosity in advertising professionals. This, of course, is another field highly dependent on curiosity for success. While this study stopped short of using one of the aforementioned inventories to measure curiosity before and after the course, the researchers did use less formal surveys to ascertain a distinguishable difference in curiosity for those who had participated in the course.

Based on all these things we can identify impactful techniques that can be employed in the education of computer security investigators encompassing formal education, shorter-term focused training, and on-the-job training. I’ve broken those into three areas:


  • When possible, encourage group interaction and thinking as much as possible. It exposes investigators to others with unique experience and ways of thinking.
  • Provide an environment that is rich in learning opportunities. It isn’t enough to expect an investigator to wade through false positive alerts all day and hope they maintain their curiosity. You have to foster it when scenario-based learning that is easily accessible.


  • Encourage challenging the status quo and solving old problems in new ways. This relates directly to data manipulation, writing custom code, and trying new tools.
  • Stimulate a hunger for knowledge by creating scenarios that allow investigators to fail fast and without negative repercussions. When an investigator is met with success, make sure they know it. Remember that experience is the thing we get when we don’t get what we wanted.
  • Pair lesser experienced investigators with mentors. This reduces the change of repetitive failure and increases positive feedback.


  • Tie learning as much as possible to real world scenarios that can be solved in multiple ways. If every scenario is solved in the same way or only provides one option, it limits the benefits of being curious, which will stifle it.
  • Create scenarios that are intriguing or mysterious. Just like reading a book, if there isn’t some desire to find out what happens next then the investigator won’t invest time it and won’t be motivated towards curiosity. The best example I can think of here is the great work being done by Counter Hack with Cyber City and the SANS Holiday Hacking Challenges.
  • Present exercises that aren’t completely beyond comprehension. This means that scenario difficulty should be appropriately established and paired correctly with the individual skill sets of investigators participating in them.


Of course, each of these thoughts presents a unique opportunity for more research, both of a practical and scientific manner. You can’t tell someone to “be more curious” and expect them to just do it any more than you can tell someone “be smarter” and expect that to happen. Curiosity is regulated by a complex array of traits and emotions that aren’t fully understood. Above all else, conditioning applies. If someone is encouraged to be curious and provided with opportunities for it, they will probably trend in that direction. If a person is discouraged or punished for being curious or isn’t provided opportunities to exhibit that characteristic, they will probably shy away from it.


Is curiosity the “X factor” that makes someone good at investigating security incidents? It certainly isn’t the only one, but most would agree that it’s in that conversation and it’s importance can’t be understated.

In this article I discussed the construct of curiosity, why it’s important, how it manifests, and what can be done to measure and promote it. Of course, beyond the literature review and application to our field, many of the things presented here are merely launching points for more research. I look forward to furthering this research myself, and hearing from those who have their own thoughts.



Koranda, D., & Sheehan, K. B. (2014). Teaching Curiosity: An Essential Advertising Skill?. Journal Of Advertising Education18(1), 14-23

Litman, J. A., & Spielberger, C. D. (2003). Measuring epistemic curiosity and its diversive and specific components. Journal of personality assessment,80(1), 75-86.

Lowenstein, G. (1994). `The Psychology of Curiosity: A Review and Reinterpretation.

Naylor, F. D. (1981). A state-trait curiosity inventory. Australian Psychologist,16(2), 172-183.

Nichols, R. G. (1963). Curiosity – The Key to Concentration. Management Of Personnel Quarterly2(1), 23-26.

Reio, T. J., Petrosko, J. M., Wiswell, A. K., & Thongsukmag, J. (2006). The Measurement and Conceptualization of Curiosity. The Journal Of Genetic Psychology: Research And Theory On Human Development167(2), 117-135. doi:10.3200/GNTP.167.2.117-135

Vidler, D. C., & Rawan, H. R. (1974). Construct validation of a scale of academic curiosity. Psychological Reports35(1), 263-266.