Accelerating Experience with Investigation Heuristics

ifthenelseWhy is someone who has been investigating security incidents for ten years so much better than someone who has only been doing it for a year?

That’s a simple question, and the simple answer is experience. As an analyst learns the fundamentals, develops a larger tool chest, and encounters more diverse scenarios they will naturally become better at their craft.

That’s straightforward, but consider these alternate scenarios. There are analysts who have been involved with security investigations for three years who are better than analysts who have been involved for ten years. Why is that? Furthermore, if there are two analysts with the same amount of experience, why would one analyst be better at investigating things than the other?

While we like to measure experience in units of time that is rarely an effective way to relate why an analyst is good at their job. Experience is related to expertise, but they don’t always directly correlate.

Today, I want to focus two elements particularly relevant to how expertise can be quantified between novice and expert analysts. These are rule-based reasoning and investigation heuristics.

Rule-Based Reasoning

I recently conducted a series of case studies where I brought in several security analysts of varying experience levels and asked them to describe a case they had worked. Through a technique known as the stimulated recall interview, I had them describe the process from beginning to end, focusing on why they took certain actions as the investigation progressed.

Once I collected a reasonable sample of these case studies, I reviewed each of them and performed a key phrase mapping exercise. I identified a list of categories based on a dual process theory model and mapped relevant statements made by the analyst to those categories. I was left with a distribution of how many responses existed in each category that I could divide based on various analyst demographics, like experience.

One category where there was a significant difference between the number of responses given be novice and expert analysts was rule-based reasoning. The expert analyst had nearly three times as many instances where rule-based reasoning was responsible for their actions.

Rule-based reasoning can be best thought of as an if-then-else statement. It’s a way that many believe humans store, retrieve, and manipulate knowledge, often leading to an action. Of course, as with several matters of the mind there are other theories too.

Regardless, it should come as no surprise that computers were designed to work using if-then-else statements, because computers are in some ways mankind’s attempt to recreate itself. It represents some of our most fundamental understanding of how we think and process information, and it can be demonstrated in all walks of life. Investigations are no different.

Consider the domain google.com. When you see that domain appear in an alert you immediately assume the alert is a false positive. This is because you’ve applied a rule like this:

  • If: Domain belongs to a well-known public company
  • Then: It’s probably not hosting malicious content
  • Else: It might have been victim of a strategic web compromise

Now consider the domain jr2jk2ndiskd.oje2kje.ru. When you see this domain in an alert you immediately assume its evil. This could be the result of a rule like this:

  • If: Domain appears to be mostly random alphanumeric characters
  • Then: It might be generated by a domain name generation algorithm and/or owned by an attacker
  • Else: It could be a coincidence, and should be documented in case I run across it again

These are simple rules that can be articulated easily. Of course, not all rules are that cut and dry.

Even if you don’t realize it, any time you review evidence in an investigation you’re evaluating a set of rules to make decisions. Some of these are very deliberate (reflective thinking) and some of them are very automatic (intuitive thinking). These two types of thinking and how they relate define dual process theory.

With that said, a rule-based system is a simplification of something that is insanely more complex. We aren’t just dealing with a linear approach to information processing, but more likely with the activation of millions of neurons in a semantic network or some other form of connectionist model. That goes well beyond the scope of this article and some levels of the current state of human understanding. Although a simplification, a rule-based system is a reasonable one for how humans might take inputs, compare them against existing knowledge (see: top-down processing), and produce outputs.

Accelerating Experience

Given this perspective on rule-based reasoning, it should come as no surprise that expert analysts have a much larger library of rules than novice analysts. These rules can be gained through experience, but as I stated earlier, experience doesn’t correlate perfectly with expertise. Gaining expertise is more about optimizing the analyst’s ability to build mental rules than arbitrarily waiting for the passage of time.

Certainly experience provides more of an opportunity to learn things, but if we can identify those things then there is little reason they can’t be taught in a more direct manner. Practically, this means that it’s possible to accelerate the rate at which an analyst gains experience by subjecting them to an environment that is more suitable for the development of rules.

That’s one reason we get analysts with the same amount of experience but varying levels of expertise (ignoring natural disposition towards the work). One environment might support the development of rules better than another. Experience is accelerated in these environments.

Investigation Heuristics

A simple way to help analysts develop a bigger library of rules is to write them down. The infosec industry has done a poor job of this, as it’s not something you’ll find publicly available. Some organizations have invested in the creation of investigation playbooks, which are a step in the right direction.

To document investigation-focused mental rules, the same if-then-else framework discussed earlier can be applied. If it ain’t broke, don’t fix it. These are more appropriately called heuristics, which are rules used to make decisions, solve problems, or draw conclusions. Better said, heuristics are mental shortcuts to finding answers to questions.

A more formalized heuristic format looks like this:

Heuristic Name
Input: $evidence_type
If:
     $evidence is/has/contains $observation
Then:
     $conclusion
Else:
     $conclusion

 

Each heuristic is given a name for quick reference. It also includes an input evidence type, because in general any investigative conclusion is drawn from some type of observation or analysis on evidence. In many cases, a heuristic could be relevant for input of multiple types of evidence, or may require multiple types.

From there, the if-then-else statement makes up the meat of the heuristic. Similar to normal if-then-else statements, these scenarios can be made infinitely more complex. Of course, the simpler they can be made the better. Humans are processing these, so they don’t have to be perfect or follow all the same guidelines as though we’d expect a computer to be able to interpret them. Here are a few examples.

Domain Fast Flux Heuristic

Input: Domain

  • If: Domain resolves to a large number of IP addresses with diverse registration ownership or geography in a very short period
  • Then: It is likely that the domain is attacker owned and exhibiting fast-flux characteristics.
  • Else: The domain could be owned by a hosting company.
  • Else: The observation could be a coincidence.

 

File Type Mismatch Heuristic

Input: File

  • If: A file received in an e-mail is identified as a specific type based on its extension, but static analysis identifies a different file type.
  • Then: It is probable that the file is malicious in nature.
  • Else: The observation could be a coincidence.

 

Isolated POST Heuristic 

Input: IP, URL

  • If: An external IP sends an HTTP POST to one of your web servers, but doesn’t send any HTTP GET requests during the same period.
  • Then: There is a possibility that the internal host has become infected with a web shell, and the communication represents malicious traffic.
  • Else: This could be normal behavior for the system.

 

These heuristics all share the fact that they probably aren’t strong enough indicators on their own to warrant detection alerts; at least, not as scale grows beyond the small business. They do make useful investigation heuristics given the appropriate input in another investigation, whether alert-driven or human-driven (as in hunting).

This is a simplified example of a structured heuristic, but there is room to add a lot of interesting metadata to this format. For example, adding reference points to specific techniques used to retrieve evidence. Another example would be adding confidence ratings to the conclusions. This is a great place to make use of words of estimative probability so analysts can approach the heuristic with the appropriate weight and scrutiny.

Ultimately, the format doesn’t matter too much as long as this fits into the investigative workflow seamlessly. If you are embracing the investigation method, this should fit well with the question-hypothesis-answer format. These heuristics serve the role of helping develop questions and hypotheses to existing questions. They can also be used to drive initial observations when the investigation takes the form of hunting.

As a Teaching Tool

In an ideal world, the industry rallies around a format for investigation heuristics that can be explained in both a narrative and programmatic form, a standard is developed, and large common bodies of knowledge could exist that teach people how to investigate things.

In reality, the information security industry isn’t great at standards, so it’s probably a bit of a pipe dream; but it’s okay to have goals. In the interim, just maintaining a simple wiki with these types of investigation shortcuts can provide a tremendous benefit to analysts in your environment attempting to gain expertise. Even in environments where you might be a one-man-army network administrator and security analysts, having the reference available and reviewing it within the context of an active investigation is a helpful. It’s a worthwhile up front time investment.

They goal of this article isn’t to give you a format for creating and storing investigation heuristics. Instead, it’s to introduce rule-based reasoning and how the familiar construct of the if-then-else statement can be used to represent investigation shortcuts. It’s up to you to find the best way you can capture and represent this information for your own development, and the nurturing of analysts on your team.

 

 

How Analysts Approach Investigations

A  challenge facing information security is our inability to effectively train new analysts. The majority of security knowledge is tacit. We have plenty of practitioners who are good at catching bad guys, but most of them can’t articulate how they do it. I believe that overcoming this issue requires a focus on fundamental thought processes underlying security investigations, which is the foundation of my doctoral research.

Every major thought-based profession has a core construct through which everything is framed. For doctors, it’s the patient case. From this stems the diagnostic process, testing frameworks, and treatment plans. For lawyers, it’s the legal case. From this stems the discovery exercise, the trial, and sentencing. These core constructs are defined as an entities whose whole is greater than the sum of their parts. Each one is a story all its own.

In information security, our core construct is the investigation case. Everything we do is based on determining if malicious activity has happened, and to what extent. I don’t think many would argue this point, but surprisingly, there is very little formal writing out there about the investigation process itself. Many texts gloss over it and merely consider in the sum of its parts, a basic container for related evidence.

I propose that the investigation is so much more.

The Investigation Method

The investigation is at the heart of information security. It is a living, beating thing through which all of our actions are motivated and framed. It is our lens. To understand the investigation you must understand how humans think.

  1. Perception is not reality. What we perceive as reality and what actually exists are two separate things separated by our ability to interpret sensory input and using higher order reasoning. The process of getting from an initial perception to an accurate depiction of reality is the basis for learning and cognition.
  2. Learning comes from questioning. Straight from the womb, humans learn by questioning their environment, themselves, and their limits. By asking questions and employing various techniques to find answers we learn to move, walk, talk, and think. These techniques range from simple experimentation to complex reasoning, and can be motivated by primal needs like food and water, or higher order needs like achievement or respect.
  3. Our biases are always present. There are countless barriers that limit our ability to get from perception to reality. The most dangerous of these is our own mindset and the biases that are inherent to it. Humans are opinionated, and the same questions that drive us toward the pursuit of reality also drive opinions. When those opinions are educated and conscious they are hypotheses, and when none of those conditions are met they are guesses, and more subject to limiting bias.

If you consider this knowledge of human psychology, it begins to paint a picture of an investigation. Instead of trying to create a framework that dictates how investigations should be done, I wanted to take an approach the uncovers how you approach investigations as a form of learning. After all, that’s basically what an investigation is. It’s all about bridging the gap between perception and reality by learning facts. This yields the following definition and method.

“An investigation is the systematic inquiry and examination of evidence and observations in an effort to gain an accurate perception of whether an incident has occurred, and to what extent.”

The Investigation Process

If this looks familiar to you, that’s because it’s not too different from the scientific method. In a similar manner, the scientific manner wasn’t thought up as some way that scientific discovery should be done; it is an identification how most scientific discovery is done based on how humans learn. Even if scientists don’t intentionally set out to use the scientific method, their subconscious mind is doing it. The scientific method is responsible for the vast majority of scientific discovery. The investigation method is similarly responsible for the discovery of network intruders.

The investigation method contains five parts. I’ll briefly cover them here, although each one is worthy of its own article which will come later.

Observation

Every investigation begins with some observation that arouses suspicion. This is often machine generated in the form of an IDS alert, but could also be human driven in the form of an observation made while hunting. It doesn’t have to be an internal observation, and may come from a third-party notification. The tactics of the investigation are often shaped by the source of the initial observation, but the general process remains the same.

  • An observation is usually based on some form of initial evidence.
  • An observation can come from anywhere, but should be supportable. Even hunches or gut feelings are supportable when framed appropriately.
  • The first goal of the investigation is usually to validate or invalidate the initial observation as the premise of the investigation. If that observation isn’t valid, the investigation may not need to progress.

Question

An investigation consists of a series of questions for which the analyst must seek answers. Based on the initial observation, the overarching questions will likely be some version of “Did a breach occur?” or “Is this malicious?” To answer those questions, more questions must be asked. Answers to one question will usually generate more questions. At any given point, an analyst should be able to articulate what question they’re trying to answer.

  • The ability to define good questions increases with experience because expert analysts have a larger pool of heuristics (rules) to draw from.
  • Most questions are centered around uncovering relationships, because ultimately it’s the relationships between devices and users that define an attack or breach.
  • Newer analysts will frequently begin answer seeking activities without clearly identifying the question they are attempting to answer. This can lead to wasted effort, but usually diminishes with experience.

Hypothesis

You’re usually already slanted towards a specific answer from the moment you define your question, even if you don’t realize it. Your opinion forms based on your mindset, and is shaped by the entirety of your experience, both personal and professional. This is also where bias lives in the investigation process. The ability to articulate a hypothesis is an ideal way to expose bias so that your assumptions can be challenged if necessary. It also provides a clear path to additional questions that can validate or invalidate your hypothesis. Collectively, this leads to better, stronger conclusions.

  • Most hypothesis generation is passive and occurs subconsciously. A trick to making this an active process is to form an “I believe” statement for a hypothesis in response to each question. I believe ______ because _______.
  • Ideally a hypothesis is an educated guess. If you cannot complete the last half of the because statement, your assumptions may be from a place of bias, inexperience, or an inability to articulate well.
  • Every question should provide opportunity for a hypothesis, even if it’s a null hypothesis stating that a scenario isn’t probable.

Answer

The area of investigation most analysts are familiar with is answer seeking. It involves familiar tasks like retrieving, manipulating, and reviewing data. Any time you analytically review data or perform research it’s because you’re seeking an answer to your questions, usually to prove or disprove a hypothesis. Traditionally, newer analysts usually learn answer seeking before anything else which explains why the learning curve is so steep. They are trying to find answers for questions they don’t fully understand.

  • The goal of every answer isn’t to solve the investigation, it’s often to provide an opportunity for more questions. The answers you find will only be as good as the question they’re trying to resolve.
  • While it may seem logical to seek answers that prove a hypothesis, seeking to disprove a hypothesis is usually a much faster route to better questions.
  • Some questions won’t be answerable due to a lack of visibility or not enough data retention. Inability to answer a question is notable, because it might have impact on the investigation later. An unanswered question does not equal an invalid hypothesis.

Conclusion

The conclusion of an investigation is its terminal point. The investigation can terminate as a false positive alert, an acceptable risk, a simple malware infection, or a large breach requiring coordinated incident response. When a terminal disposition has been made, the investigation will contain a series of questions, hypotheses, and answers that uncover a (hopefully) accurate representation of events as they have occurred.

  • The strength of conclusions should always be accurately depicted by using estimative language. Certainties should be cited as such and backed up with evidence. Analytic opinions should be weighted based on their estimated certitude and available evidence.
  • If the steps that led you to a conclusion are considered carefully and documented well throughout the process, it should ease the burden of citing supporting information when documenting conclusions.

Framing an Investigation

Let’s look at example of what an investigation looks like through the lens of the investigation method. In this case, our fictional analyst has received an alert from an intrusion detection system.

Initial Observation: IDS Alert – User account was added to a domain admin group

This alert represents activity that might be legitimate, but could be malicious if it was unauthorized. The first question that generally follows an alert of this nature is whether it is malicious or normal activity.

Question 1: Does this alert represent malicious activity?

If the analyst were in a small organization they might be aware of any changes like this that should be occurring. Our analyst works in a very large enterprise, so it’s entirely possible that someone made this change for a legitimate reason without the analyst knowing. Because of this, the analyst believes its legitimate activity.

Hypothesis 1: I believe this is legitimate activity because this is something that happens frequently within the organization. 

To answer the initial question, the analyst must prove or disprove the hypothesis. To do this, more questions must be asked. There are a number of routes the analyst could go here, but one many analysts would pursue relates to follow-up actions taken by the user account.

Question 2: What actions did the user account take after being added to the admin group?

Based on the earlier hypothesis that this is normal behavior, it’s likely the hypothesis to Q2 will be similar.

Hypothesis 2: I believe the account participated in legitimate admin activity because it supports hypothesis 1. 

Seeking an answer to Q2 should be fairly easily with adequate visibility into your system and network logs. The analyst is able to search through logs fed into his SIEM and determine that the user account in question logged into a workstation, opened Outlook, and mounted several C-level executives mailboxes from the Exchange mail server.

Answer 2: The user account logged into a workstation, opened Outlook, and mounted several C-level executives mailboxes from the Exchange mail server.

The answer to Q2 appears to disprove our hypothesis 2, which in turn disproves hypothesis 1. The activity exhibited by the user account is definitely malicious, and answers our first question.

Answer 1: The actions taken by the user account after being added to the domain admin group are malicious in nature due to unauthorized access to multiple sensitive mailboxes.

At this point, the analyst is confident a breach has occurred, and the investigation can continue with that in mind. This should bring up more questions as the investigation evolves, including:

  • Was the user account an existing user account whose credentials were compromised?
  • Are there any indicators of compromise on the workstation normally used by the user who owns this account?
  • How did the potential attacker gain enough access to be able to promote the compromised account into an admin group?
  • How did the user account gain access to the workstation used to mount the Exchange mailboxes?
  • Is there any malware installed on the workstation the mailboxes were mounted from?
  • Were any other accounts accessed from the system belonging to the owner of the compromised account?

As you can see, what I’ve articulated here is only a fraction of what could be a much larger investigation. The key takeaway is that it provides a very structured, easy to follow timeline of the investigation and how it progressed. This makes it much easier to review the investigation process from beginning to end, and to use this investigation as a teaching tool for novice analysts.

InvestgationScenario

As a Universal Method

The investigation method is a universal construct within information security. While the industry often glamorizes unique subspecialties like hunting and malware analysis, they all fit within the same scope of activities. The method still applies.

For example, consider threat hunting. It follows the same process to bridge the gap from perception to reality. The only difference is that the initial observation is usually human-driven. Instead of receiving an IDS alert or an external notification, the analyst asks broad questions based on their library of experience-derived heuristics. The goal of this questioning is for the answers to generate more questions, or lead to the discovery of evidence that represents malicious activity.

This isn’t to say that subspecialties don’t require unique skill sets. They most certainly do. A hunter is usually someone more experienced because they have a larger library of investigative heuristics to work from, which allows them to be more effective at coming up with questions that can drive the discovery of interesting observations. A novice analyst wouldn’t have nearly as many heuristics to rely on, and their efforts would be less fruitful.

The characteristics of a good analyst will vary based on specialization, but the method is universal.

Why It Matters

The investigation method isn’t provided as a framework. The truth is that this is the method you likely already use to investigate security events, even if you aren’t aware of it. That awareness is key, because it gives practitioners a language to express their knowledge. From this comes more insightful analysis, more clearly identified methods that lead to conclusions, and an ability to teach novice analysts how investigations can be performed through the lens of an expert.

If you walk into a hundred SOCs you will find a hundred ways of documenting investigations. There is no standard, and worse yet, most end up adopting whatever format their tooling provides. What happens is that ticketing systems and wikis end up defining how analysts perform investigations. This is tragic.

If you walk into those same hundreds SOC’s, you’ll also typically only find one way of teaching people how investigations should be done — through on the job observation. While observation-based training is a key component of any training program, an education that is founded entirely on observation is sure to fail. I wouldn’t want a surgeon who skipped medical school and went straight to residency to be operating on me. Sure, they might be able to get the job done, but they’ll be missing the fundamentals that make them flexible and prepared for the inevitable unknown.

This is one significant reason why defenders are so badly outpaced by attackers in information security. Our profession hasn’t gone through its cognitive revolution where we seek to understand how we approach the investigation and it’s components. If we want to get there, understanding human thought and the methods that form the investigation are key. This article seeks to shed light in some of those areas, and certainly the articles to follow will as well.

I’d encourage you to consider the method shown here and think through it as you perform your investigations. What questions are you asking? How are your hypotheses swaying your analysis? How strong are your conclusions? How do you express how you approach investigations? These are all useful questions and are pivotal in your own understanding of the craft, as well as those who will come after you.

 

Writing for Security: Action Items that Provoke Change

quillMost people don’t realize it, but the success of what you write will probably be measured by how actionable it is. I’ve read hundreds of security assessments and forensic reports that go into a perfect level of detail, only to find that they fall short of delivering what every report needs: something actionable.

Imagine watching a great movie. They’ve done a wonderful job developing complex characters, the plot engagingly builds, and you’re on the edge of your seat the entire time. Right as the climax is happening and the story is coming to it’s pivotal point…the credit start rolling. It’s over. Although you might have enjoyed the couple of hours you invested up unto that point, you’re going to walk away with a bad taste in your mouth because you were robbed of a satisfactory conclusion. We all know movies like this, and usually chalk it up to lazy writing. This is exactly what you’re doing when you write without providing something actionable.

Whether you’re writing a security assessment report or an incident response report, your purpose isn’t merely to inform, it’s also to persuade. It isn’t enough that someone knows there is a vulnerability on their network. They have to be persuaded to implement controls that mitigate the risk of that vulnerability. It doesn’t matter if your forensic report does a good job explaining how an attacker got in. It has to persuade the reader to implement the necessary process changes or install the right tools to prevent it from happening again.

There are plenty of techniques you can use to be persuasive when you write, but before you do that you must identify what you want the reader to do. These are your action items, and the ability to identify them is what makes you an expert. Plenty of people can find vulnerabilities or find evidence of an attacker, but if you can’t identify actions to mitigate the risk associated with those findings then you’re report isn’t useful.

Identifying action items is all about mitigating risk. You should give the reader advice that prevents bad guys from doing some thing, or detects when they do it. You should always do both when possible.

Prevention Action Items

Prevention is as simple as making changes that keep bad things from happening. If you can give your reader steps they can take that prevent an attacker from doing something, that’s usually a win.

In reporting, I like to conceptualize change in terms of how difficult it is to accomplish. After all, it’s a lot harder to persuade someone to make a change if it’s going to be insanely difficult. Part of good writing is being honest with your readers, so it helps if to identify the level of effort required with a requested change. If it’s going to be easy you should make that clear so the reader is compelled to do it quickly. If it’s going to be difficult, be up front about and break it into steps. Your readers will appreciate this and will trust you more.

Changes will typically be categorized in terms of people, process, and technology.

  • People: Changing mindsets, providing training, hiring new staff, replace existing staff.
  • Process: Changing the way human or tech-centric things are done, adding new processes.
  • Technology: Configuration changes, additional software, new technology.

In most case, technology changes will be the easiest and people changes will be the hardest. It’s easy to manipulate systems, but it’s very difficult to acquire new people or change the way existing people think. The latter requires a lot more political and financial capital. This hierarchy of difficulty is how you should approach identifying prevention actions in your report. You should also report them in order of easiest to hardest within each individual finding.

In a lot of cases, some changes might touch all three areas. For example, building a security operations team requires hiring new people, building new processes, and implementing new technology. These massive changes should be saved for last and you should provide plenty of ancillary resources for the reader, as they will often involve topics that need to be covered in much larger depth.

When you are ready to start identifying action items, it’s helpful to ask yourself these three questions, filling in the blanks with the pertinent details from the finding you’re addressing:

  1. Are there any changes that can be made to prevent an attacker from __________?
  2. Is there anything new that can be done to prevent an attacker from __________?
  3. Is there anything that should be stopped in order to prevent an attacker from __________?

Let’s look at some examples of common findings and their action items. Notice that some action items combine categories, and some categories aren’t present.

 

Security Assessment: Web Server – Utilizing Plaintext Authentication

  • Technology: Change authentication method

 

Security Assessment: Local Windows Admin Account – No Password Rotation

  • Technology: Purchase password management software
  • Process: Institute manual change process

 

Incident Report: Attacker Guessed VPN Password

  • Technology: Institute lockout after three failed authentication attempts. Enforce stronger password requirements and more frequent rotation.
  • Technology + Process: Implement two-factor authentication.
  • People: Train users to use passwords that can’t be easily guessed

 

Incident Report: Workstation Compromised Because User Clinked Phishing Link

  • Technology: Install an e-mail threat protection appliance.
  • Process: Force users to use non-admin accounts and escalate privileges when administrative actions are needed.
  • People: Provide phishing awareness training to users.

 

Incident Report: Attacker Moved Laterally with Ease Due to Flat Network

  • Technology: Architect network for better segmentation.

 

These are just examples, but you can see where we started with technology and moved towards people. In most cases, the technology solution is going to be the easiest to implement in terms of labor hours. Of course, this doesn’t mean that a technology solution is always the best, but it is a step in the right direction. You want to give the reader the easy wins so they are more compelled to keep working towards to bigger wins. The first step is the hardest to take.

Detection Action Items

Detective controls are designed to detect when bad things happen. The vast majority of reports you’ve read probably don’t include them, which is a shame. Whenever you are making preventive recommendations, you should also make detective recommendations. There are a few reasons why:

  • Many organizations won’t be able to implement protective changes in a timely manner, or at all due to political or budgeting restraints.
  • Prevention eventually fails, and a key tenant of security is having multiple layers of controls.
  • The findings you’ve identified may have already been exploited, and an ability to retroactively detect this can help uncover a breach.

If you’re a consultant, writing detection action items can be difficult because there are a wide array of detection technologies. It’s hard to tailor detection content exclusively for a single customer without an intimate knowledge of their detection strategy. As a place to start, consider asking your client about their detection strategy and relevant technologies so that you can tailor your recommendations to them. This can be a part of the initial scoping call.

If you’re findings are related to your own network, or you’re writing a blog post, it’s a lot easier provide detection action items based on the precise technologies your using, or at a minimum prevailing open source standards. You can start with these questions:

  1. Are there any network-based indicators that can be used for detection?
  2. Are there any network-based behaviors that can be used for detection?
  3. Are there any host-based indicators that can be used for detection?
  4. Are there any host-based behaviors that can be used for detection?

This isn’t all encompassing, but in a lot of cases you will be able to derive some type of host or network based indicators or behaviors. An indicator can be something simple like a list of MD5’s or domain names, and will usually be representative of known evil. A behavior is usually more complex and will indicate a behavior that is normally legitimate, but could be the results of an attackers actions in some cases. This might be an action like a user account being added to an administrative group, or the use of the command “ping -n 1”. Both normal activities, but something that might not be done too often and worth of investigation in relation to the identified activity as it relates to the attacker or breach you’re describing.

In all of these cases, recommendations towards specific technologies are what will differentiate you. Don’t just give someone a list of domain names, also give them a Snort or Suricata rule that will detect them, including relevant context and information links. Don’t just give someone malware characteristics, give them the YARA rule to search for it. You might think that’s time consuming, and you’re right. Don’t be lazy! If you truly want to promote change and you expect your reader to go the extra mile, you’re writing has to do it as well. Something as small as creating a 10 line bash script to detect something will endear your reader/client to you forever, and will show your hands-on expertise.

More on Writing

If the things you write require the user to take action, you’re going to have to work harder to get them to do it. Just because you’ve written a very clear and informative statement on what a problem is and how to fix it doesn’t necessarily mean someone will take the action you want. The easier you can make this for people, the more they will be likely to actually pursue your recommendations. Going the extra mile in your writing will be rewarded with actions. If you can outline some prevention and detection action items, you’ll be writing content that will get people moving.

If you’re interested in learning more about my personal systems for better technical writing, I’ll be releasing more articles in that area soon, as well as a couple of videos. You can subscribe to the mailing list below to get access to that content first, along with a few exclusives that won’t be on the site.

Sign Up for the Mailing List Here

Video: Building an NSM Lab

Building a security lab is something I get asked about really often. So often, in fact, that I decided to put some of my notes together and record a short training video on the topic. This video is only a small part of a much larger series I’m developing, so if you’re interested in learning more about that when it’s available, sign up for my mailing list.

In this one hour video I discuss the importance of an NSM lab and go through a systematic approach to building your own. I go through the following topics:

  • Analyzing your needs to define your inputs and desired outputs
  • Modeling your lab by building a list of technologies
  • The pros and cons of physical, virtual, and cloud based labs
  • Choosing the right platform for your lab
  • Designing your lab network
  • Sourcing the right hardware for your lab
  • Taking a step by step approach to designing and building the lab

Once you’re done with this video, you should have a system you can follow to build a lab that will help you test and build detection, analyze malware, and create simulations. I also provide a lot of insight to my own personal lab I use for my writing and my day job. I’ve also included some additional resources:

  • Lab planning worksheet
  • An exact parts list from my lab
  • Two example lab network diagrams
  • The network diagram for my personal lab

You can access the additional resources mentioned in the video by signing up here.

Writing for Security: Making People Give a Damn

quillIf you really want your technical content to matter for people you have to appeal to their needs. There are primary needs like food, water, sleep, or sex, but it’s difficult to tie those things to malware analysis or threat intelligence reports. If you look to secondary needs you will find things like employment, resources, morality, family, self-esteem, confidence, achievement, and respect. Hopefully, a light bulb went off when looking at this list. If you really want people to care about your content you have to appeal to one or more of these things. Let’s dig into a few of them.

Employment, Achievement, and Respect

I want to lead with employment because it is the secondary need most tied to primary needs. Everyone needs to eat, and unless your Silicon Valley startup actually made it past the second round of funding you probably need a job to buy food for yourself and your family. If your writing can appeal to someone’s need for employment, they are going to care about it.

Tangentially related are achievement and respect, because everyone wants to achieve success in the workplace and be respected while doing it. These are grouped together because most believe that being well respected and achieving positive things will lead to further career success. In most places this is definitely true.

When you’re writing something, ask yourself if it will help someone get a better job or a higher salary in their current job. You may want to think it’s much more complicated than that, but it really isn’t. You may be a person who says “Chris, I’m not in this line of work for the money, so I can’t relate to that.” If you were being completely honest with yourself, you certainly wouldn’t do your job for free, or probably even for half of your current salary. You have to eat and you have to provide for your family and so does everyone else. If you can write something that helps your reader do that, you are appealing to primal psychological needs and people will gravitate towards that.

The best way to appeal to these needs is to provide an opportunity for meaningful action. That action will vary depending on what you’re writing, but here are a few examples:

Penetration Testing Report [You want the reader to fix a finding]:

  • An example of how a finding would be exploited so it can be independently validated and recreated.
  • A news story showing how a similar finding was attacked that can be used to justify the time/resources to fix it to management.
  • A detection signature that can be applied to a Snort/Suricata/Bro IDS so the user can detect exploitation if it can’t be fixed in a timely manner.
  • A list of log types that can be ingested by a SIEM if detective controls are a primary risk reduction strategy.

Threat Intel Blog Post [You want the reader to defend against this threat actor]:

  • A diagram showing the flow of the attack and where protective/detective controls could be applied.
  • Reference links to attacks conducted by this threat group that can be used to justify the time/resources to fix it to management.
  • A detection signature that can be applied to a Snort/Suricata/Bro IDS to that can be used to detect actor activity.
  • A listing of network and host based artifacts that the user can build into their own detection infrastructure and SIEM.

Alert Investigation Ticket [You want management to provide funding for bigger sensors]:

  • A timeline showing the flow of the investigation and areas where it was stalled due to lack of visibility to justify the ask to management.
  • A hypothetical description of how the investigation could have gone and how much time might have been saved if more data was available.
  • A list of the exact type of sensor you need along with a broad cost estimate.
  • A success stories from a colleague/peer who has the level of visibility you desire.

Forensic Report [You want the company to educate users on spear phishing]:

  • A diagram showing how an attacker was able to gain an initial foothold into the network by phishing a number of users.
  • Industry reporting on statistics of users who are susceptible to phishing.
  • Links to news articles of other breaches showing how phishing was a primary attack vector.
  • A guide explaining how the IT staff could conduct a phishing test with the user base to determine how vulnerable they truly are.
  • A list of vendors (or if you’re a vendor, a price quote) on performing an external phishing test.
  • Links to free or paid phishing awareness training programs.
  • A list of tips that can be e-mailed to all users within the company.

If you give the reader a chance to take action from your writing then you’re giving them the chance to achieve something and to gain respect from their peers and boss by doing it. Doing this in a way that truly empowers them is a bit of a balancing act, which we’ll talk about next.

Confidence and Self-Esteem

Nobody likes feeling stupid. If you write something with a lot of technical detail it’s probably a good thing, but if it goes so aimlessly in–depth that it goes over the head of most people reading it, they aren’t going to connect with you. Appealing to primary and secondary needs doesn’t matter if your reader walks away thinking they aren’t smart enough to do anything about the problem you present. That’s why it’s so crucial to go the extra mile. In infosec, your goal is usually to inform, but it’s frequently to persuade. If you want someone to head down a path towards a goal you must realize that the hardest step for them to take is the first. The more work you can do for the reader up front, the more likely they are to take that first step. This means providing actionable examples and step-by-step guides that get them moving. This is more work on you up front as the author, but readers don’t reward lazy writing.

If you provide a call to action that asks the reader to write 10,000 lines of code or change the entire culture of their corporation, they aren’t going to feel confident enough to act on it. There’s a place for that type of writing, but most of the time it shows laziness on your part for not going the extra mile to give them actionable techniques for getting started down whatever path your trying to get them to take.

Figuring out where to position your material can be tricky, but there are a few things to think about when writing it:

  • What’s the lowest common denominator you are trying to appeal to?
    You don’t have to dumb everything down far enough that someone with no experience should be able to get going, but you should assume that most of your readers aren’t as smart as you. If they were, why would they need to read what you’re writing?
  • What is something the reader can do today/tomorrow/next week?
    If you can phase out your action items over the course of time it makes it can make a larger task become less overwhelming. Even something as simple as downloading a tool or sending an e-mail is a step. If the reader can accomplish that step, they are going to build confidence and be more likely to accomplish the next step. It’s a snowball effect.
  • Where can the reader learn more about the concepts they need to make this actionable?
    If you are correctly assuming the reader isn’t as knowledgable about the topic as you are, then you need to do whatever you can to minimize that gap. If you want them to take action on something they don’t know much about, you absolutely must provide reference to resources where they can learn more. If you want a user to write a signature for a malware family, link or provide supporting information about the techniques the malware uses and the libraries it relies on. If you want a user to fix an XSS vulnerability in a piece of code, link or provide examples of different types of XSS protection and libraries that demonstrate different techniques.

If you read all of this and don’t think you need to go the extra mile because your writing is to inform and not to persuade, then I’d say you’re probably fooling yourself, or you’re a lazy writer. Both will result in content that isn’t appealing to your readers, and it will be forgotten.

Morality

One of the oldest debates in history is whether mankind is inherently good or evil. I’m certainly not going to solve that debate here, but I think it’s safe to say that you probably got into information security because you have some sense of right vs. wrong. In most cases, the network you are protecting or assessing represents good, and the real or hypothetical bad guys who want to steal something from it represent evil.

Whether it’s nature or nurture, most humans have a sense of morality from a young age. Whether you realize it or not, you’ve built archetypes of the good guys and the bad guys and in most cases you probably want to be the guy with the cape saving the day. This is important to consider when you write, because if you can tap into someone’s sense of morality then you are going to reach parts of the reader that most writing can’t touch.

I want to be clear on this that I don’t want you to start making moral decisions for someone. In our field, it’s ridiculously easy to stumble into a debate about things like privacy vs. security, and you probably aren’t going to change someone’s mind there. Furthermore, a lot of people enter a way of thinking in irrational ways. Cognitive psychology tells us that someone who enters a line of thought irrationally is not likely to leave that mindset because of rational though. The goal isn’t to manipulate someone’s sense of morality; it is to appeal to it by causing the reader to ask questions.

So what if there is a new piece of malware being used to attack agriculture companies? These companies are targeted all the time. Nobody is really going to care about that unless they work at one of the targeted companies who were affected. Now, what if you consider that the malware caused a significant financial loss that led to a Q2 earnings miss resulting in layoffs of hundreds of people? That changes things a bit. Because someone used the malware to attack this organization, real people were hurt, and the reader will ask themselves whether this is morally wrong. Again, your job isn’t to tell people it’s wrong. Your job is to get them to ask themselves where this action points on their moral compass.

Getting people to ask questions about the moral disposition of something isn’t always easy, and it often requires some digging. One method for getting to this point is by using the 5 Why’s method. Take a fact that you are writing about and ask yourself why it matters, then ask yourself why that matters. For example:

 

Hypothetical Fact: A government contractor was the victim of an attack, resulting in the theft of intellectual property

  1. Why does that matter? The attacks on the government contractor was linked to group X due to similar TTPs
  2. Why does that matter? Group X is comprised of operators believe to be North Korean
  3. Why does that matter? North Korean threat actors have attacked a number of western media outlets and government contractors and are advancing their capability
  4. Why does that matter? The North Korean government has expressed interest in harming western countries through advancing weapons technology
  5. Why does that matter? If North Korea succeeds, the consequences could result in conflict or war.

 

Hypothetical Fact: A newly discovered piece of malware redirects users to a site that scrapes their social media profile if they are logged into Facebook and harvests personal information

  1. Why does that matter? An unknown attacker could gain access to your personal information.
  2. Why does that matter? The attacker could use this personal information to obtain more information about you through social engineering or password reset questions.
  3. Why does that matter? The attacker could collect enough information to steal your identity
  4. Why does that matter? The attacker could cause significant financial loss or ruin your credit score, preventing you from being able to take out a loan on a car or home.

 

In both of these examples, I’ve presented scenarios that mirrors things you’ve probably actually read at some point,  and gone through a process to translate them into their core; things that should provoke questions of morality. Is it right/wrong for North Korea to start a conflict? Is it right/wrong for someone to steal your identity? In these cases both answers are probably pretty clear-cut. In a lot of cases it won’t be so obvious. The important thing is to get people to ask the question.

More on Writing

Writing is a lot more enjoyable when people care about what you’ve written. In the current security landscape you can’t go more than a couple of days without someone writing a blog post detailing the latest threat actor campaign or malware they’ve discovered. If you’re responsible for writing content like this, whether internally or externally, appealing to primary and secondary needs will guarantee that people care more about what you have to say.

If you’re interested in learning more about my personal systems for better technical writing, I’ll be releasing more articles in that area soon, as well as a couple of videos. You can subscribe to the mailing list below to get access to that content first, along with a few exclusives that won’t be on the site.

Sign Up for the Mailing List Here