Category Archives: Network Security Monitoring

Investigating Like a Chef

Whenever I get the chance I like to try and extract lessons from practitioners in other fields. This is important because the discipline of information security is so new, while more established professions have been around, in some cases, for hundreds of years. I’ve always had a keen interest in culinary studies, mostly because I come from an area of the country where people show that they love each other by preparing meals. I’m also a bit of a BBQ connoisseur myself, as those of you who know me can solemnly attest to. While trying to enhance my BBQ craft I’ve had the opportunity to speak with and read about a few professional chefs and study how they operate. In this post I want to talk a little bit about some key lessons I took away from my observations.

If you have ever worked in food service, or have even prepared a meal for a large number of people you know that repetition is often the name of the game. It’s not trimming one rack of ribs, its trimming a dozen of them. It’s not cutting one sweet potato, its cutting a sack of them. Good chefs strive to do these things in large quantities while still maintaining enough attention to detail so that the finished product comes out pristine. There are a lot of things that go into making this happen, but none more important than a chef mastering their environment. This isn’t too different than a security analyst who investigates hundreds of alerts per day while striving to pay an appropriate amount of attention to each individual investigation. Let’s talk about how chefs master their environment and how these concepts can be applied to information security.

Chefs minimize their body movement. If you are going to be standing up in a kitchen all day performing a bunch of repetitive and time sensitive tasks, then you want to make sure every step or movement you make isn’t wasted. This prevents fatigue and increases efficiency.

As an example, take a look at Figure 1. In this image, you will see that everything the chef needs to prepare their dish is readily available without the chef having to take extra steps or turn around too often. Raw products can be moved from the grocery area, rinsed in the sink, sliced or cut on the cutting board, cooked on the stove, and plated without having to turn more than a few times or move more than a couple of feet.

chef_mentalmoves

Figure 1: A Chef’s Workspace is Optimized for Minimal Movement

Chefs learn the French phrase “mise en place” early on in their careers. This statement literally means, “put in place”, but it specifically refers to organizing and arranging all needed ingredients and tools required to prepare menu items during food service. Many culinary instructors will state that proper mise en place, or simply “mise” in shorthand, is the most important characteristic that separates a professional chef from a home cook.

There is a lot of room for mise in security investigations as well. Most analysts already practice this to some degree by making sure that their operating system is configured to their liking. They have their terminal windows configured with a font and colors the make it easy to read, they have common OSINT research sites readily accessible as browser favorites, and they have shortcut icons to all of their commonly used tools. At a higher level, some analysts even have custom scripts and tools they’ve written to minimize repetitive tasks. These things are highly encouraged.

While analysts don’t have to worry about physical movement as much, they do have to work about mental movement. In an ideal situation an analyst can get to the end of an investigation with as few steps as possible, and a strategic organization of their digital workspace can help facilitate that. I’ve seen some organizations that seek to limit the flexibility analysts have in their workspace by enforcing consistent desktop environments or limiting access to additional tools. While policies to enforce good security and analysis practices are great, every analysts learns and processes information in a different way. It isn’t only encouraged that analysts have flexibility to configure their own operating environments, it’s critical to helping them achieve success.

Beyond the individual analysts workstation, the organization can also help out by providing easy access to tool and data, and processes that support it. If an analyst has to connect to five systems to retrieve the same data, that is too much mental movement that could be better spent formulating and answering questions about the investigation. Furthermore, if organizations limit access to raw data it could force the analyst to make additional mental moves that slow down their progress.

Chefs make minimal trips to the fridge/pantry. When you are cooking dinner at home you likely make multiple trips to the fridge to get ingredients or to the pantry to retrieve spices during the course of your meal. That might look something like this:

“I think this soup needs a bit more tarragon, let me go get it. “

or…

“I forgot I need to add an egg to the carbonara at the end, I’ll go get it from the fridge.”

Building on the concept of mise en place, professional chefs minimize their trips to the fridge and pantry so that they always have the ingredients they need with as few trips as possible. This ensures they are focused on their task, and also minimizes prep and clean up time. They also ensure that they get an appropriate amount of each ingredient to minimize space, clean up, and waste.

chef_mise

Figure 2: Chef’s Gather and Lay Out Ingredients for Multiple Dishes – Mise en Place

One of the most common tasks an analyst will perform during an investigation is retrieval of data in an attempt to answering questions. This might include querying a NetFlow database, pulling full packet capture data from a sensor, or querying log data in a SIEM.

Inexperienced analysts often make two mistakes. The first is not retrieving enough data to answer their questions. This means that the analyst must continue to query the data source and retrieve more data until they get the answer they are looking for. This is equivalent to a chef not getting enough flour from the pantry when trying to make bread. On the flip side, another common pitfall is retrieving too much data, which is an even bigger problem. In these situations an analyst may not limit the time range of their query appropriately, or simply may not use enough filtering. The result is a mountain of data that takes a significant amount of time to wade through. This is equivalent to a chef walking back from the fridge with 100 eggs when they only intend to make a 3-egg omelet.

Learning how to efficiently query data sources during an investigation is product of asking the right questions, understanding the data you have available, and having the data in a place that is easily accessible and reasonably consolidated. If you can do these things you should be able to ensure you are making less trips back to the pantry.

Chefs carefully select, maintain, and master their tools. Most chefs spend a great deal of time and money purchasing and maintaining their knives. They sharpen their knives before every use, and have them professionally refinished frequently. They also spend a great deal of time practicing different types of cuts. A dull or improperly used knife can result in inconsistently cut food, which can lead to poor presentation and even cause under or overcooked food if multiple pieces of food are cooked together but are sized differently. Of course, this could also lead to you accidentally cutting yourself. These concepts go well beyond knives; a bent whisk can result in clumped batter, and an unreliable broiler can burn food. Chefs have to select, maintain, and master a variety of tools to perform their job.

chef_tools

Figure 3: A Chef’s Travel Kit Provides Well-Cared For Essential Tools

In a security investigation tools certainly aren’t everything, but they are critically important. In order analyze network communication you have to understand the protocols involved at a fundamental level, but you also need tools to sort through them, generate statistics, and work towards decision points. Whether it is a packet analysis tool like Wireshark, a flow data analysis tool like SiLK, or an IDS like Snort, you have to understand how those tools work with your data. The more ambiguity placed between you and raw data, the greater chance for assumptions that could lead to poor decisions. This is why it is critical to understand how to use tools, and how they work.

Caring for tools goes well beyond purchasing hardware and ensuring you have enough servers to crunch data. At an organization level it requires hiring the right number of people in your SOC to help manage the infrastructure. Some organizations attempt to put that burden on the analysts, but this isn’t always scalable and often results in analysts being taken away from their primary duties. This is also the “piling on” of responsibilities that results in analysts getting frustrated and leaving a job.

Beyond this, proper tool selection is important as well. I won’t delve into this too much here, but careful consideration should be given to free and open source tools, as well as the potential for developing in house tools. Enterprise solutions have their place, but that shouldn’t be the default go-to. The best work in information security in most cases is still done at the free and open source level. You should look for tools that support existing processes, and never let a tool alone dictate how you conduct an investigation.

Chefs can cook in any kitchen. When chefs master all of the previously mentioned concepts, it allows them to apply those concepts in any location. If you watch professional cooking competitions, you will see that most chefs come with only their knife kit and are able to master the environment of the kitchen they are cooking in. For example, try watching “Chopped” sometime on Food Network. These chefs are given short time constraints and challenging random ingredients. They organize their workspace, assess their tools, make very few trips to get ingredients, and are able to produce five star quality meals.

chef_chopped

Figure 4: Professional Chef’s Competing in an Unfamiliar Kitchen on Food Network’s Chopped

In security investigations, this is all about understanding the fundamentals. Yes, tools are important as I mentioned earlier, but you won’t always work in an environment that provides the same tools. If you only learn how to use Arcsight then you will only ever be successful in environments that use Arcsight. This is why understanding higher-level investigative processes that are SIEM-independent is necessary. Even at a lower level, understanding a tool like Wireshark is great, but you also need to understand how to work with packets using more fundamental and universal tools like tcpdump, as you may not always have access to a graphical desktop. Taking that step further, you should also understand TCP/IP and network protocols so that you can make better sense of the network data you are analyzing without relying on protocol dissectors. A chef’s fundamental understanding of food and cooking methods allows them to cook successfully in any kitchen. An analyst’s fundamental understanding of systems and networking allows them to investigate in any SOC.

Conclusion

Humans have been cooking food for thousands of years, and have been doing so professionally for much longer than computers have even existed. While the skills needed to be chef are dramatically different than those needed to investigate network breaches, there are certainly lessons to be learned here. Now, if you’ll excuse me, writing this has made me hungry.

* Figures 1-3 are from “The Four-Hour Chef” by Tim Ferriss. One of my favorite books.

MIRCon 2014 Slides: Applied Detection and Analysis with Flow Data

I recently had the opportunity and pleasure to speak at MIRCon 2014. The topic of the presentation was “Applied Detection and Analysis with Flow Data.” We had a great time talking about effective ways to use flow data for NSM, as well as introducing the world to FlowBAT.

 

You can view the slides from this presentation here:

BSides Augusta 2014 Slides and Video – Defeating Cognitive Bias and Developing Analytic Technique

I recently gave a presentation at BSides Augusta on the topic “Defeating Cognitive Bias and Developing Analytic Technique”.

Description:

At the center of many defensive processes is human analysis. While we spend a lot of time performing analysis, we don’t spend nearly enough time thinking about how we perform analysis. The human mind is poorly wired to deal with most complex analysis scenarios effectively. This can be attributed to the inherent complexity of solving technical issues where so many uncertainties exist, and also to the cognitive and unmotivated biases that humans unknowingly apply to their analysis. All of these things can diminish our ability to get from alert to diagnoses quickly and effectively.

In this presentation, I plan to discuss the mental challenges associated with technical defensive analysis by leveraging research associated with traditional intelligence analysis. I will discuss how complexity can overwhelm analysis, how cognitive bias can negatively influence analysis, and techniques for recognizing and overcoming these limiting factors. This will include a few fun mental exercises, as well as an overview of several strategic questioning techniques including analysis of competing hypothesis, red cell analysis, and “what if” analysis. Finally, I will discuss several structured analysis techniques, including two different techniques that can be used specifically for NSM analysis: relational investigation and differential diagnosis.

 

The video for this presentation can be found here:

The slides for this presentation can be found here: