Launching Makerspaces Across Rural America

When I first started the Rural Technology Fund in 2008 the goal was to provide a few small scholarships every year to students from rural areas pursing an education in a computer-related discipline. Coming from a very rural area myself, I know how hard it is to pursue a passion in an area where no industry exists to support it and where public schools can barely afford basic classroom supplies. As time progressed, the reach of the RTF expanded by repurposing used technical books to public school libraries, supporting high school and university computer science clubs, and donating educational equipment to classrooms. In our short existence, we’ve managed to make a tremendous impact through the efforts of a few volunteers and very few donations. You can learn more about on our impact page here.

As we look to 2016, I believe it’s time that the RTF continues to evolve and grow to impact more students in more places. We are going to continue doing everything we’ve been doing up to this point, but we are also going to challenge ourselves to commit to building 10 makerspace labs in public schools in 2016. This is a huge commitment that will positively impact a great number of students, helping to ensure that students from rural and low income areas have every chance to succeed in high demand computer-related fields.

Please watch the short video below that explains why the RTF exists, and how we are using the MARA project to address a great need.

If you are interested in contributing, you can learn more about the MARA project by visiting Technology jobs have the power to change lives and restore communities. With your help, we can empower young people to accomplish these feats!

Infosec Practitioner’s Guide to Philanthropy

elevatordownI came from a small town in western Kentucky where there wasn’t a lot of opportunity. As the son of a trucker and a sewing machine operator we struggled to get by and there were a lot of things stacked up against me as a kid who wanted to be successful in an industry that was not represented at all in my area. Despite those odds, I was fortunate to live in a place where community mattered, and several teachers and others in my life gave of themselves to ensure that I had a fighting chance. Since then, I vowed that no matter where I went or what I did that I would always remember where I came from and give of myself to others. It’s for that reason that I’m passionate about giving back to communities like the one I grew up in, and that’s also why I started the Rural Technology Fund seven years ago. As Kevin Spacey says, “If you’re lucky enough to do well, it’s your responsibility to send the elevator back down.”

Before I get too nostalgic, let me give you my word that the purpose of this post isn’t to make you feel lazy or send you on a guilt trip. While charity is important to me, I recognize that it isn’t to everyone and some people simply aren’t in a position to give their time or money. I write a lot about the importance of giving back, and people frequently ask me what they can do to contribute in a way that is relevant to the profession of information security. The purpose of this post is to discuss a few different ways you can contribute.

What Do You Care About?

Once you’ve decided you want to give back in some way, the first thing you should consider is what you really care about. If you work in infosec then you probably care about technology, but in what way? There are a lot of great technology-focused causes:

Beyond technology and security your interests might lie in other areas. When you watch the news, what stories draw out the emotion in you? What circumstances of the human condition make you angry? These are questions that can help you discover areas you care about. For example:

*Note: I don’t necessarily support all of these organizations, these are just examples. 

Even if you want to support non-technical charities there are ways you can use your technical skills to do so, which I’ll talk about later. Don’t donate your time or money to an organization just because you don’t know what else to do. Everybody cares about something and if you can think of an issue you care about then there is a good chance there is a philanthropic organization out there trying to address that issue.

What Kind of Commitment Can You Make?

Once you’ve thought about what you care about, the next step is to figure out what level commitment you can make. Do you want to donate money, time, or both? This section is organized with that in mind. While reading through this section you should try to be realistic about your commitment. If you are living paycheck to paycheck you probably don’t want to commit to donating hundreds of dollars to a charity. On the flip side, if you work eighty hours a week you probably don’t want to commit to another twenty hours of volunteer work. The last thing you want is to over commit yourself and burn out on something that should be an enjoyable experience.

I have extra cash, but limited time…

Donating to Traditional Charity

The most common method of giving back is to donate money directly to an existing charity. When you do this you are supporting the broader mission of a non-profit, or when possible a specific initiative of the organization. Some popular options include the Rural Technology Fund (shameless plug) and Hackers for Charity.


  • Very low time commitment
  • Very easy and can often be done online in a matter of minutes
  • You rely on experts to turn your donation into impact
  • A lot of people donating small amounts can enable a charity to do great things
  • Donations are tax deductible for 501(c)(3) non-profits


  • You don’t always get to see the direct benefits of your donation
  • Administrative costs can eat up some of your donation
  • You have little or no say regarding the exact use or allocation of the money
  • An organization that shares your interests may not exist
  • There aren’t a lot of information security related non-profits

Tips for Success:

  • Read as much as you can about the non-profit. Be sure you understand their mission statement, leadership, and how your money will be used.
  • Look for a statistic on what percentage of every dollar is spent on administrative costs. The less the better.
  • If you believe in an organization enough to donate, consider buying some of their swag to help spread awareness.
  • Make sure you are donating to a 501(c)(3) and track your donations for tax season.

Donating to an Interactive Charity

A modern trend in charitable organizations is the concept of allowing donors to connect directly with the people they are helping. These provide a higher level of interactivity compared to traditional non-profits so you can feel a bit more connected to the cause you are contributing to. Two popular interactive charities are Kiva and Donors Choose. Kiva allows donors to provide micro-loans to entrepreneurs in developing nations, and then reloan that money back out when it is paid back. Donors Choose provides a mechanism for teachers to request things they need for their classrooms so that donors can pick the classroom they want to support. I’m a big fan of interactive charities because it makes me feel a lot more connected to those I’m trying to help.


  • Low time commitment other than spending time selecting how your donation is used
  • You rely on experts to vet potential recipients and deliver funding
  • Higher interaction than traditional charities
  • Helps you feel much more connected to those you are helping
  • Selecting donation recipients can be a fun family activity
  • Donations are tax deductible for 501(c)(3) non-profits


  • You don’t always get to see the direct benefits of your donation
  • Administrative costs can eat up some of your donation
  • You have little or no say regarding the exact use or allocation of the money
  • An organization that shares your interests may not exist
  • There aren’t a lot of information security related non-profits

Tips for Success:

  • Consider giving a donation as a gift. Many of these organizations allow you to buy gift cards to give away so that others can try making a targeted donation. This is a great way to get others involved in an interactive charity so they can see how it works.
  • When you can try involving others. Make a night of looking through donation recipients with your spouse or kids. It’s a great way to instill a sense of charity in others and provides bonding opportunity.
  • These sites often serve a lot of causes, so try to make your donation mean something to you. On Kiva you can make microloans to individuals running Internet cafes or selling computer services. On Donors Choose you can donate computers and electronics to classrooms. On both sites you can also select the region of your donation recipient.
  • Make sure you are donating to a 501(c)(3) and track your donations for tax season.

I have a lot of free time, but not much extra cash…

While many think that the only way to give back is to donate money, the truth is that donating your time is often much more valuable. When you donate time, you have the opportunity to interact and network with other like-minded people and you often get to interact directly with those you are trying to help. There is no greater motivator than witnessing the change you’ve helped make.

Volunteer Your Skills

The thing that makes security so hard is that you have to be a programmer, sysadmin, network engineer, and auditor all in one. Guess what? Those are all skills that are very helpful for non-profit organizations that struggle to make a positive impact and manage administrative costs. Leveraging your skills gives you a great opportunity to serve a non-infosec related cause that you care about. For instance, if you are interested in housing for under privileged families, consider looking up your local Habitat for Humanity chapter and seeing if they need any help keeping their computers or network running, or working on their website. When you donate your time to help with technical tasks it means the organization doesn’t have to pay for those services, which allows more of their funding to go towards mission goals.


  • Provides a direct impact to a charitable organization
  • Allows you to use technical skills to positively impact organizations that aren’t necessarily technology focused
  • Generates opportunities for networking with like minded people and others who share your skills
  • Provides an opportunity to keep up technical skills you might not use as much


  • Can be time consuming
  • May require some level of freelance contract with the organization
  • Could require you to accept some level of responsibility for things you are helping with, like keeping a website running or getting calls when systems go down

Tips for Success:

  • Start local. There are probably a lot of organizations in your area that could use your help.
  • Think about what you are passionate about and look for organizations that support related causes.
  • Consider working with local industry professional associations like ISSA to arrange group volunteering.
  • When nothing is available locally, look for remote volunteer opportunities. Certain technical skills can be of help without having to be in the same city as the organization you’re supporting.

Share Your Knowledge

One of the things that makes the information security community great is that we have so many people who are willing to share knowledge. While giving presentations at local high schools or colleges isn’t as sexy as speaking at Defcon, it can be just as impactful. You can share your knowledge at many levels in all sorts of venues from high schools to universities to senior citizen communities. While not everybody is going to get something out of discussing exploits and detection systems, its highly likely your security experience can appeal to a much larger audience at a more fundamental level.


  • Sharing knowledge with people who care enough to receive it is very gratifying
  • Exposing younger people to information security might spark their interest in pursuing it as a career
  • We all know how important end user security is, so sharing techniques for being safer on the Internet is in the best interest of our industry
  • Speaking events like this are a great way to get more connected with your community


  • In smaller areas it may be hard to find and organize events at first until people know more about you
  • Planning and giving presentations can be very time consuming
  • Not everyone is comfortable teaching or speaking

Tips for Success:

  • Contact school administrators to gauge their interested in what you have to offer. Be professional and be prepared with a few ideas for topics and a basic outline of what you’d like to speak about and why it’s important.
  • Considering reaching out to your local city or county government to see if they could use your help. If you live in a small town then chances are that they don’t even have a security staff, so some basic user awareness training could go a long way.
  • Try organizing basic “How to Use the Internet” classes for the elderly in assisted living communities. Teach them how to get online and how to use e-mail so they can receive pictures from their family.
  • If your town has a college or university, see if they have an infosec or technology users group that might benefit from your experience. Consider giving a conference-style presentation or helping setup some hands-on labs. College students often yearn for hands on experience beyond the theory they get in the classroom.

I don’t have a lot of free time or any extra cash…

Get Your Employer Involved

Most organizations of any reasonable size have at least a few people who share common philanthropic interest. In many cases organizations are motivated to support these causes because it is tax incentivized and helps the overall company brand.


  • It can provide a way for those who typically couldn’t afford to give back the opportunity to be able to do so
  • Supporting charities in this manner is good for the company brand, and is also tax incentivized
  • Employer sponsored giving can provide great team building opportunities
  • Harnessing the giving power of an entire company can have dramatic and tangible benefits to certain causes


  • Your employer might not always be able to contribute to a cause specifically relevant to you
  • It may take a fair bit of convincing to get your employer on board initially

Tips for Success:

  • Ask your boss about providing matching contributions to charitable organizations. Many companies will provide a donation match up to a certain percent, and some will allow for those donations to come from pre-tax income.
  • Get your coworkers together and organize an after hours volunteering event. Something as simple as picking up trash on the side of the road or serving meals as a homeless shelter instills a sense of service and is great for team building.
  • Ask your employer about sponsoring a “Philanthropy Day.” This is an extra day off provided to every employee that they can use to serve the needs of an eligible philanthropic organization. This is low cost to the employer and it can be use to facilitate group volunteering events like I mentioned above. FireEye provides a philanthropy day every year and it was one of my absolute favorite parts of our benefits package.


The first step for any non-profit organization is to advocate for their cause to raise awareness. If people don’t know that a problem exists they won’t be willing to help work towards a solution. Advocacy is very important and it is happening around us all the time. For example, years of advocacy is why many of us think about breast cancer awareness when we see someone wearing pink. Spreading the word about causes you care about isn’t time consuming, usually costs little or no money, and can inspire those with free time or disposable income to take action.


  • Requires very little time investment
  • Requires little or no money
  • Inspires others to take care and take action


  • Most of the impact of your work is indirect so you won’t always see immediate benefits

Tips for Success:

  • Use social media to highlight information relation to the problem you want to address or the work of organizations who are trying to help.
  • Consider buying clothes branded by the organizations you want to support. Something as simple as a t-shirt is cheap, the money supports the organization, and when people see you wearing it that raises awareness.
  • Think about purchasing items through Amazon Smile. It allows you to direct a small percentage of your purchase to eligible non-profits and gives you an opportunity to share your actions via social media.


While this guide certainly isn’t an all inclusive listing of every way you can give back, my hope is that it has provided some ideas for you to channel your desire to give into something useful. Not everyone is in a position to give back, but if you have a desire to do so then it is possible even with very little free time or money to part with. Are you doing something cool to give back that I don’t have listed here? Consider putting it in the comments below.

Inattentional Blindness in Security Investigations

*Disclaimer: Psychology Related Blog Post*

bellJoshua woke up on a frigid Friday morning in Washington, DC and put on a black baseball cap. He walked to the L’Enfant metro station terminal and found a nice visible spot right near the door where he could expect a high level of foot traffic. Once positioned, he opened his violin case, seeded it with a handful of change and a couple of dollar bills, and then began playing for about 45 minutes.

During this time thousands of people walked by and very few paid attention to Joshua. He received several passing glances while a small handful stopped and listened for a moment. Just a coupe lingered for more than a minute or two. When he finished playing, Joshua had earned about twenty-three dollars beyond the money he put into the case himself. As luck would have it, twenty of those dollars came from one individual who recognized Joshua.

Joshua Bell is not just an ordinary violin player. He is a true virtuoso who has been described as one of the best modern violinist in the world, and he has a collection of performances and awards to back it up. Joshua walked into that metro terminal, pulled out a three hundred year old Stradivarius violin, and played some of the most beautiful music that most of us will hear in our lifetime. That leaves the glaring questions: why did nobody notice?

Inattentional Blindness

Inattentional blindness (IB) is an inability to recognize something in plain sight, and it is responsible for the scenario we just described. You may have heard this term before if you’ve had the opportunity to subject yourself to this common selective attention test:

As humans, the ability to focus our attention on something is a critical skill. You focus when you’re driving to work in the morning, when you are performing certain aspects of your job, and when you are shopping for groceries. If we didn’t have the ability to focus our attention, we would have a severely limited ability to perceive the world around us.

The tricky thing is that we have limited attention spans. We can generally only focus on a few at a time, and the more things we try to focus on, the less overall focus can be applied to any one thing. Because of this, it is easy to miss things that are right in front of us when we aren’t focused on finding them. In addition, we also tend to perceive what we expect to perceive. These factors combine to produce situations that allow us to miss things right in front of our eyes. This is why individuals in the metro station walked right by Joshua’s performance. They were focused on getting to work, and did not expect a world-class performer to be playing in the middle of the station on a Friday morning.

Manifestations in Security

As security investigators, we must deal with inattentional blindness all the time. Consider the output shown in Figure 1. This screenshot shows several TCP packets. At first glance, these might appear normal. However, an anomaly exists here. You might not see it because it exists in a place that you might not expect it to be, but it’s there.


Figure 1: HTTP Headers

In a profession where we look at data all day it is quite easy to develop expectations of normalcy. As you perform enough investigations you start to form habits based on what you expect to see. In the case of investigating the TCP packets above, you might expect to find unexpected external IP addresses, odd ports, or weird sequences of packets indicating some type of scan. As you observe and experience these occurrences and form habits related to how you discover them, you are telling your mind to build cognitive shortcuts so that you can analyze data faster. This means that your attention is focused on examining these fields and sequences, and other areas of these packets lose part of your attention. While cognitive shortcuts like these are helpful they can also promote IB.

In the example above, if you look closely at other parts of the packets, you will notice that the third packet, a TCP SYN packet initiating the communication between and actually has a data length value of 5. This is peculiar because it isn’t customary to see data present in a TCP SYN packet whose purpose is simply to establish stateful communication via the three-way handshake process. In this case, the friendly host in question was infected with malware and was using these extra 5 bytes of data in the TCP SYN to check in to a remote host and provide its status. This isn’t a very common technique, but the data is right in front of our face. You might have noticed the extra data in the context of this article because the nature of the article made you expect something weird to be there, but in practice, many analysts fail to notice this data point.

Let’s look at one more example. In Figure 2 we see a screen populated with alerts from multiple sources fed into the Sguil console. In this case, we have a screen full of anomalies waiting to be investigated. There is surely evil to be found while digging into these alerts, but one alert in particular provides a unique anomaly that we can derive immediately. Do you see it?


Figure 2: Alerts in Sguil

Our investigative habits tell us that the thing we really need to focus on when triaging alerts is the name of the signature that fired. After all, it tells us what is going on and can relay some sense of priority to our triage process. However, take a look at Alert 2.84. If you observe the internal (RFC1918) addresses reflected in all of the other alerts, they all relate to devices in the range. Alert 2.84 was generated for a device in the range. This is a small discrepancy, but if this is not on a list of approved network ranges then there is a potential for a non-approved device on the network. Of course, this could just be a case of someone plugging a cheap wireless access point into the network, but it could also be a hijacked virtual host running a new VM spun up by an attacker, or a Raspberry Pi someone plugged into a hidden wall jack to use as an entry point on to your network. Regardless of the signature name here, this alert is now something that warrants more immediate attention. This is another item that might not be spotted so easily, even by the experienced analyst.

Everyone is susceptible to IB, and it is something we battle ever day. How can we try to avoid missing things that are right in front of our eyes?

Diminishing the Effects

The unfortunate truth is that it isn’t possible to eliminate IB because it is a product of attention. As long as we have the ability to focus our attention in one area, then we will become blind to things outside of that area. With that said, there are things we can do to diminish some of these affects and improve our ability to investigate security incidents and ensure we don’t miss as much.


The easiest way to diminish some of the affects of IB is through expertise in the subject matter. In our leading example we mentioned that there were a few people who stopped to listen to Joshua play his violin in the station. It is useful to know that at least two of those people were professional musicians themselves. Hearing the music as they walked through the station triggered the right mechanisms in their brain to allow them to notice what was occurring, compelling them to stop. This was because they are experts in the field of music and probably maintain a state of awareness related to the sound of expert violin playing. Amongst the hustle and bustle of the metro station, their brain allowed them not to miss the thing that people without that expertise had missed.

In security investigations it’s clear to see IB at work in less experienced analysts. Without a higher level of expertise these junior analysts have not learned how to focus their attention in the right areas so that they don’t miss important things. If you hand a junior analyst a packet capture and ask them where they would look to find evil, chances are their list of places to look would be much shorter than a senior analyst, or it would have a number of extraneous items that aren’t worth being included. They simply haven’t tuned their ability to focus attention in the right places.

More senior analysts have developed the skill to be able to selectively apply their attention, but they rarely have the ability to codify it or explain it to another person. The more experienced analysts get at identifying and teaching this information, the better chance of younger analysts getting necessary expertise faster.

Directed Focus

While analysts spend most of their time looking at data, that data is often examined through the lens of tools like SIEMs, packet sniffers, and command line data manipulation utilities. As a young industry, many of these tools are very minimal and don’t provide a lot of visual cues related to where attention should be focused. This is beneficial in some ways because it leaves the interpretation fully open to the analyst, but without having opinionated software this sort of thing promotes IB. As an example, consider the output of tcpdump below. Tcpdump is one of the tools I use the most, but it provides no visual queues for the analysts.


Figure 3: Tcpdump provides little in the way of visual cues to direct the focus of attention

We can compare Tcpdump to a tool like Wireshark, which has all sorts of visual cues that give you an idea of things you need to look at first. This is done primarily via color coding, highlighting, and segmenting different types of data. Note that the packet capture shown in Figure 3 is the same one shown in Figure 4. Which is easier to visually process?


Figure 4: Wireshark provides a variety of visual cues to direct attention.

It is for this reason that tools developed by expert analysts are desirable. This expertise can be incorporated into the tool, and the tool can be opinionated such that it directs users towards areas where attentional focus can be beneficial. Taking this one step farther, tools that really excel in this area allow analyst users to place their own visual cues. In Wireshark for example, analysts can add packet comments, custom packet coloring rules, and mark packets that are of interest. These things can direct attention to the right places and serve as an educational tool. Developing tools in this manner is no easy task, but as our collective experience in this industry evolves this has to become a focus.

Peer Review

One last mechanism for diminishing the affects of IB that warrants mention is the use of peer review. I’ve written about the need for peer review and tools that can facilitate it multiple times. IB is ultimately a limitation that is a product of an analyst training, experience, biases, and mindset. Because of this, every analyst is subject to his or her own unique blind spots. Sometimes we can correlate these across multiple analyst who have worked in the same place for a period of time or were trained by the same person, but in general everyone is their own special snowflake. Because of this, simply putting another set of eyes on the same set of data can result in findings that vary from person to person. This level of scrutiny isn’t always feasible for every investigation, but certainly for incident response and investigations beyond the triage process, putting another analyst in the loop is probably one of the most effective ways to diminish potential misses as a result if IB.


Inattentional blindness is one of many cognitive enemies of the analyst. As long as the human analyst is at the center of the investigative process (and I hope they always are), the biggest obstacle most will have to overcome is their own self imposed biases and limitations. While we can never truly overcome these limitations without stripping away everything that makes us human, an awareness of them has been scientifically proven to increase performance in similar fields. Combining this increased level of metacognitive awareness with an arsenal of techniques we can do to minimize the effect of cognitive limitations will go a long way towards making us all better investigators and helping us catch more bad guys.

Investigations and Prospective Data Collection

confused-winnerOne of the problems we face while trying to detect and respond to adversaries is in the sheer amount of data we have to collect and parse. Twenty years ago it wasn’t as difficult to place multiple sensors in a network, collect packet and log data, and store that data for quite some time. In modern networks, that is becoming less and less feasible. Many others have written about this at length, but I want to highlight two main points.

Attackers play the long game. The average time from breach to discovery is over two hundred days. Despite media jargon about “millions of attacks a day” or attacks happening “at the speed of light”, the true nature of breaches is that they are not speedy endeavors from the attackers side. Gaining a foothold in a network, moving laterally within that network, and strategically locating and retrieving target data can take weeks or months. Structured attackers don’t win when they gain access to a network. They win once they accomplish their objective, which typically comes much later.

Long term storage isn’t economical. While some organizations are able to store PCAP or verbose log data in terms of months, that is typically reserved for incredibly well funded organizations or the gov/mil, and is becoming less common. Even on smaller networks, most can only store this data in terms of hours, or at most a few days. I typically only see long term storage for aggregate data (like flow data) or statistical data. The amount of data we generate has dramatically outgrown our capability to store and parse through that data, and this issue it only going to worsen for security purposes.

Medicine and Prospective Collection

The problem of having far too much data to collect and analyze is not unique to our domain. As I often do, let’s look towards the medical field. While the mechanics are a lot different, medical practitioners rely on a lot of the same cognitive skills to investigate afflictions to the human condition that we do to investigate afflictions to our networks. These are things like fluid ability, working memory, and source monitoring accuracy all work in the same ways to help practitioners get from a disparate set of symptoms to an underlying diagnosis, and hopefully, remediation.

Consider a doctor treating a patient experiencing undesirable symptoms. Most of the time a doctor can’t look back at the evolution of a persons health over time. They can’t take a CAT scan on a brain as it was six months ago. They can’t do an ultrasound on a pancreas as it was two weeks ago. For the most part, they have to take what they have in front of them now or what tests can tell them from very recent history.

If what is available in the short term isn’t enough to make a diagnosis, the physician can determine criteria for what data they want to observe and collect next. They can’t perform constant CAT scans, ultrasounds, or blood tests that look for everything. So, they apply their skills and define the data points they need to make decisions regarding the symptoms and the underlying condition they believe they are dealing with. This might include something like a blood test every day looking at white blood cell counts, continual EKG readings looking for cardiac anomalies, or twice daily neurological response tests. Medical tests are expensive and the amount of data can easily be overwhelming for the diagnostic process. Thus, selectively collecting data needed to support a hypothesis is employed. Physicians call this a clinical test-based approach, but I like to conceptualize it as prospective data collection. While retrospective data looks at things that have previously been collected up until a point in time, prospective data collections rely on specific criteria for what data should be collected moving forward from a fixed point in time, for a set duration. Physicians use a clinical strategy with a predominate lean towards effective use of prospective data collection because they can’t feasibly collect enough retrospective data to meet their needs. Sound familiar?

Investigating Security Incidents Clinically

As security investigators, we typically use a model based solely on past observations and retrospective data analysis. The prospective collection model is rarely leveraged, which is surprising since our field shares many similarities with medicine. We all have the same data problems, and we can all use the same clinical approach.

The symptoms our patients report are alerts. We can’t go back and look at snapshots of a devices health over the retrospective long-term because we can’t feasibly store that data. We can look back in the near term and find certain data points based on those observations, but that is severely time limited. We can also generate a potential diagnosis and observe more symptoms to find and treat the underlying cause of what is happening on our networks.

Let’s look at a scenario using this approach.

Step 1

An alert is generated for a host (System A). The symptom is that multiple failed login attempts where made on the devices administrator account from another internal system (System B). 

Step 2

The examining analyst performs an initial triage and comes up with a list of potential diagnoses. He attempts to validate or invalidate each diagnosis by examining the retrospective data that is on hand, but is unable to find any concrete evidence that a compromise has occurred. The analyst determines that System B was never able to successfully login to System A, and finds no other indication of malicious activity in the logs. More analysis is warranted, but no other data exists yet. In other scenarios, the investigation might stop here barring any other alerting. 

Step 3

The analyst adds his notes to the investigation and prunes his list of diagnoses to a few plausible candidates. Using these hypothesis diagnoses as a guide, the analyst generates a list of prospective collection criteria. These might include:

  • System A: All successful logins, newly created user accounts, flow data to/from System B.
  • System B: File downloads, attempted logins to other internal machines, websites visited, flow data to/from System A.

This is all immensely useful data in the context of the investigation, but it doesn’t break the bank in terms of storage or processing costs if the organization needs to store the data for a while in relation to this small scope. The analyst tasks these collections to the appropriate sensors or log collection devices. 

Step 4

The prospective collections record the identified data points and deliver them exclusively to the investigation container they are assigned to. The analyst collects these data points for several days, and perhaps refines them or adds new collections as data is analyzed.

Step 5

The analyst revisits and reviews the details of the investigation and the returned data, and either defines additional or refined collections, or makes a decision regarding a final diagnosis. This could be one of the following:

  • System B appears to be compromised and lateral movement to System A was being attempted.
  • No other signs of malicious activity were detected, and it was likely an anomaly resulting from a user who lost their password. 

In a purely retrospective model the later steps of this investigation might be skipped, and may lead the analyst to miss the ground truth of what is actually occurring. In this case, the analyst plays the long game and is rewarded for it.

Additional Benefits of Prospective Collection

In addition to the benefits of making better use of storage resources, a model that leverages prospective collection has a few other immediate benefits to the investigative process. These include:

Realistic-Time Detection. As I’ve written previously, when the average time from breach to detection is greater than two hundred days, attempting to discover attackers on your network the second they gain access is overly ambitious. For that matter, it doesn’t acknowledge the fact that attackers may already be inside your network. Detection can often its hardest at the time of initial compromise because attackers are typically more stealthy at this point, and because less data exists to indicate they are present on the network. This difficulty can decrease over time as attackers get sloppier and generate more data that can indicate their presence. Catching an attacker +10 days from initial compromise isn’t as sexy as “real time detection”, but it is a lot more realistic. The goal here is to stop them from completing their mission. Prospective collection supports the notion of realistic-time detection.

Cognitive Front-Loading. Research shows us that people are able to solve problems a lot more efficiently when they are aware of concepts surrounding metacognition (thinking about thinking) and are capable of applying that knowledge. This boils down to have an investigative philosophy and a strategy for generating hypotheses and having multiple approaches towards working towards a final conclusion. Using a prospective collection approach forces analysts to form hypotheses early on in the process, promoting the development of metacognition and investigation strategy.

Repeatability and Identified Assumptions. One of the biggest challenges we face is that investigative knowledge is often tacit and great investigators can’t tell others why they are so good at what they do. Defining prospective collection criteria provides insight towards what great investigators are thinking, and that can be codified and shared with less experienced analysts to increase their abilities. This also allows for more clear identification of assumptions so those can be challenged using structured analytic techniques common in both medicine and intelligence analysis. I wrote about this some here, and spoke about it last year here.


The purpose of this post isn’t to go out and tell everyone that they should stop storing data and refocus their entire SOC towards a model of prospective collection. Certainly, more research is needed there. As always, I believe there is value in examining the successes and failures of other fields that require the same level of critical thinking that security investigations also require. In this case, I think we have a lot to learn from how medical practitioners manage to get from symptoms to diagnosis while experiencing data collection problems similar to what we deal with. I’m looking forward to more research in this area.

On the Importance of Questions in an Investigation

questionsI spend a large part of my day studying cognition related to security investigations, which can ultimately be boiled down to thinking about how we learn and process information during and around our investigative processes. As part of my research, one of my professors recently pointed me towards a TEDx video by Dan Rothstein entitled “Did Socrates Get it Wrong?”. In this fourteen minute talk Rothstein questions whether Socrates approach of expert led questioning, commonly referred to as the Socratic method, was wrong. He brings up quite a few fascinating points, but ultimately concludes that Socrates was right and wrong, and that strategic questioning is of the utmost importance, but that it can also be an entirely student lead exercise. The key here is that asking the right question is critical for exploration, and of course, getting to the right answer.

This has quite a few implications to security investigations. Strategic questioning as a means towards finding and eliminating bias is something that immediately comes to mind, but not what I want to talk about here.

At a more fundamental level is questioning as the essence of the investigation process. I tend to believe that an investigation itself is simply a question. Usually something like this:

  • What happened here?
  • Did we get compromised?
  • Did APT[x] access any of our information assets?

Going one step further, I would also hypothesize that every action we take during the course of an investigation can be distilled down into a question, like these:

  • Does the activity identified in this alert match what the signature was trying to detect?
  • Did internal Host A communicate with external Host B?
  • Did the device download and execute the stage two payload of this malware family?
  • Is there a log indicating that a specific file was accessed?

Most of the time these questions don’t materialize in this form. Typically, they develop in our subconscious and analysts go forth looking for answers before they’ve articulated the question fully. I may not actually ask myself “Does the data in this PCAP match what the signature was looking for in the appropriate context?” before I go look at the signature to see what it was attempting to detect, but subconsciously that is exactly what I’m doing. Research suggests that a lot of this can be attributed to the formulation of habits or intuition (potentially in a brain structure known as the precuneus) that help us be more cognitively efficient. While this type of intuition can help us get things done faster, there is immense value in ripping these things from our subconscious into our conscious thought so that they can be articulated.

A couple things come to mind immediately when assessing the value of articulating questions consciously. First, if all of an investigation can be based on questions, we must ensure we are asking the right questions. This requires us to be consciously aware of those questions before we seek to solve them. Second, if we hope to successfully train the next generation of analysts then we have to teach them to ask the right questions, again requiring us to be consciously aware of what they are.

If you are a security investigator or are responsible for training them, consider creating a culture of articulated questions in your SOC. Before acting, attempt to determine what question you are trying to answer and share that information with your peers. I would bet that you will find this type of strategic questioning will help you ask better questions and more effectively guide your investigation towards an appropriate goal.


Dan Rothstein, “Did Socrates Get it Wrong”, TEDx Somerville –