Category Archives: Network Security

Survey: Technical Writing Pain Points

I’m currently working on a multi-part series on technical writing. Specifically, I’m addressing some of the more common things I hear from security pros who hate writing things like penetration test reports and incident reports. I’ll also be addressing informal technical writing, like blog posts.

I’d love to hear from you. What do you dislike about technical writing the most? What is it that makes you dread it or put it off until the lat minute?

Either let me know in the comments, tweet me using the hashtag #ReportingPain, or you can fill out the anonymous survey here: https://www.surveymonkey.com/r/JS9KVX2.

Research Call for Security Investigators

brainicon_blueI’m currently seeking security investigators for a research study I’m conducting on cognition and reasoning related to the investigative process. I need individuals who are willing to sit down with me over the phone and participate in an interview focused on individual investigations they’ve worked. The interviews will be focused on describing the flow of the investigation, your thought process during it, and challenges you encountered. I’ll ask you to describe what happened and how you made specific decisions. Specifically, I’m looking for investigations related to the following areas:

  • Event Analysis: You received some kind of alert and investigated it to determine whether it was a true positive or false positive.
  • Incident Response: You received notification of a breach and performed incident response to locate and/or remediate affected machines.

Ideally, these should be scenarios where you felt challenged to employ a wide range of your skills. In either domain, the scenario doesn’t have to lead to a positive confirmation of attacker activity. Failed investigations that led to a dead end are also applicable here.

A few other notes:

  • You will be kept anonymous
  • Any affected organization names are not needed, and you don’t have to give specifics there. Even if you do, I won’t use them in the research.
  • You will be asked to fill out a short (less than five minute) demographic survey
  • The phone interview will be recorded for my review
  • The phone interview should take no longer than thirty minutes
  • If you have multiple scenarios you’d like to walk through, that’s even better
  • At most, the scenario will be generalized and described at a very high level in a research paper, but it will be done in a generic manner that is not attributable to any person or organization.

If you’d like to help, please e-mail me at chris@chrissanders.org with the subject line “Investigation Case Study.”

Applied Network Security Monitoring, the book!

I’m thrilled to announce my newest project, Applied Network Security Monitoring, the book, along with my co-authors Liam Randall and Jason Smith.

Better yet, I’m excited to say that 100% of the royalties from this book will be going to support some great charities, including the Rural Technology Fund, Hackers for Charity, Hope for the Warriors, and Lighthouse Youth Services.

You can read more about the book, including a full table of contents at its companion site, here: http://www.appliednsm.com/.

Information Security Incident Morbidity and Mortality (M&M)

It may be a bit cliché, but encouraging the team dynamic within an information security group ensures mutual success over individual success. There are a lot of ways to do this, including items I’ve discussed before such as fostering the development of infosec superstars or encouraging servant leadership. Beyond these things, there is no better way to ensure team success within your group than to create a culture of learning. Creating this type of culture goes well beyond sending analysts to formalized courses or paying for certifications. It relies upon adopting the mindset that in every action an analyst takes, they should either be teaching or learning, with no exceptions. Once every analyst begins seeing every part of their daily job as an opportunity to learn something new or teach something new to their peers, then a culture of learning is flourishing.

A part of this type of organizational culture is learning from both successes and failures. The practice of Network Security Monitoring (NSM) and Incident Response (IR) are ones that are centered on technical investigations and cases, and when something bad eventually happens, incidents. This is not unlike medicine, which is also focused on medical investigations and patient cases, and when something bad eventually happens, death.

Medical M&M

When death occurs in medicine, it can usually be classified as something that was either avoidable or inevitable from both a patient standpoint and also as it related to the medical care that was provided. Whenever a death is seen as something that may have been prevented or delayed with modifications to the medical care that was provided, the treating physician will often be asked to participate in something called a Morbidity and Mortality Conference, or M&M as they are often referred to casually. In an M&M, the treating physician will present the case from the initial visit, including the presenting symptoms and the patients initial history and physical assessment. This presentation will continue through the diagnostic and treatment steps that were taken all the way through the patient’s eventual death.

The M&M presentation is given to an audience of peers, to include any other physicians who may have participated in the care of the patient in question, as well as physicians who had nothing to do with the patient. The general premise is that these peers will question the treatment process in order to uncover any mistakes that may have been made or processes that could be improved upon.

The ultimate goal of the medical M&M as a team is to learn from any complications or errors, to modify behavior and judgment based upon experiences gained, and to prevent repetition of errors leading to complications. This is something that has occurred within medicine for over one hundred years and has proven to be wildly successful.

Information Security M&M

I’ve written about how information security can learn from the medical field on multiple occasions, including recently discussing the use of Differential Diagnosis for Network Security Monitoring. The concept of M&M is also something that I think transitions very well to information security.

As information security professionals, it is very easy to miss things. I’m a firm believer that prevention eventually fails, and as a result, we can’t be expected to live in a world free from compromise. Rather, we must be positioned so that when an incident does occur, it can be detected and responded to quickly. Once that is done, we can learn from whatever mistakes occurred that allowed the intrusion, and be better prepared the next time.

When an incident occurs we want it to be because of something out of our hands, such as a very sophisticated attacker or an attacker who is using an unknown zero day. The truth of the matter is that not all incidents are that complex and often times there are ways in which detection, analysis, and response could occur faster. The information security M&M is a way to collect that information and put it to work. In order to understand how we can improve from mistakes, we have to understand why they are made. Uzi Arad summarizes this very well in the book, “Managing Strategic Surprise”, a must read for information security professionals. In this book, he cites three problems that lead to failures in intelligence management, which also apply to information security:

  • The problem of misperception of the material, which stems from the difficulty of understanding the objective reality, or the reality as it is perceived by the opponent.
  • The problems stemming form the prevalence of pre-existing mindsets among the analysts that do not allow an objective professional interpretation of the reality that emerges from the intelligence material.
  • Group pressures, groupthink, or social-political considerations that bias professional assessment and analysis.

The information security M&M aims to provide a forum for overcoming these problems through strategic questioning of incidents that have occurred.

When to Convene an M&M

In an Information Security M&M, the conference should be initiated after an incident has occurred and been remediated. Selecting which incidents are appropriate for M&M is a task that is usually handled by a team lead or member of management who has the ability to recognize when an investigation could have been handled better. This should occur reasonably soon after the incident so important details are fresh on the minds of those involved, but far enough out from the incident that those involved have time to analyze the incident as a whole, post-mortem. An acceptable time frame can usually be about a week after the incident has occurred.

M&M Presenter(s)

The presentation of the investigation will often involve multiple individuals. In medicine, this may include an initial treating emergency room physician, an operating surgeon, and a primary care physician. In information security, this could include an NSM analyst who detected the incident, the incident responder who contained and remediated the incident, the forensic investigator who performed an analysis of a compromised machine, or the malware analyst who reverse engineered the malware associated with the incident.

M&M Peers

The peers involved with the M&M should include at least one counterpart from each particular specialty, at minimum. This means that for every NSM analyst directly involved with the case, there should be at least one other NSM analyst who had nothing to do with it. This aims to get fresh outside views that aren’t tainted by feeling the need to support any actions that were taken in relation to the specific investigation. In larger organizations and more ideal situations, it is nice to have at least two counterparts from each specialty, with one being of lesser experience than the presenting individual and one being of more experience.

The Presentation

The presenting individual or group of individuals should be given at least a few days notice before their presentation. Although the M&M isn’t considered a formal affair, a reasonable presentation is expected to include a timeline overview of the incident, along with any supporting data. The presenter should go through the detection, investigation, and remediation of the incident chronologically and present new findings only as they were discovered during this progression. Once this chronological presentation is given, the incident can then be examined holistically.

During the presentation, participating peers should be expected to ask questions as they arise. Of course, this should be done respectfully by raising your hand as the presenter is speaking, but questions should NOT be saved for after the presentation. This is in order to frame the questions to the presenter as a peer would arrive at them during the investigation process.

Strategic Questioning

Questions should be asked to presenters in such a way as to determine why something was handled in a particular manner, or why it wasn’t handled in an alternative manner. As you may expect, it is very easy to offend someone when providing these types of questions, therefore, it is critical that participants enter the M&M with an open mind and both presenters and peers ask and respond to questions in a professional manner and with due respect.

Initially, it may be difficult for peers to develop questions that are entirely constructive and helpful in overcoming the three problems identified earlier. There are several methods that can be used to stimulate the appropriate type of questioning.

Devils Advocate

One method that Uzi Arad mentions in his contribution to “Managing Strategic Surprise” is the Devils Advocate method. In this method, peers attempt to oppose most every analytical conclusion made by the presenter.  This is done by first determining which conclusions can be challenged, then collecting information from the incident that supports the alternative assertion. It is then up to the presenter to support their own conclusions and debunk competing thoughts.

Alternative Analysis (AA)

R.J. Heuer presents a several of these methods in his paper, “The Limits of Intelligence Analysis”. These methods are part of a set of analytic tools called Alternative Analysis (AA).

Group A / Group B

This analysis involves two groups of experts analyzing the incident separately, based upon the same information. This requires that the presenters (Group A) provide supporting data related to the incident prior to the M&M so that the peers (Group B) can work collaboratively to come up with their own analysis to be compared and contrasted during the M&M. The goal is to establish to individual centers of thought. Whenever points arise where the two groups reach a different conclusion, additional discussion is required to find out why the conclusions differ.

Red Cell Analysis

This method focuses on the adversarial viewpoint, in which peers assume the role of the adversary involved with the particular incident. In doing this, they will question the presenter as to how their investigative steps were completed in reaction to the attackers actions. For instance, a typical defender may solely be focused on finding out how to stop malware from communicating back to the attacker, but the attacker may be more concerned with whether or not the attacker was able to decipher the communication that was occurring. This could lead to a very positive line of questioning that results in new analytic methods that help to better assess the impact of the attacker to benefit containment.

What If Analysis

This method is focused on the potential causes and effects of events that may not have actually occurred. During detection, a peer may ask a question related to how the attack might have been detected if the mechanism that did detect it didn’t do so. In the response to the event, a peer might question what the presenter would have done had the attacker been caught during the data exfiltration process rather than after it had already occurred. These questions don’t always relate directly to the incident at hand, but provide incredibly valuable thought provoking discussion that will better prepare your team for future incidents.

Analysis of Competing Hypothesis

This method is similar to what occurs during a differential diagnosis, where peers crate an exhaustive list of alternative assessments of symptoms that may have been presented. This is most effectively done by utilizing a whiteboard to list every potential diagnosis and then ruling those out based upon testing and review of additional data. You can review my article on differential diagnosis of NSM events here for a more thorough discussion of this type of questioning.

Key Assumptions Check

Most all sciences tend to make assumptions based upon generally accepted facts. This method of questioning is designed to challenge key assumptions and how they affect the investigation of a scenario. This most often pairs with the What If analysis method. As an example, in the spread of malware, it’s been the assumption that when operating within a virtual machine, the malware doesn’t have the ability to escape to the host or other virtual machines residing on it. Given an incident being presented where a virtual machine has been infected with malware, a peer might pose the question of what action might be taken if this malware did indeed escape the virtual environment and infect other virtual machines on the host, or the host itself.

Outcome

During the M&M, all participants should actively take notes. Once the M&M is completed, the presenting individuals should take their notes and combine them into a final report that accompanies their presentation materials and supporting data. This reporting should include a listing of any points which could have been handled differently, and any improvements that could be made to the organization as a whole, either technically or procedurally. This report should be attached the case file associated with the investigation of the incident.

Additional Tips

Having organized and participated in several of these conferences and reviews of similar scope, I have a few other pointers that help in ensuring they provide value.

  • M&M conferences should be held only sporadically, with no more than one per week and no more than three per month.
  • It should be stressed that the purpose of the M&M isn’t to grade or judge an individual, but rather, to encourage the culture of learning.
  • M&M conferences should be moderated by someone at a team lead or lower management level to ensure that the conversation doesn’t get too heated and to steer questions in the right direction.
  • If you make the decision to institute M&M conferences, it should be a requirement that everybody participates at some point, either as a presenter or a peer.
  • The final report that is generated from the M&M should be shared with all technical staff, as well as management.
  • Information security professionals, not unlike doctors, tend to have big egos. The first several conferences might introduce some contention and heated debates. This is to be expected initially, but will work itself out over time with proper direction and moderation.
  • The M&M should be seen as a casual event. It is a great opportunity to provide food and coordinate other activities before and after the conference to take the edge off.
  • Be wary of inviting upper management into these conferences. Their presence will often inhibit open questioning and response and they often don’t have the appropriate technical mindset to gain or provide value to the presentation.

It is absolutely critical that when initiating these conferences, it is done with care. The medical M&M was actually started in the early 1900s by a surgeon named Dr. Ernest Codman at Massachusetts General Hospital in Boston. MGH was so appalled that Dr. Codman suggested that the competence of surgeons should be evaluated that he eventually lost his staff privileges. Now, M&M is a mainstay in modern medicine and something that is done in some of the best hospitals in the world. I’ve seen instances where similar types of shunning occur in information security when these types of peer review opportunities are suggested. As information security practitioners it is crucial that we are accepting of this type of peer review and that we encourage group learning and the refinement of our skills.

References:

  • Campbell, W. (1988). “Surgical morbidity and mortality meetings“. Annals of the Royal College of Surgeons of England 70 (6): 363–365. PMC 2498614.PMID 3207327.
  • Arad, Uzi (2008). Intelligence Management as Risk Management. Paul Bracken, Ian Bremmer, David Gordon (Eds.), Managing Strategic Surprise (43-77). Cambridge: Cambridge University Press.
  • Heuer, Richards J., Jr. “Limits of Intelligence Analysis.” Orbis 49, no. 1 (2005)

4 Ideas for Operationalizing Honeypots

I’ve always thought that the concept of a honeypot was one of the most fascinating things in information security. If you aren’t familiar with honeypots, they are basically traps used to detect or deter attackers on a network. They typically come in two forms; low interaction and high interaction. A low interaction honeypot is software that emulates a set number of services that may run on a computer. When an attacker connects to a low interaction honeypot, he/she will be able to interact with that service on a limited basis, and that interaction will be logged. A high interaction honeypot is more robust and emulates all aspects of an operating system. This is most often a deployed operating system running a number of legitimate services with an extensively level of logging enabled. The thing both of these implementation methods have in common is that the honeypot doesn’t actually contain real data. Should an attacker compromise either type of honeypot, there is no real direct risk of critical data being exposed when deployed properly.

Almost every single honeypot implementation I’ve seen deployed is for research purposes. There isn’t anything wrong with a research honeypot, after all, I run a couple myself (at home) and have learned a lot from it. However, I think there is a lot of operational value that can be gained from deploying honeypots in production environments. I wanted to discuss, at a high level, a few of these strategies and the benefit that can be gained from them.

Honeypots for Prevention

There has been a fair amount of talk recently about security mechanisms designed to drive up the cost of exploiting a network by increasing the time it takes to do so. As a matter of fact, Adobe’s Senior Directory of Product Security and Privacy, Brad Arkin, even recently said that “My goal isn’t to find and fix every security bug. It’s to drive up the cost of writing exploits. We invest a lot of time in building up mitigations that increase the cost and complexity of writing exploits that will become reliable.” Of course, Arkin was referring to the exploitation of software, but the concept still applies to the network side of the house. I’m still a firm believer that your detection capability is still the most important because prevention eventually fails, but if you can drive up the cost of exploiting a network this has the potential to deter some attackers. At a minimum deterring attacks of opportunity can be achieved if you can increase the time cost of exploiting a network, and this may even work to deter attacks of choice as well.

Honeypots can do this by adding to the frustration factor. I see a couple of ways this can be done. The first of which is to utilize a  large number of low interaction honeypots with varying configurations. The important thing here is to vary their configurations as much as possible in order to prevent an attacker from characterizing them and automating them out of their window of visibility. For instance, if you deploy twenty honeypots and they all have ports 22, 80, and 3306 open and all provide the same responses to banner grabs, an attacker is going to be able to correlate this pretty quickly and will simply scan and exclude those hosts from his list of potential targets. The other method for preventive use is to deploy a significant number of high interaction honeypots. This requires a significant time investment, but the right configuration can cause an attacker to waste a significant amount of time in the right places. Again, this strategy isn’t going to prevent aggressive adversaries from reaching their goal, but it will drive up the time cost of lesser determined foes.

Honeypots for Attack Sense and Warning

This is the sacrificial lamb approach to honeypot deployment. In this scenario, honeypots are deployed based upon trust zones within your network. There are different strategies for outlining trust barriers but on a simple network you might define a low trust zone within a wireless or user space network segment, a medium trust zone in a DMZ, and a high trust zone within a server farm network segment. In that sort of topology, all three zones would contain honeypots configured with security comparable to the next step lower. The idea here is that the honeypot should be slightly more vulnerable to attack than everything else in the zone that it is currently in. This configuration provides value in a couple of ways. First, if a honeypot gets compromised, it will likely serve as a warning that other assets within that trust zone may be compromised soon as well, if they aren’t already. Taking this one step further, it is often logical to assume that if a lower trust zone honeypot becomes compromised, the next highest trust zone may be the next target. Depending on how the network is setup, if a higher trust zone includes a honeypot that gets compromised, it could mean that all of the trust zones below it could also have fallen victim to the adversary. This whole model relies on a lot of assumption, but that is the space AS&W operates in.

Honeypots for Detection Related to Critical Assets

I’m a big fan of target-based IDS deployment where instead of deploying a single IDS to your network perimeter, you user more focused IDS’s with finer tuned rule sets and place them closer to organizationally critical assets. This allows for better use of resources across the board as it usually requires less beefy hardware and ensures your analysts won’t see nearly as many false positives. For instance, if your critical data is housed in SQL servers on a single network segment, then deploy an additional IDS to that segment and only utilize SQL focused signatures there rather than on the perimeter IDS. This also allows you to prioritize IDS sensors so that alerts generated by sensors in high priority areas are given priority when it comes to investigations.

I think the same concept of target-based deployment can be tied to Honeypot deployments in the protection of critical assets. If your organization has prioritized their assets (and they should have), then the general idea behind target based honeypot deployment for the purpose of detection would be to configure and deploy honeypots that are virtually identical to the critical servers. This means that they should be running the same services,  talking to the same hosts, and vulnerable to the same types of attacks. The thought here is that if the critical server gets compromised, then so should the honeypot, and vice verse. This is valuable because it isn’t always feasible to log everything on a production server based upon its volume of traffic. This applies to both host based and network based logging. Utilizing an identically configured honeypot that doesn’t see the same amount of utilization allows you to use more aggressive logging, which may allow you to gain more visibility into an attackers movements. This can provide value in helping you determine exactly how an attacker has compromised a system, what they are utilizing the same for, if there is particular data they may be after, and if they have compromised any other systems on the network.

Reverse Honeypots for Intelligence Collection

Although the concept of a reverse honeypot is a bit radical, it really appeals to me considering the industry I work in. The concept of a traditional honeypot is that in which you fill a pot with honey and hope the attacker gets attracted to the honey and sticks his hand in the pot. A reverse honeypot is where you throw some honey in the direction of a target in such a manner as to leave a trail back to the source. The idea being that the target will notice, follow the trail, see the pot, and stick his hand in. In more practical terms, this means that you would attempt to attack a target elsewhere on the Internet. This attack doesn’t necessarily have to be successful and it may just constitute something as simple as a port scan or something as overt as a DoS attempt. During these attacks, no masking of your source IP address should occur and no third party hop points would be used, thus meaning that the target would see your true IP address when reviewing logs of your attack on his network. Given the nature of your target, this may result in his curiosity being peaking and him reciprocating your attack back at you in another form. Of course, within your network you have several vulnerable honeypots of varying interaction levels waiting for the target.

This type of honeypot is solely for the purpose of target based intelligence gathering, but has the potential to be very effective. First and foremost, should the target scan or attack your network you should be able to capture some of the tools, techniques, and procedures (TTPs) that he is using. This type of intelligence can help in recognizing, characterizing, or attributing other computer network exploitation activity to this attacker and may also lend to better detection techniques in the future. One more added value which is incredibly attractive in the modern threat landscape is the identification of hops points. Although you very purposefully did not mask your true source IP address, the attacker may choose to do so. It’s incredibly common for attackers to compromise other hosts elsewhere on the Internet to launch their attacks from, but it’s also common that they will reuse these same hop points for an extended amount of time. If you can identify these hop points then you can use that information to attribute the attacks to a particular operator or group. This is extremely valuable. Of course, this type of activity should be done from non-production networks, because it’s very possible that you might lure an attacker into launching a large scale DDoS attack on your network 10,000 bots strong.

Conclusion

I think there is a lot of room for operationalizing honeypots in production environments. The major factors prohibiting this are a lack of research in this area and a lack of production-grade tools for implementing these techniques. Unfortunately, we are still in a time that IDS is having trouble gaining traction because of the cost it entails, so a future where honeypots can be deployed for the purpose of enhancing network security seems far off. Don’t be surprised however, if you seen a job posting five years down the road for “Honeypot Administrator”. I know I’d have one if I could.