Chris Sanders

I’ve always thought that the concept of a honeypot was one of the most fascinating things in information security. If you aren’t familiar with honeypots, they are basically traps used to detect or deter attackers on a network. They typically come in two forms; low interaction and high interaction. A low interaction honeypot is software that emulates a set number of services that may run on a computer. When an attacker connects to a low interaction honeypot, he/she will be able to interact with that service on a limited basis, and that interaction will be logged. A high interaction honeypot is more robust and emulates all aspects of an operating system. This is most often a deployed operating system running a number of legitimate services with an extensively level of logging enabled. The thing both of these implementation methods have in common is that the honeypot doesn’t actually contain real data. Should an attacker compromise either type of honeypot, there is no real direct risk of critical data being exposed when deployed properly.

Almost every single honeypot implementation I’ve seen deployed is for research purposes. There isn’t anything wrong with a research honeypot, after all, I run a couple myself (at home) and have learned a lot from it. However, I think there is a lot of operational value that can be gained from deploying honeypots in production environments. I wanted to discuss, at a high level, a few of these strategies and the benefit that can be gained from them.

Honeypots for Prevention

There has been a fair amount of talk recently about security mechanisms designed to drive up the cost of exploiting a network by increasing the time it takes to do so. As a matter of fact, Adobe’s Senior Directory of Product Security and Privacy, Brad Arkin, even recently said that “My goal isn’t to find and fix every security bug. It’s to drive up the cost of writing exploits. We invest a lot of time in building up mitigations that increase the cost and complexity of writing exploits that will become reliable.” Of course, Arkin was referring to the exploitation of software, but the concept still applies to the network side of the house. I’m still a firm believer that your detection capability is still the most important because prevention eventually fails, but if you can drive up the cost of exploiting a network this has the potential to deter some attackers. At a minimum deterring attacks of opportunity can be achieved if you can increase the time cost of exploiting a network, and this may even work to deter attacks of choice as well.

Honeypots can do this by adding to the frustration factor. I see a couple of ways this can be done. The first of which is to utilize a  large number of low interaction honeypots with varying configurations. The important thing here is to vary their configurations as much as possible in order to prevent an attacker from characterizing them and automating them out of their window of visibility. For instance, if you deploy twenty honeypots and they all have ports 22, 80, and 3306 open and all provide the same responses to banner grabs, an attacker is going to be able to correlate this pretty quickly and will simply scan and exclude those hosts from his list of potential targets. The other method for preventive use is to deploy a significant number of high interaction honeypots. This requires a significant time investment, but the right configuration can cause an attacker to waste a significant amount of time in the right places. Again, this strategy isn’t going to prevent aggressive adversaries from reaching their goal, but it will drive up the time cost of lesser determined foes.

Honeypots for Attack Sense and Warning

This is the sacrificial lamb approach to honeypot deployment. In this scenario, honeypots are deployed based upon trust zones within your network. There are different strategies for outlining trust barriers but on a simple network you might define a low trust zone within a wireless or user space network segment, a medium trust zone in a DMZ, and a high trust zone within a server farm network segment. In that sort of topology, all three zones would contain honeypots configured with security comparable to the next step lower. The idea here is that the honeypot should be slightly more vulnerable to attack than everything else in the zone that it is currently in. This configuration provides value in a couple of ways. First, if a honeypot gets compromised, it will likely serve as a warning that other assets within that trust zone may be compromised soon as well, if they aren’t already. Taking this one step further, it is often logical to assume that if a lower trust zone honeypot becomes compromised, the next highest trust zone may be the next target. Depending on how the network is setup, if a higher trust zone includes a honeypot that gets compromised, it could mean that all of the trust zones below it could also have fallen victim to the adversary. This whole model relies on a lot of assumption, but that is the space AS&W operates in.

Honeypots for Detection Related to Critical Assets

I’m a big fan of target-based IDS deployment where instead of deploying a single IDS to your network perimeter, you user more focused IDS’s with finer tuned rule sets and place them closer to organizationally critical assets. This allows for better use of resources across the board as it usually requires less beefy hardware and ensures your analysts won’t see nearly as many false positives. For instance, if your critical data is housed in SQL servers on a single network segment, then deploy an additional IDS to that segment and only utilize SQL focused signatures there rather than on the perimeter IDS. This also allows you to prioritize IDS sensors so that alerts generated by sensors in high priority areas are given priority when it comes to investigations.

I think the same concept of target-based deployment can be tied to Honeypot deployments in the protection of critical assets. If your organization has prioritized their assets (and they should have), then the general idea behind target based honeypot deployment for the purpose of detection would be to configure and deploy honeypots that are virtually identical to the critical servers. This means that they should be running the same services,  talking to the same hosts, and vulnerable to the same types of attacks. The thought here is that if the critical server gets compromised, then so should the honeypot, and vice verse. This is valuable because it isn’t always feasible to log everything on a production server based upon its volume of traffic. This applies to both host based and network based logging. Utilizing an identically configured honeypot that doesn’t see the same amount of utilization allows you to use more aggressive logging, which may allow you to gain more visibility into an attackers movements. This can provide value in helping you determine exactly how an attacker has compromised a system, what they are utilizing the same for, if there is particular data they may be after, and if they have compromised any other systems on the network.

Reverse Honeypots for Intelligence Collection

Although the concept of a reverse honeypot is a bit radical, it really appeals to me considering the industry I work in. The concept of a traditional honeypot is that in which you fill a pot with honey and hope the attacker gets attracted to the honey and sticks his hand in the pot. A reverse honeypot is where you throw some honey in the direction of a target in such a manner as to leave a trail back to the source. The idea being that the target will notice, follow the trail, see the pot, and stick his hand in. In more practical terms, this means that you would attempt to attack a target elsewhere on the Internet. This attack doesn’t necessarily have to be successful and it may just constitute something as simple as a port scan or something as overt as a DoS attempt. During these attacks, no masking of your source IP address should occur and no third party hop points would be used, thus meaning that the target would see your true IP address when reviewing logs of your attack on his network. Given the nature of your target, this may result in his curiosity being peaking and him reciprocating your attack back at you in another form. Of course, within your network you have several vulnerable honeypots of varying interaction levels waiting for the target.

This type of honeypot is solely for the purpose of target based intelligence gathering, but has the potential to be very effective. First and foremost, should the target scan or attack your network you should be able to capture some of the tools, techniques, and procedures (TTPs) that he is using. This type of intelligence can help in recognizing, characterizing, or attributing other computer network exploitation activity to this attacker and may also lend to better detection techniques in the future. One more added value which is incredibly attractive in the modern threat landscape is the identification of hops points. Although you very purposefully did not mask your true source IP address, the attacker may choose to do so. It’s incredibly common for attackers to compromise other hosts elsewhere on the Internet to launch their attacks from, but it’s also common that they will reuse these same hop points for an extended amount of time. If you can identify these hop points then you can use that information to attribute the attacks to a particular operator or group. This is extremely valuable. Of course, this type of activity should be done from non-production networks, because it’s very possible that you might lure an attacker into launching a large scale DDoS attack on your network 10,000 bots strong.

Conclusion

I think there is a lot of room for operationalizing honeypots in production environments. The major factors prohibiting this are a lack of research in this area and a lack of production-grade tools for implementing these techniques. Unfortunately, we are still in a time that IDS is having trouble gaining traction because of the cost it entails, so a future where honeypots can be deployed for the purpose of enhancing network security seems far off. Don’t be surprised however, if you seen a job posting five years down the road for “Honeypot Administrator”. I know I’d have one if I could.

I’m once again going to be leading a SANS Mentor session. This time I’ll be teaching SEC 504, Hacker Techniques, Exploits, and Incident Handling in Charleston, South Carolina. The course will be starting on May 16th, running once a week for two hours, for ten weeks.

 

An excerpt from the course description:

If your organization has an Internet connection or one or two disgruntled employees (and whose doesn’t!), your computer systems will get attacked. From the five, ten, or even one hundred daily probes against your Internet infrastructure to the malicious insider slowly creeping through your most vital information assets, attackers are targeting your systems with increasing viciousness and stealth.

By helping you understand attackers’ tactics and strategies in detail, giving you hands-on experience in finding vulnerabilities and discovering intrusions, and equipping you with a comprehensive incident handling plan, the in-depth information in this course helps you turn the tables on computer attackers. This course addresses the latest cutting-edge insidious attack vectors and the “oldie-but-goodie” attacks that are still so prevalent, and everything in between. Instead of merely teaching a few hack attack tricks, this course includes a time-tested, step-by-step process for responding to computer incidents; a detailed description of how attackers undermine systems so you can prepare, detect, and respond to them; and a hands-on workshop for discovering holes before the bad guys do. Additionally, the course explores the legal issues associated with responding to computer attacks, including employee monitoring, working with law enforcement, and handling evidence.

This challenging course is particularly well suited to individuals who lead or are a part of an incident handling team. Furthermore, general security practitioners, system administrators, and security architects will benefit by understanding how to design, build, and operate their systems to prevent, detect, and respond to attacks.

 

If you are pursuing DOD 8570 certification, then the certification paired with this course, the GCIH, will satisfy the requirement for the CND-Incident Handler designation. This is a great course if you are a government employee or contractor pursuing 8570 compliance, or simply someone working in information security looking to strengthen your defensive technology skills and gain a widely accepted certification in the process.

 

If you are interested in learning more then you can visit the SANS website for this course at: http://www.sans.org/mentor/class/sec504-charleston-may-2012-sanders. Also, feel free to pass around the flyer for this course, which can be viewed here. Also, I can provide some discounts to help offset the cost a bit if you contact me directly.

I was going back through some old bookmarks when I stumbled upon on a post by Richard Bejtlich from 2007 entitled “NSM and Intrusion Detection Differences“. In this article, Richard discussed the concept of ‘immaculate collection’ versus ‘immaculate detection’. Richard’s article references IDS developers desiring immaculate detection while NSM practitioners typically vie for immaculate collection. Given this, I posed the following question to several of my colleagues: Which is more important, collection or detection?

 

The question itself is open to a bit of interpretation, but my group was split about 60/40 favoring collection over detection. I tend to agree with that majority, although the minority had some valid points as well.

 

Those favoring detection argued that a mountain of data, no matter how eloquently collected, is useless without some level of detection capability. Additionally, most in this camp agreed that your detection capability shapes how you perform collection. A few even made the point that they considered collection to be a function of network operations, and not NSM. I can’t disagree with the first of these arguments, but I’m opposed to the other two. I’ll address the argument of whether or not detection shapes collection here.

 

When I think about NSM, I typically think of it in three phases: collection, detection, and analysis. Collection is the gathering and parsing of relevant network security data, and it often performed by a combination of hardware and software. Detection is the process of finding anomalies in collected data that may represent a potential intrusion. Detection is most often done by software, but can be done by humans to a lesser extent. Analysis is the review and investigation of alert data generated during detection. Analysis is typically (and most effectively) done by humans.

 

 

Phases of Network Security Monitoring

 

 

The key takeaway from these three phases is that they form a cycle rather than a beginning to end process. Collected data feeds the detection capability, and the alert data generated from detection feeds the analysis process. What makes this process cyclical is that the investigation and research performed during the analysis process is used to define and shape what data you are collecting.

 

That said, I argue that collection is the most important phase of network security monitoring for a couple of reasons:

 

Detection Depends on Collection

Abraham Lincoln was quoted in saying that if you were to give him six hours to chop down a tree, he would spend the first four hours sharpening his ax. This analogy fits perfect here, because no matter how much thought you put into your detection tools, they are utterly useless if they aren’t digesting the right data. That nice beefy Snort sensor might just be wasting cycles if you’ve placed it on the wrong side of your firewall. Detection fails if collection isn’t done well.

 

Analysis Also Depends on Collection

I hate using the needle in the haystack analogy, but if the hay is covered in manure then you sure aren’t going to want to  spend all of that time digging through it. A human analyst interprets alert data provided by a detection mechanism and then goes out and collects more data in an effort to support his/her investigation. If this data isn’t being collected in an easily retrievable and digestible format then analysis fails. An IDS signature might tell me that a potential attacker is attempting SQL injection on my public facing web server but if I’m not collecting PCAP data and my web server/database logs aren’t accessible then I’m going to have a really hard time finding out if the attack is actually successful.

 

Analysis Feeds Collection Moreso than Detection

I’ve served in the role where I’m the guy creating the detection tools and also in the role where I’m the guy analyzing the alerts generated by the detection tools. It is absolutely true that in some cases collection software/hardware is designed and configured in such a way that it provides data in the appropriate format to a detection tool. This might lead someone to the conclusion that it is detection shaping the collection, but that argument is only made seeing a narrow view of the entire thought process. It is actually the analysis of previous alert data that typically has identified the need for the detection tool that is being created. Remember that detection is most often a task performed by software and it is analysis that is performed by individuals. Software doesn’t identify needs, people do.

 

 

Again, I think this is one of those questions that may or may not have a right answer, but for my two cents, if you gave me six hours to find the bad guys, I’d spend the first four making sure I collected the right data.

 

 

There are a lot of things that the industry does well when it comes to network security monitoring (NSM). For instance, I tend to think that we have data collection figured out reasonably well. I also think that signature-based intrusion detection is a really well developed science. However, with NSM having only existed for a short period of time there are several facets of it that aren’t too well defined. One such aspect is the actual diagnostic method that people use to analyze NSM events. That is, the process an analyst uses to connect the dots between the initial alert and the final diagnosis. In this article I’m going to discuss the use of a common medical diagnostic method called differential diagnosis and how it can be applied to NSM.

 

Understanding Normal

The first thing that was ever taught to me when I started my career as an NSM analyst was that if you know what normal looks like, then you can determine what is bad. I trusted in this concept for many years and even taught it to others. As true as this statement may be, I believe it is relied on entirely too much. This is primarily due to a failure in separating the collection, detection, and analysis processes.

 

Collection centers on the hardware and software used to collect NSM related data. Consider the collection of full content packet capture (PCAP) data. The use a network tap and DaemonLogger allow you to store this data on disk so that it may be used for the identification and analysis of network security related events. Collection occurs with a combination of hardware and software.

Detection is the process by which collected data is examined and anomalies are identified, typically through some form of signature, anomaly, or statistically based detection. Snort is software that is an example of signature-based intrusion detection that compares collected network traffic to signatures of known malicious activity in an effort to perform pattern matching to determine if something bad has occurred. Detection is typically software focused.

Analysis is what occurs when a human interprets the results of the output of an identification tool. Although Snort may detect a pattern match in a communication sequence and generate an alert, it is a human who is ultimately responsible for reviewing the alert and investigating it to an end determination on its validity. The key concept here is that analysis is human focused.

 

With those three terms more clearly defined and distinctions drawn, it would stand to reason that the concept of knowing what normal looks like in order to determine what is bad is actually more relevant to detection than analysis. Realistically speaking, it’s not feasible in the modern state of network computing to be well versed in every aspect of normal communications. Although some traffic patterns may remain fairly static, the open nature and loose standards that govern network communication protocols result in a constant evolution of traffic patterns. Don’t be mistaken, this is still an important concept that must be incorporated into the analytic approach, it’s just not strong enough to stand on its own as the singular concept new analysts should be taught. Knowing what normal looks like is best used when analyzing specific facets of a potential breach rather than as a holistic method to classify all network traffic you may be capturing.

 

A Differential Approach

The general goal of an NSM analyst is to digest the alerts generated by various detection tools and investigate multiple data sources and perform relevant tests and research to see if their findings represent a network security breach. This is very similar to that of a physician, whose goal is to digest the symptoms a human presents and investigate multiple data sources and perform relevant tests and research to see if their findings represent a breach in the person’s immune system.  Both practitioners share a similar of goal of connecting the dots to find out if something bad has happened and/or is still happening.

Although NSM has only been around a short while, medicine has been around for centuries. This means that they’ve got a head start on us when it comes to developing their diagnostic method. One of the most common diagnostic methods used in clinical medicine is one called differential diagnosis. If you’ve ever seen an episode of “House” then chances are you’ve seen this process in action. The group of doctors will be presented with a set of symptoms and they will create a list of potential diagnosis on a whiteboard. The remainder of the show is spent doing research and performing various tests to eliminate each of these potential conclusions until only one is left. Although the methods used in the show are often a bit unconventional they still fit the bill as a part of the differential diagnosis process.

The differential method is one based upon a process of elimination. It consists of five distinct steps, although in some cases only two will be necessary. The differential process exists as follows:

1. Identify and list the symptoms
   
   In medicine, symptoms are typically initially conveyed verbally by the individual experiencing them. In NSM, a symptom is most commonly in the form of an alert generated by some form of intrusion detection system or other detection software. Although this step focuses primarily on the initial symptoms, more symptoms may be added to this list as additional tests or investigations are conducted.

 

2. Consider and evaluate the most common diagnosis first
   
   A statement every medical student is taught in their first year is “If you hear hoof beats, look for horses…not zebras.” This is to state to that the most common diagnosis is likely the correct one. As a result, this diagnosis should be evaluated first. The analyst should focus his investigation on doing what is necessary to quickly confirm this diagnosis. If this common diagnosis cannot be determined to be true during this initial step then the analyst should proceed to the next step.

 

3. List all possible diagnosis for the given symptoms
   
   The next step in the differential process is to list every possible diagnosis based upon the information currently available with the initially assessed symptoms. This step requires some creative thinking is often most successful when multiple analysts participate in generating ideas. Although you may not have been able to completely confirm the most common diagnosis in the previous step, if you weren’t able to rule it out completely then it should be carried over into the list generated in this step. Each potential diagnosis on this list is referred to as a candidate condition.

 

4. Prioritize the list of candidate conditions by their severity
   
   Once a list of candidate conditions is created a physician will prioritize these listing the condition that is the largest threat to human life at the top. In the case of an NSM analyst you should also prioritize this list, but the prioritization should focus on which condition is the biggest threat to your organizations network security. This will be highly dependent upon the nature of your organization. For instance, if “MySQL Database Root Compromise” is a candidate condition then a company whose databases contains social security numbers would prioritize this condition much higher than a company who uses a simple database to store a list of its sales staffs on-call schedule.

 

5. Eliminate the candidate condition, starting with the most severe
   
   The final step is where the majority of the action occurs. Based upon the prioritized list created in the previous step the analyst should begin doing what is necessary to eliminate candidate conditions, starting with the condition that poses the greatest threat to network security. This process of elimination requires considering each candidate condition and performing tests, conducting research, and investigating other data sources in an effort to rule them out as a possibility. In some cases investigation on one candidate condition may effectively rule out multiple candidate condition, speeding up this process. Alternatively, investigation of other candidate conditions may prove inconclusive leaving one or two conditions that are unable to be definitively eliminated as possibilities. This is acceptable however as sometimes in network security monitoring (as in medicine) there are anomalies that can’t be explained that require more observation before determining a diagnosis. Ultimately, the goal of this final step is to be left with one diagnosis so that either the incident handling process may begin or the alert can be dismissed as a false positive. It’s very important to remember that “Normal Communication” is a perfectly acceptable diagnosis, and will be the most common diagnosis an NSM analyst arrives at. I also find that remembering that all packets are good unless you can prove they are bad is an important concept to remember during this step.

 

 

Let’s consider this process with a couple of broad case scenarios.

 

Scenario 1

Step 1: Identify and List the Symptoms

Symptoms:

• Internal host appears to be sending outbound traffic to a Russian IP address
• The traffic is occurring at regular intervals, every 10 minutes
• The traffic is HTTPS over port 443, and as such is encrypted and unreadable

Step 2: Consider and Evaluate the Most Common Diagnosis First

It’s been my experience that most entry level analysts will see these symptoms and automatically think that this machine is infected with some form of malware and is phoning home for further instructions. Those analysts tend to key in on that fact that the traffic is going to a Russian IP address and that it is occurring at regular 10 minute intervals. Although those things are worth noting (I wouldn’t have listed them if they weren’t), I don’t buy into the malware theory so easily. I believe entirely too much emphasis is placed on the geographic location of IP addresses, so the fact that the remote IP address is Russian means little to me. Additionally, there are a whole variety of normal communication mechanisms that talk on regular periodic intervals. This includes things like web-based chat, RSS feeds, web-based e-mail, stock tickers, software update processes, and more. Operating on the principal that all packets are good unless you can prove they are bad, I think the most common diagnosis here is that this is normal traffic.

That said, how we can confirm this potential diagnosis? Confirming something is normal can be hard. In this particular instance we could start with some open source research on the Russian IP. Although it’s located in Russia it still may be owned by a legitimate company. If we were to look up the host and find that it was registered to a popular AV vendor we might be able to use that information to conclude that this was an AV application checking for updates. I didn’t mention the URL that the HTTPS traffic is going to, but quickly Googling it may yield some useful information that will help you determine if it is a legitimate site or something that might be hosting malware or some type of botnet C2. Another technique would be to examine the host physically if you have ready access to it in an effort to see if any processes are launched on the machine at the same intervals the traffic is occurring at.

Let’s assume that we weren’t able to make a final determination on whether or not this was normal communication.

Step 3: List all Possible Diagnosis for the Given Symptoms

*There are obviously more candidate conditions in the realm of possibility, but for this and the other scenario I’ve kept it to some of the more common ones for the sake of brevity.

Candidate Conditions:

• Normal Communication
  We weren’t able to rule this out completely in the previous step so we carry it over to this step.

 

• Malware Infection / Installed Malicious Logic
  This is used as a broad category. We typically don’t care about the specific strain until we determine that malware may actually exist. If you are concerned about a specific strain then it can be listed separately. Think of this category as a doctor listing “bacterial infection” as a candidate condition knowing that they can further narrow it down later.

 

• Data Exfiltration from Compromised Host
  Potential that the host could be sending proprietary or confidential information out. This sort of thing would likely be part of a coordinated or targeted attack.

 

• Misconfiguration
  It’s well within the realm of possibilities that a system administrator fat-fingered an IP address and a piece of software that should be trying to communicate periodically with an internal IP is now trying to do so with a Russian IP. This is really quite common.

 

Step 4: Prioritize the List of Candidate Conditions by their Severity

These priorities are fairly generalized since they are dependent upon your organization.

Priority 1: Data Exfiltration from Compromised Host

Priority 2: Malware Infection / Installed Malicious Logic

Priority 3: Misconfiguration

Priority 4: Normal Communication

Step 5: Eliminate the Candidate Conditions, Starting with the Most Severe

Priority 1: Data Exfiltration from Compromised Host

This one can be a bit tricky to eliminate as a possibility. Full packet capture won’t be of the most assistance here since the traffic is encrypted, but if you can create some statistics from this traffic, or better yet, if you have netflow available, you should be able to determine the amount of data going out. If only a few bytes are going out every then minutes than it’s likely that this is not data exfiltration. The host based research you did earlier on the Russian IP address may also provide some value here in determining the reputation of this host. It would also be of value to determine if any other hosts on your network are talking to this IP address or any other IPs in the same address space. Finally, baselining normal communication for your internal host and comparing it with the potentially malicious traffic may provide some useful insight.

Priority 2: Malware Infection / Installed Malicious Logic

At this point the research you’ve already done should give you a really good idea on whether or not this condition is true. It will be likely that by examining the potential for data exfiltration you will rule this condition out as a result, or will have already been able to confirm it to be true.

Priority 3: Misconfiguration

This condition can best be approached by comparing the traffic of this host against the traffic of one or more hosts with a similar role on the network. If every other workstation on that same subnet has the same traffic pattern, but to a different IP address, then it’s likely that the wrong IP address was entered into a piece of software somewhere proving that a misconfiguration exists. Having access to host-based logs can also be useful in figuring out if a misconfiguration exists since they might exist in Windows or Unix system logs.

Priority 4: Normal Communication

If you’ve gotten this far, then the diagnosis of normal communication should be all that remains on your list of candidate conditions.

Concluding a Diagnosis

At this point you have to use your experience as an analyst and your intuition to decide if you think something malicious is really occurring. If you were able to complete the previous analysis thoroughly, then operating on the assumption that all packets are good unless you can prove they are bad would mean your final diagnosis here should be that this is normal communication. If you still have a hunch something quirky is happening though, there is no shame in monitoring the host further and reassessing once more data has been collected.

 

Scenario 2

Step 1: Identify and List the Symptoms

Symptoms:

• A web server in our DMZ is receiving massive amounts of inbound traffic
• The inbound traffic is unreadable and potentially encrypted or obfuscated
• The inbound traffic is coming to multiple destination ports on the internal host
• The inbound traffic is UDP based

Step 2: Consider and Evaluate the Most Common Diagnosis First

With the amount of traffic being received by the internal host being very large and the packets using the UDP protocol with random destination ports, my inclination would be that this is some form of denial of service attack.

The quickest way to determine whether something is a denial of service is to assess the amount of traffic being received compared with the normal amount of traffic received on that host. This is something that is really easy to do with netflow data if you have it available. If the host is only receiving 20% more traffic than it normally would then I would consider other alternatives to a DoS. However, if the host is receiving ten or one hundred times its normal amount of traffic then DoS is very likely and almost a certainty.  It’s important to remember that a DoS is still a DoS even if it is unintentional.

Once again, for the sake of this scenario we will continue as though we weren’t able to make a clear determination on whether or not a DoS condition exists.

Step 3: List all Possible Diagnosis for the Given Symptoms

Candidate Conditions:

• Denial of Service
  We weren’t able to rule this out completely in the previous step so we carry it over to this step.

 

• Normal Communication
  It doesn’t seem incredibly likely, but there is potential for this to be normal.

 

• Misdirected Attacks
  When a third party chooses to attack another they will often spoof their source address for the sake of anonymity and to prevent getting DoS’d themselves. This will result in the owner of the spoofed IP they are using seeing that traffic. This web server could be seeing the effects of this.

 

• Misconfigured External Host
  A misconfiguration can happen on somebody else’s network just as easily as it could on yours. This misconfiguration could result in an external host generating any number of types of traffic and sending them to the web server.

 

• SPAM Mail Relay
  The server could be misconfigured or compromised in a manner that allows it to be used for relaying SPAM across the Internet.

 

Step 4: Prioritize the List of Candidate Conditions by their Severity

Priority 1: Denial of Service

Priority 2: SPAM Mail Relay

Priority 3: Misconfigured External Host

Priority 4: Misdirected Attacks

Priority 5: Normal Communication

Step 5: Eliminate the Candidate Conditions, Starting with the Most Severe

Priority 1: Denial of Service

We’ve already gone through the paces on this one without being able to identify that it is the definitive diagnosis. Even though this is the most severe we would have to proceed to attempt to eliminate other candidate conditions to help in figuring out if a DoS is occurring. Of course, depending on the effect of the attack it may make the most sense to contain the issue by blocking the traffic before spending more time investigating the root cause.

Priority 2: SPAM Mail Relay

This one is relatively easy to eliminate. If the server was being used as a mail relay then you would have a proportionate amount of traffic going out as you do going in. If that’s not the case and you don’t see any abnormal traffic leaving the server then it is likely that it is not relaying SPAM. If the web server is also running mail services then you can examine the appropriate logs here as well. If it is not supposed to be running mail services you can examine the host to see if it is doing so in an unauthorized manner.

Priority 3: Misconfigured External Host

This one is typically pretty tricky. Unless you can identify the owner of the IP address and communicate with them directly then the most you can hope to do is block the traffic locally and/or report abuse at their ISP level.

Priority 4: Misdirected Attacks

This is another tricky one along the same lines as the previous candidate condition. If it’s an attacker somewhere else whose antics are causing traffic redirection to your server then the most you can do is to report the issue to the ISP responsible for the IP address and block the traffic locally.

Priority 5: Normal Communication

This doesn’t seem likely, but you can’t say this for sure without baselining the normal traffic for the host. Compare its traffic at similar times on previous days to see if you can draw any conclusions. Is the pattern normal and it’s just the amount of traffic that anomalous? Is it both the pattern and the amount that’s anomalous? Does the server ever talk to the offending IP prior to this?

 

Concluding a Diagnosis

In this scenario, it’s very possible that you are left with as many as three candidate conditions that you cannot rule out. The good thing here is that even though you can’t rule these out, the containment and remediation methods would be the same for all of them so you still have gotten to a state of diagnosis that allows the network to recover from whatever is occurring. If the amount of traffic isn’t too great then you may not need to block the activity and you may be able to monitor it further in order to attempt to collect more symptoms that may be useful in providing a more accurate diagnosis.

 

Conclusion

I’ve spent quite a bit of time doing analysis with this differential approach and also reviewing previous investigations post-mortem while applying these concepts and I’ve been really pleased with my findings. I think that if you are struggling with being able to grasp a firm analytical method then this may be a great one to start with. I’m not entirely sure that the differential method is appropriate for all organizations, but just as with medicine, there are competing approaches and I hope to examine more of those in the future so that I can draw more comparisons between the medical field and NSM. If you have any scenarios in which you’ve used this differential approach (for better or for worse), I’d love to hear about them.

I’ve been a huge fan of CloudShark ever since it was launched by QA Café back in 2010. I even wrote about CloudShark here when it was first released. I’m always finding unique uses for the web-based service. This includes everything from sharing PCAP files on my blog to viewing PCAPs on my iPad. CloudShark has the potential to make life much better anywhere you don’t have Wireshark but you do have an Internet connection.

 

The only real downside to CloudShark is that there is no guarantee of privacy for uploaded PCAP files. I recently spoke with Joe at QA Café and this is an issue they are aware of, and as a result they’ve developed the CloudShark appliance. This is a standalone instance of CloudShark that you can deploy within your organization so that you can get all of the benefits of CloudShark while keeping sensitive data contained within your packet captures private to only those who need access.

 

I wanted to briefly mention a few of my favorite CloudShark features:

 

View

This may sound a bit odd, but my absolutely favorite thing about CloudShark is the ability for me to view PCAP files on my iPad. Quite honestly, I’ve wanted an iPad for a while but could never talk myself into buying one until I figured out I could use it to view PCAP files. Now I use it for that purpose on a daily basis. CloudShark is compatible with a variety of smartphones and tablets.

 

 

Annotate

I work in a network security monitoring environment so we look at a LOT of PCAP data. Typically, this involves keeping track of a lot of notebooks where I’ve scribbled notes about the contents of capture files. One of the cooler things about CloudShark is the ability to annotate within the capture files. You can even use these annotations to link to other captures.

 

 

Share

I hate sending PCAP files around via e-mail. People rename things, filter things out and then save them, and do other weird things that may result in multiple people looking at different data thinking they may be looking at the same thing. When you upload a PCAP into CloudShark it generates a hyperlink that you can use to share your PCAPs.

 

 

Organize

When you deal with a lot of PCAP files it’s easy to lose track of what you are looking for. CloudShark has a very “Web 2.0” system for tagging packet captures so that they can be easily searched. This feature can be used for organizing in a lot of way, so you can organize your PCAPs by tagged by protocol, vulnerability, system, or even investigation case.

 

 

There are a lot of different use cases for the CloudShark appliance depending on the needs of your organization. It can be purchased as hardware with the platform pre-installed on to it, or as software that you can install on your own hardware running a Linux based OS. I saw recently that it was also updated to allow for integration with external LDAP authentication, so that adds some more flexibility to the system overall.

 

You can read more about it at http://appliance.cloudshark.org/.

 

As a lot of my frequent readers know, 100% of the profits from my books sales and the advertising revenue on this site go to support the Rural Technology Fund, a non-profit I direct that provides scholarships to students from rural areas pursuing degrees in technical fields. We’ve just announced our 2012 scholarship offerings. This includes a brand new offering, a $2500 scholarship available to any high school senior or undergraduate student who attended high school in a rural area, is pursuing a degree in a technical field, and has a strong interest in cyber security.

 

Cyber Security Scholarship

The Cyber Security scholarship is awarded annually to one high school senior or undergraduate college student who is currently or has attended high school in a rural community. In order to be considered for this scholarship, the applicant must demonstrate an interest in cyber security. The student must also be planning to attend college to pursue a degree in a computer technology related field, or already doing so as part of an undergraduate degree program. This scholarship is awarded based upon answers to a series of essay questions that are designed to gauge the student’s passion for his or her intended career in cyber security as well as the student’s sense of citizenship and pride in their rural community. This scholarship is awarded in the amount of $2500.00.

 

Judith A. Sanders Memorial Scholarship

The Judith A. Sanders Memorial Scholarship is awarded annually to a student from the rural community of Graves County in Western Kentucky. In order to be considered for this scholarship, an applicant must currently be attending Graves County High School or Mayfield High School as a senior. The student must also be planning to attend college to pursue a degree in a computer technology related field. This scholarship is awarded based upon answers to a series of essay questions that are designed to gauge the student’s passion for his or her intended career in computer technology as well as the student’s sense of citizenship and pride in their rural community. This scholarship is awarded in the amount of $1000.00.

 

Kentucky Student Technology Leadership Program (STLP) Scholarship

The Kentucky Student Technology Leadership Program (STLP) Scholarship is open to students from schools in Kentucky who have a passion for using technology skills to make a positive social change in the world or at home in their communities. In order to be considered for this scholarship, an applicant must currently be attending a rural high school as a senior in the state of Kentucky and be an active member of their schools STLP. Additionally, the student must also be planning to attend college to pursue a degree in a computer technology related field. This scholarship is awarded based upon answers to a series of essay questions that are designed to gauge the student’s passion for his or her intended career in computer technology as well as the student’s sense of citizenship and pride in their rural community. This scholarship is awarded in the amount of $1000.00.

 

The submission deadline for all three scholarships is April 15^th, 2012. In order to apply, please visit http://www.ruraltechfund.org/scholarships.

 

 

About the Rural Technology Fund

Established in 2008, the Rural Technology Fund (RTF) seeks to reduce the digital divide between rural communities and their more urban and suburban counterparts. This is done through targeted scholarship programs, community involvement, and the general promotion and advocacy of technology in rural areas.

The RTF wants to help rural students recognize the opportunity in technology careers and gain the education necessary to work in the computer industry.

One of the more useful network forensic skills is the ability to extract files from packet captures. This process, known as packet data carving, is crucial when you want to analyze malware or other artifacts of compromise that are transferred across the network. That said, packet data carving has varying degrees of difficulty depending on the type of traffic you are attempting to extract data from. Carving files from simple protocols like HTTP and FTP is something that can be done in a matter of minutes and is usually cut and dry enough that it can be done in an automated fashion with tools like Foremost and Network Miner.

There are articles all over the Internet about carving files from simple protocols so I won’t rehash those. Instead, I want to take a look at a two more complex protocols that are extremely common in production networks. Server Message Block (SMB) is the application-layer protocol that Microsoft operating systems use for file sharing and communication between networked devices. If you live on a Microsoft network (or a Unix network that utilizes SAMBA) then you are a user of SMB or SMB2, depending on your operating system version. In this article I’m going to discuss the art of carving files from SMB and SMB2 traffic. If you want to follow along you’ll need to download a copy of Wireshark (http://www.wireshark.org) and your favorite hex editor. I’ve used Cygnus Hex Editor (http://www.softcircuits.com/cygnus/fe/) for the purpose of this article since it’s simple and a free version exists.

 

Carving Files from SMB Packets

 

The first version of SMB is in use on all modern Microsoft operating systems prior to Windows Vista. In order to setup a packet capture for this scenario I took two Windows XP SP3 virtual machines running on VMWare Workstation and placed them in the same network. Once they were able to communicate with each other I setup a shared folder on one host (192.168.47.132) that is acting as the server. I then fired up Wireshark and began capturing packets as I copied an executable file from the client (192.168.47.133) to the servers shared folder. The resulting packet capture is called smb_puttyexe_xfer.pcap.

If you’ve never looked at SMB traffic then don’t get scared by all the different types of SMB packets in the capture, we will only be looking at a few of them. This article isn’t meant to be an exhaustive reference on each and every type of SMB packet (there are over a hundred of them), so if you want the gory details then take a look at the references at this end of this article.

In order to carve the file out of these packets we have to find some basic information about it. Before and after transferring a file to a server the client will attempt to open the file in order to see if it exists. This is done with an SMB NT Create AndX Request packet.  The response from the server to this is an SMB NT Create AndX Response, which contains the name, extension, and size of the file being transferred. This is everything we need to get started. You can filter for Create AndX Response packets in Wireshark with the filter (smb.cmd == 0xa2) && (smb.flags.response == 1). If we examine one of those requests that occur after the file has been transferred, we can identify that the file being transferred is putty.exe and its file size is 454,657 bytes. We will use this information later.

Figure 1: Note the file name, extension, and size.

The next step we have to take in order to extract this file is to isolate the appropriate block of traffic. Wireshark makes this pretty easy with its Follow TCP Stream functionality. Start by right-clicking any packet in the capture file and selecting Follow TCP Stream. This will bring up a window that contains all of the data being transferred in this particular communication stream concatenated together without all of the layer 2-4 headers getting in the way. We are only concerned about the traffic transferred from the client to the server so we will need to specify this in the directional drop down box by selecting 192.168.47.133 –> 192.168.47.132 (458592 bytes). Click Save As and save the file using the name putty.raw.

Figure 2: Saving the isolated traffic from Wireshark

If you were to view the properties of the data you just extracted and save you should find that its file size is 458,592 bytes. This is 3,935 bytes more than the size of the actual file that was transferred. This means that our goal is to get this raw files size down to exactly 454,657 bytes. This is where the real carving begins.

First things first, we have to delete all of the extra data that occurs before the executable data actually begins. Since we do know that the transferred file is an executable the quickest way to do this is to look for the executable header and delete everything that occurs before it. The executable header begins with the hex bytes 4D 5A (MZ in ASCII), which occurs approximately 1112 bytes into the putty.raw file. Once deleted, resave the file as putty.stage1. You should now be down to a file size of 457,480 bytes.

Figure 3: Removing added bytes from the beginning of the file

Now things get a bit trickier. SMB transmits data in blocks. This is great for reliability since a lost or damaged block can be retransmitted, but it adds some extra work for us. This is because each block must contain some bytes of SMB header data in order to be interpreted correctly by the host that is receiving it. The good thing is that the size of this data is somewhat predictable, but you have to understand a bit more about SMB in order to put the rubber to the road. The thing to know here is that the data block size in SMB is limited to 64KB, or 65536 bytes.  Of this amount, only 60KB is typically used for each block. These 61,440 bytes are combined with an additional 68 bytes of SMB header information. This means that after every 61,440 bytes of data we will have to strip out the next 68 bytes.

There is one thing to add to this that must be taken into consideration before stripping out those bytes. As a part of the normal SMB communication sequence, an additional packet is sent right after the first block. This is an NT Trans Request packet, which is packet 77 in the capture file. The SMB portion of this packet is 88 bytes, which means we will have to remove those 88 bytes in addition to the 68 bytes that make up the normal SMB block header, for a total of 156 bytes.

Now that we have all that sorted out let’s start removing bytes. In your hex editor, skip one byte past the 61,440^th byte. This will be offset 0x0F000. You should start with this byte and select a range of 156 bytes and delete them. Save this file as putty.stage2.

Figure 4: Removing the initial 156 bytes

Things get a bit easier now as we are just concerned with stripping out the 68 bytes after every block. Skip through the file in 61,440 byte increments deleting 68 bytes each time. This should occur X times in this file at offsets 0x1e000, 0x2d000, and 0x3c000, 0x4b000, 0x5a000, 0×69000. Once finished, save the file as putty.stage3.

Figure 5: Removing a 68 byte SMB header block

Go ahead and take a look at the file size of putty.stage3. We are still XXX bytes off from our target, but luckily the last part is the easiest. The data stream is actually just padded by some extra information that needs to be deleted. We know that the file should be 454,657 bytes, so browse to that byte and delete everything that occurs after it.

Figure 6: Trimming the extra bytes off the end of the file

Save the final product as putty.exe and if you did everything right, you should have a fully functioning executable.

Figure 7: Success! The executable runs!

 

The whole process can be broken down into a series of repeatable steps:

1. Record the file name, extension, and size by examining one of the SMB NT Create AndX Response packets
2. Isolate and extract the appropriate stream data from Wireshark by using the Follow TCP Stream feature and selecting the appropriate direction of traffic
3.  Remove all of the bytes occurring before the actual file header using a hex editor
4. Following the first 61,440 byte block, remove 156 bytes
5. Following each successive 61,440 byte block, remove 68 bytes
6. Trim the remaining bytes off of the file so that it matches the file size recorded in step 1

 

Carving Files from SMB2 Packets

 

Microsoft introduced SMB2 with Windows Vista and began using it with its newer operating systems moving forward. In order to setup a packet capture for this scenario I took two Windows 7 (x32) virtual machines running on VMWare Workstation and placed them in the same network. Once they were able to communicate with each other I setup a shared folder on one host (192.168.47.128) that is acting as the server. I then fired up Wireshark and began capturing packets as I copied an executable file from the client (192.168.47.129) to the servers shared folder. The resulting packet capture is called smb2_puttyexe_xfer.pcap.

You should notice that this traffic is a little bit cleaner than the SMB traffic we looked at earlier. This is because SMB2 is optimized so that there are a lot less commands. Whereas SMB had over a hundred commands and subcommands, SMB2 only has nineteen. Regardless, we still need to find the filename being transferred and the size of that file. One of the best places to do this is at one of the SMB2 Create Response File packets. This packet type serves a purpose similar to that of the SMB NT Create AndX Response packet. You can filter these out in Wireshark with the filter (smb2.cmd == 5) && (smb2.flags.response == 1). The last one of these in the capture, which is packet 81, is the one we want to look at since it occurs after the file transfer is complete. This identifies the file name as putty.exe and the file size as 454,656 bytes. This is indeed the same file as our earlier example, but it is being reported as being one byte smaller. The missing byte is just padding at the end of the file and has a null value so it’s not of any real concern to us.

Figure 8:  Once again we note the file name, extension, and size

At this point you should perform the same steps as we did earlier to isolate and extract the data stream from the capture using Wiresharks Follow TCP Stream option. Doing this should yield a new putty.raw file whose file size is 459,503 bytes. This is 4,847 too big, so it’s time to get to carving.

Once again we need to start by stripping out all of the data before the executable header. Fire up your favorite hex editor and remove everything before the bytes 4D 5A. This should account for a deletion of 1,493 bytes.

Figure 9: Removing the extra bytes found prior to the executable header

Now things change a bit. SMB2 works in a method similar to SMB, but it actually allows for more data to be transferred at once. SMB had a maximum block size of 64K because it has a limit of 16-bit data sizes. SMB2 uses either 32-bit or 64-bit data sizes, which raises the 64KB limit. In the case of the transfer taking place in the sample PCAP file, these were two 32-bit Windows 7 hosts under their default configuration which means that the block size is set at 64KB. Unlike SMB however, the full 64KB is used, so we will see data in chunks of 65,536 bytes being transferred. These 65,536 bytes combine with a 116 byte SMB2 header to form the full block.

SMB2 doesn’t include an additional initial request packet like the SMB Trans Request, so we don’t have to worry about stripping out any extra bytes right off the bat. As a matter of fact, some might say that carving data from SMB2 is a bit easier since you only have to strip out 116 bytes after each block of 65,536 bytes. You can do this now on putty.stage1. In doing so you should be deleting 116 bytes of data at offsets 0×10000, 0×20000, 0×30000, 0×40000, 0x50000and 0×60000.

Figure 10: Removing 116 bytes of data following the first 65,536 chunk

Once you’ve finished this save the file as putty.stage2. All that is left is to remove the final trailing bytes from the file. In order to do this, browse to by 454,656 and delete every byte that occurs after it.

Figure 11: Removing the final trailing bytes

Finally, save the file as putty.exe and you will have a fully functioning executable. The process of carving a file from an SMB2 data stream breaks down as follows:

1. Record the file name, extension, and size by examining one of the SMB2 Create Response File packets
2. Isolate and extract the appropriate stream data from Wireshark by using the Follow TCP Stream feature and selecting the appropriate direction of traffic
3.  Remove all of the bytes occurring before the actual file header using a hex editor
4. Following each successive 65,536 byte block (assuming a 64K block size), remove 116 bytes
5. Trim the remaining bytes off of the file so that it matches the file size recorded in step 1

 

Conclusion

 

That’s all there is to it. I’ll be the first to admit that I didn’t cover every single aspect of SMB and SMB2 here and there are a few factors that might affect your success in carving files from these streams, but this article shows the overall process. Taking this one step farther, it’s pretty reasonable to assume that this process can be automated with a quick Python script, but this is something I’ve not devoted the time to yet. If you feel like taking up that challenge then be sure to get in touch and I’ll be glad to post your code as an addendum to this post. In the mean time, happy carving!

 

References

http://msdn.microsoft.com/en-us/library/cc246231%28v=PROT.10%29.aspx

http://msdn.microsoft.com/en-us/library/cc246482%28v=PROT.10%29.aspx

http://channel9.msdn.com/Blogs/Darryl/Server-Message-Block-SMB21-Drill-down

 

In the realm of network security monitoring and intrusion analysis we are all slaves to our data. Typically speaking, we rely on two different types of data at the network layer; full content data (PCAP) and session data (Netflow). Both are pretty easy to generate given the right sensor placement, and there are a lot of great resources out there for learning how to get good value out of the data. That said, they do each have their own shortcomings as well.

 

Session Data (Netflow)

Netflow is a standard form of session data that details the ‘who, what, when, and where’ of network traffic. I tend to equate this to the call records you’ll see on your monthly cell phone bill.

 

 

 

 

Figure 1: Partial Netflow Records Exported from SiLK

 

The best thing about netflow is that it provides a lot of value with minimal disk storage overhead. It’s really a lot of bang for your buck. Most commercial grade routers and firewalls will generate netflow, and there are a lot of free and open source tools, such as SiLK, that can be used to generate and analyze netflow as well. There is even a yearly conference called FloCon where people get together and talk about cool things you can do with netflow. The only real downside to netflow data is that it doesn’t paint a complete picture, so it’s often best used as a complement to full content data.

 

Full Content Data (PCAP)

If netflow session data is equivalent to a call log, then full content data in the form of PCAP is just like having a full recording of all of your calls.

 

 

 

Figure 2: PCAP Data Investigation with Wireshark

 

 

The PCAP format has become very universal and can be collected and analyzed with a variety of free and open source applications like Dumpcap, Tcpdump, Wireshark, and more. A lot of the more popular intrusion detection systems, such as Snort, use the PCAP format as well. As an analyst, having PCAP data available tends to make the analytical process a dream come true as it provides the highest level of context when investigating an anomaly. The primary downside to full content data is that it has an incredibly high disk storage overhead, which prevents most organizations from collecting and storing any reasonable amount of it. In my experience, the organizations that are capable of collecting and storing PCAP can only measure the amount stored in hours, rather than days. In addition to this, unless you have an idea of what you are looking for within a reasonable time range, it can be a bit difficult to locate things as well, somewhat impeding flexibility in analysis.

 

Application Layer Metadata

The concept of application layer metadata originally presented itself to me in a discussion regarding additional data types that are useful within the network security monitoring function that were sort of a happy medium in between session data and full content data. It didn’t take a lot of number crunching to find that on most of the networks we monitored, the vast majority of the traffic was the application layer data of a few common protocols. The largest of these was HTTP, followed by the other usual suspects; SSL, DNS, and SMTP.

Starting with a couple of these protocols as a baseline, we quickly realized that we could save ourselves a lot of disk storage overhead by actually eliminating the stuff we didn’t need. There are an unlimited number of ways to do this, but we wanted to go with the keep it simple philosophy, so we started by using tcpdump to read in our PCAP data, outputting the ASCII formatted data to a file. Then, we ran the Unix strings command on that file to get read of any binary data that we couldn’t read anyways. We weeded out a few more things that we didn’t want through a magical combination of SED and AWK, added in the appropriate timestamps, formatted the data a bit prettier, and we had achieved our goal.

The end result of a reasonably small bash script was the ability generate application layer metadata in the form of something we call a Packet String, or PSTR file (pronounced pee-stur).  The script is ideally designed to run as a cron job where it parses continually generated PCAP files in order to generate accompanying PSTR files.

 

 

 

 

 

 

 

 

Figure 3: Sample PSTR Data

 

You can download the bash script that generates this data from PCAP files here. This is provided as a simple proof of concept and takes an input PCAP file and generates an output PSTR file. Now that we’ve got application layer metadata being generated in the form of PSTR files, let’s take a look at a few use cases.

 

Using PSTR as a Data Source

The original goal of generating PSTR files was to provide a data format with a low disk storage overhead that provided value to analysts as a secondary NSM data source. In a typical workflow, analysts would take an input from a detection capability, such as an IDS, and then PSTR would be another data source available to the analyst in order to provide supporting evidence in the analysis of a potential event or incident. I’ve written a few use cases here. Some of these are theoretical, but others are examples of actual things that have happened since implementing the PSTR data type.

 

Malware Infection Use Case

Let’s look at an example in which we’ve just received an alert from our IDS stating that an internal system has been detected as exhibiting symptoms of infection. The signature that fired did so because it saw a malicious GET request associated with a known botnet C2 server. The host was examined, and it appeared as though the GET request matches what is expected as a result of the signature that fired, so were able to determine with a pretty reasonable certainty that this box was infected.

Upon closer examination, we also notice that the infected host was also sending an HTTP POST with a very unique string. This looked like it might be an indicator of malicious activity, but it wasn’t something that any of your existing signatures fired on. In this case, an analyst was very quickly able to use GREP to quickly find other instances of this same string within the HTTP header data of all traffic on our monitored networks. PSTR data proved to be incredibly useful in finding other infected boxes across multiple networks.

 

Targeted Phishing Use Case

As a theoretical example, consider another example where several users have contacted your security team because they’ve received a very suspicious e-mail that seems to be targeted specifically at your company. This e-mail mentions a payroll adjustment and asks the client to access the provided link and log in with their employee ID number and password.

After examining the e-mail, you’ve determined that it has been sent from a spoofed e-mail address and that it uses a slightly modified subject line that is unique to each recipient. You’ve also noticed that based upon the reports you’ve received from users, these e-mails have come in over the past several weeks. One of the things you would want to do in this case would be to find who within your organizations received this e-mail. The purpose of this is to be able to warn the users not to click the link in the e-mail and also in hopes that you might be able to find a pattern as to why the selected recipients were chosen (access to certain systems, high profile employees, etc).

Typically, you might search through Exchange or Postfix logs to see if you can find who the recipients were. This of course relies on your organization having adequate logging and retention of those logs. The unique nature of the string however, makes it difficult to query these data sources. Using PSTR data, you can write a quick regular expression to match the semi unique subject lines and run a very quick query that will give you these results.

 

Using PSTR as a Detection Capability

The one thing we didn’t really anticipate when we created the PSTR file type was its use as a second level detection capability. When I refer to second level analysis and detection, I’m referring to moving past near real-time detection to the point in which analysts start reviewing traffic retrospectively to find things that signatures don’t catch. This often involves statistical and anomaly based detection with large data sets. This is something PSTR is perfect for.

 

User Agent Use Case

The user agent field within an HTTP header is always a good source for catching the low hanging fruit when it comes to malware infections on a network. Lots of malware will use a custom value in this field that deviates from standard browser identifying strings. The detection technique I’ve seen most commonly deployed to catch these types of malware infections at the network level rely on IDS/IPS signatures. As a matter of fact, if you subscribe to the common popular Snort rule sets then probably are using their user agent rules to detect known bad user agents.
The only problem with that detection scenario is that malware is now being generated at a rate much faster than the AV and ISD companies can keep up with. As a result, there are a LOT of malicious user agents out there that aren’t accounted for. In addition to this, some malware uses randomly generated user agent strings, meaning it’s much more difficult to write adequate signatures for detection.

One day, one of our analysts started playing around with PSTR data and wrote a quick script to parse all of the PSTR data for a given site, grab all of the user agent strings, and sort those by uniqueness. As expected, there were thousands of occurrences of the typical Firefox and Internet Explorer user agents, but what was really interesting was that there were several user agent strings only seen a handful of times that didn’t correlate to any particular known browsers. After a bit more analysis, we ended up finding quite a few machines that were infected with malware variants using these custom user agent strings. This one was a home run.

 

E-Mail Subject Use Case

The previous use case got us to thinking about other common fields with application layer metadata that we could do the same types of analysis on. One such field was the e-mail subject line field.  We modified our user agent parsing code to look at all PSTR data related to e-mail subject lines, and again had some very cool results.

Instead of most of the distribution being focused on one or two unique strings like we saw with user-agents, we saw that the distribution was spread very widely across thousands of different subject lines. This was expected, since most e-mails have a unique subject line. What interested us here however, was that we had a few subject lines that were used in excess. The first item of interest we found was that some sites had misconfigured applications that were mailing things to places they shouldn’t go, which was worth pursuing and getting fixed. We also found a user who was e-mailing all of his work documents to himself as a scheduled task every night, which was a policy violation.

This was all found with a very basic level of analysis, and it had some very real and useful results.

 

Additional Analytic Capabilities

The thing I love about this data format is that we can store a lot of it and it’s really quick to search through. With those things being true, there are a ton of things that can be done with it from a detection standpoint. A few immediate ideas include:

• Searching for unique values with HTTP, SMTP, DNS, and SSL headers

This is what we did in most of these examples. You can really quickly sort through the unique values within certain fields and find the outliers that warrant additional investigation.

• Byte entropy of certain fields to locate encrypted data where it shouldn’t be

It’s a common tactic to exfiltrate encrypted data through commonly used channels in an effort to hide in plain sight. Performing entropy calculations on GET and POST requests in an effort to find encrypted data would be a good way to detect where this might be occurring.

• Checking the length of certain fields for anomalies

You can do some statistical analysis and determine that certain fields will often have values that have a length falling in a particular range. Using that, you can flag on outliers that are far too short or too long in order to look for anomalies. I’ve seen good success in doing this with the various HTTP header fields, e-mail subject lines, and SSL certificate exchanges.

• Enumerating Downloads of Particular File Types

There is a great deal of value to being able to list all of the executables or PDF files downloaded within a certain time span. This is pretty easily achievable really quickly with analysis of HTTP header data within PSTR files.

Of course, all of these things CAN be done on PCAP data as well, but it would take significantly more processing power and it’s likely that you can’t store enough PCAP data at a given time to make it worth useful.

 

Conclusions

The concept of collecting and storing application layer metadata isn’t anything revolutionary. As a matter of fact, the idea isn’t even completely original as I’ve encountered other organizations that do similar things. There are even some commercial products that do this as well. However, I do know that nobody is sharing their methods and code with the world, which is the purpose of this post. Analysts live and die by their data feeds, and I think application layer metadata in whatever form it takes has its place amongst the other primary network data types. You can download the proof of concept code to generate and parse PSTR files here. I’m excited to see this data format evolve as we find more and more use for it. Look for more updates on this front as the code base continues to advance.

 

 

* A special thanks to my colleague Jason Smith for doing most of the legwork on writing the POC code.

 

I’m sitting in my hotel room after just finishing my last session at US-CERT GFIRST in Nashville, TN. This was my first time at GFIRST both as an attendee and presenter, and I really had a great time. Where I’m originally from in Kentucky isn’t too far from Nashville so I am familiar with the area and the venue choice, the Gaylord Opryland Hotel, is a beautiful facility and top-notch for this kind of conference. I wanted to take a moment to address where people can find the resources for my presentation as well as my thoughts on some of the presentations I had a chance to see and the conference as a whole.

My Presentation

Along with my friend and colleague Jason Smith, we presented a talk on Real World Security Scripting. At a bare minimum, we wanted to share some quick and dirty scripts we wrote to do some pretty neat things within our security operations center (SOC) at SPAWAR. At a higher level, we really hoped that we could encourage some people to get involved with low level BASH, Python, and PERL scripting to automate tasks within their SOC environment as well as increase capabilities of the SOC and its staff. We generated quite a bit of interest, and as a result it looks like several people were turned away because the room was filled to fire code capacity. Our sincere apologies to those who missed to talk. We got some really positive feedback from folks who did make it to the presentation.

As promised, we will be releasing our slides and source code for the presentation. The slides can be downloaded here. As for the source code, we are maintaining the distribution release on https://www.forge.mil, which requires a DOD CAC or ECA certificate to access. I understand that a lot of government folks outside of DOD don’t have access to forge.mil, so we are trying to find another place to host this code where we can control access to only people in the .gov or .mil space. In the meantime, if you would like to get copies of the code, please e-mail me at my mil address (chris.sanders.ctr@nsoc.med.osd.mil) from your mil/gov address and I will get it over to you. We are hoping to get all of that bundled up by next week.

 

Presentations I Attended

Keynote Panel Discussion – “Unplug to Save”

I started the week on Tuesday by attending the opening ceremony in which there was a panel discussion between several leaders in the government cyber defense community. The panel included Winn Schwartau, Mark Bengel, Doris Gardner, John Linkous, and John Pray, Jr and was moderated by Bobbie Stempfley. If you aren’t familiar with those individuals I’ll leave the Googling to you .

 

The discussion was centered on the concept of “unplug to save”, focusing on whether it was an acceptable solution to unplug an entity from the Internet in order to prevent a catastrophic event from occurring as a result of a cyber attack. The panel was split and brought up several good points about the interdepencies between certain aspects of government and national defense, namely citing the one that were unknown. Truth be told, sometimes we just don’t know the affect removing certain networks from the Internet would have. I’m of the opinion that in some cases hitting the kill switch is the best policy, but that is only in an extreme and I’m not sure who that authority should be put on. The panel also got into a discussion of the inherently flawed nature of the Internet and the need for an architecture redesign. That was all fine and dandy and I won’t disagree…but until some form of governing body takes on the task of redesigning the fundamental protocols of the Internet and it is taken seriously then this is just a pie in the sky dream.

 

The only thing that really irked me during the discussion was when one of the panelist mentioned how we could “solve the cyber problem” by hiring the types of hackers who can’t get clearances. It would seem to be that doing such a thing would be a prime way to generate more Bradley Manning-esque cases. Granted, Manning wasn’t a computer security expert by any means, but imagine what someone with his kind of access could do with a bit of hacking knowledge. I’d just asoon we make cyber jobs within the government more attractive to young professionals so that they stay on the straight and narrow instead of the USG resorting to hiring criminals.

 

 

Internet Blockades

This talk was presented by Dr. Earl Zmijewski from Renesys and was one of the talks I enjoyed the most. He described several types of Internet censoring, blocking, and filtering techniques used across the world citing recent examples of Egypy, Libya, North Korea, and of course, the great firewall of China. All of his examples had technical data to back them up which really left me with satisfied. Random fact – N. Korea only has 768 public IP addresses.

 

 

Using Differential Network Traffic Analysis to Find Non-Signature Threats

This talk was centered on the creation of metadata of layer 7 data on the network. This isn’t entirely a new concept, but its one that most people are just now keying in on. The general idea is that you can strip out only the layer 7 data from HTTP/DNS/EMail streams, index it, and store it so that you can perform analysis on it. The benefit here is that the amount of disk space required for storage of this type of data is much less than storing full PCAP, allowing for more long term analytics. The talk was presented by David Cavuto from Narus, who did describe a few useful analytics I hadn’t though of. For example, collecting the length of HTTP request URIs and performing a standard deviation of those to look for outliers. This could potentially find incredibly long or incredibly short URIs that might be generated by malicious code.

 

Unfortuantely, being a vendor talk, Mr. Cavuto didn’t provide anything that would help people generate layer 7 metadata, but he did have a product he was selling that would do it. Fortunately, I have some code that will generate this type of metadata from PCAP. I’m going to button that up and release it here at some point…for free

 

 

Getting Ahead of Targeted and Zero-Day Malware Using Multiple Concurrent Detection Methodologies

This was, by far, my favorite presentaiton of the week. It was given by Eddie Schwartz, the new CSO at RSA. The talk was centered around investing time in the right areas of analysis. Namely, looking across the data sources that matter and not relying on the IDS to do all the work. Once Mr. Schwartz releases his slides I would recommend checking them out. He is a man who understands intrusion detection and how to make it effective. My favorite part of his talk was something he said a couple of times: Yes, doing it this way is hard. Suck it up. It gets easier.

 

 

They Are In Your Network, Now What?

This talk was presented by Joel Esler of Sourcefire. Joel is a really smart guy and a great presenter and he didn’t disappoint. My big take away from this one was his discussion of Razorback, which I really think is going to be one of the next big things in intrusion detection. I think a lot of the crowd missed the point on this. There were a lot of complaints because of the amount of legwork required to integrate the tool, but I think most of those people were overlooking the early stage the tool was in and the potential impact of the community released nuggets and detection plugins. I played with Razorback when it was first released and look forward to digging into it again once some of the setup and configuration pains are eased. I’ve already thought of quite a few nuggets that I could possibly write for it.

 

 

Analysis Pipeline: Real-time Flow Processing

I’m a huge fan of SiLK for netflow collection and analysis so I was excited to hear Daniel Ruef from CERT|SEI talk about Analysis Pipeline, a component that adds some cool flexibility to SiLK. Overall, I was really impressed with the capability and am looking forward to playing with the next version when it comes out in a couple of months. I always say that if you aren’t collecting netflow you are missing out on some great data, and SiLK is a great way to start collecting and parsing netflow for free. If you are already using SiLK, please do yourself a favor and look into the free add-on Analysis Pipeline.

 

 

Advanced Command and Control Channels

I thought this was an awesome overview of traditional and more advanced C2 channels that malware use. I don’t think anything here was really new, but the way the presentation was broken down was very intuitive and the examples that were given were rock solid. This was given by Neal Keating, a cyber intel analyst with the Department of State.

 

 

Final Thoughts

I really enjoyed the conference and honestly consider it one of the best and most relevant conferences for folks in cyber security within the gov/mil space. My only major complaint was that a few vendors managed to sneak their way into speaking and basically giving product sales pitches rather than technical talks. I’m hoping that feedback will make it back to the US-CERT folks and more effort will go into preventing that from happening in the future. I hate showing up to a talk that I hope to learn something from and being drilled with sales junk about products I don’t want. Yes, I’m looking at you General Dynamics and Netezza.

 

Overall, the staff did a great job of organizing and I’d be happy to have the opportunity to attend and speak at GFIRST 2012 in Atlanta next year.

 

 

TL;DR – Real World Security Scripting Presentation Slides – http://chrissanders.org/pub/GFIRST2011-SandersSmith.pdf – Please e-mail me for full code.

 

I now write almost exclusively in support of the Rural Technology Fund. The RTF is a 501c3 non-profit organization I direct that is designed to provide opportunities for students in rural areas who are interested in pursuing computer related education and careers. I’m donating 100% of the profits from my book Practical Packet Analysis to that cause and nearly all of the proceeds from advertisements on this site and my other writing go there as well. Those donations mainly support the various scholarship programs the fund is involved with. I wanted to take a moment and post about another new exciting project going on with the Rural Technology Fund.

 

It’s often the case that students from rural communities have the desire to learn more about technology, but simply lack the resources. The public schools these students attend operate on shoestring budgets, and unfortunately, technical books often don’t make the cut when purchasing decisions are made. In response to this, the Rural Technology Fund has developed its Book Donation Program. This program was started in 2011 and aims to provide technical resources to students in rural areas by donating technical books to public school libraries in these communities. The ultimate goal of this program is to provide rural students with the resources they need to pursue an interest in computer technology.

 

Where do the books come from?

The books we donate to our recipient libraries are obtained through two primary resources. The program is primarily driven by several publishing companies who provide excess inventory to the RTF in support of our cause.
Individuals and businesses also play a key role in providing books for donation. The RTF gladly accepts used information technology books from individuals and businesses as tax deductible donations.

Can I donate my used computer books?

Absolutely! We rely heavily on individual donations to sustain this program. As a 501c3 organization, any donation you make is fully tax deductible. If you are interested in contributing books to the RTF, please send us an e-mail at info@ruraltechfund.org.

What kinds of books do you accept?

Generally speaking, we will accept any computer technology related book that is still relevant and reasonably up to date. This includes used books and prior versions of newer books. We’ve accepted books on a variety of topics including programming, web design, security, hardware, mobile devices, and more. When in doubt, feel free to e-mail us with any questions related to donations.

Where can I send donations?

All donations currently route through our primary office in South Carolina, where they are bundled with other books before being delivered to recipient libraries. Donations should be sent to this office at 1330 River Otter Court, Mount Pleasant, SC 29466. Shipping costs are also considered tax deductible. If your donation is large enough, we will gladly pay shipping charges in order to make sure we can get your books into the hands of students who will use them.

Where are recipient libraries located?

Currently, we service several school libraries throughout the state of Kentucky. As the program expands we hope to begin donations to recipient libraries in Tennessee and South Carolina.

How can I nominate my library as a donation recipient?

If you would like to nominate your library as a potential recipient of book donations, please send an e-mail to info@ruraltechfund.org with your needs and we will do our best to help out.

 

 

You can read more about the Rural Technology Fund at http://www.ruraltechfund.org.

I’m very excited to announce that my latest book, Practical Packet Analysis, Second Edition, has been released. Even more so, I’m thrilled that 100% of author proceeds for this book will be going to support the Rural Technology Fund to provide scholarships to students from rural areas pursuing further education in computer related sciences. You can read more about the Rural Technology Fund at http://www.ruraltechfund.org.

Book Description

I’m excited to announce that I, along with my good friend and colleague Jason Smith, will be speaking at the DHS US-CERT Government Forum of Incident Response and Security Teams (GFIRST) Conference in August. The conference is being held the week of August 7-12 at the Gaylord Opryland Hotel in Nashville, TN. We will be speaking at 1 PM on Wednesday, August 10th.

Title: Real-World Security Scripting

Abstract:

Scripting serves several purposes within a security operation center (SOC). You can write scripts to automate common tasks, to perform actions on large amounts of data, or to perform calculations and correlation on data sets. Given a bit of time an analyst can do great things with an interpreter and a little bit of elbow grease. Over the past few years our team has found that a lot of incredibly useful analysis tools can be created with only a minimal amount of programming knowledge.

In this presentation, we are going to educate our audience on how to get started with scripting for SOC related functions. Don’t be fooled though, this isn’t your typical scripting lesson. We aren’t going to ramble on about data types, expressions, and syntax formatting. Instead, we are going to look at real scripts we use in the SPAWAR SOC every single day. We will step through as many scripts as time permits while showing effective methods for automatically parsing netflow data for known malicious hosts, extracting payloads from PCAP files for content or entropy analysis, and more. We will go through the process of creating each script from inception to production.

A few specific scripts we will break down include:

• Updating a snort ruleset across multiple sensors
• Automated reporting of known malicious IP addresses and domains with netflow data
• Extracting the data payloads of packets in PCAP files
• Retrieval and concatenation of PCAP data from multiple sources
• Automatic parsing of netflow data into various graphs for visual traffic analysis

 

The tools covered will be a mix of BASH, PERL, and Python. No prior scripting knowledge is required to gain value from this presentation. We will be providing source code for all of the scripts we are discussing as well as a few extras. As a bonus, we will even provide versions of some of these scripts that can be integrated into Arcsight or other SIEM products to extend their capabilities. At the very least, attendees will walk away with scripts they can implement into their production SOC immediately. The real value of this course however, is a real world crash course in scripting for analyst-centric SOC functions.

 

You can read more about the GFIRST conference at http://www.us-cert.gov/GFIRST/.

 

 

Look forward to seeing you there!

 

I recently found myself in a situation where I had a couple dozen Snort sensors deployed in a network with no commercial software for centralized management. Due to the decentralized nature of the sensor management, one of the bigger headaches was adding new custom rules to all of the sensors. New rules had to be added to each sensor manually into a custom rules file. These rules all existed in a single file, so I wrote a bash script that automates this process. Using this script an analyst only has to add the new rules to a single file and run the script to push it out to all of the other sensors. I thought I’d share that here for those that might get some use from it.

FAQ

What does the script do?

The script does a few simple tasks. It will perform the following actions for each IP in its sensor list:

• Create a backup of the existing custom rules file on the sensor.

• Replace the current rule file with the new rule file.

• Performs a ‘diff’ on the new rule file and the old rule file and places the results in a timestamped log file in /var/log/snortrules.

• Restarts snort on the sensor to ensure the new rules are applied.

What are the requirements for running the script?

In order to execute the script, the following conditions must be met:

• A custom rule file named the same as the custom rule file on the sensor must exist in the directory the script is executed from.

• You must have SSH/SCP connectivity to the servers.

• It is necessary to have permissions to perform the actions described above on the appropriate folders.

Additionally, it helps to have certificate based authentication setup for a service account that can handle actions performed by this script. Otherwise you will have to password authenticate to each sensor.

How do I add in the addresses for my sensors?

The first line of code in the script contains the list of sensor IP addresses. Replace the following with addresses for your sensors, delimited by spaces.

What other variables do I need to modify within the script to match my environment?

There are three main variables in addition to the sensor IP addresses. These are:

• rulepath – The path on the remove server where the custom rules file exists

• customrules – The name of the custom rules file

• user – The user name to use for authentication to the sensor

Can I make modifications to the script?

Absolutely. I’m not a programmer. I’m just a guy who saw a need and wrote something to address it quickly. That said, the script could probably be setup a lot better and do a lot of cool neat things (like error checking). If you find some value you in the script and want to make some modifications or additions to it then by all means do so, I just ask that you reciprocate those changes back to me so everyone can benefit.

 

With all that out of the way, you can download the script here.

I’m once again going to be leading a SANS Mentor session. This time however, I’ll be teaching SEC 503, Intrusion Detection In-Depth in my new home of Charleston, South Carolina. The course will be starting on June 22nd, running once a week for two hours, for ten weeks. The course will be held at Honeywell, on Rivers Avenue in North Charleston.

 

An excerpt from the course description:

Learn practical hands-on intrusion detection and traffic analysis from top practitioners/authors in the field. This is the most advanced program in network intrusion detection that has ever been taught. The emphasis of this course is on increasing students’ understanding of the workings of TCP/IP, methods of network traffic analysis, and one specific network intrusion detection system (NIDS) – Snort. This is a fast-paced course, and students are expected to have a basic working knowledge of TCP/IP in order to fully understand the topics that will be discussed. Although others may benefit from this course, it is most appropriate for students who are or who will become intrusion detection analysts. Students generally range from novices with some TCP/IP background all the way to seasoned analysts. The challenging, hands-on exercises are specially designed to be valuable for all experience levels. If you want to learn the ins and outs of TCP/IP as it relates to security analysis, how to dissect packets at their most basic level, and how utilize NIDS effectively then this is the course for you.

 

If you are pursuing DOD 8570 certification, then the certification paired with this course, the GCIA, will satisfy the requirement for the CND-Analyst designation. This is a great course if you are a government employee or contractor pursuing 8570 compliance, or simply someone working in information security looking to strengthen your defensive technology skills and gain a widely accepted certification in the process.

 

If you are interested in learning more then you can visit the SANS website for this course at: http://www.sans.org/mentor/details.php?nid=24684. Also, feel free to pass around the flyer for this course, which can be viewed here. Also, I can provide some discounts to help offset the cost a bit if you contact me directly.

I had the opportunity to take the SANS FOR610: Reverse Engineering Malware course in Orlando a couple of weeks ago and I wanted to write about my experience with the course. It’s no secret that I’m a big proponent of SANS. I’ve taken SEC 503 and SEC 504 at live events and I also mentor both courses here locally in Charleston. I wanted to take FOR610 as my next course because malware analysis is something I’ve not done a significant amount of. I’ve done a fair amount of behavioral analysis but very little code analysis at the assembly level and the course syllabus appeared to be heavy on that subject so it seemed like a natural fit to help fill in some of my knowledge gaps.

Instructor

The course in Orlando was taught by Lenny Zeltser. Lenny is the primary author of the materials, and he also runs a great blog over at http://blog.zeltser.com/ that I’ve followed for quite some time. I’ve been to a lot of different training courses and have also provided courses myself so I’ve seen plenty of bad instructors and good instructors. One of the things I find most challenging when teaching is taking highly complex subject matter and breaking it down in such a way that it is understandable. Being able to do this effectively is one of my primary criteria for defining a good instructor. That said, Lenny is perhaps one of the best teachers I’ve had. He took all of the highly complex concepts and broke them down in such a way that they were understandable at some level for every one in the class. He provided clear guidance and assistance during the lab portions of the class and I don’t remember a single question that was asked that he didn’t have an immediate answer for. His depth of knowledge on the subject was very apparent and appreciated.

Difficulty

The course really has two distinct sides to it: behavioral analysis and code analysis. Depending on your background, you may find this course very difficult at times and easier at others. I have written several programs in languages including Python, PHP, and C as a function of my primary job role, so I understand programming concepts, but I’m not a professional programmer by any stretch. That being the case, I had a harder time with the code analysis portions of the course. If I didn’t have any programming experience, I think I would have been completely lost on more than a few occasions. On the other side of the coin, I had no problems whatsoever with the behavioral analysis instruction and labs, but I could tell that several other people in the class did. From what I gathered by talking to people and looking at name badges, roughly 65-85% of the folks in my class were programmers of some sort. The course is touted as not requiring any previous programming experience, but I think to get the full benefit from the class, you should at least be familiar with core programming concepts, preferably in an object oriented language.

Course Content

The course was 5 days long and covered a variety of topics. I’ve outline some of those here along with the new skills I gained or enhanced as a result of what we learned.

Day 1

The first half of the first day was devoted to the setup of the virtual malware analysis lab used in the course. This is done in such a way so that the virtual lab can be used after you leave the class to do real world malware analysis in your organization using the virtual infrastructure. The second half of day one focused on using the lab for behavioral analysis.

New Skills I Gained: Knowledge of new malware analysis tools.

Day 2

This day built upon our knowledge of behavioral analysis and introduced new concepts related to that. We were introduced to dissecting packed executables and Javascript and Flash malware.

New Skills I Gained: Automated unpacking of packed files. Tools for dissection and extraction of malicious code in Flash objects.

Day 3

This day was devoted to code analysis. We were introduced to assembly and spent a great deal of time looking at commonly identifiable assembly patterns used in malware. This was one of the most useful parts of the class for me. We also looked a bit at anti-disassembling techniques that malware authors use.

New Skills I Gained: Enhanced understanding of assembly. A plethora of anomalies to look for in assembly level code analysis of malware. Patching code at the assembly level to get a desired outcome.

Day 4

The fourth day focused on analysis of malware that was designed to prevent itself from being analyzed. We looked at packers and learned how to manually step through malware code to unpack it for analysis. The day ended with an detailed and highly valuable look into deobfuscating malware in browser scripts.

New Skills I Gained: Detailed understanding of assembly for malware analysis. Manual extraction of unpacked code from packed executables.

Day 5

The final day of the course was another one of the most useful parts of the course for me. This first half of this day focused on analysis of malicious Microsoft Office files and malicious PDFs. After lunch, we covered shellcode analysis and memory analysis.

New Skills I Gained: Tools and procedures for extracting malicious code from MS Office files and PDFs. Better understanding of PDF file structure. Extraction of malware running in memory.

Labs

The labs were an integral part of the course. In the labs we analyzed real malware samples in our virtual analysis lab. I’m incredibly happy that we looked at REAL code from REAL attackers rather than simple malware created in a lab for the purpose of the course. Doing things this way we got to see how attackers will often take shortcuts or write bad code that we have to sort through rather than just dissecting cookie cutter malware with no imperfections. The labs served their purpose, helping reinforce new concepts in a practical manner. During the course, everyone had their laptops open and two virtual machines running at all times as we would dive into them for exercises very frequently.

Although I was very pleased with the labs in some ways, I am critical of them for a few other reasons. Prior to the class, you are provided some instructions on how to setup a single Windows based VM that is destined to be infected with malware repeatedly throughout the class. In addition, the instructions said we would be given a version of Remnux, the reverse engineering malware Linux distribution created by Lenny, to use during the class when we got there. I got this all up and running without any problems, but I was pretty upset when I got to the class to find out that there was quite a bit more setup to do. As a matter of fact, almost the entire first half of the first day of instruction was taken up by additional lab configuration. We were given a CD that contained a variety of tools that were to be installed on our Windows VM. I think all in all, we had to install about 25 different tools. Several people asked why these weren’t provided prior to the class and we were told it was so that we would take more ownership over our malware analysis labs and could ask questions. Although I can respect the comments in support of this, I think providing these tools prior to the class along with the other instructions would allow for better use of time. At lunch the first day I felt a bit cheated as my company had paid for an expensive course where I was just sitting around installing software. Providing this software prior to the course and having people come prepared would have allowed for a whole half day of additional instruction which would have been incredibly valuable.

The other primary issue I had with the labs was the format in which they were laid out. In most of the labs, Lenny would teach us a concept and then step through the process on his own system. Then he would turn us loose on our systems to work on the same example he just walked through. Although somewhat helpful, it wasn’t entirely effective since we had just seen him do the same example we were working through. I would contrast this with the lab format in the SEC 503: Intrusion Detection In-Depth course. In that course, students are given a workbook with lab exercises. The instructor there would teach a concept, go through a lab on screen, and then turn students to the workbook and give them some time to work through similar, but different examples. This format provided a great deal more value because we had to do quite a bit more thinking to get through the examples on our own, rather than just recreating what the instructor did.

Summing It Up

Overall, my experience with FOR 610 was very valuable and I’m thrilled I got the chance to take the course. I walked away with a lot of new skills and am able to provide a lot of value to my organization as a result. I now feel completely comfortable performing code analysis of malicious binaries. I also learned more assembly than I ever thought I would and feel like I could even write some simple programs in assembly should I choose to punish myself in that manner. I also gained a greater understanding of lower level operating system components which will prove useful in several cases. Make no mistake, this is a very difficult course, which is why ways numbered it so high. It is the highest level forensics course they teach, and it will challenge you. However, if you are up to it, there is a lot to be learned here, and I have no doubt that it is the best malware analysis course you will find.

You can read more about this course at http://www.sans.org/security-training/reverse-engineering-malware-malware-analysis-tools-techniques-54-mid.