The 10 Commandments of Intrusion Analysis

I’ve been actively involved in the training and development of intrusion detection analysts for a few years now which includes being a SANS Mentor for SEC 503: Intrusion Detection In-Depth. One thing I find myself constantly doing is trying to evolve my philosophy on effective intrusion detection. While doing this, some themes arise that tend to stay consistent no matter how that philosophy changes. Through that, I’ve written up something I call the “10 Commandments of Intrusion Analysis” which highlight some of those themes that seem to be at the core of what I try to instill in the analysts I train and in my own analysis. They don’t really command you to anything, but there are 10 of them, so the name kind of fits. These may not fit you or your organizational goals or personal style, but they work for me!

1. Analysts, Analysts, Analysts!

The most important thing an analyst can have ingrained into them in their importance. An analyst is the first line of defense. The analyst is sitting in the crows nest watching for the icebergs. It is the analyst who can keep attacks from happening and can stop attacks from getting worse. Most security incidents begin with an analyst providing a tip based upon an IDS alert and end with an analyst putting in new signatures and developing new tools based up on intelligence gained from a declared incident. The analyst is the beginning and the end in information security. The alpha and omega. Okay, maybe that’s a bit dramatic, but the importance of an intrusion analyst can’t be understated.

2. Unless you created the packet yourself, there are no absolutes.

Analysis happens in a world of assumptions and its important to remember that. Most of the decisions you will make are centered around a packet or a log entry and then honed based upon intelligence gathered through research. The fact is that the analyst isn’t the one who generated the traffic, so every decision you will make is based upon an assumption. Don’t worry though; there is nothing wrong with that. Ask your friendly neighborhood chemist or physicist. Most of their work is based upon assumptions and they have great success. The takeaway here is that there are no absolutes. Is that IP address REALLY a known legitimate host? Does that domain REALLY belong to XYZ company? Is that DNS server REALLY supposed to be talking to that database server? There are no absolutes, merely assumptions, and because of that remember that assumptions can change. Always question yourself and stay on your toes.

3. Be mindful of how far abstracted from the data you actually are.

An analyst depends on data to perform their function. This data can come in the form of a PCAP file, an IIS log file, or SYSLOG file. Since most of your time will be spent using various tools to interact with data it’s crucial to be mindful of how that tool interacts with the data. Did you know that if you run Tcpdump without specifying otherwise, it will only capture the first 68 bytes of data in a packet? How about that Wireshark displays sequence and acknowledgement numbers within TCP packets in a relative manner by default? Tools are made by people and sometimes “features” can cloud data and prevent proper analysis. I think both of the features I described earlier are great, but I’m also mindful that they exist so I can see all of the packet data available or view the real sequence and acknowledgement numbers when needed. In a job where reliance upon data is critical, you can’t afford to not understand exactly how tools interact with that data.

4. Two sets of eyes are always better than one.

There is a reason authors have editors, policemen have partners, and there are two guys sitting in every nuclear silo. No matter how much experience you have and how good you are you will always miss things. This is to be expected because different people come from different backgrounds. I work with the government so the first thing I look at when examining network traffic is the source and destination country. I’ve worked with people who have systems administration backgrounds and as a result, will look at the port number of the traffic first. I’ve even worked with people who have a number crunching background who will look at the packet size first. This demonstrates that our experiences shape our tactics a bit differently. This means that the numbers guy might see something that the sysadmin didn’t see or that the government guy might have insight that the numbers guy didn’t. Whenever possible it’s always a good idea to have a second set of eyes look at the issue you are facing.

5. Never invite an attacker to dance.

This is something I’ve believed since the first day I ever fired up a Snort sensor, but IDS guru Mike Poor phrased it best while I was attending one of his SANS classes when he said that you should never invite an attacker to dance. As an analyst its very tempting to want to investigate a hostile IP address a bit beyond conventional means. Trust me, there have been many occasions where I’ve been tempted to port scan a hostile that kept sending me painfully obviously crafted UDP packets. Even more so, any time someone attempts to DOS a network I’m responsible for defending, I wish nothing more than to be able to unleash the full fury of a /8 network on their poor unsuspecting DSL connection. The problem with this is that 99% of the time we don’t know who or what we are dealing with. Although you may just be seeing scanning activity, the host that is originating the traffic could be operated by a large group or even a military division of another country. Even something as simple as a ping could tip off an attacker that you know they exist, prompting them to change their tactics, change source hosts, or even amplify their efforts. You don’t know who you are dealing with, what their motivation is, and what there capabilities are, so you should never invite them to dance.

6. Context!

One word can drastically change the dynamic of your monitoring and detection capabilities. In order to be effective you must have context into the network you are defending. Network diagrams, listings of servers and their roles, breakdowns of IP address allocations, and more can be your best friend. Basically any and everything that can be used to document the assets within the network, how they function, and how they relate to other assets are beneficial in running down anomalous events. Depending upon your role in the organization you may not be in a position to obtain these things and if they don’t already exist you are going to have a heck of a time getting the systems folks to put in the leg work to create them. However, as difficult as this may be, its an effort that’s worth pursuing. Whether you have to present your case to the CIO or just buy your network engineers a case of their favorite adult beverage its ultimately worth the effort.

7. Packets, in a word, are good.

The ultimate argument in life is whether or not people are inherently good or inherently evil.  This same argument can be had for packets as well. You can either be the analyst that believes all packets are inherently evil or the analyst that believes all packets are inherently good. I’ve noticed that most analysts typically start their career as for the former and quickly progress the later. That’s because its simply not feasible to approach every single piece of traffic as something that could be a potential root level compromise. If you do this, you’ll eventually get fired because you spent your entire day running down a single alert or you’ll just get burnt out. There is something to be said for being thorough but the fact of the matter is that most of the traffic that occurs on a network isn’t going to be evil, and as such, packets should be treated innocent until proven guilty.

8. Analysis is no more about tcpdump than astronomy is about a telescope.

Whenever I interview someone for any analyst position that’s above entry level I always ask them to describe how they would investigate a typical IDS alert. I get frustrated when someone gives answers along the lines of “I use  Tcpdump, Wireshark, Network Miner, Netwitness, Arcsight, Xeyes, etc” with no further clarification. Although their are processes and sciences in intrusion analysis, intrusion analysis itself is not a process or a science, but rather an art. If this wasn’t the case then it wouldn’t even be necessary to have humans in the loop when it comes to intrusion detection. An effective analyst has to understand that while different tools may be the most important part of the job, those things are merely pieces of the puzzle. Just like an astronomer’s telescope is just another tool in his arsenal that allows him to figure out what makes the planets orbit the sun, Wireshark is just another tool in an analysts arsenal that allows him to figure out what makes a packet bypass a firewall rule. Start with the science, add in a few tools and processes, stay cognizant of the big picture, keep an attention to detail, and eventually the combination of all of those things and the experience you gain over time will help you develop your own analysis philosophy. It’s at that point you have taken your analysis to the level of an art, and made it so that your worthy enough to not be replaced by a machine.

9. Sometimes, we lose.

No matter how hard you try there will come a point in which the network you are defending gets successfully attacked and compromised. In the modern security landscape its inevitable and there isn’t a lot you can do about it. In these times its likely that the analyst will take the heat over the incident. Because of this, you need to be prepared when it happens. An incident won’t be remembered for how an intrusion occurred, but rather how it was responded to, the amount of downtime that occurred, the amount of information that was lost, and ultimately the amount of money it costs the organization. What recommendations can you make to management to ensure a similar incident doesn’t occur again? What can you show your superiors to explain why the attack wasn’t detected? What shortcomings do your tools have? These are questions that can’t fully be answered until an intrusion has occurred and you have the context of an attack, but you can definitely consider the questions now and have a plan for how your information will be presented to key figures. You will get caught off guard and you will be blind sided, but its important that you don’t appear as such and you keep your game face on. This can make the difference between a promotion and a pink slip.

10. Dig deeper.

At the end of the day you have to have something to rest your laurels on and that has to be the fact that you’ve done your due diligence and that you’ve given your best. My “motto” per se when it comes to intrusion analysis is “Dig Deeper”. A defender has to control 65,535 ports. An attacker has to compromise one. A defender has to protect 10,000 users. An attacker has to deceive one. A defender has to examine millions of packets. An attacker has to hide a malicious payload in one. What can you do to increase your visibility into the data? What proficiency can you develop that gives you that edge against the attacker? You have a hunch that there is more than meets the eye, so what can you do to dig deeper?