Author Archives: Chris Sanders

Packet School 101 – Part 4

** Disclaimer to all new readers – This blog post is VERY old and not really representative of my current work. I’ve just left it up here for historical purposes. If you are interested in learning more about packet analysis I’d reccommend reading some of my newer posts or looking at my book, Practical Packet Analysis. **

In this segment of Packet School 101 we are going to take a look at the trace files of a computer infected with spyware and one suffering from an application fault.

Spyware Infection

In our first sample scenario, a user has called complaining that her computer is dropping its network connection three minutes or so after booting up. When this happens the computer also maxes out its processor utilization. After making some calls you verify that everybody else is running as they should be, so the problem is just isolated to this one computer. When looking at the computer, it is apparent that her problem does exist as told. So what do we do? We fire up our sniffer of course!

Examining the Packets

Click here to download the sample trace file (< 1MB)

In examining our trace file there are a lot of problems with many things going on but we are only going to focus on the one client machine we are looking out down. It’s IP address is 172.16.1.10. The first thing we notice at the beginning of the capture is that an outside IP address keeps trying to make a connection to our computer on port 26452. Our client doens’t know what to do with this traffic since it is not expecting it, and therefor sends repetitive RST packets effectively ending the communication. This obviously should not be happening, however it continues to do so until we get to packet 70 and see the client requesting a copy of a file called analiz.exe from this rogue IP address via TFTP. This file begins to download, and ironically enough, that is when the computer starts experiencing problems

The Resulting Analysis

After spending some time actually on the physical computer the process analiz.exe was found to be running. After terminating this process and removing the file via an autmated spyware scanning utility the computer began to run normally. What was happening in this case was that after a computer would boot up it would try to make a connection to the rogue IP address associated with this spyware. After a certain amount of time the client would then download an updated copy of this spyware file via TFTP and run it on the system.

This brings up several good security points to remember. The first one of these is the most obvious and that is to education your uses about spyware and how to avoid it. However, another key point to mention is that this infection did a lot of its dirty work through TFTP which is a UDP based protocol. A lot of the time when configuraing firewall filtering rules most administrators will completely ignore UDP and only focus on TCP. You can not neglect UDP protocols when doing this, as a lot of spyware will use this weakness to completely hammer your netork.

Application Fault

The next file we are going to examine is the trace of an appliciation fault that is causing a client computer to run incredibly slow. In this scenario we have a client who is complaining that their application is not working because the network is running to slow.

Digging Deeper

Click here to download the trace file (< 1MB)

When we open our trace file we can determine several things by looking at the first fewpackets. The first thing we see is communication taking place between our client and server machines. Looking at the packet times we can see those requests are happening at rates of less than a millisecond so that indicates that there shouldn’t really be any latency on the wire itself. Also, we see that the client and server are sending their data back at more than adequate rates so there shouldn’t be an issue with either of them acting sluggish.

When we get to packet 5 however, we begin to see something interesting. Our client machine is making a NetBIOS request to our server, and then in the following packet, our server (which is NOT running NetBIOS) returns an ICMP Destination Port Unreachable packet. So what is causing this NetBIOS traffic from our client? Well Wireshark provides a great interface for viewing data contained within a stream of data. In order to utilize this you will need to right click on packet number 4 and click “Follow TCP Stream”. This will show us the data contained in that stream of communication, including upper layer protocol information.

When we look at this stream of data it is clear to us what is going on. The user is attempting to use the AIDA32 application. Not only do we find out this information, but we can also see that they are doing this under the “Chadwick” user account which is in the administrators group of the local machine.

The Verdict

After looking at this TCP Stream data it is clear to see that the network is not at fault. What is happening is that the application is making a NetBIOS request to the server, and since the server isn’t running NetBIOS it responds to the applications request by saying it is busy. In the design of the application which is being used, there is no formal mechanism in place that lets the user know that the server is not accepting its communication. This means that the application is just sitting there doing nothing waiting for valid response, which it will never get. This being said, the proposed solution to this type of problem would be to enable NetBIOS on the server which the program is connecting to or find a program that uses an alternate means of communication.

Homework

Today we have analyzed to more trace files that have given us a further understanding of more packet related troubleshooting techniques. If you manage any type of network I would be willing to bet that you have problems with spyware on occasion. This being the case, the next time you see a computer with a spyware problem, sniff it’s communication. You would be amazed to see how often various spyware applications will silently “phone home” to download updates to the infection and perform various other tasks. In the next installment of Packet School 101 we will take a look at what a trace file looks like when a computer is the target of a port scan and attempted denial of service attack.

 

**UPDATE**

Enjoy Packets School 101? Check out Packet School 201!

You can have online degrees from well reputed institutions all over the world. Different constraints like; money, distance etc. will no more restrict you from getting valuable IT diplomas and certifications from any online university. Demand for professionals having MS 70-290 certifications has been increasing day by day. Presently, Microsoft is offering MS 70-536 certifications for the people interesting in learning the essentials of IT and MS 70-554 is more advance certification in this field.
 

Packet School 101 – Part 3.5

** Disclaimer to all new readers – This blog post is VERY old and not really representative of my current work. I’ve just left it up here for historical purposes. If you are interested in learning more about packet analysis I’d reccommend reading some of my newer posts or looking at my book, Practical Packet Analysis. **

I just wanted to make a couple of quick notes before I published the next section in Packet School 101. I have been in touch with Gerald Combs, who as many of you know is the original developer of Ethereal, and he has informed me that due to copyright and legal restrictions that Ethereal HAS been rebranded as the Wireshark project. I had heard this from several sources but wanted to make sure it was legitimate before I posted anything about it. This being said, Wireshark is basically the same program as Ethereal so all of the previous tutorials I have posted should still be valid. Also, from here on out I will no longer refer to the program as Ethereal, and all of the screenshots will also reflect the Wireshark program. You can learn more about Wireshark at http://www.wireshark.org.

Also, I want to thank you guys for all of the comments anad questions you have sent. Towards the end of the series I am going to devote one whole section to nothing but questions, so keep sending them my way.

Be sure to check back in the next couple of days for Packet School 101 – Part 4!

Packet School 101 – Part 3

** Disclaimer to all new readers – This blog post is VERY old and not really representative of my current work. I’ve just left it up here for historical purposes. If you are interested in learning more about packet analysis I’d reccommend reading some of my newer posts or looking at my book, Practical Packet Analysis. **

The response from this series has been tremendous! As of yesterday I have managed to make the front page of digg.com and make #2 on the list of the most linked sited on del.icio.us. I have seen my bandwidth grow exponentially and have logged over 5 GB of traffic in the past 24 hours. I have also had a lot of great comments regarding the first two parts that I hope to address later on in the series. I plan on ending with a Q & A so if you have any major questions feel free to e-mail me.

Troubleshooting a Slow Router

Download the sample trace file by clicking here (<1 MB)

In this section we are going to look at a client who is trying to connect to a website but it experiencing all kinds of network slowness issues. Opening the sample trace file, the first thing you will notice is a lot of ARP broadcast packets. These are typical in a lot of network environments for layer 3 to layer 2 address resolution. For the purpose of what we are doing here, we are going to remove these from out trace file so that they don’t clutter things up. You can do this by typing “!arp” in the filter text box near the top of the Ethereal window.
Getting on Time

At this point in our learning it is pertinent to take notice of the “Time” column in the main Ethereal window. You will see a time listed next to each packet, and by default, this shows the time the packet was recieved in relation to the beginning of the packet capture. This type of view has its purposes in some settings, however, for troubleshooting a slow network you will want to change this setting to display the time relative to the packet recieved previously. You can do this through by going to View > Time Display Format, and selecting “Seconds Since Previous Packet”. In this new time view you will notice that the times are now displayed as the amount of time since the previous packet was captured. For example, packet 4 was recieved 1.728522 seconds after packet 3. This will be much more handy to us in our troubleshooting of the slow network communication.

Looking for the Source of the Latency

Now that we have our time column set to display data to us in a much more helpful way, scrolling down through the trace file we see our first major network activity at packet 18 where the client (172.17.8.66) makes an HTTP request to get the website www.packet-level.com. Typically, the next packet we see should be the dns servers response to the client. In this case however, the server does not respond back to the client, so the next packet we see is over a second later and is the client attempting to request the webpage a second time. After this second request we finally see a response from the DNS server pointing us to the IP address of the web server in packet 20. In an ideal situation, as soon as this dns query is completed the client and server should begin a standard TCP/IP handshake (SYN, SYN/ACK, ACK), and in our case the client does its part sending it’s initial SYN packet out within 4 milliseconds (third place from the decimal point is milliseconds) of recieving the DNS response. The server on the other hand responds an incredible amount slower taking over half a second to send its SYN/ACK reply. At this point we can definitely begin to see that it is something other than the client causing the network latency.

Interestingly enough as we continue down into the trace, in packet number 26 we see a second DNS reply from the server. This the reply to our second DNS request that we made initially. The only problem is that it is about 5 seconds too late! Given that our client computer has already established a connection with the server there is no real need for this second connection, and it throws up an ICMP destination unreachable packet immediatly following the reciept of the DNS response.

Going back to our already established connection to the server, we begin to see problems sprouting up. After making our initial TCP/IP handshake the client requests the actual content of the webpage at packet 25. Quite some time goes by and then in packets 29 and 32 we see two TCP Retransmission packets. In this case, the client has requested the webpage, not gotten a response, waited a certain amount of time, and sent a retransmission to the server in order to make another attempt at getting the data. After the third retransmission we finally see a response from the server in packet 33. Now if you add up the times from packets 25 to 33 you will see it has taken us nearly 9 seconds to get the first bit of data from the webpage we are requesting. It doesn’t take a packet analyzing expert to realize this is entirely unacceptable.

Fixing the Problem

Given the information we have just seen we know that the client is not at fault for the slow communication. The principal rule of thought for figuring out the problem location is to move upstream along the network. In this network, the next step would be to look at the router to see if it is malfunctioning in any way. Upon rebooting the router on this particular network, the speed of data communication increased tremendously and the problem was solved. However, if the problem had not been the router then you would need to move upstream to the router of the network in which the web server you are connecting to is behind. Unfortunately when that is the case you typically do not have the power over the remote network to do anything about it.

Homework

In our next installment we are going to look at a spyware infection and its effects on a workstation. Now that you understand the concept of restransmissions and latency you may be asking yourself what exactly is a good time for something such as a website request to take place? The best thing to do in this case is to sniff the packets on your own network whenever you are not having any issues. Ideal communication times are going to vary for each and every network you are on so this is another case in which you will want to sniff your own network. Getting to know what the packets look like in your network when it is healthy will SURELY pay off in the future.

Packet School 101 – Part 4�

 

 

With the help of Internet technology, you can find wide range of available online IT courses and certifications. You can search out online schools that offer certification for MS 70-296. Online instructors help you in grasping the tricky concepts of MS 70-620. Free and fast registration for MS 70-554 is also available for business professionals. 

Packet School 101 – Part 2

** Disclaimer to all new readers – This blog post is VERY old and not really representative of my current work. I’ve just left it up here for historical purposes. If you are interested in learning more about packet analysis I’d reccommend reading some of my newer posts or looking at my book, Practical Packet Analysis. **

If you are reading this then it means you probably enjoyed the first part of my series on packet analysis, so if that is the case, welcome back! In this installment we are going to analyze our first trace file to learn some more packet analysis basics. I am going to assume that you have played around with Ethereal a bit and am going to forgo the explanation of all the different windows within it and things of that nature. Before we start you are going to need to download the sample trace file we will be working from. I obtained this trace file as well as a lot of others I will be using from Laura Chappell, Sr. Protocol Analyst for the Packet Level Protocol Analysis Institute (http://www.packet-level.com).

Our First Analysis – Downloading a File

Click here to download sample trace file (approx. 13MB)

Downloading a file is a pretty basic function when described at the network/transport layer. A client request a file from a server via HTTP and the file responds with this file. This takes the form of several packets as the file downloaded is segmented into various size packets for transport across the networks it must traverse to reach its destination. In our trace file we are going to look at an example of downloading a file from two different servers. The first server we will look at is going to be an extremely slow server so that we can look at some of the various problems that Ethereal will show us. After this we will look towards the end of the trace file for an example of what a healthy file transfer looks like.

The first thing we want to look at is the I/O graph of the packet capture. The I/O graph is customizable with a lot of various options and is a great way to visual interpret some of the data you have captured. To view the I/O graph of this packets capture click “Statistics” on the menu bar and then select “IO Graphs”. Once you have done this you are going to need to look towards the bottom right corner of the screen and change the “packets/tick” option to “bytes/tick”. This will provide the information to us in a form that is much more useful. You may feel free to change any other visual settings in this window.

Looking at the I/O graph for our packet capture we notice that traffic is slightly eleveated near the beginning of the capturing before flattening out. Then towards the end of the capture traffic spikes to a new high with a slight drop in the middle. Knowing that we are looking at an example of a fast and slow download we can easily distinguish where the two downloads take place just by looking at this graph. Let’s look at the packet capture and see what else we can determine about these downloads.

Scrolling through all of the packets you will see a lot of various information. The biggest portion of this is normal traffic and is therefor uesless to us as we are only looking for abnormal traffic. In order to filter this abnormal traffic out we are going to use a new feature in Ethereal called the expert info window. To open this window you can simply click on “Analyze” in the menu bar and select “Expert Info”. By default, the expert info window is going to show us all warning, error, note, and chat communication. Chat communication does not really constitute abnormal traffic, therefor we are going to change this settings to show only warnings, errors, and notes by changing the appropriate setting under the “Severity filter” heading.

Anatomy of a Slow Download

Looking at the expert information window there are a few things we want to discuss right away. The first thing you will see in abundance are TCP Window Update packets. If you are familiar with TCP/IP you know that the transmission rate of data is determined by the size of the TCP recieve window. When transferring data between clients they will constantly send TCP Window Update packets to each other as there ability to recieve data speeds up or slows down.

The next thing we want to look at is where we have our first problematic packets. As the download starts you being to see “Previous segment lost” packets. Basically what this is saying is that during the course of data transfer, all of a suddent a packet is dropped. In response to this the client sends what you are seeing as “Duplicate ACK” packets to the server in order to request to packet that was dropped. The client will continue sending these duplicate ACKs until it finally recieves the packet it was looking for. The retransmission of this missing packet is what you are seeing as “Fast retransmission” in the expert info window. At the start of the download you are only seeing one or two duplicate ACKs at once but as the download progresses you begins seeing more and more of them which means you are experiencing more latency. At this point you can browse through the rest of the capture and see that it is riddled with these segment losses and duplicate ACKs. If you browse to the end of the capture file however, you will notice this is not the case. In this instance there are only one or two dropped packets in the successful download.

One last thing we are going to look at is another way to visually represent the packet capture data called the “TCP Stream Graph”. You can access this graph by clicking on a packet related to the stream you wish to analyze (in the case you can select packet number 1023) and going to “Statistics”, selecting “TCP Stream Graph” and clicking on “Round Trip Time Graph”. This graph is not too visually pleasing but is a great way to compare round trip time (RTT) throughout a packet capture. You will notice near the beginning of the capture with the slow download we are seeing RTT of up to one full second. This is complete unacceptable for downloading a file. Even when downloading a file off of the internet you should see times at no more than 0.1 seconds. An ideal RTT for data transfer across the internet is no more that 0.04 seconds (40 milliseconds). Looking at the beginning of the graph as compared to the high speed download at the end of it can produce some draw jopping results.

Homework

I would encourage you at this point to try taking some packet captures of downloading various files on the internet. See if you can find a high utilization server somewhere so you can view the packets of a slow download you did yourself. This will help you to better understand what we learned today. In the next installment of Packet School 101 we are going to examine a trace file from a network in which there are router performance problems.

Packet School 101 – Part 3

 

 

Several distance learning programs allow people to have professional degrees and IT certifications, without having their tangible presence in scheduled classes. For exam training purposes MS 642-552 study guide available that features Q&A with explanation. Microsoft designed MS 70-536 certification while keeping in view the emerging trends in IT. Today, acquiring new MS 70-553 IT certifications would be a valuable increase in your certifications. 

Packet School 101 – Part 1

** Disclaimer to all new readers – This blog post is VERY old and not really representative of my current work. I’ve just left it up here for historical purposes. If you are interested in learning more about packet analysis I’d reccommend reading some of my newer posts or looking at my book, Practical Packet Analysis. **

Over the course of the next few weeks I am going to be putting out a series on network traffic analysis. This is one of the most important skills any network administrator can have and is absolutely crucial to solving a variety of network related problems.

We won’t get into any actual packet analysis in the introductory article We will, however, go ahead and make sure we have the appropriate software to proceed, along with a brief understanding of how it works and how to use it.

Getting Equipped

The software we will be using for this series is the ever popular Ethereal network sniffing application. This program is distributed freely and can be downloaded at http://www.ethereal.com. Installing this software is simple enough that I am not going to go through it here, although it is important to note that this software relies on the WinPcap driver which is included in the installation package. As a lot of you may already know, there have been some issues with WinPcap and version skew, as in different versions of the driver being required for different pieces of software to work with it. This being the case, I reccomend you to install the version of WinPcap that comes with Ethereal even if you already have a version of this driver installed. This will help to prevent any possible issues in the future.

There is no actual hardware required for doing this type of analysis, however, it makes things a LOT easier if you have an old 10/100 hub lying around. I will explain why this is here very shortly.

Ethereal Basics

Once you have installed Ethereal along with the WinPcap driver you should be ready to dive in head first. The first thing we are going to do is a simple packet capture. To do this you will first need to select your capture interface by clicking the “List avaliable capture interfaces” button to the far left hand side of the main toolbar.

Once you have done this you will be presented with a window listing all avaliable capture interfaces. Clicking the “Capture” button next to the interface you wish to use will begin capturing packets from this connection.

Wait for a few minutes until a significant amount of packets have been collected and then click the “Stop” button. You should the be returned back to the main screen with a whole bunch of new data. Congratulations! You have just completed your first successful packet capture!

Slipping On To the Wire

Now that you know how to do a basic packet capture in Ethereal it is important to learn how to capture the right traffic. Assuming you are on a switched ethernet network (which most everybody is these days), all of the traffic you just captured was your own. That is, all traffic was either coming to or going from the computer from which you intiated the packet capture. This is basically how a switched network functions. The switch only sends data to those ports in which it is destined. This brings up the question of how do you capture traffic from a computer that packet sniffing software is not installed on?

Remember earlier when I told you that it would come in handy to have an old 10/100 hub lying around? Well this is that time. I am not going to go into great detail on the differences between a switched and a hub network, but put simply, in a hub network every piece of data is sent to every single port. The hub uses a lower level of logic in that it does not know where data is supposed to go, so it sends it to every client on every port and lets those clients accept it if they want it and deny it if they don’t. This means on a hub network when you sniff packets you are seeing the packets of the entire network. So lets say you are trying to troubleshoot a computer on a switched network. What you will want to do is go to the client computer and unplug it from the network, and plug it into your hub, along with the computer you will be using to sniff the traffic. From here you will plug the hub onto the network using its uplink port. This will effectively put the two comptuers you will be using onto a hub network allowing you to read all the traffic you need to. See the diagram below for a visual explanation of the hub setup.

Homework

In the next section of this series we will look at some basic packet structure as well as some more features of Ethereal. We will also look into some basic troubleshooting using packet analysis. Between now and then I would highly reccomend playing around with Ethereal and looking at your own networks packets to see if you can try and figure out what some of them are.

Packet School 101 – Part 2

7/5/06 – If you like this series, digg it!

**UPDATE**

Enjoy Packets School 101? Check out Packet School 201!

As a lot of you may already know, Microsoft introduced several new online courses and issued certificates after successful completion of the programs. You can find various courses like 70-293, that enable you to solve concurrent problems and also 70-294, that will increase your proficiency. Online guides and CDs for Ms 70-296 certifications are also available for students’ assistance.