Network Forensics: The Tree in the Forest

Security professionals are often tasked with the unenviable position of wading through millions of bits of data, the review of thousands of systems, or the evaluation of hundreds of applications.  At the end of the day it is their job to provide the ten thousand foot view of an organization and the highest rated findings that put it at risk.  Information overload is a common theme in today’s society, and management requires the presentation of this material in a digestible manner of typically one page or less.  The ability to provide this service requires what is often referred to as “seeing the forest for the trees.”  In other words, don’t get distracted or bogged down by the minutiae of your discoveries at the risk of overlooking the big picture.

When it comes to computer forensics, however, the tables are flipped.  When an event turns into an incident and management must answer to a board or the company’s shareholders, the ten thousand foot level is no longer adequate.  At this point, every packet that ever crossed your company’s domain becomes suspect, and expectations are set whereby the answers to the questions such as, how did it happen, what damage did it do, where did it come from, when exactly did it occur, and who did it, requires the puzzle to be unraveled and presented in such excruciating detail it would make Melville  take up skim-reading.

Network Forensics:  What are we talking about?

Computer forensics involves the preservation, identification, extraction, documentation, and interpretation of computer data and has its roots in data recovery.  One of my peers recently wrote an article providing a good introductory explanation of computer forensics in his review of a SANS course.   This article dealt primarily with what we term system or file system forensics.  In this discipline, the investigative methodology is employed against a specific file system or operating system.  Analysis is used to identify one of three things:

1. Inculpatory Evidence – That which supports a given theory
2. Exculpatory Evidence – That which contradicts a given theory
3. Evidence of Tampering – That which cannot be related to any theory, but shows that someone tampered with a given system to avoid identification.

The goal of network forensics is the same as above, but it is rooted in network security and intrusion detection. Typically these investigations deal with a much broader range of systems, i.e. an enterprise.  Network forensics grew out of the investigation of cyber attacks, often stemming from malware investigations.  Because this type of investigation deals with the transfer and change of data in real-time, the challenge then becomes how to contain and eradicate the intrusion while still preserving evidence for later use. 

When looking at the descriptions above, you will quickly realize one of the major problems I’ve had with computer forensics in general in the past… it’s reactionary in nature.  The stress during any investigation is extremely high, but when you couple that with the fact that you are already a step behind before you even start, it can often become overwhelming.  The other major problem with computer forensics is the actual trigger that kicks off that reactionary response.  Whether you call it identification or detection, there has to be something in place that triggers the investigation in the first place.  Over the years I’ve seen event notification only get better and better with the implementation of firewalls, intrusion detection/prevention systems, data leak protection, end-point management, and outsourced security monitoring services.  These implementations have provided organizations with the triggers they’ve needed to respond accordingly to incidents, or at least start them down the path toward documenting or outlining an incident response plan.  Unfortunately, if you ask any CIO or CISO what they’re greatest fears are with respect to their information assets, 9 in 10 will likely respond with, “I’m worried about the incidents I don’t even know about, the ones that haven’t been identified by all of the security mechanisms I have in place.”

Moving from Reactive to Proactive

One of the standard recommendations I’ve made to clients over the years has been to implement egress filtering.  It is an easy statement to make, but I’ll be the first to admit that I never really followed up on what it entailed or the level of effort that went into bringing it to fruition.  Forced to think about this a little more, I still believe the recommendation is good, and, for the basis of this article, I think it is also the first step in becoming proactive to threats that exist within your environment.  What this step does is put you in the right mindset for embracing network forensics.  While all of the security mechanisms I mentioned above are great at logging and alerting, unless they can trigger on our biggest concern, they are really only relevant for auditing purposes.  So what is the biggest concern?  The answer is data extraction.  Don’t get me wrong, I don’t want an intrusion to happen in the first place, but going back to the fears of those C-level managers mentioned above, data extraction is top of the list.

With the advances in toolsets and processes deployed in network forensics, we finally have the ability to move from that reactive nature of waiting for security alerts and notifications to a proactive search for anomalies across our environment.  These processes address not only the C-level management fears above, but provide mechanisms for identifying data extraction.  Let’s look at an example  and determine some common things to search for when beginning this process.  For the following example I’m going to use Wireshark as my tool of choice.  Again, this is a good way to get started with network forensics.  Your goal should be able to sniff and analyze packets on the fly, but the best approach is to begin with packet capture files or PCAPs.  Keep in mind that this article isn’t about how to use Wireshark.  If you want additional information you will have to perform additional research on your own.

First we open a PCAP file in Wireshark:

A good place to start looking at your network is DNS.  There are a couple of reasons for this, but typically it deals with the patterns seen when dealing with intrusions, malware, command and control situations, and simple business practice and locations.  For instance, is there any reason a system on your network should be querying China?  For this particular case you can filter for DNS within Wireshark very easily:

As you see from this capture the DNS query is for Singapore.  Right away the hair on the back of my neck starts to rise, but I need to keep looking just to make sure I’m not crying wolf.  It looks like webmail traffic which uses the HTTP protocol, so let’s take a look at that:

When performing forensics just remember that coincidences are unlikely.  Here we have an HTTP “GET” for the same host that showed up in our DNS query, but the real kicker is the User-Agent identified within the header.  I don’t know about you, but I’ve never seen a “Windows+NT+5.1” user-agent.  If this was coming into my network I might not be all that concerned as many attackers attempt to obfuscate their User-Agent in order to see what type of response they’ll get.  But, this is an outbound connection from a system on my network performing these odd requests.  To dig a little further, let’s follow that TCP stream within Wireshark. 

This may not be all that apparent, but you should be able to see that this looks like some sort of script based on the “./”.  There is very little additional information you can glean from this stream even if you were to scroll down. In instances like this it might take more analysis and research on your part.  However, with that additional research you would be able to determine the response uses a “==” at the end of the string which signifies a Base64 encoding is taking place.  Third time is a charm; we now have proof of obfuscation during what would appear to be a normal HTTP session. 

The great thing about this exercise is that by spending a limited amount of time capturing normal traffic on our network, we were able to perform a proactive search for threats against our organization and identify those threats C-level management is concerned about most.  It is very unlikely that you would have received an alert about this particular event, because it looks like normal outbound HTTP traffic.  How many of us are configuring our firewalls and intrusion detection systems to block or alert on something like this?  Of course, Wireshark isn’t the only tool in our arsenal, and there are many others that help you limit the amount of time you spend on exercises like this.  In addition, there are commercial tools out there that provide an extremely powerful, but easy to use graphical interface that can make this job even easier.  The following screenshot shows one of those commercial tools, RSA NetWitness Investigator (there is also a freeware version of Investigator), in action as it filters down and shows a “blackhole exploitation” incident.  Make note of the File Hashes that have been generated in the event this investigation makes it way to court. 

Where to Begin

The following is a brief list of common places in which to begin your own research or investigation:

• The first place to check when performing proactive steps is your local host file.  This is the first place your system will query for a name server to connect to the Internet.
• Review “Users<usernameAppData” within Windows 7.  This is a common container for malware.
• Review sub-domains for things that just don’t appear to be legitimate.
• Pay special attention to DNS.  Determine whether the associated IP address stays consistent
• When performing filters export information for further triage, i.e. within Wireshark you can use File -> Export Objects -> HTTP.  This will  allow you to see what was downloaded and exported.
• Become familiar with common filtering options.  Example: “tcp contains “GET””
• Separate your investigation into manageable pieces.  For example, when you have separate systems or computers, you want to look at each computer separately. Example: ip.addr==172.16.255.2
• Save your filters to make the job easier.  Within Wireshark go to File -> Export Specified Packets.
• Understand common issues, e.g. whenever you see repeated POSTs, you likely have a problem. Example: tcp contains “POST”

Conclusion

This article is by no means an exhaustive “How To” Guide on network forensics and really only scratches the surface.  However, I hope that security professionals and network administrators alike see the benefit of adding just a small amount of time to their day to begin to become more proactive toward eliminating threats within their environment.  At the beginning of this article I alluded to the adage of “seeing the forest for the trees.”  However, I hope what I’ve just shown through the employment of a proper scientific methodology and the right mindset, you can utilize network forensics in a proactive manner that allows you to “find the tree within the forest.”