No one can work, everyone is angry…
What do you do? First, it doesn’t hurt to have an internal panic, then get upset at your users for being nasty and your boss for not understanding. But then what? Okay:
- Check your switches usage, are they performing?
- Have a look at your SNMP monitor, which servers are spitting out alarms?
- The Internet connection appears to be slow, better check the logs from the ISP.
Then things get worse…
Now you are getting calls from remote sites as they start to come online and notice that the system is extremely slow. Whose fault is it? It appears to be yours.
After doing your normal checks, what information do you have?
- Switches appear to have normal load, nothing seems out of the ordinary.
- SMTP monitor shows normal activity from your servers and there are no alarm bells ringing.
- Internet usage from ISP shows large utilisation of your link, it is being saturated.
What do you do now?
Get a raw packet capture?
Pull out Wireshark or tcpdump and grab a packet capture so that you can look at reams of packet capture text?
Sounds like a nightmare. But does it sound like something you have lived through? I certainly have, and I don’t enjoy recalling the stomach wrenching stress and insecurity of feeling out of my depth.
Better yet, get a permanent tool to do the job for you
A real world example
I personally love it when a client is having issues of this type, and not because I am horrible, but because I like solving problems with NetMuster. Here is how I recently solved an issue of this type a client site.
Step 1 – Plug in NetScope
Plug in NetScope to a mirror port on your managed switch. Then have a look at the traffic.
The client’s site had a 200 Mb link in both the inbound and outbound directions. We can see from the diagram that activity in the outbound direction has the link mostly flooded.
Now let’s take a look at what this Internet traffic is made up of…
Track down and isolate
Now that we have found the sources of the internal traffic that is flooding the system we can isolate them and shut them down for further investigation.
Without this level of visibility we would have to literally trawl through thousands of lines of TCP/UDP logs, and it becomes very easy to miss malicious data as identified above.
Within minutes of opening NetScope the problem is identified
- Check Internet usage.
- Drill down on unusual application traffic.
- Identify source.
- Disable source for further investigation.
If you would like to track down malicious activity on your network shoot on over to the download section of NetScope.com and grab yourself a copy.
Article written by Christopher B. Horan