Sometimes a network service is just not behaving the way it should. And the log files do not help you either. Then it is time to use the power of tcpdump and Wireshark to get a deeper look on what is actually happening on the wire.
If you have an X11 running on the host in question you may just start Wireshark and start recording the traffic. However often you need to record traffic that is running on a machine you can just login through SSH. Don’t panic – you can still analyze the traffic.
Sniff the network
Run tcpdump on the server in question. The number of options is pretty large. But most of the time the call is rather simple. Try something like this:
tcpdump -lnni eth0 -w dump -s 65535 host web01 and port 80
That will mainly record traffic on the interface eth0, write the output (in raw format) to the file named dump, record the whole packet (65535 bytes maximum) instead of just a few bytes and use the filter expression host web01 and port 80 to just listen to traffic for the server called web01 and listen only to traffic on the HTTP port (80).
Run this command and let it run while you want to record network activity. When you are done, just stop the process (Ctrl-C) and you have the raw data in the dump file.
Analyze the dumpfile in Wireshark
Now copy this dump file over to a workstation where you have X11 running – scp should do it. Then start wireshark and load the file (or just run wireshark dumpfile). You will be shown what has happened when.
In the top window you will see one line for each packet. Click on a packet that you like to inspect deeper and you will be shown the protocol stack in the lower left corner. The layers correspond to the OSI model and are used to transport the packet. Unless you are debugging your switch environment you can safely ignore all the lower layers (that are shown as the first lines) and expand (click on the ‘+’) the HTTP protocol. Wireshark is very smart and will try to interpret the network traffic so you get a clearer view at the protocol. It knows about most common protocols like HTTP, SMTP, POP3 and a few hundreds more.
Wireshark on the console
If you just need need a tad more information than what tcpdump delivers but want to see it on the console and in real-time then tshark may be for you. Give it a try.