Chapter 61 · Part 6 of 7
Monitoring with netstat and tcpdump
Observing, debugging, and tracing socket activity in Linux network applications
Why These Tools Matter
Writing network code without knowing netstat and tcpdump is like debugging C without gdb. These tools let you see exactly what your program is doing at the socket and packet level — without changing a single line of code.
netstat shows you the current state of all sockets on the system. tcpdump captures and displays the actual packets being exchanged. Together they answer: “Is my server listening?”, “Why is that connection stuck in CLOSE_WAIT?”, “Is the client actually sending data?”
netstat — Socket State Inspector
Note: On modern Linux, netstat is part of the net-tools package and is gradually being replaced by ss (from iproute2). Both commands show the same information. We cover both below.
Install if needed:
sudo apt install net-tools (for netstat)ss is built into modern Linux — no install neededKey netstat Options
Option
Meaning
-a
Show all sockets (including LISTEN state)
-n
Show numeric addresses and ports (no DNS/service lookup — faster)
-t
Show TCP sockets only
-u
Show UDP sockets only
-p
Show the PID and program name owning each socket (needs root)
-l
Show only listening sockets
-c
Continuously refresh output (like watch)
netstat in Practice
See all listening TCP servers:
netstat -tlnp
# or with ss:
ss -tlnp
# Example output:
# Proto Recv-Q Send-Q Local Address:Port Peer Address:Port State
# tcp 0 0 0.0.0.0:8080 0.0.0.0:* LISTEN
# tcp 0 0 0.0.0.0:22 0.0.0.0:* LISTEN
# tcp6 0 0 :::80 :::* LISTEN
See all established TCP connections:
netstat -tnp | grep ESTABLISHED
# Example output:
# Proto Recv-Q Send-Q Local Address Foreign Address State
# tcp 0 0 192.168.1.10:52340 93.184.216.34:80 ESTABLISHED
# tcp 0 0 192.168.1.10:8080 192.168.1.20:47891 ESTABLISHED
Look for TIME_WAIT sockets (indicates recent server restarts or high connection turnover):
netstat -an | grep TIME_WAIT | wc -l
# If this number is very high (thousands), you have a connection recycling problem
# To see them in detail:
netstat -antp | grep TIME_WAIT
Look for CLOSE_WAIT accumulation (application bug — not calling close on EOF):
netstat -antp | grep CLOSE_WAIT
# Any CLOSE_WAIT sockets lasting more than a few seconds = bug in your code
Monitor a specific port in real time:
watch -n 1 'netstat -tnp | grep :8080'
# Refreshes every second, shows all connections to/from port 8080
Count connections per state (useful for load analysis):
netstat -an | awk '/^tcp/ {print $6}' | sort | uniq -c | sort -rn
# Example output:
# 142 ESTABLISHED
# 23 TIME_WAIT
# 8 CLOSE_WAIT
# 2 LISTEN
# 1 SYN_SENT
Using ss (Modern Replacement for netstat)
ss is faster and more powerful than netstat. It reads socket info directly from the kernel via netlink, rather than parsing /proc/net/tcp.
# List all listening TCP sockets with process info:
ss -tlnp
# List all TCP connections to port 8080:
ss -tnp dst :8080
# Show sockets in TIME_WAIT:
ss -o state time-wait
# Show sockets in ESTABLISHED to a specific host:
ss -tnp dst 192.168.1.20
# Extended info (send/recv buffer sizes):
ss -tne
# Show all socket states summary:
ss -s
# Output example:
# Total: 487
# TCP: 142 (estab 98, closed 15, orphaned 3, timewait 23)
tcpdump — Packet Level Capture
While netstat shows socket states, tcpdump shows you the actual bytes flying across the network. It captures packets on a network interface and displays them in human-readable form.
tcpdump requires root (or CAP_NET_RAW capability).
# Basic syntax:
tcpdump [options] [filter-expression]
Key options:
-i eth0 — capture on interface eth0 (-i any for all interfaces)-n — don’t resolve hostnames (faster, clearer)-v / -vv — more verbose (TTL, checksums, etc.)-X — show packet data in hex and ASCII-A — show packet data in ASCII only (great for HTTP)-w file.pcap — write raw packets to file for later analysis-r file.pcap — read from a saved pcap file-c N — capture only N packets then stoptcpdump Filter Expressions
Filter expressions are the most powerful feature of tcpdump. They follow BPF (Berkeley Packet Filter) syntax.
# Capture all traffic on port 8080:
tcpdump -i lo -n port 8080
# Capture TCP traffic only:
tcpdump -i eth0 tcp
# Capture traffic to/from a specific host:
tcpdump -i eth0 host 192.168.1.20
# Capture traffic to a specific host on a specific port:
tcpdump -i eth0 host 192.168.1.20 and port 80
# Capture all SYN packets (new connection attempts):
tcpdump -i eth0 'tcp[tcpflags] & tcp-syn != 0'
# Capture SYN+ACK packets (server responding to connects):
tcpdump -i eth0 'tcp[tcpflags] == (tcp-syn|tcp-ack)'
# Capture RST packets (abrupt connection resets):
tcpdump -i eth0 'tcp[tcpflags] & tcp-rst != 0'
# Capture FIN packets (graceful closes):
tcpdump -i eth0 'tcp[tcpflags] & tcp-fin != 0'
# Exclude SSH traffic (don't clutter output when connected via SSH):
tcpdump -i eth0 not port 22
# Capture HTTP traffic and show ASCII content:
tcpdump -i eth0 -A port 80
# Capture and save to file for Wireshark analysis:
tcpdump -i eth0 port 8080 -w capture.pcap
# Read and analyse saved capture:
tcpdump -r capture.pcap -n
Practical Debugging Session: Tracing a TCP Connection
Here is a typical debugging session. We run our TCP echo server on port 9000 and use tcpdump to watch what happens.
Terminal 1 — Start tcpdump on loopback:
sudo tcpdump -i lo -n -v port 9000
Terminal 2 — Start a simple echo server:
/* echo_server.c */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/socket.h>
#include <netinet/in.h>
int main(void)
{
int listenfd = socket(AF_INET, SOCK_STREAM, 0);
int opt = 1;
setsockopt(listenfd, SOL_SOCKET, SO_REUSEADDR, &opt, sizeof(opt));
struct sockaddr_in addr = {
.sin_family = AF_INET,
.sin_addr.s_addr = INADDR_ANY,
.sin_port = htons(9000)
};
bind(listenfd, (struct sockaddr *)&addr, sizeof(addr));
listen(listenfd, 5);
printf("Echo server on port 9000\n");
int connfd = accept(listenfd, NULL, NULL);
char buf[256];
ssize_t n;
while ((n = read(connfd, buf, sizeof(buf))) > 0)
write(connfd, buf, n);
close(connfd);
close(listenfd);
return 0;
}
Terminal 3 — Connect and send data:
echo "Hello TCP" | nc 127.0.0.1 9000
tcpdump output you’ll see (annotated):
# 3-way handshake:
lo: IP 127.0.0.1.49210 > 127.0.0.1.9000: Flags [S] # SYN from client
lo: IP 127.0.0.1.9000 > 127.0.0.1.49210: Flags [S.] # SYN+ACK from server
lo: IP 127.0.0.1.49210 > 127.0.0.1.9000: Flags [.] # ACK from client
# Data transfer:
lo: IP 127.0.0.1.49210 > 127.0.0.1.9000: Flags [P.] ... "Hello TCP\n" # push
lo: IP 127.0.0.1.9000 > 127.0.0.1.49210: Flags [.] # ACK
lo: IP 127.0.0.1.9000 > 127.0.0.1.49210: Flags [P.] ... "Hello TCP\n" # echo back
# 4-way close:
lo: IP 127.0.0.1.49210 > 127.0.0.1.9000: Flags [F.] # FIN from client
lo: IP 127.0.0.1.9000 > 127.0.0.1.49210: Flags [F.] # FIN from server
lo: IP 127.0.0.1.49210 > 127.0.0.1.9000: Flags [.] # ACK
# Flags legend:
# [S] = SYN
# [S.] = SYN+ACK
# [.] = ACK (pure acknowledgment)
# [P.] = PSH+ACK (data push)
# [F.] = FIN+ACK
# [R] = RST (connection reset)
Interview Questions & Answers
Q1. How do you check which process is listening on a specific port?
Use
netstat -tlnp | grep :PORT or ss -tlnp | grep :PORT. The -p flag shows the PID and process name. Needs root (or sudo) to see processes owned by other users.Q2. How do you detect a CLOSE_WAIT accumulation using netstat?
Run
netstat -an | grep CLOSE_WAIT | wc -l. If this count grows continuously while your application runs, you have a bug where EOF is detected on a socket but close() is never called. The application is leaking sockets.Q3. What does tcpdump’s [R] flag mean and when do you see it?
[R] means RST (reset). It’s seen when: (1) a client tries to connect to a port with no listener (server sends RST), (2) one side calls close() abruptly while data is pending, (3) the OS receives a segment for a connection it doesn’t know about (e.g., after a crash). RST terminates the connection immediately without the normal 4-way close.
Q4. What is the difference between tcpdump -i lo and -i eth0?
-i lo captures loopback traffic (127.0.0.1) — traffic between processes on the same machine. -i eth0 captures traffic on the physical/virtual network interface. For debugging local client-server pairs, use -i lo. For debugging network traffic to remote hosts, use -i eth0 (or the appropriate interface).
Q5. What does Recv-Q and Send-Q mean in netstat output?
Recv-Q is the number of bytes in the kernel receive buffer waiting to be read by the application. Send-Q is the number of bytes in the kernel send buffer waiting to be transmitted (or acknowledged). If Recv-Q is large and growing, your application is too slow consuming data. If Send-Q is large, the network or peer is slow consuming your sent data.
Q6. How do you save a tcpdump capture for later analysis in Wireshark?
Use
tcpdump -i eth0 -w capture.pcap port 8080. This saves raw packets in pcap format. The file can then be opened in Wireshark for visual analysis, or read back with tcpdump -r capture.pcap. This is useful when you can’t run Wireshark directly on the target system (e.g., an embedded Linux board).