Monitoring Sockets: netstat Sockets: Advanced Topics

 

Monitoring Sockets: netstat
Chapter 61 · Sockets: Advanced Topics · Part 4 of 5
6
Output Fields
/proc
Data Source
TCP+UDP
Protocols

What is netstat?

The netstat program lets you see the current state of all sockets on your system — both Internet domain sockets (TCP, UDP) and UNIX domain sockets. It is one of the first tools you reach for when debugging network applications. Is my server listening? Are connections stuck in CLOSE_WAIT? Is the port in use? netstat answers all these questions.

On modern Linux systems, netstat is being replaced by the ss command (from the iproute2 package), which is faster. But netstat is available on virtually all UNIX systems and the output format is widely understood.

Key Terms to Know

netstat Proto Recv-Q Send-Q Local Address Foreign Address State LISTEN ESTABLISHED /proc/net ss command Wildcard address

Common netstat Command-Line Options
Option What It Does When to Use
-a Show ALL sockets including listening ones Default only shows connected sockets; use -a to see LISTEN
-n Show numeric IP addresses and port numbers (no DNS lookup) Faster output; avoids slow reverse DNS lookups
-t or --tcp Show only TCP sockets When you only care about TCP connections
-u or --udp Show only UDP sockets Debugging UDP applications
-l Show only listening sockets Quickly check which ports are open on the machine
-p Show PID and process name owning each socket “Which process is using port 8080?” (requires root for all sockets)
-c Continuously refresh display (every second) Watching TCP state changes in real time during testing
-e Extended info including socket owner’s user ID Security audit: which user owns which socket
--unix Show UNIX domain sockets only Debugging IPC between local processes

Most common combinations in real work:

# Most useful: numeric, all TCP (avoids slow DNS)
netstat -tn -a

# Who is listening on what port?
netstat -tlnp

# Watch connections change in real-time
netstat -tnc | grep :9090

# Equivalent modern commands using 'ss'
ss -tn -a         # same as netstat -tn -a
ss -tlnp          # same as netstat -tlnp

Reading netstat Output — Field by Field

Here is sample output from netstat -a --inet with each field explained:

Active Internet connections (servers and established)
Proto  Recv-Q  Send-Q  Local Address          Foreign Address        State
tcp    0       0       *:50000                *:*                    LISTEN
tcp    0       0       *:55000                *:*                    LISTEN
tcp    0       0       localhost:smtp         *:*                    LISTEN
tcp    0       0       localhost:32776        localhost:58000        TIME_WAIT
tcp    34767   0       localhost:55000        localhost:32773        ESTABLISHED
tcp    0       115680  localhost:32773        localhost:55000        ESTABLISHED
udp    0       0       localhost:61000        localhost:60000        ESTABLISHED
udp    684     0       *:60000                *:*

Field Breakdown
Proto

Protocol: tcp or udp. For IPv6 it will be tcp6 or udp6.
Recv-Q

Receive Queue: Number of bytes received by the kernel but not yet read by the application. If this is large and growing, the application is not reading fast enough (slow consumer). For UDP, includes header bytes.
Example: 34767 bytes waiting — the server app has 34KB backlog to read.
Send-Q

Send Queue: Number of bytes written by the application but not yet acknowledged by the peer. If this is large, the network is slow or the peer is not reading. For UDP, includes header bytes.
Example: 115680 bytes — this connection has 115KB of unacknowledged data.
Local Address

Local socket address: Shown as host:port. An asterisk (*) means the wildcard IP address (the socket is listening on all interfaces). Names are used by default unless -n flag is passed. Example: *:50000 means listening on port 50000 on all network interfaces.
Foreign Address

Peer socket address: The IP and port of the remote end. *:* means no peer (socket is in LISTEN state or unconnected UDP). For a connected socket, this shows the actual peer address.
State

TCP connection state: One of the states from the TCP state machine: LISTEN, SYN_SENT, SYN_RECV, ESTABLISHED, FIN_WAIT1, FIN_WAIT2, CLOSE_WAIT, LAST_ACK, TIME_WAIT, CLOSED. UDP sockets show ESTABLISHED even though UDP is connectionless — this just means connect() was called.

Interpreting the Sample Output Line by Line
Line from Output What It Means
tcp 0 0 *:50000 *:* LISTEN A TCP server is listening on port 50000, waiting for connections on all interfaces. No clients yet.
tcp 0 0 localhost:smtp *:* LISTEN A mail server is listening on the SMTP port (25) but only on the loopback interface — not accepting external mail connections.
tcp 0 0 localhost:32776 localhost:58000 TIME_WAIT A recently closed connection is in TIME_WAIT. Port 32776 was the active closer; port 58000 was the peer. This will expire in ~60 seconds.
tcp 34767 0 localhost:55000 localhost:32773 ESTABLISHED An established TCP connection. 34767 bytes received but not yet read by the app on port 55000 — the application is running behind. Send queue is 0 (all sent data acknowledged).
tcp 0 115680 localhost:32773 localhost:55000 ESTABLISHED The other end of the previous connection. This side has 115680 bytes in its send queue waiting for acknowledgement — heavy traffic or a slow peer.
udp 684 0 *:60000 *:* A UDP socket bound to port 60000 on all interfaces, with 684 bytes in its receive buffer waiting to be read. UDP has no State column (no connections).

Reading Socket Info Programmatically via /proc/net

The kernel exposes the same information as netstat through files in /proc/net/. These are useful when you need to read socket state from within a C program or shell script.

File Contents
/proc/net/tcp All IPv4 TCP sockets — state, queues, addresses, inode
/proc/net/udp All IPv4 UDP sockets
/proc/net/tcp6 All IPv6 TCP sockets
/proc/net/udp6 All IPv6 UDP sockets
/proc/net/unix All UNIX domain sockets

Reading /proc/net/tcp from a shell script:

# View raw TCP socket table (hex addresses and ports)
cat /proc/net/tcp

# Count connections in each TCP state
# Column 4 in /proc/net/tcp is the state (hex: 01=ESTABLISHED, 0A=LISTEN)
awk 'NR>1 {print $4}' /proc/net/tcp | sort | uniq -c | sort -rn

# State codes in /proc/net/tcp (hex):
# 01 = ESTABLISHED
# 02 = SYN_SENT
# 03 = SYN_RECV
# 04 = FIN_WAIT1
# 05 = FIN_WAIT2
# 06 = TIME_WAIT
# 07 = CLOSE
# 08 = CLOSE_WAIT
# 09 = LAST_ACK
# 0A = LISTEN

Reading /proc/net/tcp from a C program:

#include <stdio.h>
#include <stdlib.h>
#include <string.h>

/* TCP state codes from the kernel */
static const char *tcp_state_name(int state) {
    static const char *states[] = {
        "UNKNOWN",    /* 0 */
        "ESTABLISHED",/* 1 */
        "SYN_SENT",   /* 2 */
        "SYN_RECV",   /* 3 */
        "FIN_WAIT1",  /* 4 */
        "FIN_WAIT2",  /* 5 */
        "TIME_WAIT",  /* 6 */
        "CLOSE",      /* 7 */
        "CLOSE_WAIT", /* 8 */
        "LAST_ACK",   /* 9 */
        "LISTEN",     /* A = 10 */
    };
    if (state < 0 || state > 10) return "UNKNOWN";
    return states[state];
}

int main() {
    FILE *fp = fopen("/proc/net/tcp", "r");
    if (!fp) {
        perror("Cannot open /proc/net/tcp");
        return 1;
    }

    char line[512];
    int line_num = 0;

    printf("%-8s %-22s %-22s %-14s %-10s %-10s\n",
           "Slot", "Local Address", "Remote Address",
           "State", "Recv-Q", "Send-Q");
    printf("%s\n", "------------------------------------------------------------"
                   "------------------------------");

    while (fgets(line, sizeof(line), fp)) {
        if (line_num++ == 0) continue; /* Skip header */

        int slot, state;
        unsigned long rx_queue, tx_queue;
        unsigned int local_addr, local_port;
        unsigned int rem_addr, rem_port;

        /*
         * /proc/net/tcp format:
         * slot: local_addr:port rem_addr:port state tx_queue:rx_queue ...
         */
        sscanf(line,
               "%d: %X:%X %X:%X %X %lX:%lX",
               &slot,
               &local_addr, &local_port,
               &rem_addr, &rem_port,
               &state,
               &tx_queue, &rx_queue);

        /* Convert hex IP to dotted decimal */
        unsigned char *la = (unsigned char *)&local_addr;
        unsigned char *ra = (unsigned char *)&rem_addr;

        char local_str[24], rem_str[24];
        snprintf(local_str, sizeof(local_str),
                 "%d.%d.%d.%d:%d", la[0], la[1], la[2], la[3], local_port);
        snprintf(rem_str, sizeof(rem_str),
                 "%d.%d.%d.%d:%d", ra[0], ra[1], ra[2], ra[3], rem_port);

        printf("%-8d %-22s %-22s %-14s %-10lu %-10lu\n",
               slot, local_str, rem_str,
               tcp_state_name(state), rx_queue, tx_queue);
    }

    fclose(fp);
    return 0;
}

Compile and run:

gcc proc_net_tcp.c -o proc_net_tcp
./proc_net_tcp

Practical Debugging Scenarios with netstat
Scenario 1: Is my server listening?

netstat -tlnp | grep :9090
# Look for: tcp  0  0  0.0.0.0:9090  0.0.0.0:*  LISTEN  PID/process_name

If no output, your server hasn’t called listen() yet, or it crashed, or it’s on a different port.

Scenario 2: Why are there many CLOSE_WAIT connections?

netstat -tn | grep CLOSE_WAIT | wc -l
# If many → your server app is NOT calling close() after clients disconnect
# This is a file descriptor leak — a bug in your application

Fix: ensure your server calls close(client_fd) after each connection ends.

Scenario 3: Why does bind() fail with EADDRINUSE?

netstat -tn | grep TIME_WAIT | grep :9090
# If you see TIME_WAIT on your port, the previous connection is cooling down
# Fix: add SO_REUSEADDR to your server socket before bind()
Scenario 4: Is data stuck in the send buffer?

netstat -tn | awk '{if($3 > 0) print}'
# Look at Send-Q (column 3) — large values mean peer is slow or unresponsive
# Recv-Q (column 2) being large means your app is slow to read

Interview Questions & Answers

Q1. What does a large Recv-Q value in netstat indicate?

It means the kernel has received data from the network and buffered it, but the application has not yet read it via read() or recv(). A consistently large Recv-Q suggests the application is too slow to process incoming data — a common sign of a consumer bottleneck. For UDP, Recv-Q also includes header bytes.

Q2. What does the wildcard address (*) mean in the Local Address column?

An asterisk in the host part of the Local Address (e.g., *:9090) means the socket is bound to the wildcard IP address — it listens on all available network interfaces. A socket can also be bound to a specific interface address (e.g., 127.0.0.1:9090) to accept connections only on the loopback interface.

Q3. How do you find which process is using a specific port?

Use netstat -tlnp | grep :PORT with root privileges. The -p flag adds the PID and program name column. Alternatively, use ss -tlnp | grep :PORT or lsof -i :PORT. Without root, you can only see sockets owned by your own user.

Q4. Where does netstat get its information from on Linux?

netstat reads from the /proc/net/ virtual filesystem. The relevant files are /proc/net/tcp for IPv4 TCP, /proc/net/udp for IPv4 UDP, /proc/net/tcp6 for IPv6 TCP, /proc/net/udp6 for IPv6 UDP, and /proc/net/unix for UNIX domain sockets. These are kernel-maintained files that can also be read directly by programs.

Q5. A UDP socket shows state ESTABLISHED in netstat. Does that mean UDP is connection-oriented?

No. UDP is still connectionless. The ESTABLISHED state for UDP in netstat simply means that connect() was called on the UDP socket. This does not create a real connection — it only sets the default remote address for send() calls and restricts recv() to datagrams from that address. UDP has no handshake, no sequence numbers, and no state machine.

Q6. How is netstat different from ss?

ss (socket statistics) is the modern replacement for netstat from the iproute2 package. It is faster because it reads from the kernel’s netlink interface rather than /proc files. The command options are similar: ss -tn -a is equivalent to netstat -tn -a. On systems where netstat is not installed, ss is available. Both tools ultimately show the same information.

Continue Learning

Next: Using tcpdump to capture and analyze actual TCP packets

← Part 3: TIME_WAIT Part 5: tcpdump →

Leave a Reply

Your email address will not be published. Required fields are marked *