TCP States, TIME_WAIT, and SO_REUSEADDR

 

TCP States, TIME_WAIT, and SO_REUSEADDR
Chapter 61 · Part 6 of 7 — Understanding TCP’s state machine and the “Address already in use” problem

TCP Is a State Machine

TCP does not just “connect” and “disconnect” — it goes through a precisely defined sequence of states. Understanding these states helps you debug real-world network issues, understand why a server can’t restart immediately after being killed, and write code that cooperates correctly with the TCP stack.

TCP State Diagram

All TCP States

Connection Establishment (3-Way Handshake)
CLIENT
CLOSED
send SYN ↓
SYN_SENT
recv SYN+ACK ↓
ESTABLISHED
→ SYN →
← SYN+ACK ←
→ ACK →
SERVER
CLOSED
bind+listen ↓
LISTEN
recv SYN ↓
SYN_RCVD
recv ACK ↓
ESTABLISHED

Connection Termination (4-Way Handshake) — Active Closer on Left
ACTIVE CLOSER
ESTABLISHED
send FIN ↓
FIN_WAIT_1
recv ACK ↓
FIN_WAIT_2
recv FIN ↓ send ACK
TIME_WAIT
wait 2×MSL ↓
CLOSED
→ FIN →
← ACK ←
← FIN ←
→ ACK →
PASSIVE CLOSER
ESTABLISHED
recv FIN, send ACK ↓
CLOSE_WAIT
app calls close() ↓
LAST_ACK
recv ACK ↓
CLOSED

TIME_WAIT — The Infamous State

What Is TIME_WAIT?

After the active closer sends its final ACK, it does not go immediately to CLOSED. Instead, it enters the TIME_WAIT state and waits for 2 × MSL (Maximum Segment Lifetime) before the socket is truly closed. On Linux, MSL is typically 60 seconds, so TIME_WAIT lasts about 60 seconds (some sources say up to 120 seconds).

During TIME_WAIT, the kernel still owns that 4-tuple (local IP, local port, remote IP, remote port) and will not let another socket use it.

Why Does TIME_WAIT Exist? (Two Reasons)

Reason 1: Reliable delivery of the final ACK.

When the active closer sends the final ACK for the peer’s FIN, that ACK might be lost. The peer will then retransmit its FIN. If the active closer went straight to CLOSED, it would have no socket to respond to the retransmitted FIN and would send an RST, which could confuse the peer. By staying in TIME_WAIT, the active closer can resend the final ACK if the peer’s FIN arrives again.

Active Closer
(TIME_WAIT)
→ final ACK (lost!) →
← FIN retransmit ←
→ ACK resent → ✓
Passive Closer
(LAST_ACK)

Reason 2: Preventing “ghost” packets from old connections.

IP packets can be delayed in the network and arrive late. If a new connection reuses the same 4-tuple as a just-closed connection, a delayed packet from the old connection might arrive and corrupt the new connection’s data. TIME_WAIT ensures that all old packets have expired (their TTL has counted down) before the 4-tuple can be reused.

The Problem: “Address Already in Use”

When you kill and immediately restart a server, you get this error:

$ ./my_server
bind: Address already in use

This happens because the server was the active closer (it called close() on its end), so the just-closed connections entered TIME_WAIT under the server’s port (e.g. port 8080). The OS refuses to let the new server bind to port 8080 while those TIME_WAIT sockets still hold that port.

You can verify this with netstat:

$ netstat -tan | grep 8080
tcp  0  0  0.0.0.0:8080   0.0.0.0:*       LISTEN       # new server fails here
tcp  0  0  127.0.0.1:8080 127.0.0.1:54321  TIME_WAIT   # old connection in TIME_WAIT
tcp  0  0  127.0.0.1:8080 127.0.0.1:54322  TIME_WAIT   # another old connection

SO_REUSEADDR — The Solution

What SO_REUSEADDR Does

The SO_REUSEADDR socket option tells the kernel to allow binding to a port that has TIME_WAIT sockets on it. The TIME_WAIT sockets still exist and still serve their purpose (protecting old packets), but they no longer block new binds to the same port.

#include <sys/socket.h>
#include <netinet/in.h>
#include <stdio.h>
#include <string.h>
#include <unistd.h>

int create_server_socket(int port)
{
    int sfd, opt = 1;
    struct sockaddr_in addr;

    sfd = socket(AF_INET, SOCK_STREAM, 0);
    if (sfd == -1) { perror("socket"); return -1; }

    /*
     * SO_REUSEADDR MUST be set BEFORE bind().
     * This is the standard boilerplate for every TCP server.
     * Without it, restarting the server after kill fails with
     * "Address already in use" for ~60 seconds.
     */
    if (setsockopt(sfd, SOL_SOCKET, SO_REUSEADDR,
                   &opt, sizeof(opt)) == -1) {
        perror("setsockopt SO_REUSEADDR");
        close(sfd);
        return -1;
    }

    memset(&addr, 0, sizeof(addr));
    addr.sin_family      = AF_INET;
    addr.sin_port        = htons(port);
    addr.sin_addr.s_addr = INADDR_ANY;

    if (bind(sfd, (struct sockaddr *)&addr, sizeof(addr)) == -1) {
        perror("bind");
        close(sfd);
        return -1;
    }

    if (listen(sfd, 50) == -1) {
        perror("listen");
        close(sfd);
        return -1;
    }

    printf("Server listening on port %d\n", port);
    return sfd;
}

int main(void)
{
    int lfd = create_server_socket(8080);
    if (lfd == -1) return 1;

    /* ... accept loop ... */
    close(lfd);
    return 0;
}

Important: SO_REUSEADDR does NOT disable TIME_WAIT — the old sockets remain in TIME_WAIT and continue to protect against stale packets. It just allows the new bind to succeed in spite of them.

Diagnostic Tools: netstat and tcpdump

netstat — Viewing Socket States

netstat shows you the current state of all sockets on the system. It is indispensable for debugging.

# Show all TCP sockets (including listening) with addresses (no DNS lookup)
$ netstat -tan

# Show all TCP sockets with process IDs
$ netstat -tanp

# Show only listening sockets
$ netstat -tln

# Show sockets on a specific port
$ netstat -tan | grep :8080

# On newer Linux systems, use 'ss' (socket statistics) — faster
$ ss -tan
$ ss -tanp
$ ss -tan state time-wait   # Show only TIME_WAIT sockets
Example netstat output:
Proto  Local Address        Foreign Address      State
tcp    0.0.0.0:8080         0.0.0.0:*            LISTEN
tcp    127.0.0.1:8080       127.0.0.1:54321      ESTABLISHED
tcp    127.0.0.1:8080       127.0.0.1:54300      TIME_WAIT
tcp    127.0.0.1:8080       127.0.0.1:54299      CLOSE_WAIT

The key states to look for in netstat output are: LISTEN (server is accepting), ESTABLISHED (active connection), TIME_WAIT (waiting for old packets to expire), and CLOSE_WAIT (peer has closed but app hasn’t called close() yet — often a bug).

tcpdump — Capturing Live Packets

tcpdump captures packets on a network interface and prints them. It is essential for verifying that the 3-way handshake, data transfer, and 4-way termination are happening correctly.

# Capture all TCP traffic on loopback, show sequence numbers
$ sudo tcpdump -i lo -n tcp -S

# Capture traffic on port 8080 only
$ sudo tcpdump -i any -n 'tcp port 8080'

# Capture and show packet contents in hex + ASCII
$ sudo tcpdump -i lo -n -X 'tcp port 8080'

# Save capture to file for analysis in Wireshark
$ sudo tcpdump -i eth0 -w capture.pcap 'tcp port 8080'
# tcpdump output for a TCP connection + close
12:00:01 IP client.54321 > server.8080: Flags [S],  seq 1000       # SYN
12:00:01 IP server.8080  > client.54321: Flags [S.], seq 2000 ack 1001  # SYN-ACK
12:00:01 IP client.54321 > server.8080: Flags [.],  ack 2001       # ACK
12:00:02 IP client.54321 > server.8080: Flags [P.], seq 1001:1005  # DATA
12:00:02 IP server.8080  > client.54321: Flags [.],  ack 1005      # ACK data
12:00:03 IP server.8080  > client.54321: Flags [F.], seq 2001      # FIN (server closes)
12:00:03 IP client.54321 > server.8080: Flags [.],  ack 2002       # ACK FIN
12:00:03 IP client.54321 > server.8080: Flags [F.], seq 1005       # FIN (client closes)
12:00:03 IP server.8080  > client.54321: Flags [.],  ack 1006      # ACK FIN

TCP flags in tcpdump: [S] = SYN, [.] = ACK, [S.] = SYN+ACK, [F.] = FIN+ACK, [P.] = PUSH+ACK (data), [R] = RST.

Interview Questions

Q1. What is the TCP TIME_WAIT state and why does it exist?

Answer: TIME_WAIT is the state entered by the active closer (the side that initiates the connection termination) after sending the final ACK. It lasts for 2×MSL (about 60 seconds on Linux). It exists for two reasons: (1) to allow the final ACK to be resent if it is lost and the peer retransmits its FIN, and (2) to prevent delayed packets from an old connection from being misinterpreted by a new connection that reuses the same 4-tuple.

Q2. Why does a TCP server fail to restart immediately with “Address already in use”?

Answer: When the server closes connections, it becomes the active closer and its port enters TIME_WAIT. The kernel prevents any new socket from binding to that port while TIME_WAIT sockets still hold it, because those sockets still serve the purpose of protecting against stale packets. The solution is to set SO_REUSEADDR before calling bind().

Q3. Does SO_REUSEADDR disable TIME_WAIT? Is it safe to use?

Answer: No, SO_REUSEADDR does NOT disable TIME_WAIT. The old sockets remain in TIME_WAIT and continue to protect against stale packets. SO_REUSEADDR only tells the kernel to allow a new bind() to succeed despite the existing TIME_WAIT sockets. It is safe and is standard practice — virtually every production TCP server sets it.

Q4. What is the difference between FIN_WAIT_2 and CLOSE_WAIT?

Answer: These are on opposite sides of the connection. FIN_WAIT_2 is on the active closer’s side — it has sent its FIN and received the ACK, but is waiting for the peer to send its FIN. CLOSE_WAIT is on the passive closer’s side — it has received the peer’s FIN and sent an ACK, but the application has not yet called close(). If CLOSE_WAIT sockets accumulate in your server (visible in netstat), it almost always means the application has a bug where it is not closing connections promptly after the peer closes.

Q5. How many packets are exchanged to establish a TCP connection? To close one?

Answer: Establishment uses a 3-way handshake: SYN → SYN+ACK → ACK (3 packets). Termination typically uses a 4-way handshake: FIN → ACK → FIN → ACK (4 packets), though the middle two can be combined into one FIN+ACK if the passive closer also has no more data to send, making it 3 packets.

Q6. What netstat flags would you use to quickly check if your server is in TIME_WAIT?

Answer: Use netstat -tan | grep TIME_WAIT | grep :<port>. The -t shows TCP, -a shows all sockets including closed/time-wait, -n shows numeric addresses without DNS lookup (faster). On modern Linux, ss -tan state time-wait is faster and more powerful.

Leave a Reply

Your email address will not be published. Required fields are marked *