What is Flow Control?
Imagine filling a glass of water from a jug. If you pour too fast, water spills. TCP’s flow control is the same idea: it prevents a fast sender from sending data faster than the receiver can process it.
The receiver has a finite receive buffer. If the sender pours data into the network faster than the application reads it from the buffer, the buffer overflows and data is lost. Flow control uses a sliding window mechanism to tell the sender exactly how much data it can send at any given time.
The receiving TCP maintains a receive buffer for incoming data. Data arrives from the network, is placed in this buffer, and stays there until the application calls read() to consume it.
During connection establishment, each side tells the other the size of its receive buffer. This is the initial advertised window size. As data is received and consumed, the available window changes dynamically.
The sliding window allows the sender to have multiple segments in flight (unacknowledged) at the same time, up to a limit called the window size. This improves throughput compared to stopping and waiting for each ACK.
With each ACK, the receiver tells the sender: “I have N bytes of free space.” The sender can have at most N bytes of unacknowledged data in the network at any moment.
The window “slides” to the right as the receiver ACKs data. This is why it’s called a sliding window. The window size in each ACK tells the sender how much it can still send before it must stop and wait for more ACKs.
When the receiver’s buffer is completely full, it sends an ACK with window=0. This is called a Zero Window. The sender must completely stop transmitting.
The sender then sends periodic Window Probe segments (tiny 1-byte segments) to check if the window has reopened. Once the application reads data and frees buffer space, the receiver sends a Window Update — an ACK with a non-zero window — and the sender resumes.
You can control the receive and send buffer sizes using SO_RCVBUF and SO_SNDBUF socket options. The OS may adjust these values (typically doubling them internally for overhead), so read back the value after setting.
/* socket_buffers.c - controlling TCP send/receive buffer sizes */
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>
#include <sys/socket.h>
#include <netinet/in.h>
#define DESIRED_RCVBUF (256 * 1024) /* 256 KB receive buffer */
#define DESIRED_SNDBUF (128 * 1024) /* 128 KB send buffer */
int main(void)
{
int sockfd;
int rcvbuf, sndbuf;
socklen_t optlen;
sockfd = socket(AF_INET, SOCK_STREAM, 0);
if (sockfd == -1) { perror("socket"); exit(EXIT_FAILURE); }
/* --- Read default buffer sizes --- */
optlen = sizeof(rcvbuf);
getsockopt(sockfd, SOL_SOCKET, SO_RCVBUF, &rcvbuf, &optlen);
printf("Default recv buffer: %d bytes\n", rcvbuf);
optlen = sizeof(sndbuf);
getsockopt(sockfd, SOL_SOCKET, SO_SNDBUF, &sndbuf, &optlen);
printf("Default send buffer: %d bytes\n", sndbuf);
/* --- Set new buffer sizes --- */
rcvbuf = DESIRED_RCVBUF;
if (setsockopt(sockfd, SOL_SOCKET, SO_RCVBUF, &rcvbuf, sizeof(rcvbuf)) == -1) {
perror("setsockopt SO_RCVBUF");
}
sndbuf = DESIRED_SNDBUF;
if (setsockopt(sockfd, SOL_SOCKET, SO_SNDBUF, &sndbuf, sizeof(sndbuf)) == -1) {
perror("setsockopt SO_SNDBUF");
}
/* --- Read back actual values (OS may double them) --- */
optlen = sizeof(rcvbuf);
getsockopt(sockfd, SOL_SOCKET, SO_RCVBUF, &rcvbuf, &optlen);
printf("Actual recv buffer: %d bytes (OS may have adjusted)\n", rcvbuf);
optlen = sizeof(sndbuf);
getsockopt(sockfd, SOL_SOCKET, SO_SNDBUF, &sndbuf, &optlen);
printf("Actual send buffer: %d bytes (OS may have adjusted)\n", sndbuf);
/*
* Important: SO_RCVBUF must be set BEFORE calling listen() on a server socket
* or BEFORE calling connect() on a client socket.
* Setting it after connect() may not affect the window advertised to the peer.
*
* Linux kernel limits: check /proc/sys/net/core/rmem_max
* cat /proc/sys/net/core/rmem_max (max allowed by non-root processes)
* cat /proc/sys/net/core/rmem_default (default)
*/
printf("\n/proc hints:\n");
system("cat /proc/sys/net/core/rmem_default 2>/dev/null || echo 'not available'");
system("cat /proc/sys/net/core/rmem_max 2>/dev/null || echo 'not available'");
close(sockfd);
return 0;
}
/* Checking window size with TCP_INFO after connection */
#include <stdio.h>
#include <sys/socket.h>
#include <netinet/tcp.h>
#include <unistd.h>
void print_window_info(int sockfd)
{
struct tcp_info ti;
socklen_t len = sizeof(ti);
if (getsockopt(sockfd, IPPROTO_TCP, TCP_INFO, &ti, &len) == 0) {
/*
* tcpi_rcv_space: bytes available in receive buffer
* tcpi_snd_wnd: peer's advertised receive window (what we can send)
* tcpi_rcv_wnd: our own advertised receive window
*/
printf("Peer's advertised window (send budget): %u bytes\n", ti.tcpi_snd_wnd);
printf("Our advertised recv window: %u bytes\n", ti.tcpi_rcv_wnd);
printf("Receive buffer space: %u bytes\n", ti.tcpi_rcv_space);
}
}
System-wide buffer tuning (requires root):
# View current limits
cat /proc/sys/net/core/rmem_default # default receive buffer
cat /proc/sys/net/core/rmem_max # max receive buffer
cat /proc/sys/net/core/wmem_default # default send buffer
cat /proc/sys/net/core/wmem_max # max send buffer
# Enable auto-tuning (Linux kernel adjusts buffers based on conditions)
echo 1 > /proc/sys/net/ipv4/tcp_moderate_rcvbuf
# Set TCP-specific buffer limits
# Format: min default max (in bytes)
echo "4096 87380 16777216" > /proc/sys/net/ipv4/tcp_rmem
echo "4096 65536 16777216" > /proc/sys/net/ipv4/tcp_wmem
Flow control prevents a fast sender from overwhelming a slow receiver. Without it, the sender could flood the receiver’s buffer, causing data loss. TCP uses the advertised window (in ACKs) to tell the sender how much free space the receive buffer has.
Flow control is between sender and receiver — it protects the receiver’s buffer. Congestion control is between sender and the network — it protects intermediate routers from being overwhelmed. Flow control uses the receiver’s advertised window; congestion control uses the congestion window (cwnd) maintained by the sender based on network feedback.
A sliding window allows the sender to send multiple segments without waiting for an ACK for each one. The window defines the maximum number of unacknowledged bytes. As ACKs arrive, the window slides right, allowing more data to be sent. This avoids the “stop and wait” inefficiency and keeps the link utilized.
A zero window occurs when the receiver’s buffer is full — it advertises window=0, telling the sender to stop. The sender pauses and periodically sends 1-byte window probe segments to check if the window has reopened. Once the application reads data (freeing buffer space), the receiver sends a window update with a non-zero window.
Use setsockopt(fd, SOL_SOCKET, SO_RCVBUF, &size, sizeof(size)). A larger receive buffer increases the advertised window, allowing the sender to have more data in-flight, which improves throughput on high-latency links (BDP = bandwidth × RTT). The OS may cap the value at /proc/sys/net/core/rmem_max. Set it before listen() or connect().
BDP = bandwidth × RTT. It represents the amount of data “in flight” in the network at any moment when the pipe is full. For example, 1 Gbps × 100ms RTT = 12.5 MB. The receive buffer should be at least as large as the BDP so that the sender is never forced to wait for ACKs. Under-sized buffers on high-BDP paths cause severe throughput degradation.