Why the Linux Kernel Needs an OOM Killer

Physical memory is finite. When applications keep requesting memory and none is left — not in RAM, not in swap — the kernel is in a difficult spot. It could refuse every new allocation and let programs fail one by one, or it could grind the whole machine to a crawl while it desperately searches for free pages. Neither outcome is acceptable for a server that needs to stay responsive.

This is exactly the problem the Linux kernel OOM killer was built to solve. Instead of letting the entire system become unusable, the kernel picks a single offending process (or occasionally a small group of related ones), sends it a fatal signal, and frees its memory immediately. It is a blunt instrument, but it keeps the system alive.

How the Kernel Chooses a Victim: The OOM Score

The kernel does not pick a process at random. Every process on the system is assigned a numeric “badness” value, commonly seen as oom_score in /proc/<pid>/oom_score. The higher the score, the more likely that process is to be selected when the OOM killer activates.

The score primarily reflects how much resident memory a process is using, adjusted by a per-process tunable called oom_score_adj. This lets administrators protect critical services or mark disposable ones as expendable.

You can check or set this value yourself as root:

cat /proc/1234/oom_score
cat /proc/1234/oom_score_adj
echo 500 > /proc/1234/oom_score_adj

Memory Overcommit: Why the Kernel Waits Before Killing

One detail that confuses newcomers to this Linux kernel OOM killer tutorial is that the system rarely dies right at 100% memory usage. That’s because Linux allows a degree of “overcommit” — promising more virtual memory than physically exists, on the assumption most of it will never actually be touched.

sysctl vm.overcommit_memory
sysctl vm.overcommit_ratio

Hands-On Demo: Triggering the OOM Killer Safely

⚠ Only run this on a disposable test VM. Never run a memory-exhaustion test on a machine with anything you care about.

Below is a small original C program that steadily consumes memory in fixed-size chunks and touches every page it allocates, forcing the kernel to actually commit physical pages rather than just reserve address space.

// mem_pressure_demo.c
#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <unistd.h>

#define CHUNK_SIZE (1024 * 1024)   // 1 MB per chunk

int main(void) {
    long total_mb = 0;
    printf("mem_pressure_demo running, PID=%d\n", getpid());

    while (1) {
        char *block = malloc(CHUNK_SIZE);
        if (block == NULL) {
            fprintf(stderr, "malloc failed after %ld MB\n", total_mb);
            break;
        }
        memset(block, 0xAA, CHUNK_SIZE);   // force real page commit
        total_mb++;

        if (total_mb % 100 == 0)
            printf("Allocated %ld MB so far...\n", total_mb);
    }
    return 0;
}

Compile and run it, then watch the kernel log in another terminal:

gcc -O2 -o mem_pressure_demo mem_pressure_demo.c
./mem_pressure_demo

# In a second terminal
sudo dmesg -T | grep -i "out of memory"
sudo journalctl -k | grep -i "oom-killer"

When the process is terminated, the kernel log will show which process was selected, its calculated score, and how much memory it had committed at the time.

Modern Kernel Additions: cgroup v2 and Pressure Stall Information

Older OOM-killer material often stops at the traditional per-process model, but current kernels add container-aware and proactive tools worth knowing for any up-to-date Linux kernel OOM killer tutorial.

• cgroup v2 memory.oom.group — when set, an OOM event inside a cgroup kills every process in that group together, which matters for containerized workloads where killing one half of an application is worse than killing all of it.
• memory.max and memory.high — per-cgroup hard and soft memory ceilings, commonly used by container runtimes like containerd and CRI-O.
• PSI (Pressure Stall Information) — exposed at /proc/pressure/memory, this reports how much time processes spend stalled waiting for memory, letting user-space tools react before the kernel OOM killer has to step in.
• systemd-oomd — a user-space daemon shipped with modern systemd that watches PSI metrics and proactively terminates cgroups under pressure, often preventing the kernel’s own OOM killer from ever triggering.
• earlyoom — a lightweight, distribution-agnostic alternative that monitors free memory and swap directly and kills processes earlier and more predictably than the in-kernel mechanism.

cat /proc/pressure/memory
systemctl status systemd-oomd
oomctl

OOM Killer Workflow

Real-World Use Cases

• Kubernetes pods: a container exceeding its memory limit gets an OOMKilled status, driven by the same cgroup mechanisms discussed above.
• Database servers: DBAs commonly set a negative oom_score_adj on the database process so a runaway reporting query kills itself before the database engine.
• Cloud VM sizing: understanding overcommit ratios helps you right-size instances instead of over-provisioning RAM “just in case.”
• Build servers / CI runners: compilers like large C++ builds are classic OOM victims; adjusting parallelism avoids repeated OOM kills.

Common Mistakes and Troubleshooting

• Assuming a crash is a bug when it’s actually an OOM kill — always check dmesg or journalctl -k first.
• Disabling overcommit protection entirely (vm.overcommit_memory=2) without understanding the ratio, which can cause legitimate allocations to fail.
• Setting oom_score_adj to -1000 on too many processes, leaving the kernel with no safe victim and forcing it to kill something essential like init-adjacent processes.
• Ignoring swap entirely — a small swap file often gives the kernel and tools like systemd-oomd more breathing room to react gracefully.

Best Practices

• Enable and configure systemd-oomd or earlyoom on desktop and server systems for proactive protection.
• Set conservative memory.max limits per cgroup/container rather than relying solely on the global OOM killer.
• Monitor /proc/pressure/memory as part of your regular observability stack.
• Protect critical daemons with a modest negative oom_score_adj, not an extreme one.
• Always test memory-exhaustion scenarios in an isolated VM, never in production.

Performance and Security Considerations

Performance: aggressive overcommit settings improve throughput for memory-sparse workloads but increase the risk of sudden, unpredictable process termination under load spikes. Strict accounting (overcommit_memory=2) trades some flexibility for predictability.

Security: uncontrolled memory allocation by an untrusted process is effectively a denial-of-service vector. cgroup v2 memory limits are the modern mitigation, containing the blast radius to a single container or service instead of the whole host. Note that lowering your own oom_score_adj requires the CAP_SYS_RESOURCE capability, which prevents unprivileged processes from making themselves immune to the OOM killer.

Summary / Key Takeaways

• The OOM killer is the kernel’s last resort to keep a memory-exhausted system alive.
• Every process gets a badness score; oom_score_adj lets you influence it.
• Memory overcommit settings determine how aggressively the kernel allows allocations before intervening.
• Modern kernels add cgroup v2 group-kill semantics and PSI-based proactive tools like systemd-oomd and earlyoom.
• Always validate OOM behavior on a disposable test VM.

Conclusion

The Linux kernel OOM killer is not a bug — it’s a deliberate design decision that trades the loss of one process for the survival of the entire system. Understanding how scoring, overcommit, and the newer cgroup v2 and PSI-based tools fit together turns a scary “process was killed” message into a predictable, tunable part of system administration. With the hands-on demo and the reference tables above, you now have everything you need to reproduce, observe, and control this behavior on your own systems.

Frequently Asked Questions