Linux’s virtual memory manager (VMM) uses a global LRU page reclaim system. The decision of which pages to evict from physical RAM is made globally across all processes based on usage patterns, not per-process quotas. Implementing per-process RSS limits in a way that cooperates correctly with the global VMM is complex and has never been done in mainline Linux.

The constant RLIMIT_RSS exists so that code written on BSD which uses it can be compiled on Linux without errors. setrlimit(RLIMIT_RSS, …) succeeds silently, getrlimit(RLIMIT_RSS, …) returns the stored value, but the kernel never actually uses it.

If you need to limit a process’s physical memory usage on modern Linux, use these mechanisms instead:

• Control Groups (cgroups) — memory.limit_in_bytes in the memory cgroup subsystem enforces a hard limit on RSS + swap. This is the correct modern approach.
• RLIMIT_AS — Limits virtual address space (a superset that includes RSS). Enforced, unlike RLIMIT_RSS.

#include <stdio.h>
#include <stdlib.h>
#include <string.h>
#include <sys/resource.h>

int main(void)
{
    struct rlimit rl;

    getrlimit(RLIMIT_RSS, &rl);
    printf("RLIMIT_RSS before: soft=%lld hard=%lld\n",
           (long long)rl.rlim_cur, (long long)rl.rlim_max);

    /* Set an RSS limit — this will SUCCEED but have NO effect */
    rl.rlim_cur = 32 * 1024 * 1024;   /* 32 MB */
    rl.rlim_max = 32 * 1024 * 1024;
    if (setrlimit(RLIMIT_RSS, &rl) == 0)
        printf("setrlimit(RLIMIT_RSS, 32MB) succeeded\n");

    /* Allocate 64 MB — this will NOT be prevented by RLIMIT_RSS */
    char *buf = malloc(64 * 1024 * 1024);
    if (buf) {
        memset(buf, 1, 64 * 1024 * 1024);  /* Touch all pages */
        printf("Allocated and touched 64 MB — RSS limit not enforced\n");
        free(buf);
    }
    return 0;
}
/* Compile: gcc -o rss_demo rss_demo.c
   Lesson: setrlimit(RLIMIT_RSS) returns success but the kernel ignores
   the limit value. Use cgroups memory.limit_in_bytes for real enforcement. */