Why Ordinary Counters Break Under Concurrency

Picture a very small piece of driver state — say, a counter that tracks how many user-space processes currently have a device file open. If two CPUs run the same increment at the exact same instant, both may read the old value, both add one, and both write back the same result. One increment silently disappears. This is a classic read-modify-write race, and it’s one of the very first bugs every kernel and driver developer runs into.

The traditional fix is a lock — a mutex or a spinlock — wrapped around the increment. That works, but for something as small as a single-integer update, taking and releasing a full lock is unnecessary overhead. The kernel gives us a lighter-weight, purpose-built tool instead: atomic operators.

Understanding the atomic_t Interface

atomic_t is a special integer type the kernel provides so that increments, decrements, and compare-and-swap style updates happen as a single, indivisible CPU instruction (or an equivalent LL/SC sequence on architectures that need it). No lock, no context switch, no waiting — just one guaranteed-safe hardware operation.

Declaring and Using an atomic_t Variable

#include <linux/atomic.h>

/* Declare and initialise an atomic counter to 0 */
static atomic_t open_count = ATOMIC_INIT(0);

static int mydev_open(struct inode *inode, struct file *filp)
{
    atomic_inc(&open_count);
    pr_info("mydev: open count is now %d\n", atomic_read(&open_count));
    return 0;
}

static int mydev_release(struct inode *inode, struct file *filp)
{
    atomic_dec(&open_count);
    return 0;
}

Notice there is no lock anywhere in that snippet, yet it is completely race-free. The four workhorse operations you will use constantly are:

All of these operate on a plain signed 32-bit value under the hood. For 64-bit counters the kernel mirrors the same API under the atomic64_ prefix (atomic64_inc(), atomic64_read(), and so on), so everything you learn here transfers directly.

Read-Modify-Write (RMW) Style Operators

Beyond simple increments, the kernel also exposes RMW variants that let you fetch the old value and apply a change in one atomic step — useful when a decision depends on the counter’s previous state.

/* Returns true only when the counter actually reaches zero */
if (atomic_dec_and_test(&open_count)) {
    pr_info("mydev: last user closed the device, cleaning up\n");
    mydev_cleanup();
}

/* Classic compare-and-swap: only update if it still matches "old" */
int old_val = 1, new_val = 2;
atomic_cmpxchg(&state, old_val, new_val);

atomic_dec_and_test() is especially common in real driver code: it is exactly the pattern you need for “free this resource only when the last reference goes away.”

Why refcount_t Exists: Hardening Reference Counting

atomic_t is generic — it will happily let a counter go negative, wrap past its maximum value, or be decremented below zero without complaint. That flexibility is fine for a simple statistics counter, but it is dangerous for a reference counter that controls when a kernel object is freed. If an attacker or a buggy code path can push a reference count to zero while a pointer to the object is still in use, or wrap it back around to a positive value after it should have hit zero, the result is a classic use-after-free (UAF) vulnerability — one of the most exploited bug classes in kernel security history.

To close this gap, the kernel added a dedicated refcount_t type, purpose-built only for reference counting. It looks and behaves like atomic_t on the surface, but with much stricter rules baked in.

Using refcount_t in a Driver

#include <linux/refcount.h>

struct mydev_object {
    refcount_t refcnt;
    /* other fields */
};

static void mydev_object_init(struct mydev_object *obj)
{
    refcount_set(&obj->refcnt, 1);   /* object starts life with one reference */
}

static void mydev_object_get(struct mydev_object *obj)
{
    refcount_inc(&obj->refcnt);
}

static void mydev_object_put(struct mydev_object *obj)
{
    if (refcount_dec_and_test(&obj->refcnt))
        kfree(obj);   /* only the very last "put" actually frees it */
}

The logic reads almost identically to the atomic version, but underneath, refcount_t enforces guarantees that atomic_t never did:

• No going below zero. Decrementing an already-zero refcount triggers a loud kernel warning instead of silently corrupting memory.
• Saturation, not wraparound. Once a refcount hits its saturation ceiling, it locks there instead of overflowing back to a small (exploitable) number.
• Valid range only. A refcount is expected to live strictly between 1 and its maximum value; anything outside that range is treated as a bug and reported immediately.
• Stronger memory ordering around the final decrement, so that every CPU sees a consistent view of the object right before it is freed.

atomic_t vs refcount_t: Side-by-Side Comparison

Real-World Use Cases

• Device open/close counters — tracking how many processes currently hold a device file open (atomic_t is sufficient here).
• Kernel object lifetime management — structures such as network sockets, file objects, and driver-private structs that are shared across multiple code paths use refcount_t so they are freed exactly once, at exactly the right time.
• Module and driver statistics — packet counters, interrupt counters, and error counters exported through sysfs or debugfs.
• Workqueue and timer state flags — atomic compare-and-swap operations are used to safely transition state machines without a full lock.

Common Mistakes and Troubleshooting

Best Practices

• Default to refcount_t whenever the counter controls object lifetime — never roll your own reference counting with a raw integer.
• Reserve atomic_t for general counters, flags, and statistics that are not tied to freeing memory.
• Always initialise counters explicitly (ATOMIC_INIT() or refcount_set()) rather than relying on zeroed memory.
• Pair every “get” with exactly one “put” — audit new code paths carefully, especially error-handling branches.
• Enable kernel lock and refcount debugging options in your development kernel so mistakes surface early instead of in production.

Performance Considerations

Atomic operations are dramatically cheaper than a mutex or even a spinlock for single-variable updates, since there is no possibility of the calling thread sleeping or spinning in a loop — the CPU either completes the instruction or briefly retries at the hardware level. That said, atomics are not free: on systems with many CPU cores, a counter that is incremented and decremented extremely frequently from every core can become a cache-line bottleneck, since every CPU has to pull the same cache line into its own cache to modify it. For extremely hot counters, kernel developers sometimes switch to per-CPU counters instead, combining the totals only when a reader actually needs the full value.

Security Considerations

Reference-count bugs are not just correctness issues — they are a well-documented source of real kernel security vulnerabilities. A reference count that reaches zero too early, or wraps back to a valid-looking number after underflowing, can result in a kernel object being freed while something still holds a pointer to it. Later use of that dangling pointer is a classic use-after-free, and UAF bugs are frequently the foundation of privilege-escalation exploits. This is precisely the motivation behind refcount_t: by making underflow and overflow loudly detectable instead of silently exploitable, entire classes of these bugs are caught during testing rather than discovered by attackers.

Summary and Key Takeaways

• atomic_t gives you lock-free, safe updates to a general-purpose integer.
• refcount_t is a specialised, hardened wrapper meant only for object reference counting.
• The core difference is range enforcement: refcount_t refuses to underflow or overflow, converting silent memory-corruption bugs into loud, catchable warnings.
• Always match “get” and “put” calls, and free the object only when *_dec_and_test() reports the last reference is gone.
• Use atomics for statistics and flags; use refcounts for anything controlling when memory gets freed.

Conclusion

Atomic and refcount interfaces are two of the smallest but most heavily used building blocks anywhere in the Linux kernel. Once you internalise the difference — general-purpose counter versus hardened reference counter — you will start spotting the pattern everywhere, from simple misc-device drivers all the way up to core subsystems like the networking stack and the VFS. This lecture is part of our ongoing free Linux kernel development course and free Linux device drivers course, and the concepts here feed directly into the next topic: reader-writer spinlocks, where we protect data that is read far more often than it is written.

Frequently Asked Questions