What You Will Learn

• What the Kernel Schedulable Entity (KSE) is on Linux and why it is a thread, not a process
• How the kernel represents every thread using a task structure (task_struct)
• The difference between user threads, kernel threads, and how each is scheduled
• What a per-CPU run queue is and why modern multi-core systems need one per core
• How to count and inspect all schedulable threads on your own Linux machine right now

Prerequisites

You should be comfortable with basic Linux commands and know, at a high level, what a process and a thread are. If you are following our free embedded systems course or our earlier Linux kernel lessons on processes and threads, you already have everything you need. A Linux machine or virtual machine running any recent distribution (kernel 6.x recommended) is enough for the hands-on parts – no special hardware required.

The Big Question: What Does the Linux CPU Scheduler Schedule?

The CPU is the most contended resource in any computer. At any instant, hundreds or thousands of runnable entities may want processor time, but each CPU core can execute only one of them at a time. The scheduler is the kernel component that decides who runs next, on which core, and for how long.

So what is the “who” here? The object the scheduler operates on is called the Kernel Schedulable Entity (KSE). Different operating systems make different choices. On Linux, the answer is clear and absolute:

Every process contains at least one thread (the main thread), and it may create many more. The scheduler never looks at the process as a unit. It looks at individual threads, compares their priorities and virtual runtimes, and picks one thread per CPU core to execute. The thread is the granularity at which all scheduling decisions happen.

Why the Thread and Not the Process?

Think of a process as a container: it holds a virtual address space, open files, credentials, and one or more threads. The threads are the workers inside the container – they are the entities that actually have an instruction pointer, a stack, and CPU register state. Only something with execution state can be placed on a CPU. A process without its threads is just a passive set of resources, so there is nothing there for the scheduler to run.

This design also gives Linux a big advantage: two threads of the same process can run truly in parallel on two different CPU cores, because the scheduler treats them as fully independent schedulable units.

The task_struct: How the Kernel Sees Every Thread

Inside the kernel, every single thread on the system – whether it belongs to your web browser or to the kernel itself – is described by one instance of struct task_struct. This is one of the largest and most important structures in the entire kernel source tree (defined in include/linux/sched.h). For scheduling purposes, the interesting members include:

Along with the task structure, every user-space thread owns two stacks: a user-mode stack (used while executing application code) and a small kernel-mode stack (used when the thread enters the kernel through a system call or is interrupted). Kernel threads are simpler – they live entirely inside the kernel, so they have only a kernel stack and their address-space pointer is empty.

A Quick Counting Exercise

Imagine a small single-core embedded board running 10 user-space processes, where each process has 3 threads, and the kernel itself has spawned 5 kernel threads for housekeeping. How many schedulable entities exist?

If all of them became runnable at once (unlikely in practice, but useful for the thought experiment), the scheduler would see 35 competitors – not “10 processes plus some kernel stuff”. Internalize this: from the scheduler’s point of view, the process boundary is invisible.

Run Queues: One Waiting Room Per CPU Core

Modern systems have many CPU cores, and a global list of runnable threads would become a locking bottleneck. Linux therefore maintains a per-CPU run queue (struct rq). Each core has its own private list of runnable threads, and the scheduler on that core normally picks only from its own queue. A separate load-balancing mechanism periodically migrates threads between queues so no core sits idle while another is overloaded.

Each run queue internally holds sub-queues for the different scheduling classes (fair, real-time, deadline). We will explore those classes in the next lessons of this free Linux kernel development course.

Hands-On: Inspect the Schedulable Threads on Your System

Let’s verify everything above on a real machine. First, count all threads (not processes) on your system:

# Count every thread on the system (each row = one KSE)
ps -eLf | wc -l

# Show threads with their thread ID (LWP column = TID)
ps -eLf | head -20

The -L flag tells ps to show every thread as a separate row. The LWP (Light-Weight Process) column is the thread ID – the exact identifier the scheduler works with.

Next, spot the kernel threads. They are conventionally displayed in square brackets and are children of kthreadd (PID 2):

# List kernel threads (children of kthreadd, PID 2)
ps --ppid 2 -p 2 -o pid,ppid,comm | head -15

# Typical output shows names like:
# kthreadd, rcu_preempt, ksoftirqd/0, kworker/0:1, migration/0 ...

Finally, look at a single multi-threaded process and confirm that the kernel tracks each thread individually:

# Pick any multi-threaded process, e.g. your browser or systemd
PID=$(pgrep -o firefox || pgrep -o systemd)

# Every directory under /proc/PID/task is one schedulable thread
ls /proc/$PID/task | wc -l

# Per-thread scheduler statistics maintained by the kernel
cat /proc/$PID/task/$PID/sched | head -12

The file /proc/<pid>/task/<tid>/sched is direct evidence: the kernel keeps scheduler bookkeeping per thread, including how many times it has been switched in and how much CPU time it has consumed.

A Note on Modern Kernels

Older books and tutorials on the Linux CPU scheduler describe the Completely Fair Scheduler (CFS) as the engine behind normal threads. That was true from kernel 2.6.23 up to kernel 6.5. Starting with kernel 6.6, the fair class was rewritten around the EEVDF algorithm (Earliest Eligible Virtual Deadline First), and from kernel 6.12 onwards, Linux even allows loading custom schedulers written in BPF through the sched_ext class. The concepts in this lesson – the thread as the KSE, the task structure, per-CPU run queues – are unchanged and remain the foundation. We cover EEVDF and the modern scheduler internals in an upcoming lesson of this series.

Common Mistakes and Misconceptions

• “Linux schedules processes” – No. Linux schedules threads. A single-threaded process is scheduled only because it contains one thread.
• “Kernel threads are special-cased by the scheduler” – No. A kernel thread has a task structure like any other thread and competes through the same scheduling classes.
• “Threads of one process must run on the same core” – No. Each thread is independent; two threads of one process can run in parallel on different cores.
• “There is one global run queue” – Not on modern Linux. Each CPU core has its own run queue, plus load balancing between them.

Interview Questions and Answers

Q1: What is the Kernel Schedulable Entity on Linux?

A: The thread. Every scheduling decision – priority comparison, time-slice accounting, CPU placement – happens at thread granularity. Processes are only resource containers.

Q2: How does the kernel represent a thread internally?

A: Through struct task_struct, which stores the thread’s state, scheduling policy, priorities, scheduler-entity bookkeeping, and a pointer to the process address space. User threads additionally own a user stack and a kernel stack.

Q3: How can you distinguish a kernel thread from a user thread inside the kernel?

A: A kernel thread has no user address space – its mm pointer is NULL. From user space, kernel threads appear as children of kthreadd (PID 2) and are shown in square brackets by ps.

Q4: Why does Linux use per-CPU run queues instead of one global queue?

A: Scalability. A single global queue would require a global lock touched by every core on every scheduling decision. Per-CPU queues keep decisions local and lock-free in the common case; a load balancer redistributes threads when queues become uneven.

Q5: If a process has 4 threads and the machine has 4 cores, can the process use 100% of the machine?

A: Yes. Because each thread is an independent KSE, the scheduler can place all four threads on four different cores simultaneously.

FAQ

Is this Linux CPU scheduler tutorial free?

Yes. This entire series is part of the free Linux kernel development course on EmbeddedPathashala, alongside our free Linux device drivers course and free embedded systems course.

Which kernel version does this tutorial target?

All concepts here apply to modern 6.x kernels, including the EEVDF-based fair scheduler introduced in kernel 6.6. Where older material (like CFS) has been replaced, we say so explicitly.

Do I need special hardware to practice?

No. Any Linux machine or virtual machine works. Even a Raspberry Pi is enough for every hands-on exercise in this series.

What is the difference between PID and TID?

The TID identifies one thread and is what the scheduler uses. The PID identifies the process; for the main thread, PID and TID are equal. ps -eLf shows TIDs in the LWP column.

Do kernel threads have user stacks?

No. Kernel threads execute only kernel code, so they have a kernel stack and a task structure but no user stack and no user address space.

Key Takeaways

• On Linux, the thread is the Kernel Schedulable Entity – all scheduling happens at thread granularity.
• Every thread is represented by a task_struct; user threads own both a user stack and a kernel stack.
• Kernel threads are ordinary schedulable threads that simply have no user address space.
• Each CPU core maintains its own run queue, with load balancing between cores.
• Modern kernels (6.6+) use EEVDF instead of CFS for normal threads – covered later in this series.