L22: Comparing Kernel Configs and Kernel Security Hardening



📘 Free Linux Kernel Programming Course | Chapter 2

Chapter 2 · Lecture 22

Comparing Kernel Configs and Kernel Security Hardening

Learn the right way to see what changed in your kernel .config, and understand why kernel-space security hardening matters for every Linux kernel developer.

2
Key Topics
6.x
Kernel Version
Free
EmbeddedPathashala

Topics Covered in This Lecture

diffconfig script
scripts/diffconfig
.config comparison
Linux kernel security
KSPP
kernel-hardening-checker
KASLR
CONFIG_STACKPROTECTOR
kernel config audit
free linux kernel course

🎯 What You Will Learn

  • Why plain diff fails on large .config files and what to use instead
  • How to use scripts/diffconfig and read its output symbols
  • Why kernel-space security hardening is still catching up to userspace security
  • What the Kernel Self Protection Project (KSPP) is and what it protects against
  • How to audit your own kernel config using kernel-hardening-checker
  • How to auto-generate a hardening config fragment for your architecture

📸 Suggested Image for This Section

Screenshot of a terminal showing scripts/diffconfig .config.old .config output with color-highlighted lines — red for removed, green for added, white for changed. Alt text: “Linux kernel diffconfig script comparing two .config files in terminal”

1. Why Plain diff Fails on Kernel Config Files

When you configure the Linux kernel, the output is a file named .config at the root of your kernel source tree. On a modern kernel 6.x build, this file contains around 10,000 lines. If you changed a handful of settings and want to see exactly what you modified, your first instinct might be to run:

$ diff .config.old .config

The problem is that plain diff compares files line by line. A .config file contains comment lines starting with #, blank lines, and symbols in a certain order that may shift when the kernel version changes. The result is an overwhelming wall of text where your five actual changes are buried under hundreds of irrelevant lines.

💡 The Core Problem: Out of 10,000 config entries, you changed 5. Plain diff gives you hundreds of lines. You cannot easily find your 5 changes in that noise.

2. The diffconfig Script — Built Into the Kernel

The Linux kernel ships its own solution inside the scripts/ folder: a small utility called diffconfig. It was written specifically for this problem. Instead of a raw file comparison, it parses both config files, extracts the symbol names and their values, and then compares only the values. The result is sorted, minimal, and immediately useful.

How scripts/diffconfig Processes Your Config Files
📄 .config.old
Your previous config file
+ 📄 .config
Your updated config file
✅ Clean Output
Sorted, meaningful changes only

How to Use diffconfig

The workflow is always the same: save a backup of your current config before making changes, apply your changes, then compare the two files:

# Step 1 — back up before you change anything
$ cp .config .config.old

# Step 2 — make your changes
$ make menuconfig

# Step 3 — compare cleanly
$ scripts/diffconfig .config.old .config

Reading the diffconfig Output

The output is compact and follows a strict three-symbol format. Here is a realistic example of what you might see:

-AX25 n
-HAMRADIO y
 DEBUG_STACK_USAGE n -> y
 IKCONFIG m -> y
 IKCONFIG_PROC n -> y
 LOCALVERSION "" -> "-mykernel01"
+UIO_AEC n

diffconfig Output Symbols — What Each One Means
Symbol Meaning Example
-OPTION val Option was in the old config but is gone in the new one -AX25 n
+OPTION val Option is brand new — did not exist in old config +UIO_AEC n
OPTION a -> b Option changed value from a to b IKCONFIG m -> y

Quick Reminder — The Three Config Values:

  • y — built directly into the kernel image, always present at boot
  • m — compiled as a separate loadable module (.ko file), loaded on demand
  • n — not compiled at all, takes zero space in the final image
📌 Kernel 6.x Note: scripts/diffconfig is present and unchanged in all modern Linux 6.x kernels. Run scripts/diffconfig --help to see all available options.

3. Kernel Security — More Complex Than You Think

There is a widespread belief that Linux is secure by default. The reality is more nuanced. The kernel has hundreds of Kconfig options that directly affect how resistant it is to being exploited. Distributions like Ubuntu or Fedora make their own choices about these options, balancing compatibility, performance, and security — and security does not always win.

More importantly, most attention in the Linux security world has historically focused on userspace security — tools like SELinux, AppArmor, and seccomp that control what running processes are allowed to do. These technologies have matured over two decades. Kernel-space hardening — protecting the kernel itself from being exploited — is a younger field and still catching up.

Linux Security — Two Distinct Layers
🛡️ Userspace Security
SELinux, AppArmor, seccomp — control what processes can do. Mature and widely deployed across all major Linux distributions.
✅ Well established
⚙️ Kernel-Space Security
Protecting the kernel itself from exploitation. Many options exist in Kconfig but are often off by default — this area is actively being improved.
⚠️ Still catching up

4. The Kernel Self Protection Project — KSPP

The Kernel Self Protection Project (KSPP) is a community initiative by kernel security engineers whose goal is simple: get hardening features merged directly into the mainline Linux kernel. Rather than maintaining out-of-tree patches that distributions have to apply separately, KSPP pushes these improvements upstream where everyone benefits automatically.

What KSPP Focuses On — Key Protection Areas
Stack Overflow Protection
Insert canary values on the kernel stack to detect and stop stack-based buffer overflow attacks before they can hijack execution.
Read-Only Kernel Memory
Mark kernel code sections and read-only data as non-writable after initialization so they cannot be tampered with at runtime.
KASLR
Randomize the base address at which the kernel is loaded so attackers cannot rely on fixed memory addresses when writing exploits.
Heap Protections
Randomize the slab allocator free lists, sanitize freed memory, and harden heap metadata against use-after-free and heap spray attacks.

Many KSPP improvements are now in mainline Linux and enabled by default in kernel 6.x. But a significant number of hardening options are still n by default because they carry a performance cost or compatibility trade-off. This is exactly where tools like kernel-hardening-checker help.

5. Auditing Your Config With kernel-hardening-checker

There are well over a hundred security-relevant Kconfig options. Checking each one by hand is not realistic. Security engineer Alexander Popov created a tool called kernel-hardening-checker (formerly known as kconfig-hardened-check) that automates this audit completely.

You give it your kernel’s .config file and it checks each security-relevant option against recommendations from four well-respected sources:

Sources That Drive kernel-hardening-checker Recommendations
KSPP
Kernel Self Protection Project
CLIP OS
French government security OS
Android Security
Google kernel hardening team
Lockdown LSM
Linux kernel lockdown module

Installing and Running the Tool

# Install via pip (Python package)
$ pip install kernel-hardening-checker

# Check your currently running kernel's config
$ kernel-hardening-checker -c /boot/config-$(uname -r)

# Check a custom kernel config you are building
$ kernel-hardening-checker -c /path/to/kernel/source/.config

# Just print what the recommendations are for x86_64 (no config needed)
$ kernel-hardening-checker -p X86_64

Reading the Output Table

option name                      | desired | decision  | reason          | result
CONFIG_BUG                       |    y    | defconfig  | self_protection | OK
CONFIG_STRICT_KERNEL_RWX         |    y    | defconfig  | self_protection | OK
CONFIG_STACKPROTECTOR_STRONG     |    y    | defconfig  | self_protection | OK
CONFIG_DEBUG_LIST                |    y    | kspp       | self_protection | FAIL: "is not set"
CONFIG_INIT_ON_ALLOC_DEFAULT_ON  |    y    | kspp       | self_protection | OK

What Each Column in the Output Means
Column What It Tells You
option name The kernel Kconfig symbol being evaluated
desired val The value the security guidelines recommend
decision Which source recommends this — defconfig, kspp, clipos, android
reason The protection category — self_protection, cut_attack_surface, etc.
check result OK — your config meets this recommendation | FAIL — it does not
⚠️ Important: Never blindly enable every FAIL option. Some have real performance costs. Some break compatibility with software your system needs. Some do not apply to your specific threat model at all. Use this tool to make informed decisions — not as a mandatory checklist.

6. Modern Feature: Auto-Generating a Hardening Fragment

The original kconfig-hardened-check tool could only audit your existing config. The modern kernel-hardening-checker gained a powerful new feature: with the -g flag, it generates a Kconfig fragment you can feed directly into your build process.

# Generate a ready-to-use hardening fragment for x86_64
$ kernel-hardening-checker -g X86_64 > /tmp/hardening.fragment

# Merge it with your existing .config using the kernel's own merge script
$ cd /path/to/your/kernel/source
$ scripts/kconfig/merge_config.sh .config /tmp/hardening.fragment

This is very useful if you maintain custom kernel builds for embedded products or production systems. You can automate security hardening as part of your CI/CD pipeline instead of relying on someone manually checking options.

Automated Hardening Workflow With kernel-hardening-checker -g
Your
.config
+ kernel-hardening-checker -g X86_64
generates hardening.fragment
merge_config.sh
hardened .config ready to build

🛠️ Practical Tips for Kernel Developers

Always Back Up
Copy .config to .config.bak before any config session. One second of effort, hours saved when things go wrong.
Use diffconfig in CI
Add scripts/diffconfig to your build pipeline to automatically catch accidental config drift between kernel version upgrades.
Match Your Threat Model
An isolated embedded device does not need the same hardening as a public cloud server. Apply options that match your actual risk.

📋 Lecture Summary

  • scripts/diffconfig .config.old .config gives you a clean, sorted list of exactly what changed — use it every time
  • The three output prefixes: - means removed, + means new, no prefix with -> means value changed
  • Kernel-space security hardening is still an evolving area — many protective options are available but not enabled by default
  • KSPP (Kernel Self Protection Project) is pushing hardening features into mainline Linux so everyone benefits
  • kernel-hardening-checker audits your .config against KSPP, CLIP OS, Android, and Lockdown LSM recommendations
  • The -g flag generates a merge-ready config fragment to automate hardening in your build pipeline

❓ Frequently Asked Questions

Is scripts/diffconfig available in Linux kernel 6.x?

Yes. The scripts/diffconfig utility is present in all modern Linux 6.x kernel source trees. It is a stable part of the kernel developer toolset and is not deprecated. You can verify it with ls scripts/diffconfig from your kernel source root.

What is the difference between kconfig-hardened-check and kernel-hardening-checker?

They are the same tool. The project was renamed from kconfig-hardened-check to kernel-hardening-checker as it matured and gained new capabilities. The modern version supports checking Kconfig options, kernel boot parameters, and sysctl settings, and can now generate hardening config fragments with the -g flag.

Should I apply all KSPP recommendations on my embedded Linux kernel?

Not blindly. Each recommendation needs to be evaluated against your device’s threat model, performance requirements, and software compatibility needs. An embedded device with no network interface and no untrusted user input has very different needs from a networked server. Use the tool’s output as an informed starting point, not a mandatory checklist.

Does enabling KASLR have a performance impact?

The impact is minimal on modern hardware. KASLR randomizes the kernel load address at boot time, which is a one-time operation. The ongoing runtime overhead is negligible. It is generally considered worth enabling on any system that faces untrusted input. It is already enabled by default in most distributions using kernel 6.x.

Can I use diffconfig to compare configs from different kernel versions?

Yes, and this is one of the most useful applications. When you upgrade from, say, kernel 6.6 to kernel 6.9, your old .config may have symbols that no longer exist or new symbols that were not present before. Running diffconfig old_config new_config immediately shows you exactly what is new, what was removed, and what changed — making the upgrade review much faster.

🎯 Interview Questions

Q1. What is the difference between plain diff and scripts/diffconfig for comparing kernel .config files?

Plain diff compares files line by line and includes all differences — comment lines, ordering changes, blank lines — producing very noisy output for large .config files. scripts/diffconfig parses both files, extracts the config symbol names and values, ignores formatting noise, and presents a sorted list of only the meaningful changes. It distinguishes removed options (prefix -), new options (prefix +), and value changes (shown as old -> new).

Q2. What does CONFIG_STACKPROTECTOR_STRONG=y do from a security perspective?

This option enables the GCC stack protector at its strongest setting for kernel functions. It inserts a randomly chosen canary value on the stack frame between the local variables and the return address. If a stack overflow corrupts the canary, the kernel detects the mismatch and panics rather than allowing the hijacked return address to execute. This mitigates a wide class of stack-based buffer overflow exploits inside the kernel.

Q3. What is KSPP and how does it differ from SELinux or AppArmor?

SELinux and AppArmor are Linux Security Modules operating at the userspace-kernel boundary — they control what actions processes are permitted to perform. KSPP (Kernel Self Protection Project) operates at a deeper level: it hardens the kernel itself against being exploited, regardless of what the attacker is running in userspace. Features like KASLR, read-only kernel memory, and heap hardening are KSPP contributions. The goal is to make it much harder to turn a kernel vulnerability into a working exploit.

Q4. You run kernel-hardening-checker on an embedded kernel config and see 30 FAIL entries. Should you fix all of them?

No. Each FAIL is a recommendation, not a mandate. You must evaluate each one against your device’s specific threat model. An embedded device without network access has very different exposure than a public server. Some options have performance costs that are unacceptable for real-time systems. Some may break compatibility with existing software. The correct approach is to understand what each failing option protects against, assess whether that threat applies to your system, and then make an informed decision.

Q5. What is KASLR and why must it be implemented in the kernel rather than userspace?

KASLR (Kernel Address Space Layout Randomization) randomizes the base address at which the kernel is loaded into memory at every boot. Many kernel exploits depend on knowing the fixed address of kernel functions or data structures. With KASLR enabled, attackers cannot assume fixed addresses. It must be done in the kernel because the kernel controls its own virtual address space — userspace processes have no access to or control over how the kernel loads itself. It is controlled by CONFIG_RANDOMIZE_BASE.

← Previous Lecture
EmbeddedPathashala · Free Linux Kernel Programming Course · Ch.2 Lecture 22
 Next Lecture→

Leave a Reply

Your email address will not be published. Required fields are marked *