PREV_LEC  |  NEXT_LEC

What You Will Learn

• How a complete async-registration flow ties together notifier init, sub-device matching, and completion in one place
• Why routing video data through multiple on-chip IP blocks breaks the simple bridge-plus-sensor model
• The real engineering problems SoC vendors hit before the media controller framework existed
• A first look at the entity/pad/link abstraction that the rest of this chapter builds on

Prerequisites

You should already be comfortable with struct v4l2_async_notifier, fwnode graph parsing, and the bound/complete/unbind callback flow covered earlier in this free linux device drivers course. If any of that feels shaky, revisit the earlier V4L2 async lectures in this series before continuing.

Recap: A Complete Async Binding Flow

Before moving on, let’s put every piece of the async model together in one original example, so the transition to the media controller framework makes sense. Imagine a bridge driver called ep_bridge that talks to one off-chip camera sensor over an MIPI CSI-2 endpoint described in the device tree.

static int ep_bridge_probe(struct platform_device *pdev)
{
    struct ep_bridge_dev *epb;
    int ret;

    epb = devm_kzalloc(&pdev->dev, sizeof(*epb), GFP_KERNEL);
    if (!epb)
        return -ENOMEM;

    epb->dev = &pdev->dev;
    v4l2_async_nf_init(&epb->notifier, &epb->v4l2_dev);

    ret = ep_bridge_parse_fwnode(epb);
    if (ret)
        return ret;

    epb->notifier.ops = &ep_bridge_notifier_ops;

    ret = v4l2_async_nf_register(&epb->notifier);
    if (ret) {
        v4l2_async_nf_cleanup(&epb->notifier);
        return ret;
    }

    return 0;
}

The ep_bridge_parse_fwnode() helper walks the endpoint graph using fwnode_graph_get_next_endpoint(), resolves the remote sub-device fwnode, and calls v4l2_async_nf_add_fwnode_remote() to register a match entry. Once the matching sensor driver probes, .bound() fires, and once every expected sub-device is bound, .complete() fires and the bridge finally calls video_register_device() to expose /dev/videoX to user space.

Where This Model Breaks Down

This flow works beautifully when the picture is “one bridge, one sensor, one straight-line data path.” But real SoCs rarely look like that. Consider a chip with a CSI-2 receiver, a hardware scaler, and a separate hardware image converter, all sitting between the sensor and the final capture buffer. Now ask yourself three questions the async model was never designed to answer:

• Should the raw sensor frame go straight to the scaler, or through the image converter first?
• Can user space switch that routing at runtime, without reloading any driver?
• How does user space even discover that a scaler and an image converter exist on this particular chip, since not every board wires them the same way?

The async notifier only knows about “bound” and “not yet bound.” It has no concept of a data path, no concept of “this output feeds that input,” and no API for a user-space application to query or change routing. Early SoC vendors tried to paper over this using custom sysfs attributes or private ioctl numbers. That approach created real problems:

Problem	Why It Hurt
No unified user-space API	Every vendor’s routing controls looked different, so no common application (like a camera stack) could be written once for all hardware
Vendor lock-in inside the kernel	Custom ioctls and sysfs knobs duplicated logic across drivers instead of sharing one framework
No topology discovery	User space had no generic way to ask “what blocks exist on this chip and how are they wired?”
Fragile ABI	Private ioctls are not covered by the stable V4L2 ABI guarantees, so they could break across kernel versions

The Missing Piece: A Graph Abstraction

What was actually needed was a generic, reusable way to describe any hardware pipeline as a graph: a set of processing blocks, each with input and output connection points, wired together by explicit links that can be enabled, disabled, or in some cases reconfigured at runtime — all exposed through one consistent, documented user-space API. That is precisely what the Linux media controller framework provides, and it is the subject of the rest of this chapter in this free embedded systems course.

Three terms will anchor everything from here on, and it’s worth memorizing them now because every later lecture in this chapter builds directly on top of them:

• Entity — any functional block in the pipeline: a sensor, a CSI receiver, a scaler, or the final capture device node itself.
• Pad — a connection point on an entity, either a sink (data comes in) or a source (data goes out).
• Link — a point-to-point connection joining one entity’s source pad to another entity’s sink pad.

Common Mistakes To Avoid

• Trying to solve multi-block routing with private ioctls instead of registering entities with the media controller — this is exactly the anti-pattern the framework was built to eliminate.
• Assuming the media controller replaces V4L2 async binding — it does not; the two work together. Async binding still discovers and connects sub-devices, the media controller then describes and controls the data path between them.
• Forgetting that a single-sensor, single-bridge driver may never need the media controller at all — it earns its complexity only when there is real topology to describe.

Best Practices

• Only pull in CONFIG_MEDIA_CONTROLLER support when your hardware genuinely has more than one routable block — don’t add unnecessary complexity to a simple driver.
• Keep async sub-device binding and media controller entity registration as two separate, clearly documented steps in your driver’s probe/complete sequence.
• Use media-ctl from userspace-mediactl tools during development to visualize the graph you registered — it will save you hours of debugging.

Summary

Async sub-device binding solves discovery and probe-order problems, but it says nothing about how data actually flows once every sub-device is bound. The moment a chip has more than one routable processing block, you need a graph model: entities, pads, and links. That is the exact gap the media controller framework fills, and starting with the next lecture in this free linux kernel development course, we will build that model piece by piece with original, current-kernel code.

FAQ

Does every V4L2 driver need the media controller framework?

No. A simple USB webcam driver with one sensor and one capture node usually doesn’t need it. It becomes necessary once a chip has multiple routable blocks whose connections need to be discovered or changed.

Does the media controller replace V4L2 async notifiers?

No, they solve different problems and work together. Async notifiers handle discovering and binding sub-devices; the media controller describes and manages the data path between the entities once they exist.

What tool lets me inspect a registered media graph from user space?

The media-ctl command-line tool, part of the v4l-utils / media-ctl userspace packages, can print the full entity/pad/link topology and change link states.

Can links between entities be changed while streaming?

Some can. Whether a link’s state can change at runtime depends on flags set by the driver, which we cover in detail later in this chapter.

Is the media controller specific to camera drivers?

No. It was born out of camera/video pipeline needs but is a generic graph framework, also used by some audio and DVB-related complex devices.

PREV_LEC  |  NEXT_LEC