Why Media Pipelines Need the V4L2 Async Framework

Every camera pipeline on an SoC is really a chain of small hardware blocks: an image sensor talks over I2C, a MIPI CSI-2 receiver pulls the bytes in, and an ISP block reshapes the pixels before they reach memory. In driver terms, each of these blocks becomes its own struct v4l2_subdev, and a bridge driver is expected to stitch them together into one working /dev/videoX node.

On older, statically described systems, a bridge driver could simply call into each sub-device driver directly, in a fixed order, during its own probe. That works fine when every driver is guaranteed to be present and ready. It falls apart the moment the system is described by a device tree, because device tree nodes are not probed in any guaranteed order. The I2C sensor driver might not even have its module loaded yet when the CSI bridge driver’s probe function runs. If the bridge driver just gave up at that point, the camera would never come up.

This is exactly the gap the V4L2 async framework fills. Instead of the bridge driver reaching out and calling sub-devices directly, it registers a list of *descriptors* describing which sub-devices it is waiting for, along with a notifier and a set of callbacks. Whenever a sub-device driver successfully probes and registers itself asynchronously, the V4L2 async core checks it against every notifier’s descriptor list. If there’s a match, the core invokes the bridge driver’s callback right then — regardless of which driver happened to load first.

Because probing order is now irrelevant, it does not matter whether the sensor module, the CSI receiver module, or the ISP module finishes loading first — the pipeline only comes together once every piece the bridge driver asked for has shown up.

Graph Binding: Ports and Endpoints

Knowing that a sub-device eventually shows up is only half the problem. The bridge driver also needs to know *which* sub-device connects to *which* physical input on the hardware, especially when a block like an ISP has several camera inputs. A simple parent/child device tree relationship cannot express that kind of wiring, so the kernel borrows a small, standard modelling convention often called graph binding (originally the Open Firmware graph binding, now generalised across device tree and ACPI through the fwnode graph API).

Graph binding introduces two building blocks:

• Port — represents a physical interface on a device, such as a CSI-2 input or a parallel video output. A device can expose more than one port.
• Endpoint — represents one link out of a port. A single port can fan out to more than one endpoint if the same physical interface connects to more than one remote device.

Two endpoints describe opposite ends of the same physical wire. Each endpoint node carries a remote-endpoint phandle pointing at its counterpart, and the pair together forms one link between two ports. This is what lets a bridge driver walk from its own port node, across to the remote endpoint, and discover exactly which sub-device sits on the other side of the cable — all without hardcoding any assumptions about board layout into the driver.

A Graph Binding Example

Here is an original, simplified device tree fragment describing a CSI-2 bridge with a single input port connected to a camera sensor. Node and label names are written from scratch for this course rather than reused from any book:

&csi2_bridge {
    port {
        csi2_in: endpoint {
            remote-endpoint = <&sensor_out>;
            data-lanes = <1 2>;
        };
    };
};

&ep_camsensor_i2c {
    port {
        sensor_out: endpoint {
            remote-endpoint = <&csi2_in>;
        };
    };
};

The two remote-endpoint phandles point at each other, so the kernel graph API can start at either endpoint and resolve straight to the matching one on the other device.

The Modern fwnode Graph API

The original device-tree-only graph API (of_graph_*) still exists, but current V4L2 drivers are expected to use the fwnode-based equivalents so the same driver code works whether the platform describes hardware with device tree or ACPI. The table below lines up the concept from the older API against its modern fwnode counterpart.

Concept	Older of_graph API	Current fwnode graph API
Get first/next endpoint of a device	of_graph_get_next_endpoint()	fwnode_graph_get_next_endpoint()
Get endpoint by port/endpoint id	of_graph_get_endpoint_by_regs()	fwnode_graph_get_endpoint_by_id()
Get remote endpoint from local one	of_graph_get_remote_endpoint()	fwnode_graph_get_remote_endpoint()
Get remote device node itself	of_graph_get_remote_port_parent()	fwnode_graph_get_remote_port_parent()
Parse port/endpoint id + data lanes	manual of_property_read_u32() calls	fwnode_graph_parse_endpoint() filling struct v4l2_fwnode_endpoint

A bridge driver typically walks every endpoint on its own device, resolves each one to a remote fwnode, and hands that fwnode to the V4L2 async notifier as a descriptor to watch for:

struct fwnode_handle *ep = NULL;
struct fwnode_handle *remote;

while ((ep = fwnode_graph_get_next_endpoint(dev_fwnode(dev), ep))) {
    remote = fwnode_graph_get_remote_endpoint(ep);
    if (!remote)
        continue;

    /* 'remote' now identifies the exact sub-device wired to this port */
    fwnode_handle_put(remote);
}

Registering a V4L2 Async Notifier

Once the bridge driver knows which remote fwnodes it cares about, it builds a struct v4l2_async_notifier and adds one descriptor per expected sub-device. The current kernel API for this is:

struct v4l2_async_notifier notifier;

v4l2_async_nf_init(&notifier, &ep_v4l2_dev);

conn = v4l2_async_nf_add_fwnode_remote(&notifier, ep, struct v4l2_async_connection);
if (IS_ERR(conn))
    return PTR_ERR(conn);

notifier.ops = &ep_bridge_notifier_ops;

ret = v4l2_async_nf_register(&notifier);

The three callbacks a bridge driver typically implements on ep_bridge_notifier_ops are:

• bound() — called the moment a specific sub-device matches one of the descriptors; this is where the driver usually saves a pointer to the matched struct v4l2_subdev.
• unbind() — called if a matched sub-device goes away (module unload, hot-unplug); used to clean up references safely.
• complete() — called once every descriptor in the notifier’s list has matched, which is the correct place to register the media device and create the actual video links.

Hands-On: An Original Async Bridge Demo

The demo below builds on the ep_camsensor sub-device driver from earlier lectures in this free Linux device drivers course. Here we add a small original bridge driver, ep_graphbridge, that discovers its endpoint, resolves the remote sub-device, and registers a V4L2 async notifier for it.

#include <linux/module.h>
#include <linux/platform_device.h>
#include <linux/property.h>
#include <media/v4l2-device.h>
#include <media/v4l2-async.h>

struct ep_graphbridge {
    struct v4l2_device v4l2_dev;
    struct v4l2_async_notifier notifier;
    struct v4l2_subdev *remote_sd;
};

static int ep_graphbridge_bound(struct v4l2_async_notifier *notifier,
                                 struct v4l2_subdev *subdev,
                                 struct v4l2_async_connection *asc)
{
    struct ep_graphbridge *gb =
        container_of(notifier, struct ep_graphbridge, notifier);

    gb->remote_sd = subdev;
    dev_info(gb->v4l2_dev.dev, "ep_graphbridge: bound sub-device %s\n",
             subdev->name);
    return 0;
}

static void ep_graphbridge_unbind(struct v4l2_async_notifier *notifier,
                                   struct v4l2_subdev *subdev,
                                   struct v4l2_async_connection *asc)
{
    struct ep_graphbridge *gb =
        container_of(notifier, struct ep_graphbridge, notifier);

    dev_info(gb->v4l2_dev.dev, "ep_graphbridge: unbound sub-device %s\n",
             subdev->name);
    gb->remote_sd = NULL;
}

static int ep_graphbridge_complete(struct v4l2_async_notifier *notifier)
{
    struct ep_graphbridge *gb =
        container_of(notifier, struct ep_graphbridge, notifier);

    dev_info(gb->v4l2_dev.dev, "ep_graphbridge: pipeline complete, ready to stream\n");
    return v4l2_device_register_subdev_nodes(&gb->v4l2_dev);
}

static const struct v4l2_async_notifier_operations ep_graphbridge_notifier_ops = {
    .bound    = ep_graphbridge_bound,
    .unbind   = ep_graphbridge_unbind,
    .complete = ep_graphbridge_complete,
};

static int ep_graphbridge_probe(struct platform_device *pdev)
{
    struct ep_graphbridge *gb;
    struct fwnode_handle *ep, *remote;
    struct v4l2_async_connection *conn;
    int ret;

    gb = devm_kzalloc(&pdev->dev, sizeof(*gb), GFP_KERNEL);
    if (!gb)
        return -ENOMEM;

    ret = v4l2_device_register(&pdev->dev, &gb->v4l2_dev);
    if (ret)
        return ret;

    ep = fwnode_graph_get_next_endpoint(dev_fwnode(&pdev->dev), NULL);
    if (!ep) {
        dev_err(&pdev->dev, "ep_graphbridge: no endpoint found\n");
        return -ENODEV;
    }

    remote = fwnode_graph_get_remote_endpoint(ep);
    fwnode_handle_put(ep);
    if (!remote)
        return -ENODEV;

    v4l2_async_nf_init(&gb->notifier, &gb->v4l2_dev);

    conn = v4l2_async_nf_add_fwnode_remote(&gb->notifier, remote,
                                            struct v4l2_async_connection);
    fwnode_handle_put(remote);
    if (IS_ERR(conn))
        return PTR_ERR(conn);

    gb->notifier.ops = &ep_graphbridge_notifier_ops;

    ret = v4l2_async_nf_register(&gb->notifier);
    if (ret)
        dev_err(&pdev->dev, "ep_graphbridge: notifier register failed: %d\n", ret);

    return ret;
}

static int ep_graphbridge_remove(struct platform_device *pdev)
{
    /* v4l2_async_nf_unregister() + v4l2_async_nf_cleanup() go here */
    return 0;
}

static const struct of_device_id ep_graphbridge_of_match[] = {
    { .compatible = "ep,graphbridge" },
    { }
};
MODULE_DEVICE_TABLE(of, ep_graphbridge_of_match);

static struct platform_driver ep_graphbridge_driver = {
    .probe  = ep_graphbridge_probe,
    .remove = ep_graphbridge_remove,
    .driver = {
        .name = "ep_graphbridge",
        .of_match_table = ep_graphbridge_of_match,
    },
};
module_platform_driver(ep_graphbridge_driver);

MODULE_LICENSE("GPL");
MODULE_DESCRIPTION("EmbeddedPathashala async graph binding demo bridge driver");

Build and Run

$ make -C /lib/modules/$(uname -r)/build M=$(pwd) modules
$ sudo insmod ep_camsensor.ko
$ sudo insmod ep_graphbridge.ko

Expected dmesg trace once both drivers are loaded, regardless of which one loads first:

$ dmesg | tail
[ 122.001233] ep_camsensor: probed sub-device on i2c bus 2
[ 122.004501] ep_graphbridge: bound sub-device ep_camsensor
[ 122.004622] ep_graphbridge: pipeline complete, ready to stream
[ 122.004890] video4linux video0: Registered as /dev/video0

Notice that the “bound” and “pipeline complete” messages only appear after both modules exist in the kernel — try reversing the insmod order and the sequence still ends the same way, which is the whole point of the V4L2 async framework.

Real-World Use Cases

• Mobile and embedded camera pipelines — sensor, CSI-2 receiver, and ISP are usually separate drivers from separate vendors, loaded in unpredictable order.
• Multi-camera boards — a single ISP with several input ports uses graph binding to know exactly which physical connector maps to which sensor.
• HDMI capture and TV tuner cards — bridge chips wait asynchronously for tuner or demodulator sub-devices that may load from separate kernel modules.

Common Mistakes and Troubleshooting

Mistake	Why it breaks	Fix
Forgetting fwnode_handle_put() after graph lookups	Leaks a reference on every probe, eventually pins nodes in memory	Always pair a get with a put, including on early-return error paths
Registering the notifier before calling v4l2_async_nf_init()	The internal descriptor list is never initialised, causing a crash on add	Always call v4l2_async_nf_init() first
Doing pipeline/media-link setup inside bound()	Not every expected sub-device has matched yet at that point	Do final pipeline setup in complete() instead
Hardcoding the remote sub-device’s name or i2c address	Defeats the purpose of graph binding and breaks on other boards	Resolve everything through the fwnode graph, not string/address matching

Best Practices

• Always release fwnode references you acquire — treat every _get_ call as needing a matching fwnode_handle_put().
• Prefer the fwnode-based graph API over the older of_graph_* calls so the driver also works on ACPI-described platforms.
• Keep bound() and unbind() lightweight and symmetrical — anything allocated in bound() should be freed in unbind().
• Do heavier setup, like registering media links, only inside complete() once the whole pipeline is present.

Summary and Key Takeaways

• The V4L2 async framework exists because device-tree based systems probe drivers in an unpredictable order.
• Graph binding uses ports and endpoints, linked by remote-endpoint phandles, to describe exactly how hardware blocks are wired together.
• Modern drivers should use the fwnode graph API (fwnode_graph_*) rather than the older device-tree-only of_graph_* calls.
• A bridge driver registers a notifier with descriptors for the sub-devices it expects, and implements bound(), unbind(), and complete() to react as those sub-devices appear.

Conclusion

The V4L2 async framework, paired with graph binding, is what makes it possible for a camera pipeline built from several independently-probed drivers to reliably come together into one working video device. Once you can read a graph binding correctly and wire up a notifier with bound()/unbind()/complete(), you have the foundation needed for the next step in this free Linux kernel development course: the Linux media controller framework, which wraps this whole pipeline into a single, configurable media device.

Frequently Asked Questions

What problem does the V4L2 async framework actually solve?

It lets a bridge driver wait for sub-device drivers to probe in any order, instead of requiring them all to be ready synchronously during the bridge driver’s own probe function.

What is the difference between a port and an endpoint in graph binding?

A port represents a physical interface on a device, while an endpoint represents one specific link leaving that port to a remote device. A port can contain more than one endpoint.

Should new drivers use of_graph_* or fwnode_graph_* functions?

New drivers should use the fwnode_graph_* API, since it works across both device tree and ACPI-described platforms, while of_graph_* only works with device tree.

When should pipeline setup happen — in bound() or complete()?

Pipeline and media-link setup belongs in complete(), because that callback only fires after every sub-device the notifier is waiting for has matched.

What happens if a matched sub-device’s driver is unloaded later?

The V4L2 async core calls the notifier’s unbind() callback so the bridge driver can drop its reference and mark that part of the pipeline as unavailable.

Is graph binding specific to V4L2, or used elsewhere in the kernel?

Graph binding (ports and endpoints) is a general device tree/fwnode convention. V4L2 is one of its biggest consumers, but the same pattern shows up wherever hardware blocks need graph-style wiring described.

Do I need the media controller framework to use V4L2 async?

No — the async framework only handles matching sub-devices. The media controller framework, covered in the next lecture, is what represents the resulting pipeline as a configurable graph of entities.

Where can I practice this if I don’t have camera hardware?

You can build both the sub-device and bridge drivers as platform drivers bound through a device tree overlay on any board, or test the fwnode graph calls against dummy compatible strings without real video hardware attached.