What You Will Learn

• Why a video pipeline needs an explicit format negotiation step at all
• The difference between a TRY format and an ACTIVE format
• How struct v4l2_subdev_state replaced the older per-pad config structure
• Implementing get_fmt and set_fmt pad operations on a current kernel
• Where and when a driver should call media_device_register()
• A complete, original demo sub-device you can build and test yourself

Prerequisites

• Comfortable with entities, pads, and links from the earlier lectures in this free embedded linux course
• Basic familiarity with struct v4l2_subdev and struct media_entity
• A Linux kernel build environment (native or cross-compiled) to try the demo module

Why Format Negotiation Exists

A camera sensor can usually output several resolutions and several pixel encodings — raw Bayer, YUV, RGB and so on. The very next entity in the chain, say a MIPI receiver or a video mux, only accepts a subset of those combinations. Without a negotiation step, a userspace application would have to know the internal capabilities of every hardware block on every board, which does not scale. V4L2 solves this with a pad-level format API: every pad on every sub-device exposes a get_fmt and set_fmt operation, and userspace (or a pipeline manager like media-ctl) walks the graph pad by pad, propagating a compatible format from the sensor all the way to the capture node.

TRY vs ACTIVE: Two Copies of the Same Format

Every pad format request carries a which field describing whether the caller wants to touch the device’s real, currently streaming configuration, or a private scratch copy used only to test “what if” combinations.

This two-state design is what lets a userspace configuration tool probe several candidate resolutions without disturbing a pipeline that might already be streaming on another file descriptor.

The Modern Subdev State API

Older kernels stored the try format in a structure called v4l2_subdev_pad_config, allocated per file handle. Current mainline kernels replaced this with a single, richer object: struct v4l2_subdev_state. This structure holds the try format (and try crop/compose rectangles) for every pad, plus a lock, and — on sub-devices that opt in — it doubles as the “active state” that mirrors the real hardware configuration. The subsystem manages allocation and locking for you once the driver calls v4l2_subdev_init_finalize() during probe.

Implementing get_fmt and set_fmt

Below is an original demo sub-device, ep_camsensor, extended from earlier lectures in this course with real pad format support. It is intentionally simple: one source pad, one supported pixel code, and a small table of supported resolutions.

#include <media/v4l2-subdev.h>

struct ep_camsensor {
    struct v4l2_subdev sd;
    struct v4l2_mbus_framefmt fmt;
};

static inline struct ep_camsensor *to_ep_cam(struct v4l2_subdev *sd)
{
    return container_of(sd, struct ep_camsensor, sd);
}

static int ep_cam_get_fmt(struct v4l2_subdev *sd,
                           struct v4l2_subdev_state *state,
                           struct v4l2_subdev_format *fmt)
{
    struct ep_camsensor *cam = to_ep_cam(sd);
    struct v4l2_mbus_framefmt *mf;

    if (fmt->which == V4L2_SUBDEV_FORMAT_TRY)
        mf = v4l2_subdev_state_get_format(state, fmt->pad);
    else
        mf = &cam->fmt;

    fmt->format = *mf;
    return 0;
}

static int ep_cam_set_fmt(struct v4l2_subdev *sd,
                           struct v4l2_subdev_state *state,
                           struct v4l2_subdev_format *fmt)
{
    struct ep_camsensor *cam = to_ep_cam(sd);
    struct v4l2_mbus_framefmt *mf;

    /* Clamp to the only mode this demo sensor supports */
    fmt->format.code = MEDIA_BUS_FMT_UYVY8_2X8;
    fmt->format.width = clamp_t(u32, fmt->format.width, 640, 1920);
    fmt->format.height = clamp_t(u32, fmt->format.height, 480, 1080);
    fmt->format.field = V4L2_FIELD_NONE;

    if (fmt->which == V4L2_SUBDEV_FORMAT_TRY)
        mf = v4l2_subdev_state_get_format(state, fmt->pad);
    else
        mf = &cam->fmt;

    *mf = fmt->format;
    return 0;
}

static const struct v4l2_subdev_pad_ops ep_cam_pad_ops = {
    .get_fmt = ep_cam_get_fmt,
    .set_fmt = ep_cam_set_fmt,
};

Notice that the driver no longer needs a separate init_cfg callback to seed the try format. Instead, the sub-device calls v4l2_subdev_init_finalize() once in probe(), and the core allocates and initializes the state for every pad automatically:

static int ep_camsensor_probe(struct i2c_client *client)
{
    struct ep_camsensor *cam;
    int ret;

    cam = devm_kzalloc(&client->dev, sizeof(*cam), GFP_KERNEL);
    if (!cam)
        return -ENOMEM;

    v4l2_i2c_subdev_init(&cam->sd, client, &ep_cam_subdev_ops);
    cam->sd.flags |= V4L2_SUBDEV_FL_HAS_DEVNODE;

    cam->fmt.code = MEDIA_BUS_FMT_UYVY8_2X8;
    cam->fmt.width = 1280;
    cam->fmt.height = 720;
    cam->fmt.field = V4L2_FIELD_NONE;

    ret = v4l2_subdev_init_finalize(&cam->sd);
    if (ret)
        return ret;

    return v4l2_async_register_subdev(&cam->sd);
}

Registering the Media Device

Once every sub-device in the graph has bound, the bridge driver’s async notifier .complete callback fires. That is exactly the moment to call media_device_register() — before this point the topology is incomplete and userspace has nothing meaningful to query yet. Continuing the ep_bridge driver from an earlier lecture:

static int ep_bridge_notifier_complete(struct v4l2_async_notifier *notifier)
{
    struct ep_bridge *bridge =
        container_of(notifier, struct ep_bridge, notifier);
    int ret;

    ret = v4l2_device_register_subdev_nodes(&bridge->v4l2_dev);
    if (ret)
        return ret;

    return media_device_register(&bridge->mdev);
}

static const struct v4l2_async_notifier_operations ep_bridge_notifier_ops = {
    .bound    = ep_bridge_notifier_bound,
    .complete = ep_bridge_notifier_complete,
};

On successful registration the kernel creates a new /dev/mediaN character device with a dynamically assigned minor number. From this point on, tools like media-ctl can open that node and walk the full pipeline — which is exactly the subject of the next lecture in this free linux device drivers course.

Format Negotiation Flow

[ Sensor pad set_fmt ] → [ Receiver pad set_fmt ] → [ Capture pad set_fmt ] → media_device_register()

Common Mistakes and Troubleshooting

• Forgetting v4l2_subdev_init_finalize(): without it, v4l2_subdev_state_get_format() dereferences an uninitialized state and the kernel oopses on the first TRY format request.
• Applying a TRY format to hardware: a driver that writes registers inside set_fmt regardless of the which field will corrupt a running stream the moment a userspace tool probes an alternate resolution.
• Registering the media device too early: calling media_device_register() from probe() instead of the notifier’s .complete callback exposes a topology with entities still missing.
• Ignoring driver-clamped values: userspace applications must always re-read the format after a set_fmt call, since the driver is free to snap the request to the nearest supported mode.

Best Practices for Subdev Format Negotiation

• Always honor the which field and never touch real hardware for a TRY request
• Keep the ACTIVE format cached in your driver-private structure for fast reads
• Use v4l2_subdev_init_finalize() instead of hand-rolling state allocation
• Return a driver-supported format rather than an error when clamping an out-of-range request
• Register the media device only after the notifier confirms the graph is complete

Real-World Use Case

On a typical embedded camera board, a userspace configuration script calls set_fmt on the sensor pad first, reads back the clamped result, then propagates that exact format to every downstream pad using the same call on each entity in order. This “sensor-first” propagation pattern is what tools such as media-ctl automate, and it is the reason format negotiation and media device registration belong in the same lecture — one is meaningless without the other actually being reachable from userspace.

Summary and Key Takeaways

• V4L2 subdev format negotiation lets independent hardware blocks agree on a common image format pad by pad
• struct v4l2_subdev_state is the modern replacement for the old per-file try-format config
• get_fmt/set_fmt must respect the TRY vs ACTIVE distinction at all times
• media_device_register() belongs in the async notifier’s .complete callback

Conclusion

Subdev format negotiation is the quiet, unglamorous plumbing that makes every camera pipeline on Linux actually work end to end. Once you understand that every pad speaks the same TRY/ACTIVE language, and that the media device only becomes visible to userspace after registration, the rest of the media controller framework — which we explore hands-on in the next lecture of this free linux kernel development course — starts to feel a lot less mysterious.

Frequently Asked Questions