PREV_LEC  |  NEXT_LEC

What You Will Learn

• What media_entity_operations is for and when a driver actually needs it
• The current-kernel v4l2_subdev_pad_ops callback table for negotiating the media bus format
• Why struct v4l2_subdev_pad_config was replaced by struct v4l2_subdev_state, and how to migrate old code
• A full original sub-device implementing get_fmt/set_fmt on the current API

Prerequisites

You should already understand entities, pads, and links from the previous two lectures in this free linux device drivers course, plus general V4L2 sub-device basics from earlier in this chapter.

media_entity_operations: Reacting To Link Changes

An entity can optionally provide link-related callbacks that the media framework invokes on link creation and validation:

struct media_entity_operations {
    int (*get_fwnode_pad)(struct fwnode_endpoint *endpoint);
    int (*link_setup)(struct media_entity *entity,
                       const struct media_pad *local,
                       const struct media_pad *remote,
                       u32 flags);
    int (*link_validate)(struct media_link *link);
};

• get_fwnode_pad — maps a firmware-described endpoint to a pad number on this entity; returns the pad index or a negative error code. Optional.
• link_setup — called whenever a link touching this entity changes state (enabled/disabled). Returning an error here cancels the requested link change. Optional.
• link_validate — called by media_pipeline_start() for every link the entity participates in, to confirm the link is actually usable before streaming begins. If a driver doesn’t implement this, sub-devices fall back to v4l2_subdev_link_validate_default(), which checks that the source and sink pad width, height, and media bus code all agree — a mismatch here returns an error and streaming refuses to start.

Most simple drivers never need to implement this structure at all — it exists for entities that genuinely need custom validation logic beyond the default width/height/code check.

The Concept Of A Media Bus

Before two entities can actually exchange data, their pad configurations need to agree — same resolution, same pixel format, on both sides of the link. This agreement happens over what the framework calls a media bus: the physical or logical connection (MIPI CSI-2, parallel, BT.656, and so on) carrying frames between blocks. User-space applications are responsible for configuring compatible formats across the whole pipeline; the kernel checks for mismatches at VIDIOC_STREAMON time using exactly the link_validate mechanism above.

v4l2_subdev_pad_ops: Negotiating The Format

A sub-device that participates in the media framework implements pad-level getters and setters through struct v4l2_subdev_pad_ops. This is where the biggest API change from older books shows up. On current kernels, the second parameter to every one of these callbacks is struct v4l2_subdev_state *state, not the older struct v4l2_subdev_pad_config *cfg you may see in outdated references:

struct v4l2_subdev_pad_ops {
    int (*init_cfg)(struct v4l2_subdev *sd,
                     struct v4l2_subdev_state *state);
    int (*enum_mbus_code)(struct v4l2_subdev *sd,
                           struct v4l2_subdev_state *state,
                           struct v4l2_subdev_mbus_code_enum *code);
    int (*enum_frame_size)(struct v4l2_subdev *sd,
                            struct v4l2_subdev_state *state,
                            struct v4l2_subdev_frame_size_enum *fse);
    int (*get_fmt)(struct v4l2_subdev *sd,
                    struct v4l2_subdev_state *state,
                    struct v4l2_subdev_format *format);
    int (*set_fmt)(struct v4l2_subdev *sd,
                    struct v4l2_subdev_state *state,
                    struct v4l2_subdev_format *format);
#ifdef CONFIG_MEDIA_CONTROLLER
    int (*link_validate)(struct v4l2_subdev *sd,
                          struct media_link *link,
                          struct v4l2_subdev_format *source_fmt,
                          struct v4l2_subdev_format *sink_fmt);
#endif
};

• init_cfg — initializes the “try” format state to sane defaults; on current kernels this is the right place to seed the try-format storage that state now owns.
• enum_mbus_code — backs the VIDIOC_SUBDEV_ENUM_MBUS_CODE ioctl, enumerating which pixel formats this pad currently supports.
• enum_frame_size — backs VIDIOC_SUBDEV_ENUM_FRAME_SIZE, enumerating supported resolutions.
• get_fmt / set_fmt — back VIDIOC_SUBDEV_G_FMT / VIDIOC_SUBDEV_S_FMT, reading or applying the pad’s current media bus format.
• link_validate (pad_ops version) — used specifically by the media controller to check whether a link belonging to a pipeline is safe to stream through; this is what backs the default width/height/code check mentioned earlier.

Why v4l2_subdev_pad_config Was Replaced

Older kernel versions (and most existing books) pass a bare struct v4l2_subdev_pad_config *cfg holding just a “try” format and a “try” crop rectangle per pad. Current kernels replace this with struct v4l2_subdev_state, which is richer: it centralizes try-format, try-crop, try-compose, and (on kernels supporting streams) per-stream format state behind a single locked structure, accessed through helpers like v4l2_subdev_state_get_format() instead of reaching into a raw cfg->try_fmt field. If you’re reading an older tutorial that still uses cfg, mentally substitute state and the corresponding accessor helper — the underlying idea (a place to stage a “would-be” format before committing it) hasn’t changed, only the container has.

The Format Structures Themselves

struct v4l2_subdev_format {
    __u32 which;   /* V4L2_SUBDEV_FORMAT_TRY or V4L2_SUBDEV_FORMAT_ACTIVE */
    __u32 pad;
    struct v4l2_mbus_framefmt format;
};

struct v4l2_mbus_framefmt {
    __u32 width;
    __u32 height;
    __u32 code;
    __u32 field;
    __u32 colorspace;
};

which tells the driver whether user space is asking about the “try” format (a proposal, never actually applied to hardware) or the “active” format (what’s really configured right now). pad identifies which pad on the sub-device the request applies to. format carries the actual media bus format: resolution, pixel code, field order, and colorspace.

Original Example: A Sub-Device With get_fmt/set_fmt

Here’s an original sub-device driver, ep_camsensor2, implementing format negotiation on the current API for a single source pad.

struct ep_camsensor2 {
    struct v4l2_subdev sd;
    struct media_pad pad;
    struct v4l2_mbus_framefmt active_fmt;
};

static int ep_cs2_get_fmt(struct v4l2_subdev *sd,
                           struct v4l2_subdev_state *state,
                           struct v4l2_subdev_format *format)
{
    struct ep_camsensor2 *cs = container_of(sd, struct ep_camsensor2, sd);

    if (format->which == V4L2_SUBDEV_FORMAT_TRY) {
        format->format = *v4l2_subdev_state_get_format(state, format->pad);
        return 0;
    }

    format->format = cs->active_fmt;
    return 0;
}

static int ep_cs2_set_fmt(struct v4l2_subdev *sd,
                           struct v4l2_subdev_state *state,
                           struct v4l2_subdev_format *format)
{
    struct ep_camsensor2 *cs = container_of(sd, struct ep_camsensor2, sd);

    /* Clamp to what this original sensor model actually supports */
    format->format.width  = clamp_t(u32, format->format.width, 320, 1920);
    format->format.height = clamp_t(u32, format->format.height, 240, 1080);
    format->format.code   = MEDIA_BUS_FMT_UYVY8_2X8;
    format->format.field  = V4L2_FIELD_NONE;

    if (format->which == V4L2_SUBDEV_FORMAT_TRY) {
        *v4l2_subdev_state_get_format(state, format->pad) = format->format;
    } else {
        cs->active_fmt = format->format;
    }

    return 0;
}

static const struct v4l2_subdev_pad_ops ep_cs2_pad_ops = {
    .get_fmt = ep_cs2_get_fmt,
    .set_fmt = ep_cs2_set_fmt,
};

Querying this from user space with v4l2-ctl against the sub-device node shows the negotiated format:

$ v4l2-ctl -d /dev/v4l-subdev0 --get-subdev-fmt pad=0
Format Video Capture:
        Width/Height      : 1920/1080
        Mediabus Code      : UYVY8_2X8 (0x2007)
        Field              : None
        Colorspace         : sRGB

Common Mistakes To Avoid

• Copying old code that dereferences cfg->try_fmt directly — on current kernels that field doesn’t exist on the parameter you’re given; use v4l2_subdev_state_get_format() against the state parameter instead.
• Applying a set_fmt request to real hardware registers even when which == V4L2_SUBDEV_FORMAT_TRY — try-format requests must never touch actual hardware state.
• Implementing a custom link_validate that’s looser than the default width/height/code check without a genuinely good reason — it exists to catch real pipeline misconfigurations.

Best Practices

• Always implement both get_fmt and set_fmt together — a pad that can be queried but never configured, or vice versa, breaks most generic camera applications.
• Use init_cfg to seed a sensible default try-format so v4l2-ctl and camera stacks don’t see garbage on first query.
• Keep pixel-format clamping logic (like the width/height clamp above) centralized in one place rather than duplicated across every callback.

Summary

This lecture closed the loop on the media controller framework: media_entity_operations lets an entity react to and validate link changes, and v4l2_subdev_pad_ops — now built around struct v4l2_subdev_state rather than the older v4l2_subdev_pad_config — handles negotiating the actual media bus format across a link. Together with entities, pads, and links from the earlier lectures, you now have the complete picture needed to build a real multi-block camera pipeline driver. That wraps up this chapter of the free linux kernel development course — the next chapter moves on to a new subsystem entirely.

FAQ

What replaced struct v4l2_subdev_pad_config in current kernels?

struct v4l2_subdev_state, accessed through helpers like v4l2_subdev_state_get_format() instead of a raw cfg->try_fmt field.

What’s the difference between a try format and an active format?

A try format is a proposal that is never applied to real hardware, used for negotiation. An active format is what’s actually configured on the hardware right now.

What happens if link_validate isn’t implemented by a sub-device?

It falls back to v4l2_subdev_link_validate_default(), which checks that source and sink pad width, height, and media bus code match.

When does link mismatch checking actually happen?

At VIDIOC_STREAMON time, when media_pipeline_start() walks every link in the pipeline and calls link_validate on each.

Is media_entity_operations mandatory for every sub-device driver?

No, it’s optional. Most simple drivers rely entirely on the default link_validate behavior and never implement it.

PREV_LEC  |  NEXT_LEC