path: root/Documentation/userspace-api/media/v4l/dev-decoder.rst

.. SPDX-License-Identifier: GPL-2.0

.. _decoder:

*************************************************
Memory-to-Memory Stateful Video Decoder Interface
*************************************************

A stateful video decoder takes complete chunks of the bytestream (e.g. Annex-B
H.264/HEVC stream, raw VP8/9 stream) and decodes them into raw video frames in
display order. The decoder is expected not to require any additional information
from the client to process these buffers.

Performing software parsing, processing etc. of the stream in the driver in
order to support this interface is strongly discouraged. In case such
operations are needed, use of the Stateless Video Decoder Interface (in
development) is strongly advised.

Conventions and Notations Used in This Document
===============================================

1. The general V4L2 API rules apply if not specified in this document
   otherwise.

2. The meaning of words "must", "may", "should", etc. is as per `RFC
   2119 <https://tools.ietf.org/html/rfc2119>`_.

3. All steps not marked "optional" are required.

4. :c:func:`VIDIOC_G_EXT_CTRLS` and :c:func:`VIDIOC_S_EXT_CTRLS` may be used
   interchangeably with :c:func:`VIDIOC_G_CTRL` and :c:func:`VIDIOC_S_CTRL`,
   unless specified otherwise.

5. Single-planar API (see :ref:`planar-apis`) and applicable structures may be
   used interchangeably with multi-planar API, unless specified otherwise,
   depending on decoder capabilities and following the general V4L2 guidelines.

6. i = [a..b]: sequence of integers from a to b, inclusive, i.e. i =
   [0..2]: i = 0, 1, 2.

7. Given an ``OUTPUT`` buffer A, then A' represents a buffer on the ``CAPTURE``
   queue containing data that resulted from processing buffer A.

.. _decoder-glossary:

Glossary
========

CAPTURE
   the destination buffer queue; for decoders, the queue of buffers containing
   decoded frames; for encoders, the queue of buffers containing an encoded
   bytestream; ``V4L2_BUF_TYPE_VIDEO_CAPTURE`` or
   ``V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE``; data is captured from the hardware
   into ``CAPTURE`` buffers.

client
   the application communicating with the decoder or encoder implementing
   this interface.

coded format
   encoded/compressed video bytestream format (e.g. H.264, VP8, etc.); see
   also: raw format.

coded height
   height for given coded resolution.

coded resolution
   stream resolution in pixels aligned to codec and hardware requirements;
   typically visible resolution rounded up to full macroblocks;
   see also: visible resolution.

coded width
   width for given coded resolution.

coding tree unit
   processing unit of the HEVC codec (corresponds to macroblock units in
   H.264, VP8, VP9),
   can use block structures of up to 64×64 pixels.
   Good at sub-partitioning the picture into variable sized structures.

decode order
   the order in which frames are decoded; may differ from display order if the
   coded format includes a feature of frame reordering; for decoders,
   ``OUTPUT`` buffers must be queued by the client in decode order; for
   encoders ``CAPTURE`` buffers must be returned by the encoder in decode order.

destination
   data resulting from the decode process; see ``CAPTURE``.

display order
   the order in which frames must be displayed; for encoders, ``OUTPUT``
   buffers must be queued by the client in display order; for decoders,
   ``CAPTURE`` buffers must be returned by the decoder in display order.

DPB
   Decoded Picture Buffer; an H.264/HEVC term for a buffer that stores a decoded
   raw frame available for reference in further decoding steps.

EOS
   end of stream.

IDR
   Instantaneous Decoder Refresh; a type of a keyframe in an H.264/HEVC-encoded
   stream, which clears the list of earlier reference frames (DPBs).

keyframe
   an encoded frame that does not reference frames decoded earlier, i.e.
   can be decoded fully on its own.

macroblock
   a processing unit in image and video compression formats based on linear
   block transforms (e.g. H.264, VP8, VP9); codec-specific, but for most of
   popular codecs the size is 16x16 samples (pixels). The HEVC codec uses a
   slightly more flexible processing unit called coding tree unit (CTU).

OUTPUT
   the source buffer queue; for decoders, the queue of buffers containing
   an encoded bytestream; for encoders, the queue of buffers containing raw
   frames; ``V4L2_BUF_TYPE_VIDEO_OUTPUT`` or
   ``V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE``; the hardware is fed with data
   from ``OUTPUT`` buffers.

PPS
   Picture Parameter Set; a type of metadata entity in an H.264/HEVC bytestream.

raw format
   uncompressed format containing raw pixel data (e.g. YUV, RGB formats).

resume point
   a point in the bytestream from which decoding may start/continue, without
   any previous state/data present, e.g.: a keyframe (VP8/VP9) or
   SPS/PPS/IDR sequence (H.264/HEVC); a resume point is required to start decode
   of a new stream, or to resume decoding after a seek.

source
   data fed to the decoder or encoder; see ``OUTPUT``.

source height
   height in pixels for given source resolution; relevant to encoders only.

source resolution
   resolution in pixels of source frames being source to the encoder and
   subject to further cropping to the bounds of visible resolution; relevant to
   encoders only.

source width
   width in pixels for given source resolution; relevant to encoders only.

SPS
   Sequence Parameter Set; a type of metadata entity in an H.264/HEVC bytestream.

stream metadata
   additional (non-visual) information contained inside encoded bytestream;
   for example: coded resolution, visible resolution, codec profile.

visible height
   height for given visible resolution; display height.

visible resolution
   stream resolution of the visible picture, in pixels, to be used for
   display purposes; must be smaller or equal to coded resolution;
   display resolution.

visible width
   width for given visible resolution; display width.

State Machine
=============

.. kernel-render:: DOT
   :alt: DOT digraph of decoder state machine
   :caption: Decoder State Machine

   digraph decoder_state_machine {
       node [shape = doublecircle, label="Decoding"] Decoding;

       node [shape = circle, label="Initialization"] Initialization;
       node [shape = circle, label="Capture\nsetup"] CaptureSetup;
       node [shape = circle, label="Dynamic\nResolution\nChange"] ResChange;
       node [shape = circle, label="Stopped"] Stopped;
       node [shape = circle, label="Drain"] Drain;
       node [shape = circle, label="Seek"] Seek;
       node [shape = circle, label="End of Stream"] EoS;

       node [shape = point]; qi
       qi -> Initialization [ label = "open()" ];

       Initialization -> CaptureSetup [ label = "CAPTURE\nformat\nestablished" ];

       CaptureSetup -> Stopped [ label = "CAPTURE\nbuffers\nready" ];

       Decoding -> ResChange [ label = "Stream\nresolution\nchange" ];
       Decoding -> Drain [ label = "V4L2_DEC_CMD_STOP" ];
       Decoding -> EoS [ label = "EoS mark\nin the stream" ];
       Decoding -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ];
       Decoding -> Stopped [ label = "VIDIOC_STREAMOFF(CAPTURE)" ];
       Decoding -> Decoding;

       ResChange -> CaptureSetup [ label = "CAPTURE\nformat\nestablished" ];
       ResChange -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ];

       EoS -> Drain [ label = "Implicit\ndrain" ];

       Drain -> Stopped [ label = "All CAPTURE\nbuffers dequeued\nor\nVIDIOC_STREAMOFF(CAPTURE)" ];
       Drain -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ];

       Seek -> Decoding [ label = "VIDIOC_STREAMON(OUTPUT)" ];
       Seek -> Initialization [ label = "VIDIOC_REQBUFS(OUTPUT, 0)" ];

       Stopped -> Decoding [ label = "V4L2_DEC_CMD_START\nor\nVIDIOC_STREAMON(CAPTURE)" ];
       Stopped -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ];
   }

Querying Capabilities
=====================

1. To enumerate the set of coded formats supported by the decoder, the
   client may call :c:func:`VIDIOC_ENUM_FMT` on ``OUTPUT``.

   * The full set of supported formats will be returned, regardless of the
     format set on ``CAPTURE``.
   * Check the flags field of :c:type:`v4l2_fmtdesc` for more information
     about the decoder's capabilities with respect to each coded format.
     In particular whether or not the decoder has a full-fledged bytestream
     parser and if the decoder supports dynamic resolution changes.

2. To enumerate the set of supported raw formats, the client may call
   :c:func:`VIDIOC_ENUM_FMT` on ``CAPTURE``.

   * Only the formats supported for the format currently active on ``OUTPUT``
     will be returned.

   * In order to enumerate raw formats supported by a given coded format,
     the client must first set that coded format on ``OUTPUT`` and then
     enumerate formats on ``CAPTURE``.

3. The client may use :c:func:`VIDIOC_ENUM_FRAMESIZES` to detect supported
   resolutions for a given format, passing desired pixel format in
   :c:type:`v4l2_frmsizeenum` ``pixel_format``.

   * Values returned by :c:func:`VIDIOC_ENUM_FRAMESIZES` for a coded pixel
     format will include all possible coded resolutions supported by the
     decoder for given coded pixel format.

   * Values returned by :c:func:`VIDIOC_ENUM_FRAMESIZES` for a raw pixel format
     will include all possible frame buffer resolutions supported by the
     decoder for given raw pixel format and the coded format currently set on
     ``OUTPUT``.

4. Supported profiles and levels for the coded format currently set on
   ``OUTPUT``, if applicable, may be queried using their respective controls
   via :c:func:`VIDIOC_QUERYCTRL`.

Initialization
==============

1. Set the coded format on ``OUTPUT`` via :c:func:`VIDIOC_S_FMT`.

   * **Required fields:**

     ``type``
         a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``.

     ``pixelformat``
         a coded pixel format.

     ``width``, ``height``
         coded resolution of the stream; required only if it cannot be parsed
         from the stream for the given coded format; otherwise the decoder will
         use this resolution as a placeholder resolution that will likely change
         as soon as it can parse the actual coded resolution from the stream.

     ``sizeimage``
         desired size of ``OUTPUT`` buffers; the decoder may adjust it to
         match hardware requirements.

     other fields
         follow standard semantics.

   * **Returned fields:**

     ``sizeimage``
         adjusted size of ``OUTPUT`` buffers.

   * The ``CAPTURE`` format will be updated with an appropriate frame buffer
     resolution instantly based on the width and height returned by
     :c:func:`VIDIOC_S_FMT`.
     However, for coded formats that include stream resolution information,
     after the decoder is done parsing the information from the stream, it will
     update the ``CAPTURE`` format with new values and signal a source change
     event, regardless of whether they match the values set by the client or
     not.

   .. important::

      Changing the ``OUTPUT`` format may change the currently set ``CAPTURE``
      format. How the new ``CAPTURE`` format is determined is up to the decoder
      and the client must ensure it matches its needs afterwards.

2.  Allocate source (bytestream) buffers via :c:func:`VIDIOC_REQBUFS` on
    ``OUTPUT``.

    * **Required fields:**

      ``count``
          requested number of buffers to allocate; greater than zero.

      ``type``
          a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``.

      ``memory``
          follows standard semantics.

    * **Returned fields:**

      ``count``
          the actual number of buffers allocated.

    .. warning::

       The actual number of allocated buffers may differ from the ``count``
       given. The client must check the updated value of ``count`` after the
       call returns.

    Alternatively, :c:func:`VIDIOC_CREATE_BUFS` on the ``OUTPUT`` queue can be
    used to have more control over buffer allocation.

    * **Required fields:**

      ``count``
          requested number of buffers to allocate; greater than zero.

      ``type``
          a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``.

      ``memory``
          follows standard semantics.

      ``format``
          follows standard semantics.

    * **Returned fields:**

      ``count``
          adjusted to the number of allocated buffers.

    .. warning::

       The actual number of allocated buffers may differ from the ``count``
       given. The client must check the updated value of ``count`` after the
       call returns.

3.  Start streaming on the ``OUTPUT`` queue via :c:func:`VIDIOC_STREAMON`.

4.  **This step only applies to coded formats that contain resolution information
    in the stream.** Continue queuing/dequeuing bytestream buffers to/from the
    ``OUTPUT`` queue via :c:func:`VIDIOC_QBUF` and :c:func:`VIDIOC_DQBUF`. The
    buffers will be processed and returned to the client in order, until
    required metadata to configure the ``CAPTURE`` queue are found. This is
    indicated by the decoder sending a ``V4L2_EVENT_SOURCE_CHANGE`` event with
    ``changes`` set to ``V4L2_EVENT_SRC_CH_RESOLUTION``.

    * It is not an error if the first buffer does not contain enough data for
      this to occur. Processing of the buffers will continue as long as more
      data is needed.

    * If data in a buffer that triggers the event is required to decode the
      first frame, it will not be returned to the client, until the
      initializatio