On 6/28/19 8:09 PM, Nicolas Dufresne wrote:
> Le vendredi 28 juin 2019 à 16:34 +0200, Hans Verkuil a écrit :
>> Hi all,
>>
>> I hope I Cc-ed everyone with a stake in this issue.
>>
>> One recurring question is how a stateful encoder fills buffers and how a
>> stateful
>> decoder consumes buffers.
>>
>> The most generic case is that an encoder produces a bitstream and just fills
>> each
>> CAPTURE buffer to the brim before continuing with the next buffer.
>>
>> I don't think there are drivers that do this, I believe that all drivers just
>> output a single compressed frame. For interlaced formats I understand it is
>> either
>> one compressed field per buffer, or two compressed fields per buffer (this is
>> what I heard, I don't know if this is true).
>>
>> In any case, I don't think this is specified anywhere. Please correct me if
>> I am
>> wrong.
>>
>> The latest stateful codec spec is here:
>>
>> https://hverkuil.home.xs4all.nl/codec-api/uapi/v4l/dev-mem2mem.html
>>
>> Assuming what I described above is indeed the case, then I think this should
>> be documented. I don't know enough if a flag is needed somewhere to describe
>> the behavior for interlaced formats, or can we leave this open and have
>> userspace
>> detect this?
>>
>>
>> For decoders it is more complicated. The stateful decoder spec is written
>> with
>> the assumption that userspace can just fill each OUTPUT buffer to the brim
>> with
>> the compressed bitstream. I.e., no need to split at frame or other
>> boundaries.
>>
>> See section 4.5.1.7 in the spec.
>>
>> But I understand that various HW decoders *do* have limitations. I would
>> really
>> like to know about those, since that needs to be exposed to userspace
>> somehow.
>>
>> Specifically, the venus decoder needs to know the resolution of the coded
>> video
>> beforehand and it expects a single frame per buffer (how does that work for
>> interlaced formats?).
>>
>> Such requirements mean that some userspace parsing is still required, so
>> these
>> decoders are not completely stateful.
>>
>> Can every codec author give information about their decoder/encoder?
>>
>> I'll start off with my virtual codec driver:
>>
>> vicodec: the decoder fully parses the bitstream. The encoder produces a
>> single
>> compressed frame per buffer. This driver doesn't yet support interlaced
>> formats,
>> but when that is added it will encode one field per buffer.
>>
>> Let's see what the results are.
>
> Hans though a summary of what existing userspace expects / assumes
> would be nice.
>
> GStreamer:
> ==========
> Encodes:
> fwht, h263, h264, hevc, jpeg, mpeg4, vp8, vp9
> Decodes:
> fwht, h263, h264, hevc, jpeg, mpeg2, mpeg4, vc1, vp8, vp9
>
> It assumes that each encoded v4l2_buffer contains exactly one frame
> (any format, two fields for interlaced content). It may still work
> otherwise, but some issues will appear, timestamp shift, lost of
> metadata (e.g. timecode, cc, etc.).
When you say 'each encoded v4l2_buffer contains exactly on frame',
does that include H.264 SPS/PPS headers? Or are those passed in
a separate v4l2_buffer? Ditto for FFMPEG.
Regards,
Hans
>
> FFMpeg:
> =======
> Encodes:
> h263, h264, hevc, mpeg4, vp8
> Decodes:
> h263, h264, hevc, mpeg2, mpeg4, vc1, vp8, vp9
>
> Similarly to GStreamer, it assumes that one AVPacket will fit one
> v4l2_buffer. On the encoding side, it seems less of a problem, but they
> don't fully implement the FFMPEG CODEC API for frame matching, which I
> suspect would create some ambiguity if it was.
>
> Chromium:
> =========
> Decodes:
> H264, VP8, VP9
> Encodes:
> H264
>
> That is the code I know the less, but the encoder does not seem
> affected by the nal alignment. The keyframe flag and timestamps seems
> to be used and are likely expected to correlate with the input, so I
> suspect that there exist some possible ambiguity if the output is not
> full frame. For the decoder, I'll have to ask someone else to comment,
> the code is hard to follow and I could not get to the place where
> output buffers are filled. I thought the GStreamer code was tough, but
> this is quite similarly a mess.
>
> Nicolas
>
>
>
>
>
>
