PR #25586 opened by jayaimzzz URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25586 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25586.patch
`qsv_h264_set_encode_ctrl()` writes the A53 caption SEI's `payloadSize` as one byte (`sei_data[1] = (mfxU8)sei_size;`). Caption data over 255 bytes wraps mod 256, and the SEI message is malformed. In practice this hits the first frame of most live broadcast transcodes. When decoding starts mid-stream, the MPEG-2/H.264 decoder attaches the captions of all frames it couldn't output to the first output frame, which is often 300–1,800 bytes. `trace_headers` then reports `Failed to read unit 3 (type 6): Invalid data`, and Apple's decoder rejects the stream (live TV in Jellyfin on iOS fails to start). libx264 writes these sizes correctly, and `h264_vaapi` doesn't write A53 at all. The fix codes `payloadSize` as `0xFF` bytes plus a final byte (H.264 7.3.2.3.1). The A53 data goes into a temporary buffer, and the `mfxPayload` is allocated once with the variable-length header, so `free_encoder_ctrl()` still frees it with a single `av_freep()`. **Testing** (i5-12400 / UHD 730, iHD 26.3.5, oneVPL 2.17, Linux 6.12): - Payload sizes 0–5000 round-trip through the H.264 `payloadSize` parsing rule (e.g. 255 → `FF 00`, 731 → `FF FF DD`). - The FATE sample `sub/Closedcaption_rollup.m2v` cut mid-stream, `-init_hw_device vaapi=va:/dev/dri/renderD128 -init_hw_device qsv=qs@va -vf format=nv12,hwupload -c:v h264_qsv -a53cc 1`: - before: the first caption SEI declares 219 bytes but carries 731, with 1 `trace_headers` error; - after (this patch on master e4b32e7's parent 914e454): it declares 731 bytes (3-byte size), with 0 errors, 30 caption SEIs, and `movie=…[out+subcc]` extracts the same captions as libx264; - `-a53cc 0` is unchanged. - The same change has run in production in jellyfin-ffmpeg (https://github.com/jellyfin/jellyfin-ffmpeg/pull/800) on live OTA ATSC channels: 0 parse errors, captions intact, and iOS playback works. Code assistance: Claude Code (Claude Opus) helped find the bug, write the patch and run the tests; I reviewed the change and the results. 🤖 Generated with [Claude Code](https://claude.com/claude-code) From e4b32e7c0559eb3d7d05320c62919a557618d93b Mon Sep 17 00:00:00 2001 From: James Horton <[email protected]> Date: Thu, 29 Oct 2026 12:49:48 -0400 Subject: [PATCH] avcodec/qsvenc_h264: fix A53 caption SEI payload size over 255 bytes The SEI payloadSize was written as a single byte, so A53 caption data larger than 255 bytes produced a malformed SEI message (the size wraps modulo 256). This happens on the first frame of most live broadcast transcodes: when decoding starts mid-stream, the decoder attaches the caption data of all frames it could not output to the first frame, typically several hundred bytes or more. Some decoders, Apple's included, reject the resulting stream. Write payloadSize as 0xFF bytes followed by a final byte, as specified in H.264 7.3.2.3.1. Signed-off-by: James Horton <[email protected]> --- libavcodec/qsvenc_h264.c | 31 ++++++++++++++++++++++--------- 1 file changed, 22 insertions(+), 9 deletions(-) diff --git a/libavcodec/qsvenc_h264.c b/libavcodec/qsvenc_h264.c index 7d39338d6ae..65598097ded 100644 --- a/libavcodec/qsvenc_h264.c +++ b/libavcodec/qsvenc_h264.c @@ -27,6 +27,7 @@ #include <mfxvideo.h> #include "libavutil/common.h" +#include "libavutil/mem.h" #include "libavutil/opt.h" #include "avcodec.h" @@ -49,20 +50,32 @@ static int qsv_h264_set_encode_ctrl(AVCodecContext *avctx, if (q->a53_cc && frame) { mfxPayload* payload; mfxU8* sei_data; - size_t sei_size; + void *a53_data; + size_t sei_size, hdr_size; int res; - res = ff_alloc_a53_sei(frame, sizeof(mfxPayload) + 2, (void**)&payload, &sei_size); - if (res < 0 || !payload) + res = ff_alloc_a53_sei(frame, 0, &a53_data, &sei_size); + if (res < 0 || !a53_data) return res; - sei_data = (mfxU8*)(payload + 1); - // SEI header - sei_data[0] = 4; - sei_data[1] = (mfxU8)sei_size; // size of SEI data - // SEI data filled in by ff_alloc_a53_sei + // SEI header: payloadType, then payloadSize as 0xFF bytes followed by + // a final byte < 255 (H.264 7.3.2.3.1). Caption data can be larger + // than 255 bytes, e.g. on the first frame after tuning in mid-stream. + hdr_size = 2 + sei_size / 255; + payload = av_mallocz(sizeof(mfxPayload) + hdr_size + sei_size); + if (!payload) { + av_free(a53_data); + return AVERROR(ENOMEM); + } - payload->BufSize = sei_size + 2; + sei_data = (mfxU8*)(payload + 1); + sei_data[0] = 4; + memset(sei_data + 1, 0xFF, hdr_size - 2); + sei_data[hdr_size - 1] = sei_size % 255; + memcpy(sei_data + hdr_size, a53_data, sei_size); + av_free(a53_data); + + payload->BufSize = hdr_size + sei_size; payload->NumBit = payload->BufSize * 8; payload->Type = 4; payload->Data = sei_data; -- 2.54.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
