PR #25373 opened by roffe URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25373 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25373.patch
# Summary of changes Seeking in open-GOP H.264 (and HEVC with CRA keyframes) on VideoToolbox shows broken blocks until the next IDR. avcodec_flush_buffers() never reached the hwaccel with frame threads, and VideoToolbox keeps the session's reference frames across a flush anyway. Two patches: 1. pthread_frame: a hwaccel without HWACCEL_CAP_THREAD_SAFE keeps its state in the stash between frames, so ff_thread_flush() never found a flush callback to run. Call it on the stashed state. 2. videotoolbox: give the h264 and hevc hwaccels a flush callback, and mark the next sample with kCMSampleBufferAttachmentKey_ResetDecoderBeforeDecoding unless its picture is an IDR (or BLA for HEVC). That is what Apple advised Chromium for the same problem, https://crbug.com/451536366. Tested on an M2 Pro with a 3-ref open-GOP H.264 file where 66 of 118 keyframes are recovery points: the blocks are gone, and a seek that lands on such a keyframe costs about 3.5 ms more than a plain flush. Patch 1 was checked on Linux with the Vulkan H.264 hwaccel and 2 frame threads, where the flush callback never ran before and runs now. fate is unchanged. >From f79e320dc6742e3a6cfcbbeffff7527c33699b43 Mon Sep 17 00:00:00 2001 From: roffe <[email protected]> Date: Thu, 22 Oct 2026 21:59:09 +0200 Subject: [PATCH 1/2] avcodec/pthread_frame: call the stashed hwaccel's flush callback Between frames, a hwaccel without HWACCEL_CAP_THREAD_SAFE keeps its state in the frame thread context's stash, and every worker context has its hwaccel pointers cleared. ff_thread_flush() flushes the worker contexts, which is where the decoders look for the hwaccel's flush callback, so with frame threading the callback never ran for such a hwaccel. Call it on the stashed state. Signed-off-by: roffe <[email protected]> --- libavcodec/pthread_frame.c | 14 ++++++++++++++ 1 file changed, 14 insertions(+) diff --git a/libavcodec/pthread_frame.c b/libavcodec/pthread_frame.c index 8e88dda2ce0..884899c46e2 100644 --- a/libavcodec/pthread_frame.c +++ b/libavcodec/pthread_frame.c @@ -1007,6 +1007,20 @@ av_cold void ff_thread_flush(AVCodecContext *avctx) avcodec_flush_buffers(p->avctx); } + + /* A thread-unsafe hwaccel keeps its state in the stash between frames, + * so the flushes above found no hwaccel to call. */ + if (fctx->stash_hwaccel && ffhwaccel(fctx->stash_hwaccel)->flush) { + AVCodecContext *ctx = fctx->threads[0].avctx; + + FFSWAP(const AVHWAccel*, ctx->hwaccel, fctx->stash_hwaccel); + FFSWAP(void*, ctx->hwaccel_context, fctx->stash_hwaccel_context); + FFSWAP(void*, ctx->internal->hwaccel_priv_data, fctx->stash_hwaccel_priv); + FF_HW_SIMPLE_CALL(ctx, flush); + FFSWAP(const AVHWAccel*, ctx->hwaccel, fctx->stash_hwaccel); + FFSWAP(void*, ctx->hwaccel_context, fctx->stash_hwaccel_context); + FFSWAP(void*, ctx->internal->hwaccel_priv_data, fctx->stash_hwaccel_priv); + } } int ff_thread_can_start_frame(AVCodecContext *avctx) -- 2.54.0 >From 8e1ff4453fd111e3c515d7307aa97d62a284b9e8 Mon Sep 17 00:00:00 2001 From: roffe <[email protected]> Date: Thu, 22 Oct 2026 22:30:55 +0200 Subject: [PATCH 2/2] avcodec/videotoolbox: reset the decoder after a flush VideoToolbox keeps the session's reference frames across avcodec_flush_buffers(), and a keyframe that is not an IDR picture (an H.264 recovery point, an HEVC CRA) does not clear them. After a seek to such a keyframe, the pictures up to the next IDR predict from frames left from before the seek and show broken blocks. The software decoder, whose flush drops its DPB, plays the same seek cleanly. VideoToolbox has no reset call, but a sample buffer can carry kCMSampleBufferAttachmentKey_ResetDecoderBeforeDecoding, which Apple advises for the first sample after a seek (see https://crbug.com/451536366 and Chromium's media/gpu/mac/video_toolbox_h264_accelerator.cc). Give the h264 and hevc hwaccels a flush callback that marks a reset pending, and set the attachment on the next sample unless its picture is an IDR, or a BLA for HEVC, which drops the references by itself. The other VideoToolbox hwaccels decode codecs whose keyframes reset the references in-band, so they are left alone. On an M2 Pro the reset adds about 3.5 ms to a seek that lands on such a keyframe. Signed-off-by: roffe <[email protected]> --- libavcodec/videotoolbox.c | 25 +++++++++++++++++++++++++ libavcodec/vt_internal.h | 4 ++++ 2 files changed, 29 insertions(+) diff --git a/libavcodec/videotoolbox.c b/libavcodec/videotoolbox.c index 8df8a3214ce..191249790b7 100644 --- a/libavcodec/videotoolbox.c +++ b/libavcodec/videotoolbox.c @@ -424,6 +424,9 @@ int ff_videotoolbox_h264_start_frame(AVCodecContext *avctx, VTContext *vtctx = avctx->internal->hwaccel_priv_data; H264Context *h = avctx->priv_data; + if (h->picture_idr) + vtctx->reset_pending = false; + if (h->is_avc == 1) { return ff_videotoolbox_buffer_copy(vtctx, buffer, size); } @@ -522,6 +525,16 @@ static void videotoolbox_stop(AVCodecContext *avctx) } } +// The session keeps its reference frames through a flush, and a keyframe +// that is not an IDR picture leaves them in place, so the first picture +// decoded after the flush has the decoder reset instead. +static void videotoolbox_flush(AVCodecContext *avctx) +{ + VTContext *vtctx = avctx->internal->hwaccel_priv_data; + + vtctx->reset_pending = true; +} + int ff_videotoolbox_uninit(AVCodecContext *avctx) { VTContext *vtctx = avctx->internal->hwaccel_priv_data; @@ -757,6 +770,12 @@ static OSStatus videotoolbox_session_decode_frame(AVCodecContext *avctx) if (!sample_buf) return -1; + if (vtctx->reset_pending) { + CMSetAttachment(sample_buf, kCMSampleBufferAttachmentKey_ResetDecoderBeforeDecoding, + kCFBooleanTrue, kCMAttachmentMode_ShouldNotPropagate); + vtctx->reset_pending = false; + } + status = VTDecompressionSessionDecodeFrame(videotoolbox->session, sample_buf, 0, // decodeFlags @@ -1102,6 +1121,10 @@ static int videotoolbox_hevc_start_frame(AVCodecContext *avctx, { HEVCContext *h = avctx->priv_data; AVFrame *frame = h->cur_frame->f; + VTContext *vtctx = avctx->internal->hwaccel_priv_data; + + if (IS_IDR(h) || IS_BLA(h)) + vtctx->reset_pending = false; frame->crop_right = 0; frame->crop_left = 0; @@ -1407,6 +1430,7 @@ const FFHWAccel ff_hevc_videotoolbox_hwaccel = { .frame_params = ff_videotoolbox_frame_params, .init = ff_videotoolbox_common_init, .uninit = ff_videotoolbox_uninit, + .flush = videotoolbox_flush, .priv_data_size = sizeof(VTContext), }; @@ -1423,6 +1447,7 @@ const FFHWAccel ff_h264_videotoolbox_hwaccel = { .frame_params = ff_videotoolbox_frame_params, .init = ff_videotoolbox_common_init, .uninit = ff_videotoolbox_uninit, + .flush = videotoolbox_flush, .priv_data_size = sizeof(VTContext), }; diff --git a/libavcodec/vt_internal.h b/libavcodec/vt_internal.h index b1eb07310af..dbaff44caab 100644 --- a/libavcodec/vt_internal.h +++ b/libavcodec/vt_internal.h @@ -46,6 +46,10 @@ typedef struct VTContext { uint8_t sps[3]; bool reconfig_needed; + // Set by a flush: the next sample resets the decoder, unless its picture + // is an IDR. + bool reset_pending; + void *logctx; } VTContext; -- 2.54.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
