PR #24993 opened by Kilian (lusoris) URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24993 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24993.patch
This fixes #24972: hevc_nvenc with `-udu_sei 1` stops with "Failed locking bitstream buffer: out of memory" once the user data SEI of a picture gets large. NVENC's HEVC encoder accepts at most 1024 bytes of non-VCL NAL units per picture: VPS, SPS, PPS, AUD and every SEI, in Annex B form with emulation prevention. Above that, `nvEncLockBitstream()` returns `NV_ENC_ERR_OUT_OF_MEMORY`. Neither nvEncodeAPI.h 13.1 nor the programming guide states the limit. On the last picture of a stream, only an error line is printed and the picture is lost. What the patch does: - gives the SEI in `seiPayloadArray` 768 bytes per HEVC picture, less the A/53 and timecode SEI already queued, and keeps 256 bytes for the encoder's own NAL units (at most 173 bytes measured, with HDR10 metadata and AUD) - drops a user data unregistered SEI that does not fit: a warning for the first, verbose after that, and a total when the encoder closes - leaves h264_nvenc (64 KiB payloads encode) and av1_nvenc (no udu_sei) unchanged Testing, RTX 4090, driver 615.78.08, nv-codec-headers 13.1.15, on top of n9.0.2: - streams of 1 IDR and 5 P pictures with payloads around the budget, two payloads per picture, and with A/53 and timecode SEI: every frame yields a packet, where stock fails with ENOMEM - under-budget hevc_nvenc streams and all h264_nvenc output are byte-identical to stock - on master (ba987fe24d) the commit applies cleanly, nvenc.c, nvenc_h264.c and nvenc_hevc.c build without warnings (GCC 16.2.1), and the same RTX 4090 check passes: a 700-byte payload is written and decodes byte-exact, a 1000-byte payload is dropped with the warning, packets == frames; no FATE test covers NVENC For review: the 256-byte reserve, and the per-NAL byte count (start code, NAL header, payload type and size, payload with emulation prevention, trailing bits, plus one byte of margin across the size and payload boundary). >From c70852716a8f9db9e1210d7e8f5ab06a86823a91 Mon Sep 17 00:00:00 2001 From: lusoris <[email protected]> Date: Fri, 9 Oct 2026 22:41:51 +0200 Subject: [PATCH] avcodec/nvenc: keep hevc_nvenc user data SEI within the picture limit hevc_nvenc with udu_sei=1 stops the encode with "Failed locking bitstream buffer: out of memory" once the user data unregistered SEI of a picture is large. NVENC's HEVC encoder accepts at most 1024 bytes of non-VCL NAL units per picture: VPS, SPS, PPS, access unit delimiter and every SEI NAL unit, in Annex B form with start codes and emulation prevention bytes. Above that, nvEncLockBitstream() returns NV_ENC_ERR_OUT_OF_MEMORY, which nvenc.c maps to AVERROR(ENOMEM). On the last picture of a stream only an error line is printed and the picture is lost. Neither nvEncodeAPI.h 13.1 nor the NVENC programming guide 13.1 states the limit; h264_nvenc carries payloads of 64 KiB. For HEVC, give the SEI passed in seiPayloadArray 768 bytes per picture, less the A/53 and timecode SEI already queued, and keep 256 bytes for the NAL units NVENC writes on its own (128 bytes measured at 8K Main10 with VUI and AUD). A user data unregistered SEI that does not fit is not written. The first one is a warning, later ones are verbose, and the total is a warning when the encoder closes. Measured on an RTX 4090 with driver 615.71.09: the largest payload that encodes is 898 bytes on an IDR picture and 999 bytes on a P picture at 1080p, which is 1024 bytes of non-VCL NAL units in both cases. Checked with driver 615.78.08 on streams of one IDR and five P pictures at 1080p: payloads within the budget are written, larger ones are dropped with the messages above, every frame yields a packet, and the h264_nvenc output is unchanged. With HDR10 mastering display and content light level metadata, NVENC writes at most 173 bytes of its own non-VCL NAL units on an IDR picture, within the 256-byte reserve. Fixes #24972 Signed-off-by: lusoris <[email protected]> --- libavcodec/nvenc.c | 75 ++++++++++++++++++++++++++++++++++++++++++++++ libavcodec/nvenc.h | 1 + 2 files changed, 76 insertions(+) diff --git a/libavcodec/nvenc.c b/libavcodec/nvenc.c index 6e7331405f..258954b014 100644 --- a/libavcodec/nvenc.c +++ b/libavcodec/nvenc.c @@ -44,6 +44,15 @@ #include "encode.h" #include "internal.h" +/* hevc_nvenc fails nvEncLockBitstream() with NV_ENC_ERR_OUT_OF_MEMORY when the + * non-VCL NAL units of one picture, VPS, SPS, PPS, access unit delimiter and + * every SEI in Annex B form, exceed 1024 bytes. The SEI passed in + * seiPayloadArray get what is left after 256 bytes for the NAL units the + * encoder writes on its own (at most 128 bytes measured, plus the HDR and + * 3D reference display SEI it builds from the picture parameters). */ +#define NVENC_HEVC_NON_VCL_LIMIT 1024 +#define NVENC_HEVC_SEI_BUDGET (NVENC_HEVC_NON_VCL_LIMIT - 256) + #define CHECK_CU(x) FF_CUDA_CHECK_DL(avctx, dl_fn->cuda_dl, x) #define NVENC_CAP 0x30 @@ -2084,6 +2093,12 @@ av_cold int ff_nvenc_encode_close(AVCodecContext *avctx) av_freep(&ctx->surfaces); ctx->nb_surfaces = 0; + if (ctx->udu_sei_dropped) + av_log(avctx, AV_LOG_WARNING, "%"PRId64" user data unregistered SEI " + "payload(s) were not written: hevc_nvenc fails a picture whose " + "parameter sets and SEI exceed %d bytes.\n", + ctx->udu_sei_dropped, NVENC_HEVC_NON_VCL_LIMIT); + av_frame_free(&ctx->frame); av_freep(&ctx->sei_data); @@ -2726,9 +2741,57 @@ static int output_ready(AVCodecContext *avctx, int flush) return (nb_ready > 0) && (nb_ready + nb_pending >= ctx->async_depth); } +/* Annex B size of the SEI NAL unit NVENC writes for one payload: start code, + * NAL unit header, payload type and size, the payload with its emulation + * prevention bytes and the trailing bits, plus one byte for emulation + * prevention across the size and payload boundary. The payload is scanned only + * when the rest fits into limit. */ +static size_t nvenc_sei_nal_size(const NV_ENC_SEI_PAYLOAD *sei, size_t limit) +{ + const uint8_t *p = sei->payload; + size_t size = 4 + 2 + sei->payloadType / 255 + 1 + sei->payloadSize / 255 + 1 + + sei->payloadSize + 1 + 1; + int zeros = 0; + + if (size > limit) + return size; + + for (uint32_t i = 0; i < sei->payloadSize; i++) { + if (zeros >= 2 && p[i] <= 3) { + size++; + zeros = 0; + } + zeros = p[i] ? 0 : zeros + 1; + } + return size; +} + +/* Charge one user data unregistered SEI against the picture's budget, or log + * that it is not written. */ +static int nvenc_hevc_sei_fits(AVCodecContext *avctx, const AVFrame *frame, + const NV_ENC_SEI_PAYLOAD *sei, size_t *budget) +{ + NvencContext *ctx = avctx->priv_data; + size_t size = nvenc_sei_nal_size(sei, *budget); + + if (size <= *budget) { + *budget -= size; + return 1; + } + + av_log(avctx, ctx->udu_sei_dropped ? AV_LOG_VERBOSE : AV_LOG_WARNING, + "Not writing a %"PRIu32"-byte user data unregistered SEI at pts " + "%"PRId64": hevc_nvenc fails a picture whose parameter sets and SEI " + "exceed %d bytes.\n", sei->payloadSize, frame->pts, + NVENC_HEVC_NON_VCL_LIMIT); + ctx->udu_sei_dropped++; + return 0; +} + static int prepare_sei_data_array(AVCodecContext *avctx, const AVFrame *frame) { NvencContext *ctx = avctx->priv_data; + size_t sei_budget = NVENC_HEVC_SEI_BUDGET; int sei_count = 0; int i, res; @@ -2803,6 +2866,12 @@ static int prepare_sei_data_array(AVCodecContext *avctx, const AVFrame *frame) if (!ctx->udu_sei) return sei_count; + /* The A/53 and timecode SEI above come out of the same budget. */ + if (avctx->codec->id == AV_CODEC_ID_HEVC) { + for (i = 0; i < sei_count; i++) + sei_budget -= FFMIN(sei_budget, nvenc_sei_nal_size(&ctx->sei_data[i], sei_budget)); + } + for (i = 0; i < frame->nb_side_data; i++) { AVFrameSideData *side_data = frame->side_data[i]; void *tmp; @@ -2827,6 +2896,12 @@ static int prepare_sei_data_array(AVCodecContext *avctx, const AVFrame *frame) goto error; } + if (avctx->codec->id == AV_CODEC_ID_HEVC && + !nvenc_hevc_sei_fits(avctx, frame, &ctx->sei_data[sei_count], &sei_budget)) { + av_freep(&ctx->sei_data[sei_count].payload); + continue; + } + sei_count++; } } diff --git a/libavcodec/nvenc.h b/libavcodec/nvenc.h index acde2a42c1..52ff40b7da 100644 --- a/libavcodec/nvenc.h +++ b/libavcodec/nvenc.h @@ -198,6 +198,7 @@ typedef struct NvencContext NV_ENC_SEI_PAYLOAD *sei_data; int sei_data_size; + int64_t udu_sei_dropped; ///< user data unregistered SEI not written, see prepare_sei_data_array() struct { void *ptr; -- 2.52.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
