PR #24994 opened by Kilian (lusoris)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24994
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24994.patch
Depends on #24993: it reuses the emulation prevention counter that pull request
adds. Only the last commit is new here.
NVENC writes some user data unregistered SEI payloads truncated. The NAL unit
ends inside the payload without rbsp trailing bits, decoders drop it ("Skipping
invalid undecodable NALU: 39"), and the encode reports success. Stock FFmpeg
hits this with h264_nvenc and hevc_nvenc.
The trigger is exact. With P the payload size including the UUID, Pmax the
largest payload of the picture and E the emulation prevention bytes its RBSP
needs, a payload is truncated when P + 3 + E > floor(4 * (Pmax + 5) / 3). In
practice only zero-heavy payloads, with E above about P/3, reach it.
What the patch does:
- checks every user data unregistered SEI against that rule before encoding and
does not write the ones NVENC would truncate: a warning for the first, verbose
after that, and a total when the encoder closes
- checks again after each drop, since dropping the largest payload shrinks the
room of the others
- leaves every other payload and picture unchanged
Testing, RTX 4090, driver 615.78.08, h264_nvenc and hevc_nvenc, IDR and P
pictures, on top of n9.0.2:
- the rule matches 76870 single-payload and 1600 multi-payload cases; the
boundaries at P = 100, 300, 700 and 2866 are exact
- patched: exactly the marked payloads are dropped, every written SEI NAL unit
is complete and decodes, A/53 captions are unaffected, and output without such
payloads is byte-identical to stock
- on master (ba987fe24d) the commit applies and builds without warnings, and
the RTX 4090 check passes on h264_nvenc and hevc_nvenc: payloads the rule marks
are dropped with the warning, payloads within it are written complete and
decode byte-exact, packets == frames; no FATE test covers NVENC
For review: whether dropping is the right default over writing a NAL unit that
decoders discard, and whether to gate it on the driver version once NVIDIA
fixes the encoder.
>From c70852716a8f9db9e1210d7e8f5ab06a86823a91 Mon Sep 17 00:00:00 2001
From: lusoris <[email protected]>
Date: Fri, 9 Oct 2026 22:41:51 +0200
Subject: [PATCH 1/2] avcodec/nvenc: keep hevc_nvenc user data SEI within the
picture limit
hevc_nvenc with udu_sei=1 stops the encode with "Failed locking
bitstream buffer: out of memory" once the user data unregistered SEI of
a picture is large. NVENC's HEVC encoder accepts at most 1024 bytes of
non-VCL NAL units per picture: VPS, SPS, PPS, access unit delimiter and
every SEI NAL unit, in Annex B form with start codes and emulation
prevention bytes. Above that, nvEncLockBitstream() returns
NV_ENC_ERR_OUT_OF_MEMORY, which nvenc.c maps to AVERROR(ENOMEM). On the
last picture of a stream only an error line is printed and the picture
is lost. Neither nvEncodeAPI.h 13.1 nor the NVENC programming guide
13.1 states the limit; h264_nvenc carries payloads of 64 KiB.
For HEVC, give the SEI passed in seiPayloadArray 768 bytes per picture,
less the A/53 and timecode SEI already queued, and keep 256 bytes for
the NAL units NVENC writes on its own (128 bytes measured at 8K Main10
with VUI and AUD). A user data unregistered SEI that does not fit is not
written. The first one is a warning, later ones are verbose, and the
total is a warning when the encoder closes.
Measured on an RTX 4090 with driver 615.71.09: the largest payload that
encodes is 898 bytes on an IDR picture and 999 bytes on a P picture at
1080p, which is 1024 bytes of non-VCL NAL units in both cases.
Checked with driver 615.78.08 on streams of one IDR and five P pictures
at 1080p: payloads within the budget are written, larger ones are
dropped with the messages above, every frame yields a packet, and the
h264_nvenc output is unchanged. With HDR10 mastering display and
content light level metadata, NVENC writes at most 173 bytes of its own
non-VCL NAL units on an IDR picture, within the 256-byte reserve.
Fixes #24972
Signed-off-by: lusoris <[email protected]>
---
libavcodec/nvenc.c | 75 ++++++++++++++++++++++++++++++++++++++++++++++
libavcodec/nvenc.h | 1 +
2 files changed, 76 insertions(+)
diff --git a/libavcodec/nvenc.c b/libavcodec/nvenc.c
index 6e7331405f..258954b014 100644
--- a/libavcodec/nvenc.c
+++ b/libavcodec/nvenc.c
@@ -44,6 +44,15 @@
#include "encode.h"
#include "internal.h"
+/* hevc_nvenc fails nvEncLockBitstream() with NV_ENC_ERR_OUT_OF_MEMORY when the
+ * non-VCL NAL units of one picture, VPS, SPS, PPS, access unit delimiter and
+ * every SEI in Annex B form, exceed 1024 bytes. The SEI passed in
+ * seiPayloadArray get what is left after 256 bytes for the NAL units the
+ * encoder writes on its own (at most 128 bytes measured, plus the HDR and
+ * 3D reference display SEI it builds from the picture parameters). */
+#define NVENC_HEVC_NON_VCL_LIMIT 1024
+#define NVENC_HEVC_SEI_BUDGET (NVENC_HEVC_NON_VCL_LIMIT - 256)
+
#define CHECK_CU(x) FF_CUDA_CHECK_DL(avctx, dl_fn->cuda_dl, x)
#define NVENC_CAP 0x30
@@ -2084,6 +2093,12 @@ av_cold int ff_nvenc_encode_close(AVCodecContext *avctx)
av_freep(&ctx->surfaces);
ctx->nb_surfaces = 0;
+ if (ctx->udu_sei_dropped)
+ av_log(avctx, AV_LOG_WARNING, "%"PRId64" user data unregistered SEI "
+ "payload(s) were not written: hevc_nvenc fails a picture whose "
+ "parameter sets and SEI exceed %d bytes.\n",
+ ctx->udu_sei_dropped, NVENC_HEVC_NON_VCL_LIMIT);
+
av_frame_free(&ctx->frame);
av_freep(&ctx->sei_data);
@@ -2726,9 +2741,57 @@ static int output_ready(AVCodecContext *avctx, int flush)
return (nb_ready > 0) && (nb_ready + nb_pending >= ctx->async_depth);
}
+/* Annex B size of the SEI NAL unit NVENC writes for one payload: start code,
+ * NAL unit header, payload type and size, the payload with its emulation
+ * prevention bytes and the trailing bits, plus one byte for emulation
+ * prevention across the size and payload boundary. The payload is scanned only
+ * when the rest fits into limit. */
+static size_t nvenc_sei_nal_size(const NV_ENC_SEI_PAYLOAD *sei, size_t limit)
+{
+ const uint8_t *p = sei->payload;
+ size_t size = 4 + 2 + sei->payloadType / 255 + 1 + sei->payloadSize / 255
+ 1 +
+ sei->payloadSize + 1 + 1;
+ int zeros = 0;
+
+ if (size > limit)
+ return size;
+
+ for (uint32_t i = 0; i < sei->payloadSize; i++) {
+ if (zeros >= 2 && p[i] <= 3) {
+ size++;
+ zeros = 0;
+ }
+ zeros = p[i] ? 0 : zeros + 1;
+ }
+ return size;
+}
+
+/* Charge one user data unregistered SEI against the picture's budget, or log
+ * that it is not written. */
+static int nvenc_hevc_sei_fits(AVCodecContext *avctx, const AVFrame *frame,
+ const NV_ENC_SEI_PAYLOAD *sei, size_t *budget)
+{
+ NvencContext *ctx = avctx->priv_data;
+ size_t size = nvenc_sei_nal_size(sei, *budget);
+
+ if (size <= *budget) {
+ *budget -= size;
+ return 1;
+ }
+
+ av_log(avctx, ctx->udu_sei_dropped ? AV_LOG_VERBOSE : AV_LOG_WARNING,
+ "Not writing a %"PRIu32"-byte user data unregistered SEI at pts "
+ "%"PRId64": hevc_nvenc fails a picture whose parameter sets and SEI
"
+ "exceed %d bytes.\n", sei->payloadSize, frame->pts,
+ NVENC_HEVC_NON_VCL_LIMIT);
+ ctx->udu_sei_dropped++;
+ return 0;
+}
+
static int prepare_sei_data_array(AVCodecContext *avctx, const AVFrame *frame)
{
NvencContext *ctx = avctx->priv_data;
+ size_t sei_budget = NVENC_HEVC_SEI_BUDGET;
int sei_count = 0;
int i, res;
@@ -2803,6 +2866,12 @@ static int prepare_sei_data_array(AVCodecContext *avctx,
const AVFrame *frame)
if (!ctx->udu_sei)
return sei_count;
+ /* The A/53 and timecode SEI above come out of the same budget. */
+ if (avctx->codec->id == AV_CODEC_ID_HEVC) {
+ for (i = 0; i < sei_count; i++)
+ sei_budget -= FFMIN(sei_budget,
nvenc_sei_nal_size(&ctx->sei_data[i], sei_budget));
+ }
+
for (i = 0; i < frame->nb_side_data; i++) {
AVFrameSideData *side_data = frame->side_data[i];
void *tmp;
@@ -2827,6 +2896,12 @@ static int prepare_sei_data_array(AVCodecContext *avctx,
const AVFrame *frame)
goto error;
}
+ if (avctx->codec->id == AV_CODEC_ID_HEVC &&
+ !nvenc_hevc_sei_fits(avctx, frame, &ctx->sei_data[sei_count],
&sei_budget)) {
+ av_freep(&ctx->sei_data[sei_count].payload);
+ continue;
+ }
+
sei_count++;
}
}
diff --git a/libavcodec/nvenc.h b/libavcodec/nvenc.h
index acde2a42c1..52ff40b7da 100644
--- a/libavcodec/nvenc.h
+++ b/libavcodec/nvenc.h
@@ -198,6 +198,7 @@ typedef struct NvencContext
NV_ENC_SEI_PAYLOAD *sei_data;
int sei_data_size;
+ int64_t udu_sei_dropped; ///< user data unregistered SEI not written, see
prepare_sei_data_array()
struct {
void *ptr;
--
2.52.0
>From eb39da4ea5ed300c4e5d448671bb77ee9266e685 Mon Sep 17 00:00:00 2001
From: lusoris <[email protected]>
Date: Fri, 9 Oct 2026 23:53:57 +0200
Subject: [PATCH 2/2] avcodec/nvenc: drop user data SEI that NVENC would write
truncated
NVENC writes the user data unregistered SEI of a picture through an
escape buffer of 4/3 of the largest payload of the picture plus 5
bytes. A payload whose emulation prevention bytes do not fit is written
truncated: the NAL unit stops inside the payload, has no rbsp trailing
bits, and decoders discard it ("Skipping invalid undecodable NALU: 39").
The encode itself succeeds, so the payload is lost without an error.
Only payloads that need more emulation prevention bytes than about a
third of their size are affected, which in practice means long runs of
zero bytes. With P the payload size including the UUID, Pmax the
largest payload of the picture and E the emulation prevention bytes of
the payload's RBSP, the payload is truncated exactly when
P + 3 + E > floor(4 * (Pmax + 5) / 3); for a single payload that is
E > ceil(P / 3) + 3.
Check every user data unregistered SEI against that before encoding and
do not write the ones that would be truncated, so no undecodable NAL
unit reaches the stream. The first one is a warning, later ones are
verbose, and the total is a warning when the encoder closes. Dropping
the largest payload shrinks the room of the others, so the check
repeats until it drops nothing.
Measured on an RTX 4090 with driver 615.78.08 for h264_nvenc and
hevc_nvenc, on IDR and P pictures: the rule matches all 76870
single-payload and 1600 multi-payload cases.
Checked with the same driver on h264_nvenc and hevc_nvenc, IDR and P
pictures: payloads the rule marks are not written, every SEI NAL unit
that is written is complete and decodes, A/53 closed captions in the
same pictures are unaffected, and output without such payloads is
byte-identical to the unpatched encoder.
Signed-off-by: lusoris <[email protected]>
---
libavcodec/nvenc.c | 94 +++++++++++++++++++++++++++++++++++++++-------
libavcodec/nvenc.h | 1 +
2 files changed, 82 insertions(+), 13 deletions(-)
diff --git a/libavcodec/nvenc.c b/libavcodec/nvenc.c
index 258954b014..5ab2cd4da2 100644
--- a/libavcodec/nvenc.c
+++ b/libavcodec/nvenc.c
@@ -2098,6 +2098,11 @@ av_cold int ff_nvenc_encode_close(AVCodecContext *avctx)
"payload(s) were not written: hevc_nvenc fails a picture whose "
"parameter sets and SEI exceed %d bytes.\n",
ctx->udu_sei_dropped, NVENC_HEVC_NON_VCL_LIMIT);
+ if (ctx->udu_sei_truncated)
+ av_log(avctx, AV_LOG_WARNING, "%"PRId64" user data unregistered SEI "
+ "payload(s) were not written: NVENC writes a payload truncated "
+ "when it needs more emulation prevention bytes than about a "
+ "third of its size.\n", ctx->udu_sei_truncated);
av_frame_free(&ctx->frame);
@@ -2746,24 +2751,31 @@ static int output_ready(AVCodecContext *avctx, int
flush)
* prevention bytes and the trailing bits, plus one byte for emulation
* prevention across the size and payload boundary. The payload is scanned only
* when the rest fits into limit. */
-static size_t nvenc_sei_nal_size(const NV_ENC_SEI_PAYLOAD *sei, size_t limit)
+/* Emulation prevention bytes size bytes of payload p need, after zeros zero
+ * bytes. */
+static size_t nvenc_sei_epb_count(const uint8_t *p, uint32_t size, int zeros)
{
- const uint8_t *p = sei->payload;
- size_t size = 4 + 2 + sei->payloadType / 255 + 1 + sei->payloadSize / 255
+ 1 +
- sei->payloadSize + 1 + 1;
- int zeros = 0;
+ size_t epb = 0;
- if (size > limit)
- return size;
-
- for (uint32_t i = 0; i < sei->payloadSize; i++) {
+ for (uint32_t i = 0; i < size; i++) {
if (zeros >= 2 && p[i] <= 3) {
- size++;
+ epb++;
zeros = 0;
}
zeros = p[i] ? 0 : zeros + 1;
}
- return size;
+ return epb;
+}
+
+static size_t nvenc_sei_nal_size(const NV_ENC_SEI_PAYLOAD *sei, size_t limit)
+{
+ size_t size = 4 + 2 + sei->payloadType / 255 + 1 + sei->payloadSize / 255
+ 1 +
+ sei->payloadSize + 1 + 1;
+
+ if (size > limit)
+ return size;
+
+ return size + nvenc_sei_epb_count(sei->payload, sei->payloadSize, 0);
}
/* Charge one user data unregistered SEI against the picture's budget, or log
@@ -2788,11 +2800,66 @@ static int nvenc_hevc_sei_fits(AVCodecContext *avctx,
const AVFrame *frame,
return 0;
}
+/* NVENC escapes the user data unregistered SEI of a picture into a buffer of
+ * 4/3 of the largest payload of the picture plus 5 bytes. A payload whose
+ * emulation prevention bytes do not fit is written truncated, without its end
+ * and trailing bits, and decoders discard the NAL unit. Only a payload that
+ * needs more emulation prevention bytes than about a third of its size, as
+ * long runs of zero bytes do, can overflow it. Measured with driver
+ * 615.78.08 for h264_nvenc and hevc_nvenc. */
+static int nvenc_udu_sei_truncated(const NV_ENC_SEI_PAYLOAD *sei, uint32_t
pmax)
+{
+ /* The last payload size byte is zero when the size is a multiple of 255.
*/
+ size_t epb = nvenc_sei_epb_count(sei->payload, sei->payloadSize,
+ !(sei->payloadSize % 255));
+ size_t room = 4 * ((size_t)pmax + 5) / 3;
+
+ return sei->payloadSize + 3 + epb > room;
+}
+
+/* Drop the user data unregistered SEI in sei_data[first, count) that NVENC
+ * would truncate and return the new count. Dropping the largest payload
+ * shrinks the room of the others, so repeat until nothing is dropped. */
+static int nvenc_drop_truncated_udu_sei(AVCodecContext *avctx, const AVFrame
*frame,
+ int first, int count)
+{
+ NvencContext *ctx = avctx->priv_data;
+ int dropped;
+
+ do {
+ uint32_t pmax = 0;
+ int kept = first;
+
+ for (int i = first; i < count; i++)
+ pmax = FFMAX(pmax, ctx->sei_data[i].payloadSize);
+
+ for (int i = first; i < count; i++) {
+ NV_ENC_SEI_PAYLOAD *sei = &ctx->sei_data[i];
+
+ if (!nvenc_udu_sei_truncated(sei, pmax)) {
+ ctx->sei_data[kept++] = *sei;
+ continue;
+ }
+
+ av_log(avctx, ctx->udu_sei_truncated ? AV_LOG_VERBOSE :
AV_LOG_WARNING,
+ "Not writing a %"PRIu32"-byte user data unregistered SEI at
pts "
+ "%"PRId64": NVENC would write it truncated, and decoders "
+ "discard such a NAL unit.\n", sei->payloadSize, frame->pts);
+ ctx->udu_sei_truncated++;
+ av_freep(&sei->payload);
+ }
+ dropped = count - kept;
+ count = kept;
+ } while (dropped && count > first);
+
+ return count;
+}
+
static int prepare_sei_data_array(AVCodecContext *avctx, const AVFrame *frame)
{
NvencContext *ctx = avctx->priv_data;
size_t sei_budget = NVENC_HEVC_SEI_BUDGET;
- int sei_count = 0;
+ int sei_count = 0, udu_first;
int i, res;
if (ctx->a53_cc && av_frame_get_side_data(frame, AV_FRAME_DATA_A53_CC)) {
@@ -2872,6 +2939,7 @@ static int prepare_sei_data_array(AVCodecContext *avctx,
const AVFrame *frame)
sei_budget -= FFMIN(sei_budget,
nvenc_sei_nal_size(&ctx->sei_data[i], sei_budget));
}
+ udu_first = sei_count;
for (i = 0; i < frame->nb_side_data; i++) {
AVFrameSideData *side_data = frame->side_data[i];
void *tmp;
@@ -2906,7 +2974,7 @@ static int prepare_sei_data_array(AVCodecContext *avctx,
const AVFrame *frame)
}
}
- return sei_count;
+ return nvenc_drop_truncated_udu_sei(avctx, frame, udu_first, sei_count);
error:
for (i = 0; i < sei_count; i++)
diff --git a/libavcodec/nvenc.h b/libavcodec/nvenc.h
index 52ff40b7da..5cc0a5de3a 100644
--- a/libavcodec/nvenc.h
+++ b/libavcodec/nvenc.h
@@ -199,6 +199,7 @@ typedef struct NvencContext
NV_ENC_SEI_PAYLOAD *sei_data;
int sei_data_size;
int64_t udu_sei_dropped; ///< user data unregistered SEI not written, see
prepare_sei_data_array()
+ int64_t udu_sei_truncated; ///< user data unregistered SEI NVENC would
truncate, not written
struct {
void *ptr;
--
2.52.0
_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
