PR #25797 opened by Marti2203
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25797
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25797.patch
The VC-2 encoder's `wavelet_type` option is declared with the numeric range
`[0, VC2_TRANSFORMS_NB]`, but `ff_vc2enc_init_transforms()` only installs four
of the seven transform function pointers (9_7, 5_3, Haar, Haar-with-shift).
The 13_7, Fidelity and Daubechies-9_7 entries stay NULL, and
`VC2_TRANSFORMS_NB` itself is one past the end of the `vc2_subband_dwt[]`
array.
`dwt_plane()` indexes that array with the selected value and calls it, so:
ffmpeg -f lavfi -i testsrc=s=64x64 -c:v vc2 -wavelet_type 2 -frames:v 1 -f
null -
calls through a NULL pointer (SIGSEGV at 0x0), and `-wavelet_type 7` reads a
function pointer out of bounds and calls it. The named option values (9_7,
5_3, haar, haar_noshift) are unaffected; only the out-of-set numeric values
reach the crash.
The patch rejects an unsupported wavelet type at init with `AVERROR(EINVAL)`.
With it, `-wavelet_type 2/5/6/7` fail cleanly and 0/1/3/4 still encode; the
vc2 FATE encode tests (including vc2-thaar and vc2-t5_3) pass.
The issue was found with an LLM-assisted static analysis tool and verified
manually.
>From 64eb34472a2c63439dabcaa34104cb2b4f1d7e15 Mon Sep 17 00:00:00 2001
From: Martin Mirchev <[email protected]>
Date: Fri, 6 Nov 2026 11:40:23 +0800
Subject: [PATCH] avcodec/vc2enc: reject wavelet types the encoder does not
implement
The wavelet_type option is declared with the numeric range [0,
VC2_TRANSFORMS_NB], but ff_vc2enc_init_transforms() only installs four of
the transform function pointers (9_7, 5_3, Haar and Haar-with-shift). The
other in-range values leave the corresponding vc2_subband_dwt[] entry NULL
(13_7, Fidelity, Daubechies 9_7), and VC2_TRANSFORMS_NB itself is one past
the end of the array.
dwt_plane() then calls t->vc2_subband_dwt[s->wavelet_idx](), so e.g.
"-c:v vc2 -wavelet_type 2" calls through a NULL pointer and
"-wavelet_type 7" reads a function pointer out of bounds and calls it,
crashing the encoder.
Reject an unsupported wavelet type at init with AVERROR(EINVAL) instead.
Found with an LLM-assisted static analysis tool and verified manually.
Signed-off-by: Martin Mirchev <[email protected]>
---
libavcodec/vc2enc.c | 8 ++++++++
1 file changed, 8 insertions(+)
diff --git a/libavcodec/vc2enc.c b/libavcodec/vc2enc.c
index 2c155c01a16..8cb4a056755 100644
--- a/libavcodec/vc2enc.c
+++ b/libavcodec/vc2enc.c
@@ -1093,6 +1093,14 @@ static av_cold int vc2_encode_init(AVCodecContext *avctx)
return AVERROR(EINVAL);
}
+ if (s->wavelet_idx != VC2_TRANSFORM_9_7 &&
+ s->wavelet_idx != VC2_TRANSFORM_5_3 &&
+ s->wavelet_idx != VC2_TRANSFORM_HAAR &&
+ s->wavelet_idx != VC2_TRANSFORM_HAAR_S) {
+ av_log(avctx, AV_LOG_ERROR, "Unsupported wavelet type %d\n",
s->wavelet_idx);
+ return AVERROR(EINVAL);
+ }
+
if (s->base_vf <= 0) {
if (avctx->strict_std_compliance < FF_COMPLIANCE_STRICT) {
s->strict_compliance = s->base_vf = 0;
--
2.54.0
_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
