PR #25308 opened by Marti2203 URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25308 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25308.patch
In `mode=backward`, vf_vignette stores `1. / get_natural_factor(s, x, y)` as the gain of each pixel. `get_natural_factor()` returns 0 for pixels farther than `dmax` from the center, which happens as soon as the center is moved with `x0`/`y0` (or the aspect stretches the distance), so the gain is inf. The filter then computes `srcp[0] * f + dither` (or `f * (src - 127) + 127` for chroma) and converts it to int for `av_clip_uint8()`: inf, or NaN when the sample is 0 (or chroma 127), is out of range for that conversion, which is undefined behaviour. On x86 these pixels come out black, so the brightness ramps up towards the edge and then drops to 0: ``` ffmpeg -f lavfi -i color=gray:s=64x64 -vf format=gray,vignette=x0=0:y0=0:mode=backward -frames:v 1 -f rawvideo out.gray ``` Luma on the diagonal x = y at x = 0, 10, 20, 30, 35, 40, 50, 63: ``` master: [128, 138, 176, 255, 0, 0, 0, 0] patched: [128, 138, 176, 255, 255, 255, 255, 255] ``` rgb24 (packed path) and the yuv formats behave the same. The same overflow also happens inside `dmax` for an angle at or near PI/2 (the maximum the filter allows): at the corner `cos(PI/2)^4` is about 1e-65, its inverse overflows float, and `vignette=a=PI/2:mode=backward` turns the corner pixel black (0) while its neighbour is 255. The patch caps the gain at 256 instead of letting it become inf. 256 already saturates any non-zero 8-bit sample (and any chroma value other than 127), so the output is the same as before wherever it was defined, and pixels beyond `dmax` now saturate, as the inverse of forward mode (which makes them black) suggests. Clamping the distance to `dmax` instead would be another option, happy to change it if that is preferred. The second commit adds `fate-filter-vignette-backward-yuv444p` and `fate-filter-vignette-backward-rgb24`. They use `a=0`, so the gain is exactly 1 inside `dmax` and the output does not depend on libm rounding, and `x0=0:y0=0`, so part of the frame lies beyond `dmax`. Both fail on master and pass with the fix (THREADS=1 and 4, gcc and clang builds); `make fate-vfilter` (504 tests without samples) passes. There was no FATE coverage for vignette before. This issue was found by an LLM-Enhanced Static Analysis Engine and verified manually. 🤖 Generated with [Claude Code](https://claude.com/claude-code) From 73bfe5a79853c94542505cf21ccc3955357f6c2d Mon Sep 17 00:00:00 2001 From: Martin Mirchev <[email protected]> Date: Wed, 21 Oct 2026 19:06:54 +0800 Subject: [PATCH 1/2] avfilter/vf_vignette: keep the backward gain finite In backward mode the gain is 1 / factor, but the factor is 0 for pixels farther than dmax from the center (possible with x0/y0 or aspect), so the gain is inf. inf * sample, or inf * 0 = NaN, is then converted to int, which is undefined behaviour; on x86 these pixels come out black instead of saturated. The same happens for an angle close to PI/2, where cos^4 underflows and 1 / factor overflows float. Cap the gain at 256, which already saturates any non-zero 8-bit sample, so the output is unchanged wherever it was well defined before. Signed-off-by: Martin Mirchev <[email protected]> --- libavfilter/vf_vignette.c | 9 +++++++-- 1 file changed, 7 insertions(+), 2 deletions(-) diff --git a/libavfilter/vf_vignette.c b/libavfilter/vf_vignette.c index da4c3079648..ff208855cf5 100644 --- a/libavfilter/vf_vignette.c +++ b/libavfilter/vf_vignette.c @@ -175,8 +175,13 @@ static void update_context(VignetteContext *s, AVFilterLink *inlink, AVFrame *fr if (s->backward) { for (y = 0; y < inlink->h; y++) { - for (x = 0; x < inlink->w; x++) - dst[x] = 1. / get_natural_factor(s, x, y); + for (x = 0; x < inlink->w; x++) { + const double f = get_natural_factor(s, x, y); + /* f is 0 beyond dmax and tiny for an angle close to PI/2: + * cap the gain to keep it finite, 256 already saturates any + * non-zero 8-bit sample */ + dst[x] = f > 1. / 256 ? 1. / f : 256; + } dst += dst_linesize; } } else { -- 2.54.0 From 8f9edc69e20e43f55c470fbeba349b5c42181d2b Mon Sep 17 00:00:00 2001 From: Martin Mirchev <[email protected]> Date: Wed, 21 Oct 2026 19:06:54 +0800 Subject: [PATCH 2/2] tests/fate/filter-video: add tests for vignette backward mode Use angle 0, so the gain is exactly 1 inside dmax, and move the center to the top left corner so that part of the frame lies beyond dmax and must saturate. Cover the planar (yuv444p) and packed (rgb24) paths. Signed-off-by: Martin Mirchev <[email protected]> --- tests/fate/filter-video.mak | 3 +++ tests/ref/fate/filter-vignette-backward-rgb24 | 6 ++++++ tests/ref/fate/filter-vignette-backward-yuv444p | 6 ++++++ 3 files changed, 15 insertions(+) create mode 100644 tests/ref/fate/filter-vignette-backward-rgb24 create mode 100644 tests/ref/fate/filter-vignette-backward-yuv444p diff --git a/tests/fate/filter-video.mak b/tests/fate/filter-video.mak index 4a065ec0320..95254ff3fcf 100644 --- a/tests/fate/filter-video.mak +++ b/tests/fate/filter-video.mak @@ -480,6 +480,9 @@ fate-filter-chromashift-wrap: CMD = framecrc -lavfi testsrc2=r=5:d=1,chromashif FATE_FILTER-$(call FILTERFRAMECRC, TESTSRC2 FPS DECIMATE) += fate-filter-decimate fate-filter-decimate: CMD = framecrc -lavfi testsrc2=r=24:d=10,fps=60,decimate=5,decimate=4,decimate=3 -pix_fmt yuv420p +FATE_FILTER-$(call FILTERFRAMECRC, TESTSRC2 FORMAT VIGNETTE) += fate-filter-vignette-backward-yuv444p fate-filter-vignette-backward-rgb24 +fate-filter-vignette-backward-%: CMD = framecrc -lavfi testsrc2=s=64x64:r=1:d=1,format=$(@:fate-filter-vignette-backward-%=%),vignette=a=0:x0=0:y0=0:mode=backward -pix_fmt $(@:fate-filter-vignette-backward-%=%) + FATE_FILTER-$(call FILTERFRAMECRC, TESTSRC2 FPS MPDECIMATE) += fate-filter-mpdecimate fate-filter-mpdecimate: CMD = framecrc -lavfi testsrc2=r=2:d=10,fps=3,mpdecimate -pix_fmt yuv420p diff --git a/tests/ref/fate/filter-vignette-backward-rgb24 b/tests/ref/fate/filter-vignette-backward-rgb24 new file mode 100644 index 00000000000..26c62146a23 --- /dev/null +++ b/tests/ref/fate/filter-vignette-backward-rgb24 @@ -0,0 +1,6 @@ +#tb 0: 1/1 +#media_type 0: video +#codec_id 0: rawvideo +#dimensions 0: 64x64 +#sar 0: 1/1 +0, 0, 0, 1, 12288, 0x282440f2 diff --git a/tests/ref/fate/filter-vignette-backward-yuv444p b/tests/ref/fate/filter-vignette-backward-yuv444p new file mode 100644 index 00000000000..b559a9e7559 --- /dev/null +++ b/tests/ref/fate/filter-vignette-backward-yuv444p @@ -0,0 +1,6 @@ +#tb 0: 1/1 +#media_type 0: video +#codec_id 0: rawvideo +#dimensions 0: 64x64 +#sar 0: 1/1 +0, 0, 0, 1, 12288, 0xf08ed17c -- 2.54.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
