PR #24048 opened by Jake (jakefineman)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24048
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24048.patch
The `AV_PIX_FMT_RGB24`/`AV_PIX_FMT_BGR24` output case in
`ff_proc_from_dnn_to_frame()` (`libavfilter/dnn/dnn_io_proc.c`) hardcodes the
`sws_getContext`/`sws_scale` width arguments and strides to `frame->width * 3`,
unconditional on the model's real output channel count. For any model whose
output has fewer than 3 channels — e.g. a single-channel alpha-matte output
from an RGB-in background-removal model such as MODNet — that mismatch is
memory-unsafe on both layouts:
- The first `sws_scale` runs for **both** `DL_NCHW` and `DL_NHWC` output,
before the planar-to-packed branch, and reads `frame->width * 3 *
src_datatype_size * frame->height` bytes from `output->data`; a 1-channel
tensor holds a third of that: a ~3x source-side over-read.
- On `DL_NCHW`, `middle_data` is correctly sized for the real channel count
(`plane_size * output->dims[1]`), so the same hardcoded width also makes
`sws_scale` write `frame->width * 3 * frame->height` bytes into a buffer sized
for only `frame->width * frame->height * output->dims[1]` bytes: a heap write
3x past the allocation for `dims[1] == 1`. glibc's allocator catches this on a
subsequent `free()`/malloc consistency check, producing `SIGABRT`, not a clean
FFmpeg error return.
## Fix
Reject channel-mismatched model output at the top of the RGB24/BGR24 case,
layout-agnostically: read the channel dim via
`dnn_get_channel_idx_by_layout(output->layout)` (the same helper this file
already uses on the input side, `dnn_io_proc.c:370`), and if
`output->dims[channel_idx] != 3`, log a clear error and return
`AVERROR(ENOSYS)` before any of the mismatched-stride arithmetic runs.
One check covers all three hazards: the source-side over-read (both layouts),
the `DL_NCHW` destination over-write, and the `DL_NCHW` planar-to-packed GBRP
repack a few lines further down, which likewise unconditionally assumes 3
planes and is never reached once channel count is checked.
## Repro
Build: FFmpeg `n9.0` (pinned commit `d32b387`), debian trixie, ONNX Runtime
1.28.0, `--enable-libonnxruntime`, native linux/arm64. Model:
`onnx-community/modnet-webnn` `onnx/model.onnx`, sha256
`07c308cf0fc7e6e8b2065a12ed7fc07e1de8febb7dc7839d7b7f15dd66584df9` (output
`[1,1,H,W]`, i.e. `DL_NCHW` with `dims[1] == 1`).
**Before** — identical build with this patch omitted:
```
$ ffmpeg -f lavfi -i testsrc2=size=320x320:rate=1:duration=1 \
-vf
'format=rgb24,dnn_processing=dnn_backend=onnx:model=modnet.onnx:input=input:output=output'
\
-frames:v 1 -f null -
[dnn_base] Using CPU execution provider
free(): invalid pointer
$ echo $?
134
```
**After** — same command, same model, patched build:
```
[dnn_base] dnn_processing to a rgb24 frame requires a 3-channel model output,
got 1 channels;
channel-reducing/expanding models (e.g. single-channel matte output) are not
supported by this filter
$ echo $?
0
```
Scope note on the receipt: the runtime repro above exercises the `DL_NCHW`
destination over-write, which is the path that aborts. The `DL_NHWC`
source-side over-read is identified from source rather than separately
reproduced — I did not have a channel-reducing NHWC model to hand — but it runs
through the same hardcoded-stride `sws_scale` call above the layout branch,
which is why the guard is keyed on the layout-resolved channel index rather
than on `dims[1]`.
## Deliberate scope
This patch rejects channel-reducing model output cleanly; it does **not** add
support for actually rendering a 1-channel (or 2-channel) model output through
this filter. Real support for that would need a new pixel-format/plane-count
target on the output side (e.g. gray/ya8 for a matte, or a caller-supplied
channel mapping) — a bigger design decision than a minimal heap-safety fix
should make unilaterally, and out of scope here.
I'm sending the minimal safety fix first because the crash is the more urgent
problem: a SIGABRT driven by model-controlled output shape is a hardening bug
independent of whether anyone wants channel-reducing rendering. Happy to follow
up with a rendering-support patch, or take direction on it, if a maintainer
prefers that be done in the same series instead of split.
One further caveat, so the receipt does not overclaim: `vf_dnn_processing.c`'s
own caller does not hard-abort the filter pipeline on this specific
`AVERROR(ENOSYS)` return (pre-existing behaviour of that file, untouched here)
— for a real (non-`-f null`) pipeline the practical effect is an
unfiltered/passthrough frame rather than a full pipeline failure.
## Checks
Applies clean with `git am` on both `n9.0` and current `master` (`master`'s
`dnn_io_proc.c` still hardcodes `frame->width * 3` unconditionally as of this
writing — not yet fixed upstream). Builds with `--enable-libonnxruntime`.
`tools/patcheck` reports only the expected "missing changelog entry" advisory
for a one-function bugfix.
No existing trac ticket or patchwork series found for this bug in a bounded
search (`dnn_processing sws_scale`, `dnn_io_proc heap`, `background removal
matte crash dnn`).
From b3f3be7585bd7ac299c044316843acfd047c8626 Mon Sep 17 00:00:00 2001
From: Jake Fineman <[email protected]>
Date: Fri, 7 Aug 2026 17:57:03 -0400
Subject: [PATCH] avfilter/dnn: reject channel-mismatched model output in
dnn_processing
ff_proc_from_dnn_to_frame()'s AV_PIX_FMT_RGB24/BGR24 case assumes the
model's output has exactly 3 channels, but never checks it.
The first sws_scale() in that case reads from output->data with a
hardcoded source stride of frame->width * 3 * src_datatype_size over
frame->height rows, i.e. frame->width * 3 * frame->height *
src_datatype_size bytes. For a model whose output has fewer channels
than that -- e.g. the single-channel alpha matte an RGB-in
background-removal model produces -- the real tensor only holds
frame->width * frame->height * channels * src_datatype_size bytes, so
the call reads roughly 3x past the end of it. This affects both
DL_NCHW and DL_NHWC output, since it happens before the layout-specific
planar-to-packed branch.
For DL_NCHW there is a second, independent overflow on the write side:
middle_data is allocated plane_size * output->dims[1] bytes, but
linesize[0] is set to frame->width * 3 unconditionally, so the same
sws_scale() also writes past that allocation. This reproduces as a
SIGABRT (glibc "free(): invalid pointer", or a malloc.c assertion,
depending on allocator state) inside dnn_processing, after inference
has already completed successfully in the backend.
The input side is guarded -- check_modelinput_inlink() in
vf_dnn_processing.c -- but the output side is not. dnn_processing only
supports channel-preserving models for these pixel formats by design,
so reject a mismatched channel count with a clear error instead of
reading and writing out of bounds. The channel dimension is resolved
via dnn_get_channel_idx_by_layout() so the check covers both layouts.
This does not add support for rendering a 1- or 2-channel model output
through this filter; that would need a different pixel-format target
and is left as future work.
Signed-off-by: Jake Fineman <[email protected]>
---
libavfilter/dnn/dnn_io_proc.c | 12 ++++++++++++
1 file changed, 12 insertions(+)
diff --git a/libavfilter/dnn/dnn_io_proc.c b/libavfilter/dnn/dnn_io_proc.c
index 826110dab0..55981c64da 100644
--- a/libavfilter/dnn/dnn_io_proc.c
+++ b/libavfilter/dnn/dnn_io_proc.c
@@ -50,6 +50,7 @@ int ff_proc_from_dnn_to_frame(AVFrame *frame, DNNData
*output, void *log_ctx)
int plane_size = frame->width * frame->height * sizeof(uint8_t);
enum AVPixelFormat src_fmt = AV_PIX_FMT_NONE;
int src_datatype_size = get_datatype_size(output->dt);
+ int channel_idx;
int bytewidth = av_image_get_linesize(frame->format, frame->width, 0);
if (bytewidth < 0) {
@@ -83,6 +84,17 @@ int ff_proc_from_dnn_to_frame(AVFrame *frame, DNNData
*output, void *log_ctx)
switch (frame->format) {
case AV_PIX_FMT_RGB24:
case AV_PIX_FMT_BGR24:
+ channel_idx = dnn_get_channel_idx_by_layout(output->layout);
+ if (output->dims[channel_idx] != 3) {
+ av_log(log_ctx, AV_LOG_ERROR,
+ "dnn_processing to a %s frame requires a 3-channel model "
+ "output, got %d channels; channel-reducing/expanding "
+ "models (e.g. single-channel matte output) are not "
+ "supported by this filter\n",
+ av_get_pix_fmt_name(frame->format),
output->dims[channel_idx]);
+ ret = AVERROR(ENOSYS);
+ goto err;
+ }
sws_ctx = sws_getContext(frame->width * 3,
frame->height,
src_fmt,
--
2.52.0
_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
