PR #25545 opened by Jake (jakefineman) URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25545 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25545.patch
Three fixes in dnn_io_proc.c, where the frame/tensor helpers copy a fixed number of bytes per pixel whatever the backend allocated for the model. 1. ff_frame_to_dnn_classify() always writes three channels. A 1-channel classifier overflows the input tensor: on ONNX by 2*H*W floats per bbox, and on OpenVINO 2.0 (u8 NHWC tensor) by 2*H*W bytes. A 4-channel model on ONNX runs with an uninitialized fourth plane. The check now requires three channels. 2. ff_proc_from_dnn_to_frame() converts gray8/YUV/NV12 output as GRAYF32 even when the output tensor is UINT8. OpenVINO with scale=1 gives a U8 output, so the output is garbage and swscale reads past the tensor. grayf32 frames copy 4*W bytes per row in both directions; on the input side OpenVINO always uses a U8 tensor, so every grayf32 frame corrupts the heap. The patch uses the real source format for the output conversion and rejects non-float tensors for grayf32 frames. 3. ff_frame_to_dnn_detect() skips its 3-channel check for packed uint8 inputs, which take the direct sws_scale() path. A uint8 [1,-1,-1,1] TensorFlow detector, or a 1-channel OpenVINO detector, gets 2*H*W bytes written past the tensor on every frame. The check now applies to all inputs. The OpenVINO and TensorFlow backends ignore the return value of the input helpers, so on those backends a rejected model logs an error on every frame and inference still runs on an unfilled tensor. The heap is no longer corrupted, but the filter does not fail. Propagating these errors touches the same paths as #24702, #24760 and #24761, so I'll send it separately once those are in. #24702 already makes the backends propagate ff_proc_from_dnn_to_frame() errors. #25537 also rejects the 1-channel OpenVINO classify and detect models, in fill_model_input_ov(). Behaviour change: OpenVINO classify with an NCHW model and no layout= option now logs an error, because the model's W is read as the channel count. Before, it ran on a wrongly laid out input without overflowing. #25537 rejects this case too. **Testing** macOS arm64. Base 215351c1c3. Each case was run on the base and the patched build. "gmalloc" means libgmalloc with guard pages, so any access past the end of a heap block faults. Most rows come from one OpenVINO + ONNX Runtime + TensorFlow build of all three commits. The ASan row used a minimal ASan build of the classify commit, and the default-scale and yuv420p scale=1 rows were run on the gray commit alone. | case | base | this PR | |---|---|---| | ONNX classify, 1-channel model (ASan) | heap-buffer-overflow in detect_write_tensor | exit 251, "requires a 3-channel input, but the model has 1 channels" | | ONNX classify, 4-channel model | exit 0, plane 3 never written | exit 251, same error | | ONNX classify, 3-channel model | | framemd5 identical to base | | OpenVINO classify, 1-channel, layout=nchw, gmalloc | SIGSEGV | error logged on every frame, exit 0 | | OpenVINO classify, 3-channel, layout=nchw, gmalloc | | bbox side data identical to base | | OpenVINO detect, 1-channel, gmalloc | SIGSEGV | error logged on every frame, exit 0 | | OpenVINO detect, 3-channel | | bbox side data identical to base | | TF detect, uint8 [1,-1,-1,1], gmalloc and plain | SIGBUS in sws_scale from ff_frame_to_dnn_detect | error logged on every frame, exit 0 | | TF detect, uint8 3-channel | no detections, exit 0 | same, no channel error | | ONNX detect, 3-channel | 5 bboxes | identical | | OpenVINO gray8, 64x64 identity model, scale=1 | every frame differs from the input; SIGSEGV with gmalloc | output identical to the input, also with gmalloc | | OpenVINO gray8/yuv420p/rgb24, default scale | | framemd5 identical to base | | OpenVINO yuv420p, scale=1 | garbage Y plane | identical to the input | | OpenVINO grayf32 frames, gmalloc | SIGSEGV (heap corruption without gmalloc) | "requires a float input tensor" logged, exit 0 | The grayf32 output-side check is not reachable with ffmpeg on these backends, because OpenVINO rejects grayf32 on the input side first. It was tested by calling ff_proc_from_dnn_to_frame() directly with a UINT8 tensor. Not tested: Torch (it only uses float tensors for dnn_processing, so the changed branches do not apply), a TF detector that actually detects something, ASan with OpenVINO, and real-world 1-channel models. All the models here are small synthetic ones. There is no FATE coverage for the DNN filters. >From fd6b7343f875caae3e4fe652ade63271488db27e Mon Sep 17 00:00:00 2001 From: Jake Fineman <[email protected]> Date: Tue, 27 Oct 2026 15:09:47 -0400 Subject: [PATCH 1/3] avfilter/dnn_io_proc: reject non-3-channel inputs in classify ff_frame_to_dnn_classify() writes three values per pixel into the input tensor. For NCHW inputs it scales the bbox crop to RGB24 and deinterleaves it with detect_write_tensor(), which always writes three planes; otherwise it scales the crop with sws_scale() straight into the tensor as packed BGR24 or RGB24. The tensor is sized by the backend from the model's own channel count, though. The ONNX backend allocates N*C*H*W floats in fill_model_input_onnx(), so a float [1,1,H,W] classifier gets 2*H*W floats written past the end of the heap buffer for every classified bbox, and a 4-channel model is run with its fourth plane left uninitialized. With OpenVINO 2.0, the u8 NHWC tensor created by the preprocessing keeps the model's channel count, so a [1,H,W,1] model, or a [1,1,H,W] model with layout=nchw, gets an H*W byte tensor that sws_scale() fills with 3*H*W bytes. Check that the input has three channels before writing to it, and return ENOSYS with an error naming the channel count instead of corrupting the heap. With the ONNX backend the filter then fails. The OpenVINO backend ignores the return value of ff_frame_to_dnn_classify(), so there the error is logged and inference runs on an input tensor that was not filled. Signed-off-by: Jake Fineman <[email protected]> --- libavfilter/dnn/dnn_io_proc.c | 9 ++++++++- 1 file changed, 8 insertions(+), 1 deletion(-) diff --git a/libavfilter/dnn/dnn_io_proc.c b/libavfilter/dnn/dnn_io_proc.c index 7c58d5cd02d..1562841a95e 100644 --- a/libavfilter/dnn/dnn_io_proc.c +++ b/libavfilter/dnn/dnn_io_proc.c @@ -366,7 +366,7 @@ int ff_frame_to_dnn_classify(AVFrame *frame, DNNData *input, uint32_t bbox_index int ret = 0; enum AVPixelFormat fmt; int left, top, width, height; - int width_idx, height_idx; + int width_idx, height_idx, channel_idx; const AVDetectionBBoxHeader *header; const AVDetectionBBox *bbox; AVFrameSideData *sd = av_frame_get_side_data(frame, AV_FRAME_DATA_DETECTION_BBOXES); @@ -383,6 +383,13 @@ int ff_frame_to_dnn_classify(AVFrame *frame, DNNData *input, uint32_t bbox_index width_idx = dnn_get_width_idx_by_layout(input->layout); height_idx = dnn_get_height_idx_by_layout(input->layout); + channel_idx = dnn_get_channel_idx_by_layout(input->layout); + if (input->dims[channel_idx] != 3) { + av_log(log_ctx, AV_LOG_ERROR, "dnn_classify requires a 3-channel input, " + "but the model has %d channels\n", + input->dims[channel_idx]); + return AVERROR(ENOSYS); + } header = (const AVDetectionBBoxHeader *)sd->data; bbox = av_get_detection_bbox(header, bbox_index); -- 2.54.0 >From d3b8dd3994b24b4c9c9c4ae0a1e5389ee3519ec6 Mon Sep 17 00:00:00 2001 From: Jake Fineman <[email protected]> Date: Tue, 27 Oct 2026 15:13:23 -0400 Subject: [PATCH 2/3] avfilter/dnn_io_proc: honour the tensor data type for gray frames ff_proc_from_dnn_to_frame() picks src_fmt from output->dt (GRAY8 for DNN_UINT8, GRAYF32 for DNN_FLOAT), but the GRAY8/YUV/NV12 case then ignores it and always hands AV_PIX_FMT_GRAYF32 to swscale, while the source stride is computed from the real data type. With a UINT8 output tensor every row is read as 4 * width bytes from a width * height buffer: the output is garbage and the last rows read up to 3 * width bytes past the end of the tensor. This is reached with the OpenVINO backend and scale=1 (mean=0), where the output tensor is set to U8, e.g. -vf format=gray,dnn_processing=dnn_backend=openvino:model=m.onnx:scale=1 With a 64x64 gray identity model every output frame differs from the input (framemd5), and with libgmalloc the run crashes in float_y_to_uint_y_wrapper(). Calling ff_proc_from_dnn_to_frame() directly with a 64x64 UINT8 tensor, all 64 rows of the gray8 frame differ from the tensor. The GRAYF32 frame cases in both directions have the same assumption: they copy 4 * width bytes per row whatever the tensor type is. On the input side this is reached with OpenVINO for every grayf32 frame, since the backend always sets the input tensor to U8 and leaves the scaling to its preprocessing; 4 * width * height bytes are written into a width * height tensor, which corrupts the heap. Use src_fmt for the GRAY8/YUV/NV12 output conversion, and reject a non-float tensor for grayf32 frames on both sides with ENOSYS instead of copying the wrong number of bytes. Signed-off-by: Jake Fineman <[email protected]> --- libavfilter/dnn/dnn_io_proc.c | 14 +++++++++++++- 1 file changed, 13 insertions(+), 1 deletion(-) diff --git a/libavfilter/dnn/dnn_io_proc.c b/libavfilter/dnn/dnn_io_proc.c index 1562841a95e..136608383c2 100644 --- a/libavfilter/dnn/dnn_io_proc.c +++ b/libavfilter/dnn/dnn_io_proc.c @@ -151,6 +151,12 @@ int ff_proc_from_dnn_to_frame(AVFrame *frame, DNNData *output, void *log_ctx) } break; case AV_PIX_FMT_GRAYF32: + if (output->dt != DNN_FLOAT) { + av_log(log_ctx, AV_LOG_ERROR, "dnn_processing to a grayf32 frame " + "requires a float output tensor\n"); + ret = AVERROR(ENOSYS); + goto err; + } av_image_copy_plane(frame->data[0], frame->linesize[0], output->data, bytewidth, bytewidth, frame->height); @@ -164,7 +170,7 @@ int ff_proc_from_dnn_to_frame(AVFrame *frame, DNNData *output, void *log_ctx) case AV_PIX_FMT_NV12: sws_ctx = sws_getContext(frame->width, frame->height, - AV_PIX_FMT_GRAYF32, + src_fmt, frame->width, frame->height, AV_PIX_FMT_GRAY8, @@ -293,6 +299,12 @@ int ff_proc_from_frame_to_dnn(AVFrame *frame, DNNData *input, void *log_ctx) sws_freeContext(sws_ctx); break; case AV_PIX_FMT_GRAYF32: + if (input->dt != DNN_FLOAT) { + av_log(log_ctx, AV_LOG_ERROR, "dnn_processing from a grayf32 frame " + "requires a float input tensor\n"); + ret = AVERROR(ENOSYS); + goto err; + } av_image_copy_plane(input->data, bytewidth, frame->data[0], frame->linesize[0], bytewidth, frame->height); -- 2.54.0 >From c6a5607e3713ee641392b70718926d926839cb1e Mon Sep 17 00:00:00 2001 From: Jake Fineman <[email protected]> Date: Tue, 27 Oct 2026 15:53:36 -0400 Subject: [PATCH 3/3] avfilter/dnn_io_proc: check the channel count of packed detect inputs ff_frame_to_dnn_detect() only checks for a 3-channel input when the tensor is not uint8 packed (uint8 and not NCHW). That branch scales the frame with sws_scale() straight into input->data as packed BGR24 or RGB24, three bytes per pixel, whatever channel count the backend sized the tensor for. With TensorFlow, get_input_tf() accepts TF_UINT8 inputs and always sets DL_NHWC, and allocate_input_tensor() sizes the tensor H*W*C from the model's channel count. A uint8 [1,-1,-1,1] detection model therefore gets 2*H*W bytes written past the end of its input tensor for every frame. With OpenVINO 2.0 the u8 NHWC tensor created by the preprocessing keeps the model's channel count, so a 1-channel model hits the same overflow. That case is also rejected in the backend, in fill_model_input_ov(), by pending pull request #25537. Check the channel count for all inputs. packed_u8 still selects the direct sws_scale() path. The TensorFlow and OpenVINO backends ignore the return value of ff_frame_to_dnn_detect(), so on those backends the check prevents the overflow and logs an error, but inference still runs on an input tensor that was not filled. Signed-off-by: Jake Fineman <[email protected]> --- libavfilter/dnn/dnn_io_proc.c | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libavfilter/dnn/dnn_io_proc.c b/libavfilter/dnn/dnn_io_proc.c index 136608383c2..fca3354c36f 100644 --- a/libavfilter/dnn/dnn_io_proc.c +++ b/libavfilter/dnn/dnn_io_proc.c @@ -574,7 +574,7 @@ int ff_frame_to_dnn_detect(AVFrame *frame, DNNData *input, void *log_ctx) } packed_u8 = (input->dt == DNN_UINT8) && (input->layout != DL_NCHW); - if (!packed_u8 && input->dims[channel_idx] != 3) { + if (input->dims[channel_idx] != 3) { av_log(log_ctx, AV_LOG_ERROR, "dnn_detect requires a 3-channel input, " "but the model has %d channels\n", input->dims[channel_idx]); -- 2.54.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
