PR #25537 opened by Jake (jakefineman)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25537
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25537.patch
Two fixes for dnn_classify with the OpenVINO backend. They are complementary to
#24736 and apply cleanly together with it.
**Problem**
1. With OpenVINO 2.0, fill_model_input_ov() always describes the input tensor
as NHWC and takes its dimensions from the model input after preprocessing.
dnn_classify does not call ff_dnn_get_input(), so without layout= the model
layout is never resolved and an NCHW model keeps its shape. For googlenet-12,
[1,3,224,224], the bounding box is scaled into a 224x3 packed BGR image, which
writes 2016 of the 150528 bytes of the tensor from ov_tensor_create(). The rest
is uninitialized heap memory, so the labels and confidences change from run to
run for the same input. A single-channel NHWC model is worse: dnn_detect and
dnn_classify write three bytes per pixel and overrun the tensor.
2. dnn_execute_model_ov() refuses synchronous classification with ENOSYS, which
vf_dnn_classify turns into EIO, so dnn_classify with async=0 fails on the first
frame on master, for every model.
**Fix**
1. In fill_model_input_ov(), fail with a hint to set layout=nchw when the NHWC
channel dimension of a dnn_detect or dnn_classify input is not 3. Both filters
always write 3-channel packed pixels. dnn_processing is unchanged.
2. Run synchronous requests until the inference queue is empty, as the ONNX
Runtime backend does, since classification queues one inference per matching
bounding box, possibly none. execute_model_ov() now checks that its own
inference completed rather than the whole task.
**Relation to #24736**
#24736 makes dnn_classify call ff_dnn_get_input(), so the layout is resolved
from the shape and googlenet-12 without layout= is filled correctly. That fixes
the default case. The first patch here covers what is left: an explicit
layout=nhwc on an NCHW model, a shape the dims[1] <= 3 guess gets wrong, and
the single-channel overrun. With the check, the filter either writes the whole
input tensor or fails. The second patch is independent of #24736: async=0
classification fails with EIO on master and with #24736.
**Testing**
OpenVINO 2026.4, macOS arm64. dnn_detect uses a synthetic [1,3,64,64] SSD model
with a constant output (one box per frame above the threshold). dnn_classify
uses googlenet-12 or a synthetic [1,3,64,64] classifier with a constant output.
The input is 25 identical frames:
ffmpeg -nostdin -nostats -hide_banner -f lavfi -i
smptebars=s=160x120:r=25:d=1,format=bgr24 \
-vf
"dnn_detect=dnn_backend=openvino:model=ssd.onnx:input=data:output=det:layout=nchw:async=A,\
dnn_classify=dnn_backend=openvino:model=googlenet-12.onnx:input=data_0:output=prob_1:confidence=0[:layout=nchw]:async=A,showinfo"
\
-f null -
Each case ran 10 times. The result of a run is a hash of the per-frame bounding
box and classify lines from showinfo.
| build | classifier | layout= | async | result |
|---|---|---|---|---|
| master | googlenet-12 | unset | 1 | 10 different results in 10 runs, 14
different per-frame results in one run |
| master | googlenet-12 | nchw | 1 | identical in all runs and frames (A) |
| master | any | any | 0 | exit 251, "classify for sync execution is not
implemented" |
| master + #24736 | googlenet-12 | unset | 1 | identical to A |
| this PR | googlenet-12 | unset | 1, 0 | exit 251, "Model input has 224
channels in NHWC layout, expected 3; set layout=nchw for an NCHW model." |
| this PR | googlenet-12 | nchw | 1, 0 | identical to A |
| this PR | synthetic | nchw | 1, 0 | identical to master with async=1 |
| this PR | synthetic | unset | 1, 0 | exit 251, "Model input has 64 channels
..." |
The #24736 row comes from a build that also has other pending dnn patches. The
synthetic classifier's output does not depend on its input, so on master it
gives the same result with and without layout=.
Builds without new warnings; make fate-source and git diff --check pass.
Not tested: async=0 with several boxes per frame, IR models without layout
information, and the pre-2.0 OpenVINO API.
>From d1ef24dee4d0caa6c0ecb69cae4f7095da974bd5 Mon Sep 17 00:00:00 2001
From: Jake Fineman <[email protected]>
Date: Tue, 27 Oct 2026 13:06:40 -0400
Subject: [PATCH 1/2] avfilter/dnn_backend_openvino: reject non-NHWC RGB
analytics input
With OpenVINO 2.0, fill_model_input_ov() always describes the input
tensor as NHWC and takes its dimensions from the model input after
preprocessing. dnn_classify does not call ff_dnn_get_input(), so
without layout= the model layout stays unknown and an NCHW model keeps
its shape. For googlenet-12, [1,3,224,224], dnn_classify reads it as
H=3, W=224, C=224 and scales the bounding box into a 224x3 packed BGR
image. That writes 2016 of the 150528 bytes of the tensor from
ov_tensor_create(); the rest is uninitialized heap memory, and the
labels and confidences change from run to run.
The same happens with layout=nhwc on an NCHW model, or when the layout
get_input_ov() guesses from the shape is wrong. For a single-channel
NHWC model, both dnn_detect and dnn_classify write three bytes per
pixel and overrun the tensor.
Both filters always write 3-channel packed pixels, so fail with a hint
to set layout=nchw when the NHWC channel dimension is not 3.
dnn_processing is unchanged.
Tested with dnn_detect,dnn_classify and googlenet-12 on 25 identical
frames, 10 runs per case: without layout=, master gives 10 different
results in 10 runs, and this patch fails with the hint. With
layout=nchw, the results match master with layout=nchw.
Signed-off-by: Jake Fineman <[email protected]>
---
libavfilter/dnn/dnn_backend_openvino.c | 7 +++++++
1 file changed, 7 insertions(+)
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index 82fd145a860..e23436ce8b7 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -237,6 +237,13 @@ static int fill_model_input_ov(OVModel *ov_model,
OVRequestItem *request)
input.dims[i] = dims[i];
input.layout = DL_NHWC;
input.dt = precision_to_datatype(precision);
+ // detect and classify write packed 3-channel pixels
+ if (ov_model->model.func_type != DFT_PROCESS_FRAME && input.dims[3] != 3) {
+ av_log(ctx, AV_LOG_ERROR, "Model input has %d channels in NHWC layout,
expected 3; "
+ "set layout=nchw for an NCHW model.\n", input.dims[3]);
+ ov_shape_free(&input_shape);
+ return AVERROR(EINVAL);
+ }
#else
status = ie_infer_request_get_blob(request->infer_request,
task->input_name, &input_blob);
if (status != OK) {
--
2.54.0
>From 6cd9439b0899a86ebad716b72ab2101b8e06924f Mon Sep 17 00:00:00 2001
From: Jake Fineman <[email protected]>
Date: Tue, 27 Oct 2026 13:06:46 -0400
Subject: [PATCH 2/2] avfilter/dnn_backend_openvino: support dnn_classify with
async=0
dnn_execute_model_ov() refused synchronous classification with ENOSYS,
which vf_dnn_classify turns into EIO, so dnn_detect,dnn_classify with
async=0 always failed on the first frame that reached dnn_classify.
The synchronous path ran exactly one request per frame. That fits
processing and detection, which queue one inference per frame, but
classification queues one per matching bounding box, possibly none.
With several boxes the frame's task would be left incomplete, and
execute_model_ov() would report an error because inference_done had
not reached inference_todo yet; with none, execute_model_ov() would be
called on an empty queue, where it frees the request instead of
returning it to the pool.
Run requests until the inference queue is empty, as the ONNX Runtime
backend does, and make execute_model_ov() check that its own inference
completed rather than the whole task.
Tested with dnn_detect,dnn_classify on 25 frames, 10 runs per case:
async=0 output is identical to async=1, for googlenet-12 with
layout=nchw and for a synthetic classifier, with one box per frame.
Signed-off-by: Jake Fineman <[email protected]>
---
libavfilter/dnn/dnn_backend_openvino.c | 29 +++++++++++++-------------
1 file changed, 15 insertions(+), 14 deletions(-)
diff --git a/libavfilter/dnn/dnn_backend_openvino.c
b/libavfilter/dnn/dnn_backend_openvino.c
index e23436ce8b7..f2bb22c27d4 100644
--- a/libavfilter/dnn/dnn_backend_openvino.c
+++ b/libavfilter/dnn/dnn_backend_openvino.c
@@ -1023,6 +1023,7 @@ static int execute_model_ov(OVRequestItem *request, Queue
*inferenceq)
}
return 0;
} else {
+ uint32_t done = task->inference_done;
status = ov_infer_request_infer(request->infer_request);
if (status != OK) {
av_log(NULL, AV_LOG_ERROR, "Failed to start synchronous model
inference for OV2\n");
@@ -1030,7 +1031,7 @@ static int execute_model_ov(OVRequestItem *request, Queue
*inferenceq)
goto err;
}
infer_completion_callback(request);
- return (task->inference_done == task->inference_todo) ? 0 :
DNN_GENERIC_ERROR;
+ return task->inference_done > done ? 0 : DNN_GENERIC_ERROR;
}
#else
if (task->async) {
@@ -1048,6 +1049,7 @@ static int execute_model_ov(OVRequestItem *request, Queue
*inferenceq)
}
return 0;
} else {
+ uint32_t done = task->inference_done;
status = ie_infer_request_infer(request->infer_request);
if (status != OK) {
av_log(ctx, AV_LOG_ERROR, "Failed to start synchronous model
inference\n");
@@ -1055,7 +1057,7 @@ static int execute_model_ov(OVRequestItem *request, Queue
*inferenceq)
goto err;
}
infer_completion_callback(request);
- return (task->inference_done == task->inference_todo) ? 0 :
DNN_GENERIC_ERROR;
+ return task->inference_done > done ? 0 : DNN_GENERIC_ERROR;
}
#endif
err:
@@ -1550,24 +1552,23 @@ static int dnn_execute_model_ov(const DNNModel *model,
DNNExecBaseParams *exec_p
return 0;
}
else {
- if (model->func_type == DFT_ANALYTICS_CLASSIFY) {
- // Classification filter has not been completely
- // tested with the sync mode. So, do not support now.
- avpriv_report_missing_feature(ctx, "classify for sync execution");
- return AVERROR(ENOSYS);
- }
-
if (ctx->batch_size > 1) {
avpriv_report_missing_feature(ctx, "batch mode for sync
execution");
return AVERROR(ENOSYS);
}
- request = ff_safe_queue_pop_front(ov_model->request_queue);
- if (!request) {
- av_log(ctx, AV_LOG_ERROR, "unable to get infer request.\n");
- return AVERROR(EINVAL);
+ // classify queues one inference per bounding box, possibly none
+ while (ff_queue_size(ov_model->lltask_queue) > 0) {
+ request = ff_safe_queue_pop_front(ov_model->request_queue);
+ if (!request) {
+ av_log(ctx, AV_LOG_ERROR, "unable to get infer request.\n");
+ return AVERROR(EINVAL);
+ }
+ ret = execute_model_ov(request, ov_model->lltask_queue);
+ if (ret != 0)
+ return ret;
}
- return execute_model_ov(request, ov_model->lltask_queue);
+ return 0;
}
fail:
av_frame_free(&exec_params->in_frame);
--
2.54.0
_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
