PR #24049 opened by Jake (jakefineman)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24049
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24049.patch
The ONNX backend (`dnn_backend_onnx.c`) switches on `model.func_type` in both
`fill_model_input_onnx()` and `infer_completion_callback()`, but only handles
`DFT_PROCESS_FRAME` (plus `DFT_ANALYTICS_DETECT` on the input side).
`DFT_ANALYTICS_CLASSIFY` falls through to the default
`avpriv_report_missing_feature()` branch unconditionally, so `dnn_classify`
cannot use the ONNX backend at all:
```
model function type 3 is not implemented
```
## 1/2 — implement `DFT_ANALYTICS_CLASSIFY`
Adds the classify case to both switches, mirroring the pattern the OpenVINO
backend already uses (`dnn_backend_openvino.c`): the input side calls
`ff_frame_to_dnn_classify()` with the bbox index carried on the task item; the
output side invokes the filter's `classify_post_proc` callback (set by
`vf_dnn_classify.c`) with the same bbox index.
The completion callback ignores `classify_post_proc`'s return value. That is
deliberate — it matches the OpenVINO backend's own precedent exactly, which
calls `classify_post_proc(...)` as a bare statement at
`dnn_backend_openvino.c:473` (and `detect_post_proc` likewise at :463).
Changing that convention is out of scope here; if maintainers want the return
propagated it should be done for both backends in one change.
## 2/2 — register the `onnx` named constant for `dnn_classify`
`dnn_classify` registered only `tensorflow`/`openvino` named `AVOption`
constants for `dnn_backend`, so after 1/2 the ONNX backend was reachable only
by numeric value (`dnn_backend=8`). This registers the `onnx` constant behind
`CONFIG_LIBONNXRUNTIME` — same macro and same position in the per-backend `#if`
chain as `vf_dnn_processing.c` — and adds `DNN_ONNX` to `dnn_classify`'s
`AVFILTER_DNN_DEFINE_CLASS` backend mask so `-h filter=dnn_classify` lists the
ONNX sub-options.
**`dnn_detect` is deliberately not included.** An earlier revision of this
series also registered the constant for `dnn_detect`, added `DNN_ONNX` to its
backend mask, and allowlisted `DNN_ONNX` in its `check_output_nb()`. That was
wrong and has been dropped: `infer_completion_callback()` has no
`DFT_ANALYTICS_DETECT` case, so the ONNX backend cannot produce bounding boxes.
Admitting `dnn_detect` past its init-time rejection would only move the failure
later, from a clear "does not support current backend" at filter init to "model
function type 2 is not implemented" at inference time. Detect support belongs
in a separate change that implements the output-side dispatch first. This
series leaves `vf_dnn_detect.c` byte-identical to master.
## Repro
Build: FFmpeg `n9.0` (pinned commit `d32b387`), debian trixie, ONNX Runtime
1.28.0, `--enable-libonnxruntime`, native linux/arm64.
Before 1/2 — bare code-path check, no model needed:
```
$ ffmpeg -f lavfi -i testsrc2=size=64x64:rate=1:duration=1 \
-vf 'dnn_classify=dnn_backend=8:model=/tmp/x.onnx' -frames:v 1 -f null -
[dnn_backend_onnx] model function type 3 is not implemented
```
After both patches — the named constant parses and the filter reaches ONNX
session creation, erroring only on the missing model file:
```
$ ffmpeg -f lavfi -i testsrc2=size=64x64:rate=1:duration=1 \
-vf 'dnn_classify=dnn_backend=onnx:model=/tmp/nonexistent.onnx' -frames:v 1
-f null -
[dnn_base] Using CPU execution provider
[dnn_base] Failed to create ONNX session: Load model from /tmp/nonexistent.onnx
failed. File doesn't exist
[Parsed_dnn_classify_0] could not load DNN model
```
`-h filter=dnn_classify` now lists it, and `dnn_detect` is unchanged:
```
$ ffmpeg -h filter=dnn_classify | grep onnx
onnx 8 ..FV....... onnx backend flag
$ ffmpeg -h filter=dnn_detect | grep onnx # no output — unchanged by
this series
```
`dnn_detect` with the ONNX backend selected numerically still fails closed at
init, as it does on master:
```
[dnn_detect] Dnn detect filter does not support current backend
```
With a real model (MobileNetV2, `onnx/models` zoo, Apache-2.0, sha256
`c1c513582d56afceff8516c73804e484c81c6a830712ab6d682253f4a3cd042f`) an ONNX
Runtime session loads and the classify path is entered. To be precise about
what that does and does not prove: session creation is reached, but a
standalone `dnn_classify` run then stops at the pre-existing `av_assert0(sd)`
in `ff_frame_to_dnn_classify()` (`dnn_io_proc.c:355`), because `dnn_classify`
expects `AV_FRAME_DATA_DETECTION_BBOXES` side data — it is architecturally
designed to run chained after `dnn_detect`, and a bare `testsrc2` frame has
none. So this is a code-path-entry receipt, not an end-to-end inference receipt.
That assertion is pre-existing, backend-agnostic code that this series does not
touch: `ff_frame_to_dnn_classify()` is called identically from
`dnn_backend_openvino.c:315`, so a standalone `dnn_classify` behaves the same
way on the OpenVINO backend today. This series makes ONNX match the established
behaviour rather than introducing a new failure mode. Whether that bare-frame
case deserves a clean error instead of an assert is a real question, but it
applies to every backend and belongs in its own patch — happy to send one if
maintainers agree.
## Checks
Both patches apply clean with `git am` on `n9.0` and on current `master`, build
with `--enable-libonnxruntime`, and pass `tools/patcheck` with only the
expected "missing changelog entry" advisory.
No existing trac ticket or patchwork series was found for this gap (searched
`dnn_classify`, `dnn_backend_onnx classify`, and the in-flight 20-part
`dnn_classify`-as-multimedia-filter series on patchwork — that series targets
the Torch backend's zero-shot-classification path and does not overlap this
fix).
From 33fb19c89f88fb158b876337a1ed6090d5147f35 Mon Sep 17 00:00:00 2001
From: Jake Fineman <[email protected]>
Date: Fri, 7 Aug 2026 17:55:34 -0400
Subject: [PATCH 1/2] avfilter/dnn_backend_onnx: implement
DFT_ANALYTICS_CLASSIFY
The ONNX backend switch()es on model.func_type in both the input-prep
path (fill_model_input_onnx) and the output-extraction path
(infer_completion_callback), but only DFT_PROCESS_FRAME and (on the
input side) DFT_ANALYTICS_DETECT were handled; DFT_ANALYTICS_CLASSIFY
fell through to the default avpriv_report_missing_feature() branch
unconditionally, regardless of model shape ("model function type 3 is
not implemented").
Add the classify case to both switches, mirroring the pattern already
used by the OpenVINO backend (dnn_backend_openvino.c): the input path
calls ff_frame_to_dnn_classify() with the per-bounding-box index
carried on the LastLevelTaskItem, and the output path invokes the
filters classify_post_proc callback (set by vf_dnn_classify.c) with
that same bbox index.
Signed-off-by: Jake Fineman <[email protected]>
---
libavfilter/dnn/dnn_backend_onnx.c | 14 ++++++++++++++
1 file changed, 14 insertions(+)
diff --git a/libavfilter/dnn/dnn_backend_onnx.c
b/libavfilter/dnn/dnn_backend_onnx.c
index 6c75d6eb24..1bdd4ea1da 100644
--- a/libavfilter/dnn/dnn_backend_onnx.c
+++ b/libavfilter/dnn/dnn_backend_onnx.c
@@ -383,6 +383,9 @@ static int fill_model_input_onnx(ONNXModel *onnx_model,
ONNXRequestItem *request
case DFT_ANALYTICS_DETECT:
ff_frame_to_dnn_detect(task->in_frame, &input, ctx);
break;
+ case DFT_ANALYTICS_CLASSIFY:
+ ff_frame_to_dnn_classify(task->in_frame, &input, lltask->bbox_index,
ctx);
+ break;
default:
avpriv_report_missing_feature(ctx, "model function type %d",
onnx_model->model.func_type);
ret = AVERROR(ENOSYS);
@@ -599,6 +602,17 @@ static void infer_completion_callback(void *args)
task->out_frame->height =
outputs.dims[dnn_get_height_idx_by_layout(outputs.layout)];
}
break;
+ case DFT_ANALYTICS_CLASSIFY:
+ if (!onnx_model->model.classify_post_proc) {
+ av_log(ctx, AV_LOG_ERROR, "classify filter needs to provide post
proc\n");
+ av_free(dims);
+ g_ort->ReleaseTensorTypeAndShapeInfo(tensor_info);
+ goto err;
+ }
+ onnx_model->model.classify_post_proc(task->in_frame, &outputs,
+ lltask->bbox_index,
+ onnx_model->model.filter_ctx);
+ break;
default:
avpriv_report_missing_feature(ctx, "model function type %d",
onnx_model->model.func_type);
av_free(dims);
--
2.52.0
From 3c615716f60a33aa2a1cf4d750deb42f1164224a Mon Sep 17 00:00:00 2001
From: Jake Fineman <[email protected]>
Date: Fri, 7 Aug 2026 17:55:34 -0400
Subject: [PATCH 2/2] avfilter/vf_dnn_classify: register named onnx backend
constant
dnn_classify only registered "tensorflow"/"openvino" named AVOption
constants for dnn_backend, even though the ONNX execution provider is
reachable by numeric value (dnn_backend=8, DNN_ONNX). This left "onnx"
undefined as a symbolic option value ("Undefined constant or missing
(in onnx") and excluded DNN_ONNX from the backend mask that gates the
filter's child-class option iteration, so -h filter=dnn_classify never
listed the ONNX-backend sub-options.
Register the "onnx" named constant behind CONFIG_LIBONNXRUNTIME,
matching the existing pattern in vf_dnn_processing.c, and add DNN_ONNX
to dnn_classify's AVFILTER_DNN_DEFINE_CLASS backend mask.
dnn_detect is deliberately left alone. The ONNX backend's
infer_completion_callback() has no DFT_ANALYTICS_DETECT case, so it
cannot produce bounding boxes: advertising the backend there would
admit dnn_detect past its init-time check_output_nb() rejection only
to fail later at inference with "model function type 2 is not
implemented", replacing a clear early error with a late one. Detect
support belongs in a separate change that implements the output-side
dispatch first.
Signed-off-by: Jake Fineman <[email protected]>
---
libavfilter/vf_dnn_classify.c | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
diff --git a/libavfilter/vf_dnn_classify.c b/libavfilter/vf_dnn_classify.c
index f92c41ab76..8faf6685ee 100644
--- a/libavfilter/vf_dnn_classify.c
+++ b/libavfilter/vf_dnn_classify.c
@@ -48,6 +48,9 @@ static const AVOption dnn_classify_options[] = {
{ "dnn_backend", "DNN backend", OFFSET(backend_type),
AV_OPT_TYPE_INT, { .i64 = DNN_OV }, INT_MIN, INT_MAX, FLAGS, .unit =
"backend" },
#if (CONFIG_LIBOPENVINO == 1)
{ "openvino", "openvino backend flag", 0,
AV_OPT_TYPE_CONST, { .i64 = DNN_OV }, 0, 0, FLAGS, .unit = "backend" },
+#endif
+#if (CONFIG_LIBONNXRUNTIME == 1)
+ { "onnx", "onnx backend flag", 0,
AV_OPT_TYPE_CONST, { .i64 = DNN_ONNX }, 0, 0, FLAGS, .unit = "backend" },
#endif
{ "confidence", "threshold of confidence", OFFSET2(confidence),
AV_OPT_TYPE_FLOAT, { .dbl = 0.5 }, 0, 1, FLAGS},
{ "labels", "path to labels file", OFFSET2(labels_filename),
AV_OPT_TYPE_STRING, { .str = NULL }, 0, 0, FLAGS },
@@ -55,7 +58,7 @@ static const AVOption dnn_classify_options[] = {
{ NULL }
};
-AVFILTER_DNN_DEFINE_CLASS(dnn_classify, DNN_OV);
+AVFILTER_DNN_DEFINE_CLASS(dnn_classify, DNN_OV | DNN_ONNX);
static int dnn_classify_post_proc(AVFrame *frame, DNNData *output, uint32_t
bbox_index, AVFilterContext *filter_ctx)
{
--
2.52.0
_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
