PR #24702 opened by Jake (jakefineman) URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24702 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24702.patch
Follow-up to #24048, addressing the review point that its new channel-mismatch error was dropped. **Problem.** Every DNN backend ignored, or jumped past, the result of post-processing: - `dnn_processing` got `DAST_SUCCESS` and forwarded its separately allocated output frame with no pixels written. So a post-processing failure, such as the channel mismatch rejected by #24048, produced an uninitialised frame while the filter reported success. - On a failed `detect_post_proc()`, the ONNX backend skipped `inference_done++`, so on an async backend the task never completed and the result queue stalled. OpenVINO and TensorFlow ignored `detect_post_proc()`/`classify_post_proc()` failures altogether. **Commits** 1. `avfilter/dnn: fix single-pointer arrays passed as 4-plane sws_scale buffers`: both conversion directions passed `&middle_data` (one pointer) where `sws_scale()` reads a four-entry plane array. They now use zero-initialised 4-entry arrays. This predates #24048 and was noted in its review. 2. `avfilter/dnn: propagate post-processing failures through async results`: - `TaskItem` gains a `status` field, set from the default and custom frame post-processing in all four backends. - `ff_dnn_get_result_common()` pops the task, frees both frames and the task, and returns `DAST_FAIL` with the status through a new out-parameter. - `dnn_processing` returns that error instead of forwarding a frame. 3. `avfilter/vf_derain, vf_sr: return the post-processing error` instead of collapsing every failure into `AVERROR(EINVAL)`. 4. `avfilter/dnn: complete analytics tasks whose post-processing failed`: - A failed `detect_post_proc()`/`classify_post_proc()` is recorded in `task->status`, and the lltask still counts as done, so the task completes and reports `DAST_FAIL` instead of stalling. - `dnn_detect` and `dnn_classify` return that error. **Testing** - ONNX Runtime 1.30.0 on macOS arm64. #24048 was applied on top for the mismatch case, with a tiny ONNX model whose output has fewer channels than its input (RGB in, 1 channel out): - Before this series: exit 0, 5 frames forwarded. - After: `Terminating thread with error: Function not implemented`, exit code 178, 0 frames. - A valid 3→3 model still processes all 5 frames, exit 0. - `dnn_detect` (ONNX, Tiny YOLOv2) output is unchanged: 12 frames, same boxes. - Compile-checked with OpenVINO 2026.3.1 and TensorFlow 2.21.0: every changed backend and filter object, no new warnings. - Torch was not available locally. Its change (commit 2 only) is the same 4-line pattern as the ONNX and TF hunks, reviewed by hand. Torch has no analytics post-processing, so commit 4 does not touch it. **Known limitation, not changed here:** Torch's CUDA output path (`fill_model_output_th_cuda()`) returns `void`, so a failure there is not reported through `status`. I can't build or test CUDA here, so I left it rather than send untested code. 🤖 Generated with [Claude Code](https://claude.com/claude-code) From e7d2141e59c2606797d69426efdecf6d5a965807 Mon Sep 17 00:00:00 2001 From: Jake Fineman <[email protected]> Date: Fri, 25 Sep 2026 14:48:37 -0400 Subject: [PATCH 1/4] avfilter/dnn: fix single-pointer arrays passed as 4-plane sws_scale buffers ff_proc_from_dnn_to_frame() and ff_proc_from_frame_to_dnn() take the address of a single `void *middle_data` local (`&middle_data`) for the DL_NCHW intermediate buffer and pass it to sws_scale() cast to a 4-element plane pointer array (`uint8_t * const*` / `const uint8_t **`). &middle_data is the address of one pointer-sized stack slot, not an array of four; sws_scale() indexes dst[1..3]/src[1..3] even for a single-plane GRAY8 conversion (to determine plane strides/validity), so this reads three pointer-sized values past the stack slot holding middle_data. This happens to be harmless in practice on common ABIs, since the adjacent stack slots are read but never written or dereferenced for these single-plane conversions, but it is undefined behavior and an ASan/UBSan stack-buffer-overflow flag waiting to fire, and makes the surrounding buffer-safety work in these two functions harder to reason about. Replace the single pointer with a real, zero-initialized 4-entry `void *middle_data_planes[4]` array, storing the allocated buffer in slot 0 and leaving the unused plane slots explicitly NULL, matching how frame->data (a genuine 4-element array) is used everywhere else in this file. Signed-off-by: Jake Fineman <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> --- libavfilter/dnn/dnn_io_proc.c | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) diff --git a/libavfilter/dnn/dnn_io_proc.c b/libavfilter/dnn/dnn_io_proc.c index 0a18c56b84..08c663be04 100644 --- a/libavfilter/dnn/dnn_io_proc.c +++ b/libavfilter/dnn/dnn_io_proc.c @@ -46,6 +46,7 @@ int ff_proc_from_dnn_to_frame(AVFrame *frame, DNNData *output, void *log_ctx) int linesize[4] = { 0 }; void **dst_data = NULL; void *middle_data = NULL; + void *middle_data_planes[4] = { NULL, NULL, NULL, NULL }; uint8_t *planar_data[4] = { 0 }; int plane_size = frame->width * frame->height * sizeof(uint8_t); enum AVPixelFormat src_fmt = AV_PIX_FMT_NONE; @@ -76,7 +77,8 @@ int ff_proc_from_dnn_to_frame(AVFrame *frame, DNNData *output, void *log_ctx) ret = AVERROR(ENOMEM); goto err; } - dst_data = &middle_data; + middle_data_planes[0] = middle_data; + dst_data = middle_data_planes; linesize[0] = frame->width * 3; } @@ -186,6 +188,7 @@ int ff_proc_from_frame_to_dnn(AVFrame *frame, DNNData *input, void *log_ctx) int linesize[4] = { 0 }; void **src_data = NULL; void *middle_data = NULL; + void *middle_data_planes[4] = { NULL, NULL, NULL, NULL }; uint8_t *planar_data[4] = { 0 }; int plane_size = frame->width * frame->height * sizeof(uint8_t); enum AVPixelFormat dst_fmt = AV_PIX_FMT_NONE; @@ -215,7 +218,8 @@ int ff_proc_from_frame_to_dnn(AVFrame *frame, DNNData *input, void *log_ctx) ret = AVERROR(ENOMEM); goto err; } - src_data = &middle_data; + middle_data_planes[0] = middle_data; + src_data = middle_data_planes; linesize[0] = frame->width * 3; } -- 2.52.0 From 7584dbe1d9969531e604eb254fbb8290ae6a1b50 Mon Sep 17 00:00:00 2001 From: Jake Fineman <[email protected]> Date: Fri, 25 Sep 2026 14:48:37 -0400 Subject: [PATCH 2/4] avfilter/dnn: propagate post-processing failures through async results Every backend's default post-processing call site (ONNX, OpenVINO, TensorFlow, Torch, in their inference-completion callbacks) calls ff_proc_from_dnn_to_frame() -- or a filter-supplied FramePrePostProc callback -- to convert the raw inference output into out_frame, but discards the returned AVERROR. The task is then unconditionally marked complete (inference_done++), so ff_dnn_get_result_common() always returns DAST_SUCCESS with out_frame handed back as though it had been validly written, even when post-processing failed and out_frame is untouched or partially written. vf_dnn_processing.c allocates out_frame separately from in_frame (av_frame_copy_props() only copies metadata, not pixel data), so on a post-processing failure it forwards this unwritten buffer downstream as a normal successful frame. This is exactly the class of bug that motivates rejecting channel-mismatched model output in ff_proc_from_dnn_to_frame() (see the AVERROR(ENOSYS) check added there): rejecting bad output at the post-proc call site only helps if callers actually see the failure instead of a synthetic success. Fix this generically, independent of any one post-proc failure mode: - Add TaskItem.status (dnn_backend_common.h), initialized to 0 by ff_dnn_fill_task(). All four backends now assign the return value of ff_proc_from_dnn_to_frame()/frame_post_proc() into task->status instead of discarding it. - ff_dnn_get_result_common() checks task->status before handing back the frames: on failure it releases both in_frame and out_frame, sets *in/*out to NULL, and returns DAST_FAIL (already defined in DNNAsyncStatusType, previously unused by this path) instead of DAST_SUCCESS. The task is popped and freed either way, so a failed post-proc can't stall async draining. - ff_dnn_get_result_common(), the get_result vtable entry in DNNModule, and the ff_dnn_get_result() wrapper all gain an optional `int *status` out-parameter carrying the stored AVERROR back to the caller when DAST_FAIL is returned; NULL is safe to pass when the specific code isn't needed. - vf_dnn_processing.c's activate() and flush_frame() check for DAST_FAIL and return the stored AVERROR (or AVERROR_EXTERNAL as a fallback) instead of silently continuing. - vf_dnn_classify.c, vf_dnn_detect.c (DFT_ANALYTICS_* function types, which never touch this post-proc path, so task->status stays 0) and vf_derain.c/vf_sr.c (which already treat any async_state != DAST_SUCCESS as AVERROR(EINVAL)) are updated only for the new ff_dnn_get_result() parameter and pass NULL; their existing behavior is unchanged, and derain/sr now also correctly fail instead of forwarding a bad frame if their post-proc callback ever errors. The OpenVINO, TensorFlow and Torch backends are not build-tested in this environment (only libonnxruntime is installed) and are updated by careful reading for consistency with the ONNX backend; the CUDA zero-copy output path in dnn_backend_torch.cpp (fill_model_output_th_cuda) is a separate conversion path that never calls ff_proc_from_dnn_to_frame/frame_post_proc and is left as-is. Signed-off-by: Jake Fineman <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> --- libavfilter/dnn/dnn_backend_common.c | 23 +++++++++++++++++++++-- libavfilter/dnn/dnn_backend_common.h | 18 +++++++++++++++++- libavfilter/dnn/dnn_backend_onnx.c | 8 ++++---- libavfilter/dnn/dnn_backend_openvino.c | 8 ++++---- libavfilter/dnn/dnn_backend_tf.c | 8 ++++---- libavfilter/dnn/dnn_backend_torch.cpp | 12 ++++++------ libavfilter/dnn_filter_common.c | 4 ++-- libavfilter/dnn_filter_common.h | 2 +- libavfilter/dnn_interface.h | 7 +++++-- libavfilter/vf_derain.c | 2 +- libavfilter/vf_dnn_classify.c | 4 ++-- libavfilter/vf_dnn_detect.c | 4 ++-- libavfilter/vf_dnn_processing.c | 18 ++++++++++++++++-- libavfilter/vf_sr.c | 2 +- 14 files changed, 86 insertions(+), 34 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_common.c b/libavfilter/dnn/dnn_backend_common.c index 82627c7a96..1684c222f0 100644 --- a/libavfilter/dnn/dnn_backend_common.c +++ b/libavfilter/dnn/dnn_backend_common.c @@ -63,6 +63,7 @@ int ff_dnn_fill_task(TaskItem *task, DNNExecBaseParams *exec_params, void *backe task->model = backend_model; task->nb_output = exec_params->nb_output; task->output_names = exec_params->output_names; + task->status = 0; return 0; } @@ -141,7 +142,7 @@ int ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_module) return 0; } -DNNAsyncStatusType ff_dnn_get_result_common(Queue *task_queue, AVFrame **in, AVFrame **out) +DNNAsyncStatusType ff_dnn_get_result_common(Queue *task_queue, AVFrame **in, AVFrame **out, int *status) { TaskItem *task = ff_queue_peek_front(task_queue); @@ -153,9 +154,27 @@ DNNAsyncStatusType ff_dnn_get_result_common(Queue *task_queue, AVFrame **in, AVF return DAST_NOT_READY; } + ff_queue_pop_front(task_queue); + + if (task->status < 0) { + // Post-processing of the inference output into out_frame failed + // (either the default ff_proc_from_dnn_to_frame() or a custom + // FramePrePostProc callback). out_frame was never validly + // written, so it must not be handed back as a result; release + // both frames here since the caller has no valid pointer to + // free them through once DAST_FAIL is returned. + if (status) + *status = task->status; + av_frame_free(&task->in_frame); + av_frame_free(&task->out_frame); + *in = NULL; + *out = NULL; + av_freep(&task); + return DAST_FAIL; + } + *in = task->in_frame; *out = task->out_frame; - ff_queue_pop_front(task_queue); av_freep(&task); return DAST_SUCCESS; diff --git a/libavfilter/dnn/dnn_backend_common.h b/libavfilter/dnn/dnn_backend_common.h index f0b8b47397..b91f78166c 100644 --- a/libavfilter/dnn/dnn_backend_common.h +++ b/libavfilter/dnn/dnn_backend_common.h @@ -52,6 +52,15 @@ typedef struct TaskItem { uint32_t nb_output; uint32_t inference_todo; uint32_t inference_done; + /** + * Result of post-processing the inference output into out_frame + * (either the default ff_proc_from_dnn_to_frame() or a custom + * FramePrePostProc callback). 0 on success (the default set by + * ff_dnn_fill_task()), a negative AVERROR if post-processing + * failed. Consumed and reset by ff_dnn_get_result_common(), which + * must not hand back out_frame as valid when this is negative. + */ + int status; } TaskItem; // one task might have multiple inferences @@ -146,12 +155,19 @@ int ff_dnn_start_inference_async(void *ctx, DNNAsyncExecModule *async_module); * @param task_queue pointer to the task queue of the backend * @param in double pointer to the input frame * @param out double pointer to the output frame + * @param status optional pointer to receive the negative AVERROR + * stored on the task when post-processing failed. + * Only written when DAST_FAIL is returned; may be NULL + * if the caller does not need the specific error code. * * @retval DAST_EMPTY_QUEUE if task queue is empty * @retval DAST_NOT_READY if inference not completed yet. + * @retval DAST_FAIL if inference completed but post-processing of the + * output into out_frame failed; *in and *out are + * set to NULL and both frames are released. * @retval DAST_SUCCESS if result successfully extracted */ -DNNAsyncStatusType ff_dnn_get_result_common(Queue *task_queue, AVFrame **in, AVFrame **out); +DNNAsyncStatusType ff_dnn_get_result_common(Queue *task_queue, AVFrame **in, AVFrame **out, int *status); /** * Allocate input and output frames and fill the Task diff --git a/libavfilter/dnn/dnn_backend_onnx.c b/libavfilter/dnn/dnn_backend_onnx.c index 009387b36d..405739c300 100644 --- a/libavfilter/dnn/dnn_backend_onnx.c +++ b/libavfilter/dnn/dnn_backend_onnx.c @@ -699,9 +699,9 @@ static void infer_completion_callback(void *args) if (task->do_ioproc) { outputs[0].scale = 255; if (onnx_model->model.frame_post_proc != NULL) { - onnx_model->model.frame_post_proc(task->out_frame, outputs, onnx_model->model.filter_ctx); + task->status = onnx_model->model.frame_post_proc(task->out_frame, outputs, onnx_model->model.filter_ctx); } else { - ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx); + task->status = ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx); } } else { task->out_frame->width = outputs[0].dims[dnn_get_width_idx_by_layout(outputs[0].layout)]; @@ -1189,10 +1189,10 @@ static int dnn_execute_model_onnx(const DNNModel *model, DNNExecBaseParams *exec return execute_model_onnx(request, onnx_model->lltask_queue); } -static DNNAsyncStatusType dnn_get_result_onnx(const DNNModel *model, AVFrame **in, AVFrame **out) +static DNNAsyncStatusType dnn_get_result_onnx(const DNNModel *model, AVFrame **in, AVFrame **out, int *status) { ONNXModel *onnx_model = (ONNXModel *)model; - return ff_dnn_get_result_common(onnx_model->task_queue, in, out); + return ff_dnn_get_result_common(onnx_model->task_queue, in, out, status); } static int dnn_flush_onnx(const DNNModel *model) diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index 0df4d585e8..651e480df8 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -444,9 +444,9 @@ static void infer_completion_callback(void *args) case DFT_PROCESS_FRAME: if (task->do_ioproc) { if (ov_model->model.frame_post_proc != NULL) { - ov_model->model.frame_post_proc(task->out_frame, outputs, ov_model->model.filter_ctx); + task->status = ov_model->model.frame_post_proc(task->out_frame, outputs, ov_model->model.filter_ctx); } else { - ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx); + task->status = ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx); } } else { task->out_frame->width = @@ -1558,10 +1558,10 @@ static int dnn_execute_model_ov(const DNNModel *model, DNNExecBaseParams *exec_p } } -static DNNAsyncStatusType dnn_get_result_ov(const DNNModel *model, AVFrame **in, AVFrame **out) +static DNNAsyncStatusType dnn_get_result_ov(const DNNModel *model, AVFrame **in, AVFrame **out, int *status) { OVModel *ov_model = (OVModel *)model; - return ff_dnn_get_result_common(ov_model->task_queue, in, out); + return ff_dnn_get_result_common(ov_model->task_queue, in, out, status); } static int dnn_flush_ov(const DNNModel *model) diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index f8a4d9bc62..ef0df1097d 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -722,9 +722,9 @@ static void infer_completion_callback(void *args) { //it only support 1 output if it's frame in & frame out if (task->do_ioproc) { if (tf_model->model.frame_post_proc != NULL) { - tf_model->model.frame_post_proc(task->out_frame, outputs, tf_model->model.filter_ctx); + task->status = tf_model->model.frame_post_proc(task->out_frame, outputs, tf_model->model.filter_ctx); } else { - ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx); + task->status = ff_proc_from_dnn_to_frame(task->out_frame, outputs, ctx); } } else { task->out_frame->width = @@ -847,10 +847,10 @@ static int dnn_execute_model_tf(const DNNModel *model, DNNExecBaseParams *exec_p return execute_model_tf(request, tf_model->lltask_queue); } -static DNNAsyncStatusType dnn_get_result_tf(const DNNModel *model, AVFrame **in, AVFrame **out) +static DNNAsyncStatusType dnn_get_result_tf(const DNNModel *model, AVFrame **in, AVFrame **out, int *status) { TFModel *tf_model = (TFModel *)model; - return ff_dnn_get_result_common(tf_model->task_queue, in, out); + return ff_dnn_get_result_common(tf_model->task_queue, in, out, status); } static int dnn_flush_tf(const DNNModel *model) diff --git a/libavfilter/dnn/dnn_backend_torch.cpp b/libavfilter/dnn/dnn_backend_torch.cpp index 1ecf95e938..bee179800b 100644 --- a/libavfilter/dnn/dnn_backend_torch.cpp +++ b/libavfilter/dnn/dnn_backend_torch.cpp @@ -453,11 +453,11 @@ static void infer_completion_callback(void *args) { outputs.scale = 255; outputs.data = out_slice.data_ptr(); if (th_model->model.frame_post_proc != NULL) { - th_model->model.frame_post_proc(task->out_frame, &outputs, - th_model->model.filter_ctx); + task->status = th_model->model.frame_post_proc(task->out_frame, &outputs, + th_model->model.filter_ctx); } else { - ff_proc_from_dnn_to_frame(task->out_frame, &outputs, - th_model->ctx); + task->status = ff_proc_from_dnn_to_frame(task->out_frame, &outputs, + th_model->ctx); } #if CONFIG_CUDA } @@ -752,10 +752,10 @@ static int dnn_execute_model_th(const DNNModel *model, DNNExecBaseParams *exec_p return 0; } -static DNNAsyncStatusType dnn_get_result_th(const DNNModel *model, AVFrame **in, AVFrame **out) +static DNNAsyncStatusType dnn_get_result_th(const DNNModel *model, AVFrame **in, AVFrame **out, int *status) { THModel *th_model = (THModel *)model; - return ff_dnn_get_result_common(th_model->task_queue, in, out); + return ff_dnn_get_result_common(th_model->task_queue, in, out, status); } static int dnn_flush_th(const DNNModel *model) diff --git a/libavfilter/dnn_filter_common.c b/libavfilter/dnn_filter_common.c index b644a6280f..b123a4c608 100644 --- a/libavfilter/dnn_filter_common.c +++ b/libavfilter/dnn_filter_common.c @@ -216,9 +216,9 @@ int ff_dnn_execute_model_classification(DnnContext *ctx, AVFrame *in_frame, AVFr return (ctx->dnn_module->execute_model)(ctx->model, &class_params.base); } -DNNAsyncStatusType ff_dnn_get_result(DnnContext *ctx, AVFrame **in_frame, AVFrame **out_frame) +DNNAsyncStatusType ff_dnn_get_result(DnnContext *ctx, AVFrame **in_frame, AVFrame **out_frame, int *status) { - return (ctx->dnn_module->get_result)(ctx->model, in_frame, out_frame); + return (ctx->dnn_module->get_result)(ctx->model, in_frame, out_frame, status); } int ff_dnn_flush(DnnContext *ctx) diff --git a/libavfilter/dnn_filter_common.h b/libavfilter/dnn_filter_common.h index 1e89ab42ae..596cab55a4 100644 --- a/libavfilter/dnn_filter_common.h +++ b/libavfilter/dnn_filter_common.h @@ -59,7 +59,7 @@ int ff_dnn_get_input(DnnContext *ctx, DNNData *input); int ff_dnn_get_output(DnnContext *ctx, int input_width, int input_height, int *output_width, int *output_height); int ff_dnn_execute_model(DnnContext *ctx, AVFrame *in_frame, AVFrame *out_frame); int ff_dnn_execute_model_classification(DnnContext *ctx, AVFrame *in_frame, AVFrame *out_frame, const char *target); -DNNAsyncStatusType ff_dnn_get_result(DnnContext *ctx, AVFrame **in_frame, AVFrame **out_frame); +DNNAsyncStatusType ff_dnn_get_result(DnnContext *ctx, AVFrame **in_frame, AVFrame **out_frame, int *status); int ff_dnn_flush(DnnContext *ctx); void ff_dnn_uninit(DnnContext *ctx); #if CONFIG_CUDA diff --git a/libavfilter/dnn_interface.h b/libavfilter/dnn_interface.h index 632ae6be84..810d3c4bdf 100644 --- a/libavfilter/dnn_interface.h +++ b/libavfilter/dnn_interface.h @@ -191,8 +191,11 @@ struct DNNModule { DNNModel *(*load_model)(DnnContext *ctx, DNNFunctionType func_type, AVFilterContext *filter_ctx); // Executes model with specified input and output. Returns the error code otherwise. int (*execute_model)(const DNNModel *model, DNNExecBaseParams *exec_params); - // Retrieve inference result. - DNNAsyncStatusType (*get_result)(const DNNModel *model, AVFrame **in, AVFrame **out); + // Retrieve inference result. status, if non-NULL, receives the + // negative AVERROR stored on the task when DAST_FAIL is returned + // (i.e. inference completed but post-processing the output into + // *out failed); *in and *out are set to NULL in that case. + DNNAsyncStatusType (*get_result)(const DNNModel *model, AVFrame **in, AVFrame **out, int *status); // Flush all the pending tasks. int (*flush)(const DNNModel *model); // Frees memory allocated for model. diff --git a/libavfilter/vf_derain.c b/libavfilter/vf_derain.c index 690a797bdb..c4910d9a1e 100644 --- a/libavfilter/vf_derain.c +++ b/libavfilter/vf_derain.c @@ -75,7 +75,7 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *in) return dnn_result; } do { - async_state = ff_dnn_get_result(&dr_context->dnnctx, &in, &out); + async_state = ff_dnn_get_result(&dr_context->dnnctx, &in, &out, NULL); } while (async_state == DAST_NOT_READY); if (async_state != DAST_SUCCESS) diff --git a/libavfilter/vf_dnn_classify.c b/libavfilter/vf_dnn_classify.c index f92c41ab76..5751e7c4bc 100644 --- a/libavfilter/vf_dnn_classify.c +++ b/libavfilter/vf_dnn_classify.c @@ -216,7 +216,7 @@ static int dnn_classify_flush_frame(AVFilterLink *outlink, int64_t pts, int64_t do { AVFrame *in_frame = NULL; AVFrame *out_frame = NULL; - async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame); + async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame, NULL); if (async_state == DAST_SUCCESS) { ret = ff_filter_frame(outlink, in_frame); if (ret < 0) @@ -259,7 +259,7 @@ static int dnn_classify_activate(AVFilterContext *filter_ctx) do { AVFrame *in_frame = NULL; AVFrame *out_frame = NULL; - async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame); + async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame, NULL); if (async_state == DAST_SUCCESS) { ret = ff_filter_frame(outlink, in_frame); if (ret < 0) diff --git a/libavfilter/vf_dnn_detect.c b/libavfilter/vf_dnn_detect.c index 99dca8546c..846e4c11fc 100644 --- a/libavfilter/vf_dnn_detect.c +++ b/libavfilter/vf_dnn_detect.c @@ -882,7 +882,7 @@ static int dnn_detect_flush_frame(AVFilterLink *outlink, int64_t pts, int64_t *o do { AVFrame *in_frame = NULL; AVFrame *out_frame = NULL; - async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame); + async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame, NULL); if (async_state == DAST_SUCCESS) { ret = ff_filter_frame(outlink, in_frame); if (ret < 0) @@ -925,7 +925,7 @@ static int dnn_detect_activate(AVFilterContext *filter_ctx) do { AVFrame *in_frame = NULL; AVFrame *out_frame = NULL; - async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame); + async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame, NULL); if (async_state == DAST_SUCCESS) { ret = ff_filter_frame(outlink, in_frame); if (ret < 0) diff --git a/libavfilter/vf_dnn_processing.c b/libavfilter/vf_dnn_processing.c index 16904a2c20..5f88734e44 100644 --- a/libavfilter/vf_dnn_processing.c +++ b/libavfilter/vf_dnn_processing.c @@ -276,7 +276,14 @@ static int flush_frame(AVFilterLink *outlink, int64_t pts, int64_t *out_pts) do { AVFrame *in_frame = NULL; AVFrame *out_frame = NULL; - async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame); + int proc_status = 0; + async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame, &proc_status); + if (async_state == DAST_FAIL) { + av_log(ctx, AV_LOG_ERROR, + "DNN post-processing of inference output failed while flushing: %s\n", + av_err2str(proc_status)); + return proc_status < 0 ? proc_status : AVERROR_EXTERNAL; + } if (out_frame) { if (isPlanarYUV(in_frame->format)) copy_uv_planes(ctx, out_frame, in_frame); @@ -328,7 +335,14 @@ static int activate(AVFilterContext *filter_ctx) do { AVFrame *in_frame = NULL; AVFrame *out_frame = NULL; - async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame); + int proc_status = 0; + async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame, &proc_status); + if (async_state == DAST_FAIL) { + av_log(ctx, AV_LOG_ERROR, + "DNN post-processing of inference output failed: %s\n", + av_err2str(proc_status)); + return proc_status < 0 ? proc_status : AVERROR_EXTERNAL; + } if (out_frame) { if (isPlanarYUV(in_frame->format)) copy_uv_planes(ctx, out_frame, in_frame); diff --git a/libavfilter/vf_sr.c b/libavfilter/vf_sr.c index 8f1ccee56a..7eac5ca18d 100644 --- a/libavfilter/vf_sr.c +++ b/libavfilter/vf_sr.c @@ -142,7 +142,7 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *in) } do { - async_state = ff_dnn_get_result(&ctx->dnnctx, &in, &out); + async_state = ff_dnn_get_result(&ctx->dnnctx, &in, &out, NULL); } while (async_state == DAST_NOT_READY); if (async_state != DAST_SUCCESS) -- 2.52.0 From 044d1d9c1570ab144b2ebdab90a16dc15c25a85f Mon Sep 17 00:00:00 2001 From: Jake Fineman <[email protected]> Date: Fri, 25 Sep 2026 18:42:27 -0400 Subject: [PATCH 3/4] avfilter/vf_derain, vf_sr: return the post-processing error ff_dnn_get_result() now reports DAST_FAIL together with the error returned by the frame post-processing step, but vf_derain and vf_sr still passed NULL for the status and collapsed every non-success result into AVERROR(EINVAL). A failed conversion of the model output (for example an output shape that does not match the output frame) was therefore reported as a generic invalid-argument error with no hint of what went wrong. Pass a status pointer, log the failure and return the real error, as vf_dnn_processing already does. Fall back to AVERROR_EXTERNAL if the backend did not store a negative status. On DAST_FAIL the frames have already been released by ff_dnn_get_result(), so nothing else needs freeing here. Signed-off-by: Jake Fineman <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> --- libavfilter/vf_derain.c | 9 ++++++++- libavfilter/vf_sr.c | 9 ++++++++- 2 files changed, 16 insertions(+), 2 deletions(-) diff --git a/libavfilter/vf_derain.c b/libavfilter/vf_derain.c index c4910d9a1e..4242e2f518 100644 --- a/libavfilter/vf_derain.c +++ b/libavfilter/vf_derain.c @@ -54,6 +54,7 @@ AVFILTER_DNN_DEFINE_CLASS(derain, DNN_TF); static int filter_frame(AVFilterLink *inlink, AVFrame *in) { DNNAsyncStatusType async_state = 0; + int proc_status = 0; AVFilterContext *ctx = inlink->dst; AVFilterLink *outlink = ctx->outputs[0]; DRContext *dr_context = ctx->priv; @@ -75,9 +76,15 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *in) return dnn_result; } do { - async_state = ff_dnn_get_result(&dr_context->dnnctx, &in, &out, NULL); + async_state = ff_dnn_get_result(&dr_context->dnnctx, &in, &out, &proc_status); } while (async_state == DAST_NOT_READY); + if (async_state == DAST_FAIL) { + av_log(ctx, AV_LOG_ERROR, + "DNN post-processing of inference output failed: %s\n", + av_err2str(proc_status)); + return proc_status < 0 ? proc_status : AVERROR_EXTERNAL; + } if (async_state != DAST_SUCCESS) return AVERROR(EINVAL); diff --git a/libavfilter/vf_sr.c b/libavfilter/vf_sr.c index 7eac5ca18d..13d3f22919 100644 --- a/libavfilter/vf_sr.c +++ b/libavfilter/vf_sr.c @@ -112,6 +112,7 @@ static int config_output(AVFilterLink *outlink) static int filter_frame(AVFilterLink *inlink, AVFrame *in) { DNNAsyncStatusType async_state = 0; + int proc_status = 0; AVFilterContext *context = inlink->dst; SRContext *ctx = context->priv; AVFilterLink *outlink = context->outputs[0]; @@ -142,9 +143,15 @@ static int filter_frame(AVFilterLink *inlink, AVFrame *in) } do { - async_state = ff_dnn_get_result(&ctx->dnnctx, &in, &out, NULL); + async_state = ff_dnn_get_result(&ctx->dnnctx, &in, &out, &proc_status); } while (async_state == DAST_NOT_READY); + if (async_state == DAST_FAIL) { + av_log(context, AV_LOG_ERROR, + "DNN post-processing of inference output failed: %s\n", + av_err2str(proc_status)); + return proc_status < 0 ? proc_status : AVERROR_EXTERNAL; + } if (async_state != DAST_SUCCESS) return AVERROR(EINVAL); -- 2.52.0 From f1db4d7ff6b7fab6f69ab22676c54827fb2d4403 Mon Sep 17 00:00:00 2001 From: Jake Fineman <[email protected]> Date: Fri, 25 Sep 2026 18:42:27 -0400 Subject: [PATCH 4/4] avfilter/dnn: complete analytics tasks whose post-processing failed When detect_post_proc() failed, the ONNX backend jumped past task->inference_done++, so the task could never satisfy inference_done == inference_todo. On a backend that runs requests asynchronously, ff_dnn_get_result_common() then kept returning DAST_NOT_READY for that task and the queue stalled. The OpenVINO and TensorFlow backends ignored the return values of detect_post_proc() and classify_post_proc() altogether, so failures there went unnoticed. Record the first failure in task->status and still count the lltask as done, so the task completes and the result path reports DAST_FAIL with the error. dnn_detect and dnn_classify now pass a status to ff_dnn_get_result() and return that error, like dnn_processing does, instead of silently forwarding the frame. The Torch backend has no analytics post-processing and is unchanged. Signed-off-by: Jake Fineman <[email protected]> Co-Authored-By: Claude Opus 5.5 <[email protected]> --- libavfilter/dnn/dnn_backend_onnx.c | 4 ++-- libavfilter/dnn/dnn_backend_openvino.c | 20 +++++++++++++------- libavfilter/dnn/dnn_backend_tf.c | 5 ++++- libavfilter/vf_dnn_classify.c | 18 ++++++++++++++++-- libavfilter/vf_dnn_detect.c | 18 ++++++++++++++++-- 5 files changed, 51 insertions(+), 14 deletions(-) diff --git a/libavfilter/dnn/dnn_backend_onnx.c b/libavfilter/dnn/dnn_backend_onnx.c index 405739c300..fb52931c8b 100644 --- a/libavfilter/dnn/dnn_backend_onnx.c +++ b/libavfilter/dnn/dnn_backend_onnx.c @@ -712,8 +712,8 @@ static void infer_completion_callback(void *args) ret = onnx_model->model.detect_post_proc(task->in_frame, outputs, infer_request->nb_outputs, onnx_model->model.filter_ctx); - if (ret < 0) - goto err; + if (ret < 0 && task->status >= 0) + task->status = ret; break; default: avpriv_report_missing_feature(ctx, "model function type %d", onnx_model->model.func_type); diff --git a/libavfilter/dnn/dnn_backend_openvino.c b/libavfilter/dnn/dnn_backend_openvino.c index 651e480df8..5497bfeaee 100644 --- a/libavfilter/dnn/dnn_backend_openvino.c +++ b/libavfilter/dnn/dnn_backend_openvino.c @@ -338,6 +338,7 @@ static void infer_completion_callback(void *args) OVModel *ov_model = task->model; SafeQueue *requestq = ov_model->request_queue; DNNData *outputs; + int ret; DnnContext *ctx = ov_model->ctx; #if HAVE_OPENVINO2 size_t* dims; @@ -460,19 +461,24 @@ static void infer_completion_callback(void *args) av_log(ctx, AV_LOG_ERROR, "detect filter needs to provide post proc\n"); goto end; } - ov_model->model.detect_post_proc(task->in_frame, outputs, - ov_model->nb_outputs, - ov_model->model.filter_ctx); + ret = ov_model->model.detect_post_proc(task->in_frame, outputs, + ov_model->nb_outputs, + ov_model->model.filter_ctx); + if (ret < 0 && task->status >= 0) + task->status = ret; break; case DFT_ANALYTICS_CLASSIFY: if (!ov_model->model.classify_post_proc) { av_log(ctx, AV_LOG_ERROR, "classify filter needs to provide post proc\n"); goto end; } - for (int output_i = 0; output_i < ov_model->nb_outputs; output_i++) - ov_model->model.classify_post_proc(task->in_frame, outputs, - request->lltasks[i]->bbox_index, - ov_model->model.filter_ctx); + for (int output_i = 0; output_i < ov_model->nb_outputs; output_i++) { + ret = ov_model->model.classify_post_proc(task->in_frame, outputs, + request->lltasks[i]->bbox_index, + ov_model->model.filter_ctx); + if (ret < 0 && task->status >= 0) + task->status = ret; + } break; default: av_assert0(!"should not reach here"); diff --git a/libavfilter/dnn/dnn_backend_tf.c b/libavfilter/dnn/dnn_backend_tf.c index ef0df1097d..3fb5b5b4c1 100644 --- a/libavfilter/dnn/dnn_backend_tf.c +++ b/libavfilter/dnn/dnn_backend_tf.c @@ -696,6 +696,7 @@ static void infer_completion_callback(void *args) { LastLevelTaskItem *lltask = request->lltask; TaskItem *task = lltask->task; DNNData *outputs; + int ret; TFInferRequest *infer_request = request->infer_request; TFModel *tf_model = task->model; DnnContext *ctx = tf_model->ctx; @@ -738,7 +739,9 @@ static void infer_completion_callback(void *args) { av_log(ctx, AV_LOG_ERROR, "Detect filter needs provide post proc\n"); return; } - tf_model->model.detect_post_proc(task->in_frame, outputs, task->nb_output, tf_model->model.filter_ctx); + ret = tf_model->model.detect_post_proc(task->in_frame, outputs, task->nb_output, tf_model->model.filter_ctx); + if (ret < 0 && task->status >= 0) + task->status = ret; break; default: av_log(ctx, AV_LOG_ERROR, "Tensorflow backend does not support this kind of dnn filter now\n"); diff --git a/libavfilter/vf_dnn_classify.c b/libavfilter/vf_dnn_classify.c index 5751e7c4bc..ab55816b1a 100644 --- a/libavfilter/vf_dnn_classify.c +++ b/libavfilter/vf_dnn_classify.c @@ -216,7 +216,14 @@ static int dnn_classify_flush_frame(AVFilterLink *outlink, int64_t pts, int64_t do { AVFrame *in_frame = NULL; AVFrame *out_frame = NULL; - async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame, NULL); + int proc_status = 0; + async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame, &proc_status); + if (async_state == DAST_FAIL) { + av_log(ctx, AV_LOG_ERROR, + "DNN post-processing of inference output failed while flushing: %s\n", + av_err2str(proc_status)); + return proc_status < 0 ? proc_status : AVERROR_EXTERNAL; + } if (async_state == DAST_SUCCESS) { ret = ff_filter_frame(outlink, in_frame); if (ret < 0) @@ -259,7 +266,14 @@ static int dnn_classify_activate(AVFilterContext *filter_ctx) do { AVFrame *in_frame = NULL; AVFrame *out_frame = NULL; - async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame, NULL); + int proc_status = 0; + async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame, &proc_status); + if (async_state == DAST_FAIL) { + av_log(ctx, AV_LOG_ERROR, + "DNN post-processing of inference output failed: %s\n", + av_err2str(proc_status)); + return proc_status < 0 ? proc_status : AVERROR_EXTERNAL; + } if (async_state == DAST_SUCCESS) { ret = ff_filter_frame(outlink, in_frame); if (ret < 0) diff --git a/libavfilter/vf_dnn_detect.c b/libavfilter/vf_dnn_detect.c index 846e4c11fc..2f8a5b55c2 100644 --- a/libavfilter/vf_dnn_detect.c +++ b/libavfilter/vf_dnn_detect.c @@ -882,7 +882,14 @@ static int dnn_detect_flush_frame(AVFilterLink *outlink, int64_t pts, int64_t *o do { AVFrame *in_frame = NULL; AVFrame *out_frame = NULL; - async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame, NULL); + int proc_status = 0; + async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame, &proc_status); + if (async_state == DAST_FAIL) { + av_log(ctx, AV_LOG_ERROR, + "DNN post-processing of inference output failed while flushing: %s\n", + av_err2str(proc_status)); + return proc_status < 0 ? proc_status : AVERROR_EXTERNAL; + } if (async_state == DAST_SUCCESS) { ret = ff_filter_frame(outlink, in_frame); if (ret < 0) @@ -925,7 +932,14 @@ static int dnn_detect_activate(AVFilterContext *filter_ctx) do { AVFrame *in_frame = NULL; AVFrame *out_frame = NULL; - async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame, NULL); + int proc_status = 0; + async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame, &proc_status); + if (async_state == DAST_FAIL) { + av_log(ctx, AV_LOG_ERROR, + "DNN post-processing of inference output failed: %s\n", + av_err2str(proc_status)); + return proc_status < 0 ? proc_status : AVERROR_EXTERNAL; + } if (async_state == DAST_SUCCESS) { ret = ff_filter_frame(outlink, in_frame); if (ret < 0) -- 2.52.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
