PR #25509 opened by Jake (jakefineman) URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25509 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25509.patch
dnn_processing always outputs frames in the same pixel format as its input. A model that takes RGB and returns one channel, such as a background matte, a depth map or a saliency map, therefore has no usable output format: with rgb24 the output is rejected as a channel mismatch, and with gray8 the input no longer matches the model. This adds a `format` option that sets the output pixel format, named as in mergeplanes and the hardware scale filters. Only gray8 and grayf32 are accepted. When it is unset (the default), negotiation and output are unchanged. When it is set, the input keeps the full format list and the output is negotiated separately. The filter checks that the model returns exactly one channel and converts it with the existing gray paths of ff_proc_from_dnn_to_frame(). The manual gets an example that feeds the result to alphamerge. The model's output channel count is not known before the first frame, since get_output() only reports width and height, so a mismatch is reported on the first frame and the filter returns the error instead of passing frames on. Testing, with 3-channel-in, 1-channel-out ONNX models whose output is the mean of R, G and B: - Default path: framemd5 identical to master for rgb24, bgr24, gray8, grayf32, yuv420p, yuv444p and nv12 input, with ONNX Runtime 1.30, and with OpenVINO 2026.4 for rgb24. - format=gray8: maximum error 0.33 against the expected values, from rgb24 and bgr24 with ONNX Runtime and from rgb24 with OpenVINO. format=grayf32: maximum error 8.5e-08 (ONNX Runtime). - A 3-channel model with format=gray8 fails with "the output format gray requires a 1-channel model output, but the model output has 3 channels" and writes no frames. - The alphamerge example from the documentation runs as written. There is no DNN coverage in FATE, since every backend is an external library, so no test is added. >From 5e622febbc5cccdfcb427b77ee3f337faa9e40e0 Mon Sep 17 00:00:00 2001 From: Jake Fineman <[email protected]> Date: Mon, 26 Oct 2026 22:39:27 -0400 Subject: [PATCH] avfilter/vf_dnn_processing: add option to set the output pixel format dnn_processing negotiates a single pixel format list shared by its input and output, so the output frames always have the input pixel format. Since check_modelinput_inlink() ties the input format to the channel count of the model input, a model whose output has a different channel count than its input cannot be used. This rules out a common class of models that take an RGB frame and produce a single-channel map, such as background removal mattes, depth or saliency maps. Feeding such a model an rgb24 frame today only gets the output rejected by the channel check in ff_proc_from_dnn_to_frame(). Add a "format" option, following the naming used by other filters for the output pixel format. When it is not set, the formats are negotiated from the same shared list as before, so the default behaviour does not change. When it is set, the input keeps the existing list and the output is restricted to the requested format, using a query_formats callback with separate input and output lists. Only gray8 and grayf32 are accepted for now. The output frames of these formats go through the existing GRAY8 and GRAYF32 branches of ff_proc_from_dnn_to_frame(), so no new conversion code is needed. The input side is still validated against the model input by check_modelinput_inlink(). The model output channel count is not available at configuration time, as the backend get_output() callback only reports the output width and height. It is therefore validated in a frame post-processing callback that the filter installs only when the option is set. The backends ignore the return value of that callback, so an error is latched in the filter context, and activate() returns it instead of passing the frame on. A mismatched model thus fails on the first frame with an explicit error, instead of producing garbage output. With an output format set, the chroma planes of planar YUV input are no longer copied, since the output has no chroma planes. Hardware frame input is rejected, as the output would be a software frame. There is no FATE coverage for the DNN filters. All the backends are external libraries and there are no test models in the FATE suite, so no test is added here. This was tested with the ONNX Runtime backend and synthetic models. With the default settings, the framemd5 output matches master for rgb24, bgr24, gray8, grayf32, yuv420p, yuv444p and nv12. A 3-channel to 1-channel model gives the expected gray8 and grayf32 values and composes with alphamerge. Signed-off-by: Jake Fineman <[email protected]> --- doc/filters.texi | 25 +++++++++ libavfilter/vf_dnn_processing.c | 95 +++++++++++++++++++++++++++++++-- 2 files changed, 116 insertions(+), 4 deletions(-) diff --git a/doc/filters.texi b/doc/filters.texi index 37f9123df49..0e5c3a6fa27 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -12382,6 +12382,21 @@ optional for the ONNX Runtime backend. Set the output name of the dnn network. Required for the TensorFlow backend; optional for the ONNX Runtime backend. +@item format +Set the pixel format of the output frames. By default the output frames +have the same pixel format as the input frames. + +Currently only @code{gray8} and @code{grayf32} are supported. This allows +using a model whose output has a different channel count than its input, +e.g. a model that takes an @code{rgb24} frame and produces a single-channel +alpha matte, depth map or saliency map. The input frames are still checked +against the model input as usual, and the model output must have exactly +one channel; the latter is checked when the first frame is processed. For +@code{gray8} the model output is expected in the range [0, 1] and is scaled +to [0, 255], for @code{grayf32} it is stored unchanged. For planar YUV input +only the processed luma plane is output. Hardware frame input is not +supported with this option. + @item backend_configs Set the configs to be passed into backend. To use async execution, set async (default: set). Roll back to sync execution if the backend does not support async. @@ -12447,6 +12462,16 @@ please use tools/python/tf_sess_config.py to get the configs of TensorFlow backe ./ffmpeg -i 480p.jpg -vf format=yuv420p,dnn_processing=dnn_backend=tensorflow:model=espcn.pb:input=x:output=y:backend_configs=sess_config=0x10022805320e09cdccccccccccec3f20012a01303801 -y tmp.espcn.jpg @end example +@item +Run a background removal model with an RGB input and a single-channel +matte output, and use the matte as the alpha channel of the input with +the alphamerge filter: +@example +./ffmpeg -i input.mp4 -filter_complex \ + "[0:v]split[main][in];[in]format=rgb24,dnn_processing=dnn_backend=onnx:model=matte.onnx:format=gray8[alpha];[main][alpha]alphamerge" \ + -c:v ffv1 -y output.mkv +@end example + @item Perform zero-copy GPU inference with the Libtorch backend. Because there is currently no GPU-side filter to convert from YUV to RGB, diff --git a/libavfilter/vf_dnn_processing.c b/libavfilter/vf_dnn_processing.c index 96c72e09a62..07d6772a974 100644 --- a/libavfilter/vf_dnn_processing.c +++ b/libavfilter/vf_dnn_processing.c @@ -33,6 +33,7 @@ #include "filters.h" #include "formats.h" #include "dnn_filter_common.h" +#include "dnn/dnn_io_proc.h" #include "video.h" #include "libswscale/swscale.h" #include "libavutil/time.h" @@ -47,9 +48,12 @@ typedef struct DnnProcessingContext { struct SwsContext *sws_uv_scale; int sws_uv_height; AVBufferRef *hw_frames_ctx_out; + enum AVPixelFormat out_fmt; + int post_proc_err; } DnnProcessingContext; #define OFFSET(x) offsetof(DnnProcessingContext, dnnctx.x) +#define OFFSET2(x) offsetof(DnnProcessingContext, x) #define FLAGS AV_OPT_FLAG_FILTERING_PARAM | AV_OPT_FLAG_VIDEO_PARAM static const AVOption dnn_processing_options[] = { { "dnn_backend", "DNN backend", OFFSET(backend_type), AV_OPT_TYPE_INT, { .i64 = DNN_TF }, INT_MIN, INT_MAX, FLAGS, .unit = "backend" }, @@ -65,6 +69,7 @@ static const AVOption dnn_processing_options[] = { #if (CONFIG_LIBONNXRUNTIME == 1) { "onnx", "onnx backend flag", 0, AV_OPT_TYPE_CONST, { .i64 = DNN_ONNX }, 0, 0, FLAGS, "backend" }, #endif + { "format", "output pixel format", OFFSET2(out_fmt), AV_OPT_TYPE_PIXEL_FMT, { .i64 = AV_PIX_FMT_NONE }, AV_PIX_FMT_NONE, INT_MAX, FLAGS }, { NULL } }; @@ -72,6 +77,20 @@ AVFILTER_DNN_DEFINE_CLASS(dnn_processing, DNN_TF | DNN_OV | DNN_TH | DNN_ONNX); static av_cold int init(AVFilterContext *context) { + DnnProcessingContext *ctx = context->priv; + + switch (ctx->out_fmt) { + case AV_PIX_FMT_NONE: + case AV_PIX_FMT_GRAY8: + case AV_PIX_FMT_GRAYF32: + break; + default: + av_log(context, AV_LOG_ERROR, "Unsupported output format %s, " + "only gray8 and grayf32 are supported\n", + av_get_pix_fmt_name(ctx->out_fmt)); + return AVERROR(EINVAL); + } + /* * Model loading is deferred to config_input(): the ONNX Runtime * backend's D3D12/DirectML path must bind its execution provider to @@ -96,6 +115,50 @@ static const enum AVPixelFormat pix_fmts[] = { AV_PIX_FMT_NONE }; +static int query_formats(const AVFilterContext *context, + AVFilterFormatsConfig **cfg_in, + AVFilterFormatsConfig **cfg_out) +{ + const DnnProcessingContext *ctx = context->priv; + int ret; + + if (ctx->out_fmt == AV_PIX_FMT_NONE) + return ff_set_pixel_formats_from_list2(context, cfg_in, cfg_out, pix_fmts); + + ret = ff_formats_ref(ff_make_pixel_format_list(pix_fmts), &cfg_in[0]->formats); + if (ret < 0) + return ret; + + return ff_formats_ref(ff_make_formats_list_singleton(ctx->out_fmt), + &cfg_out[0]->formats); +} + +/* + * Replaces the backend's default post-processing when an output format is + * set. The channel count of the model output is only known once inference + * has run, so it is checked here; a failure is latched so that activate() + * returns an error instead of passing the frame on. + */ +static int post_proc(AVFrame *frame, DNNData *output, AVFilterContext *filter_ctx) +{ + DnnProcessingContext *ctx = filter_ctx->priv; + int channels = output->dims[dnn_get_channel_idx_by_layout(output->layout)]; + int ret; + + if (channels != 1) { + av_log(filter_ctx, AV_LOG_ERROR, "the output format %s requires a " + "1-channel model output, but the model output has %d channels\n", + av_get_pix_fmt_name(frame->format), channels); + ctx->post_proc_err = AVERROR(EINVAL); + return ctx->post_proc_err; + } + + ret = ff_proc_from_dnn_to_frame(frame, output, &ctx->dnnctx); + if (ret < 0) + ctx->post_proc_err = ret; + return ret; +} + #define LOG_FORMAT_CHANNEL_MISMATCH() \ av_log(ctx, AV_LOG_ERROR, \ "the frame's format %s does not match " \ @@ -225,6 +288,12 @@ static int config_input(AVFilterLink *inlink) return result; } + if (ctx->out_fmt != AV_PIX_FMT_NONE) { + result = ff_dnn_set_frame_proc(&ctx->dnnctx, NULL, post_proc); + if (result < 0) + return result; + } + result = ff_dnn_get_input(&ctx->dnnctx, &model_input); if (result != 0) { av_log(ctx, AV_LOG_ERROR, "could not get input from the model\n"); @@ -253,7 +322,7 @@ static int prepare_uv_scale(AVFilterLink *outlink) AVFilterLink *inlink = context->inputs[0]; enum AVPixelFormat fmt = inlink->format; - if (isPlanarYUV(fmt)) { + if (isPlanarYUV(fmt) && outlink->format == fmt) { if (inlink->w != outlink->w || inlink->h != outlink->h) { if (fmt == AV_PIX_FMT_NV12) { ctx->sws_uv_scale = sws_getContext(inlink->w >> 1, inlink->h >> 1, AV_PIX_FMT_YA8, @@ -351,6 +420,14 @@ static int config_output(AVFilterLink *outlink) FilterLink *inl = ff_filter_link(inlink); FilterLink *outl = ff_filter_link(outlink); + if (outlink->format != inlink->format && + av_pix_fmt_desc_get(inlink->format)->flags & AV_PIX_FMT_FLAG_HWACCEL) { + av_log(context, AV_LOG_ERROR, "the output format %s cannot be used " + "with %s input frames\n", av_get_pix_fmt_name(outlink->format), + av_get_pix_fmt_name(inlink->format)); + return AVERROR(ENOSYS); + } + // have a try run in case that the dnn model resize the frame result = ff_dnn_get_output(&ctx->dnnctx, inlink->w, inlink->h, &outlink->w, &outlink->h); if (result != 0) { @@ -421,7 +498,12 @@ static int flush_frame(AVFilterLink *outlink, int64_t pts, int64_t *out_pts) AVFrame *out_frame = NULL; async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame); if (out_frame) { - if (isPlanarYUV(in_frame->format)) + if (ctx->post_proc_err < 0) { + av_frame_free(&in_frame); + av_frame_free(&out_frame); + return ctx->post_proc_err; + } + if (isPlanarYUV(out_frame->format)) copy_uv_planes(ctx, out_frame, in_frame); av_frame_free(&in_frame); ret = ff_filter_frame(outlink, out_frame); @@ -479,7 +561,12 @@ static int activate(AVFilterContext *filter_ctx) AVFrame *out_frame = NULL; async_state = ff_dnn_get_result(&ctx->dnnctx, &in_frame, &out_frame); if (out_frame) { - if (isPlanarYUV(in_frame->format)) + if (ctx->post_proc_err < 0) { + av_frame_free(&in_frame); + av_frame_free(&out_frame); + return ctx->post_proc_err; + } + if (isPlanarYUV(out_frame->format)) copy_uv_planes(ctx, out_frame, in_frame); av_frame_free(&in_frame); ret = ff_filter_frame(outlink, out_frame); @@ -542,7 +629,7 @@ const FFFilter ff_vf_dnn_processing = { .uninit = uninit, FILTER_INPUTS(dnn_processing_inputs), FILTER_OUTPUTS(dnn_processing_outputs), - FILTER_PIXFMTS_ARRAY(pix_fmts), + FILTER_QUERY_FUNC2(query_formats), .activate = activate, .p.flags = AVFILTER_FLAG_HWDEVICE, .flags_internal = FF_FILTER_FLAG_HWFRAME_AWARE, -- 2.54.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
