PR #24693 opened by maryla-uc URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24693 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24693.patch
Add a new video filter smpte2094app5gen that generates SMPTE ST 2094-50 metadata by sampling tone mapping curves. The specification is available at https://pub.smpte.org/doc/st2094-50/20260824-pub/ Supports tone mapping curves from vf_tonemap and libplacebo when available. The filter attaches AV_FRAME_DATA_DYNAMIC_HDR_SMPTE_2094_APP5 side data to each frame. The filter does not modify frame pixels, so this could also be implemented as a bitstream filter. The advantage of a video filter is that it's codec agnostic and that it could later support other metadata generation methods based on frame pixels. The drawback is that it requires video reencoding. >From cd6b70313f1b96ad76e9e96cf073e8b4d7d984ad Mon Sep 17 00:00:00 2001 From: Maryla Ustarroz-Calonge <[email protected]> Date: Thu, 3 Sep 2026 17:22:36 +0200 Subject: [PATCH] avfilter/vf_smpte2094app5gen: add SMPTE ST 2094-50 metadata generator filter Add a new video filter smpte2094app5gen that generates SMPTE ST 2094-50 metadata by sampling tone mapping curves. The specification is available at https://pub.smpte.org/doc/st2094-50/20260824-pub/ Supports tone mapping curves from vf_tonemap and libplacebo when available. The filter attaches AV_FRAME_DATA_DYNAMIC_HDR_SMPTE_2094_APP5 side data to each frame. The filter does not modify frame pixels, so this could also be implemented as a bitstream filter. The advantage of a video filter is that it's codec agnostic and that it could later support other metadata generation methods based on frame pixels. The drawback is that it requires video reencoding. --- doc/filters.texi | 98 +++ libavfilter/Makefile | 1 + libavfilter/allfilters.c | 1 + libavfilter/vf_smpte2094app5gen.c | 729 ++++++++++++++++++ tests/fate/filter-video.mak | 5 + tests/ref/fate/filter-smpte2094app5gen-linear | 31 + .../ref/fate/filter-smpte2094app5gen-reinhard | 44 ++ 7 files changed, 909 insertions(+) create mode 100644 libavfilter/vf_smpte2094app5gen.c create mode 100644 tests/ref/fate/filter-smpte2094app5gen-linear create mode 100644 tests/ref/fate/filter-smpte2094app5gen-reinhard diff --git a/doc/filters.texi b/doc/filters.texi index b7379d3060..0e86ed8074 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -16953,6 +16953,7 @@ ffmpeg -i input.mov -vf lensfun=make=Canon:model="Canon EOS 100D":lens_model="Ca Low Complexity Enhancement Video Codec filter based on liblcevc_dec (@url{https://github.com/v-novaltd/LCEVCdec}). +@anchor{libplacebo} @section libplacebo Flexible GPU-accelerated processing filter based on libplacebo @@ -23123,6 +23124,103 @@ in [-30,0] will filter edges. Default value is @option{luma_threshold}. If a chroma or alpha option is not explicitly set, the corresponding luma value is set. +@section smpte2094app5gen + +Generate SMPTE ST 2094-50 metadata side data for video frames. + +The filter creates SMPTE ST 2094-50 metadata for HDR videos based on the +specified tone mapping algorithm, and attaches it as side data to each video +frame. + +The filter accepts the following options: + +@table @option +@item algorithm +Set the tone mapping algorithm used to generate the metadata. See also the @ref{tonemap} and @ref{libplacebo} filters. +Accepted values are: +@table @samp +@item reinhard +Reinhard tone mapping algorithm (default). +@item linear +Linear scaling tone mapping. +@item gamma +Gamma power curve tone mapping. +@item clip +Hard clipping tone mapping. +@item hable +Hable filmic tone mapping. +@item mobius +Mobius rational polynomial tone mapping. +@item libplacebo_st2094-40 +SMPTE ST 2094-40 tone mapping curve via libplacebo (requires building with @code{--enable-libplacebo}). Converts HDR10+ metadata present in frame side data to ST 2094-50 metadata. +@item libplacebo_st2094-10 +SMPTE ST 2094-10 tone mapping curve via libplacebo (requires building with @code{--enable-libplacebo}). +@item libplacebo_bt2390 +ITU-R BT.2390 EETF tone mapping curve via libplacebo (requires building with @code{--enable-libplacebo}). +@item libplacebo_bt2446a +ITU-R BT.2446 Method A tone mapping curve via libplacebo (requires building with @code{--enable-libplacebo}). +@item libplacebo_spline +Single-pivot cubic spline tone mapping curve via libplacebo (requires building with @code{--enable-libplacebo}). +@end table + +@item param +Tune the tone mapping algorithm. Same as @code{param} option in @ref{tonemap}. + +@item peak +Signal peak luminance in nits. Default is 0.0 (auto-detect from frame side data such as Content Light Level or Mastering Display Metadata; falls back to the transfer function max e.g. 10000 nits for PQ). + +@item ref_white +Reference diffuse white luminance in nits, used to convert nits to SDR-relative. Controls the overall brightness of the image. Default is 203.0 nits. + +@item num_points +Number of control points per gain curve, from 2 to 32. Default is 10. More points means more precise curves but larger metadata size. + +@item gain_space +Gain application space chromaticities. Accepted values are: +@table @samp +@item bt709 +ITU-R BT.709 color primaries. +@item p3 +SMPTE EG 432-1 (P3) color primaries. +@item bt2020 +ITU-R BT.2020 color primaries (default). +@end table + +@item mixing +Component mixing type for applying gains. Controls how tone mapping affects colors. +@table @samp +@item maxrgb +Apply gain computed from max(R, G, B) (default). +@item rgb +Apply independent gains to RGB channels. +@item standard_balanced +Apply gain computed from 50% max(R, G, B) and 50% Mean(R, G, B). +@item balanced +Apply gain computed from 50% max(R, G, B) and 50% Luma (Y). +@item luma +Apply gain computed from Luma channel. +@end table + +@item mode +Operating mode: +@table @samp +@item static +Compute tone mapping metadata once on the first frame and use it for all subsequent frames. +@item dynamic +Recompute tone mapping metadata if frame side data (peak luminance or dynamic HDR metadata) changes across frames (default). +@end table +@end table + +@subsection Examples + +@itemize +@item +Convert HDR10+ metadata to ST 2094-50 metadata using libplacebo: +@example +ffmpeg -i hdr10plus_input.mp4 -vf "smpte2094app5gen=algorithm=libplacebo_st2094-40" output.mp4 +@end example +@end itemize + @section sobel Apply sobel operator to input video stream. diff --git a/libavfilter/Makefile b/libavfilter/Makefile index ceea328e67..76f6f16eec 100644 --- a/libavfilter/Makefile +++ b/libavfilter/Makefile @@ -509,6 +509,7 @@ OBJS-$(CONFIG_SIDEDATA_FILTER) += f_sidedata.o OBJS-$(CONFIG_SIGNALSTATS_FILTER) += vf_signalstats.o OBJS-$(CONFIG_SIGNATURE_FILTER) += vf_signature.o OBJS-$(CONFIG_SMARTBLUR_FILTER) += vf_smartblur.o +OBJS-$(CONFIG_SMPTE2094APP5GEN_FILTER) += vf_smpte2094app5gen.o OBJS-$(CONFIG_SOBEL_FILTER) += vf_convolution.o OBJS-$(CONFIG_SOBEL_OPENCL_FILTER) += vf_convolution_opencl.o opencl.o \ opencl/convolution.o diff --git a/libavfilter/allfilters.c b/libavfilter/allfilters.c index 3b93fe12f6..d9c956bd53 100644 --- a/libavfilter/allfilters.c +++ b/libavfilter/allfilters.c @@ -483,6 +483,7 @@ extern const FFFilter ff_vf_signalstats; extern const FFFilter ff_vf_signature; extern const FFFilter ff_vf_siti; extern const FFFilter ff_vf_smartblur; +extern const FFFilter ff_vf_smpte2094app5gen; extern const FFFilter ff_vf_sobel; extern const FFFilter ff_vf_sobel_opencl; extern const FFFilter ff_vf_split; diff --git a/libavfilter/vf_smpte2094app5gen.c b/libavfilter/vf_smpte2094app5gen.c new file mode 100644 index 0000000000..c8d3c6dcac --- /dev/null +++ b/libavfilter/vf_smpte2094app5gen.c @@ -0,0 +1,729 @@ +/* + * Copyright 2026 Google LLC + * + * This file is part of FFmpeg. + * + * FFmpeg is free software; you can redistribute it and/or + * modify it under the terms of the GNU Lesser General Public + * License as published by the Free Software Foundation; either + * version 2.1 of the License, or (at your option) any later version. + * + * FFmpeg is distributed in the hope that it will be useful, + * but WITHOUT ANY WARRANTY; without even the implied warranty of + * MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the GNU + * Lesser General Public License for more details. + * + * You should have received a copy of the GNU Lesser General Public + * License along with FFmpeg; if not, write to the Free Software + * Foundation, Inc., 51 Franklin Street, Fifth Floor, Boston, MA 02110-1301 USA + */ + +#include <float.h> + +#include "libavutil/avassert.h" +#include "libavutil/buffer.h" +#include "libavutil/common.h" +#include "libavutil/frame.h" +#include "libavutil/hdr_dynamic_metadata.h" +#include "libavutil/internal.h" +#include "libavutil/libm.h" +#include "libavutil/mastering_display_metadata.h" +#include "libavutil/mathematics.h" +#include "libavutil/mem.h" +#include "libavutil/opt.h" +#include "libavutil/pixdesc.h" + +#include "avfilter.h" +#include "filters.h" +#include "video.h" + +#if CONFIG_LIBPLACEBO +#include <libplacebo/colorspace.h> +#include <libplacebo/tone_mapping.h> +#include <libplacebo/utils/libav.h> +#endif + +enum Smpte2094App5Algorithm { + TONEMAP_REINHARD, + TONEMAP_LINEAR, + TONEMAP_GAMMA, + TONEMAP_CLIP, + TONEMAP_HABLE, + TONEMAP_MOBIUS, + TONEMAP_LIBPLACEBO_ST2094_40, + TONEMAP_LIBPLACEBO_ST2094_10, + TONEMAP_LIBPLACEBO_BT2390, + TONEMAP_LIBPLACEBO_BT2446A, + TONEMAP_LIBPLACEBO_SPLINE, + TONEMAP_NB +}; + +enum Smpte2094App5GainSpace { + GAIN_SPACE_BT709, + GAIN_SPACE_P3, + GAIN_SPACE_BT2020, + GAIN_SPACE_NB +}; + +enum Smpte2094App5MixingType { + MIXING_MAXRGB, /* Type 0: Max(R,G,B) */ + MIXING_RGB, /* Type 1: Separate RGB */ + MIXING_STANDARD_BALANCED,/* Type 2: Standard balanced (0.5 Max + 0.5 Mean) */ + MIXING_BALANCED, /* Type 3: Custom (0.5 MaxRGB + 0.5 Luma) */ + MIXING_LUMA, /* Type 3: Custom (Pure Luma Y) */ + MIXING_NB +}; + +typedef struct Smpte2094App5GenContext { + const AVClass *class; + + int algorithm; + double param; + double peak; + double ref_white; + int num_points; + int mixing_type; + int gain_space; + int mode; + + /* Cached metadata for static mode */ + AVDynamicHDRSmpte2094App5 cached_metadata; + int has_cached_metadata; +} Smpte2094App5GenContext; + +#define MAX_CONTROL_POINTS 32 +#define OFFSET(x) offsetof(Smpte2094App5GenContext, x) +#define FLAGS AV_OPT_FLAG_FILTERING_PARAM | AV_OPT_FLAG_VIDEO_PARAM + +static const AVOption smpte2094app5gen_options[] = { + { "algorithm", "set tone mapping algorithm", OFFSET(algorithm), AV_OPT_TYPE_INT, {.i64 = TONEMAP_REINHARD}, 0, TONEMAP_NB - 1, FLAGS, "algorithm" }, + { "reinhard", "Reinhard tone mapping", 0, AV_OPT_TYPE_CONST, {.i64 = TONEMAP_REINHARD}, 0, 0, FLAGS, "algorithm" }, + { "linear", "Linear scaling", 0, AV_OPT_TYPE_CONST, {.i64 = TONEMAP_LINEAR}, 0, 0, FLAGS, "algorithm" }, + { "gamma", "Gamma curve", 0, AV_OPT_TYPE_CONST, {.i64 = TONEMAP_GAMMA}, 0, 0, FLAGS, "algorithm" }, + { "clip", "Hard clipping", 0, AV_OPT_TYPE_CONST, {.i64 = TONEMAP_CLIP}, 0, 0, FLAGS, "algorithm" }, + { "hable", "Hable filmic", 0, AV_OPT_TYPE_CONST, {.i64 = TONEMAP_HABLE}, 0, 0, FLAGS, "algorithm" }, + { "mobius", "Mobius rational polynomial", 0, AV_OPT_TYPE_CONST, {.i64 = TONEMAP_MOBIUS}, 0, 0, FLAGS, "algorithm" }, +#if CONFIG_LIBPLACEBO + { "libplacebo_st2094-40", "SMPTE ST 2094-40 (libplacebo)", 0, AV_OPT_TYPE_CONST, {.i64 = TONEMAP_LIBPLACEBO_ST2094_40}, 0, 0, FLAGS, "algorithm" }, + { "libplacebo_st2094-10", "SMPTE ST 2094-10 (libplacebo)", 0, AV_OPT_TYPE_CONST, {.i64 = TONEMAP_LIBPLACEBO_ST2094_10}, 0, 0, FLAGS, "algorithm" }, + { "libplacebo_bt2390", "ITU-R BT.2390 EETF (libplacebo)", 0, AV_OPT_TYPE_CONST, {.i64 = TONEMAP_LIBPLACEBO_BT2390}, 0, 0, FLAGS, "algorithm" }, + { "libplacebo_bt2446a", "ITU-R BT.2446 Method A (libplacebo)", 0, AV_OPT_TYPE_CONST, {.i64 = TONEMAP_LIBPLACEBO_BT2446A}, 0, 0, FLAGS, "algorithm" }, + { "libplacebo_spline", "Single-pivot spline (libplacebo)", 0, AV_OPT_TYPE_CONST, {.i64 = TONEMAP_LIBPLACEBO_SPLINE}, 0, 0, FLAGS, "algorithm" }, +#endif + + { "param", "tune algorithm parameter (gamma exponent, mobius knee, etc.), see the tonemap filter", OFFSET(param), AV_OPT_TYPE_DOUBLE, {.dbl = NAN}, DBL_MIN, DBL_MAX, FLAGS, NULL }, + { "peak", "source peak luminance in nits (0 = auto from metadata)", OFFSET(peak), AV_OPT_TYPE_DOUBLE, {.dbl = 0.0}, 0.0, 10000.0, FLAGS, NULL }, + { "ref_white", "reference diffuse white in nits", OFFSET(ref_white), AV_OPT_TYPE_DOUBLE, {.dbl = 203.0}, 10.0, 1000.0, FLAGS, NULL }, + + { "num_points", "number of control points per curve (2 to 32)", OFFSET(num_points), AV_OPT_TYPE_INT, {.i64 = 10}, 2, MAX_CONTROL_POINTS, FLAGS, NULL }, + + { "gain_space", "gain application space chromaticities", OFFSET(gain_space), AV_OPT_TYPE_INT, {.i64 = GAIN_SPACE_BT2020}, 0, GAIN_SPACE_NB - 1, FLAGS, "gain_space" }, + { "bt709", "ITU-R BT.709", 0, AV_OPT_TYPE_CONST, {.i64 = GAIN_SPACE_BT709}, 0, 0, FLAGS, "gain_space" }, + { "p3", "SMPTE EG 432-1 (P3)", 0, AV_OPT_TYPE_CONST, {.i64 = GAIN_SPACE_P3}, 0, 0, FLAGS, "gain_space" }, + { "bt2020", "ITU-R BT.2020", 0, AV_OPT_TYPE_CONST, {.i64 = GAIN_SPACE_BT2020}, 0, 0, FLAGS, "gain_space" }, + + { "mixing", "component mixing type", OFFSET(mixing_type), AV_OPT_TYPE_INT, {.i64 = MIXING_MAXRGB}, 0, MIXING_NB - 1, FLAGS, "mixing" }, + { "maxrgb", "Max(R,G,B)", 0, AV_OPT_TYPE_CONST, {.i64 = MIXING_MAXRGB}, 0, 0, FLAGS, "mixing" }, + { "rgb", "Separate RGB", 0, AV_OPT_TYPE_CONST, {.i64 = MIXING_RGB}, 0, 0, FLAGS, "mixing" }, + { "standard_balanced", "0.5 Max + 0.5 Mean", 0, AV_OPT_TYPE_CONST, {.i64 = MIXING_STANDARD_BALANCED}, 0, 0, FLAGS, "mixing" }, + { "balanced", "0.5 MaxRGB + 0.5 Luma", 0, AV_OPT_TYPE_CONST, {.i64 = MIXING_BALANCED}, 0, 0, FLAGS, "mixing" }, + { "luma", "Luma (Y)", 0, AV_OPT_TYPE_CONST, {.i64 = MIXING_LUMA}, 0, 0, FLAGS, "mixing" }, + + { "mode", "operating mode", OFFSET(mode), AV_OPT_TYPE_INT, {.i64 = 1}, 0, 1, FLAGS, "mode" }, + { "static", "generate once from first frame/options", 0, AV_OPT_TYPE_CONST, {.i64 = 0}, 0, 0, FLAGS, "mode" }, + { "dynamic", "update if frame side-data peak/metadata changes", 0, AV_OPT_TYPE_CONST, {.i64 = 1}, 0, 0, FLAGS, "mode" }, + + { NULL } +}; + +AVFILTER_DEFINE_CLASS(smpte2094app5gen); + +static av_cold int init(AVFilterContext *ctx) +{ + Smpte2094App5GenContext *s = ctx->priv; + + if (!isnan(s->param) && s->param <= 0.0) { + av_log(ctx, AV_LOG_ERROR, "param must be > 0.0\n"); + return AVERROR(EINVAL); + } + + switch (s->algorithm) { + case TONEMAP_GAMMA: + if (isnan(s->param)) + s->param = 1.8; + break; + case TONEMAP_MOBIUS: + if (isnan(s->param)) + s->param = 0.3; + break; + case TONEMAP_REINHARD: + if (isnan(s->param)) { + s->param = 0.5; + } else if (s->param >= 1.0) { + av_log(ctx, AV_LOG_ERROR, "reinhard param must be < 1.0\n"); + return AVERROR(EINVAL); + } + break; + case TONEMAP_LIBPLACEBO_ST2094_40: + case TONEMAP_LIBPLACEBO_ST2094_10: + case TONEMAP_LIBPLACEBO_BT2390: + case TONEMAP_LIBPLACEBO_BT2446A: + case TONEMAP_LIBPLACEBO_SPLINE: +#if !CONFIG_LIBPLACEBO + av_log(ctx, AV_LOG_ERROR, + "Selected algorithm requires libplacebo, but FFmpeg was built without --enable-libplacebo\n"); + return AVERROR(ENOSYS); +#endif + break; + default: + if (isnan(s->param)) + s->param = 1.0; + break; + } + + return 0; +} + +static double get_signal_peak_nits(Smpte2094App5GenContext *s, const AVFrame *in) +{ + const AVFrameSideData *sd; + + /* For libplacebo algorithms, get peak from HDR10+ metadata if available. */ + if (s->algorithm >= TONEMAP_LIBPLACEBO_ST2094_40 && + s->algorithm <= TONEMAP_LIBPLACEBO_SPLINE) { + sd = av_frame_get_side_data(in, AV_FRAME_DATA_DYNAMIC_HDR_PLUS); + if (sd) { + const AVDynamicHDRPlus *dhp = (const AVDynamicHDRPlus *)sd->data; + if (dhp->num_windows > 0) { + const AVHDRPlusColorTransformParams *pars = &dhp->params[0]; + uint8_t n = pars->num_distribution_maxrgb_percentiles; + if (n > 0) { + double hist_max = av_q2d(pars->distribution_maxrgb[n - 1].percentile) * 10000.0; + if (hist_max > 0.0) + return hist_max; + } + } + } + } + + /* For other algorithms, match vf_tonemap behavior and use static metadata. */ + + sd = av_frame_get_side_data(in, AV_FRAME_DATA_CONTENT_LIGHT_LEVEL); + if (sd) { + const AVContentLightMetadata *clm = (const AVContentLightMetadata *)sd->data; + if (clm->MaxCLL > 0) + return (double)clm->MaxCLL; + } + + sd = av_frame_get_side_data(in, AV_FRAME_DATA_MASTERING_DISPLAY_METADATA); + if (sd) { + const AVMasteringDisplayMetadata *metadata = (const AVMasteringDisplayMetadata *)sd->data; + if (metadata->has_luminance && metadata->max_luminance.num > 0) + return av_q2d(metadata->max_luminance); + } + + if (in->color_trc == AVCOL_TRC_SMPTE2084) + return 10000.0; + if (in->color_trc == AVCOL_TRC_ARIB_STD_B67) + return 1000.0; + + return 203.0; +} + +static inline double hable(double in) +{ + double a = 0.15, b = 0.50, c = 0.10, d = 0.20, e = 0.02, f = 0.30; + return (in * (in * a + b * c) + d * e) / (in * (in * a + b) + d * f) - e / f; +} + +static inline double mobius(double in, double j, double peak) +{ + double a, b; + if (in <= j) + return in; + + a = -j * j * (peak - 1.0) / (j * j - 2.0 * j + peak); + b = (j * j - 2.0 * j * peak + peak) / FFMAX(peak - 1.0, 1e-6); + + return (b * b + 2.0 * b * j + j * j) / (b - a) * (in + a) / (in + b); +} + +static double eval_tonemap(Smpte2094App5GenContext *s, const AVFrame *frame, + double x, double content_headroom_linear, double target_headroom_linear) +{ + double y = x; + + if (target_headroom_linear >= content_headroom_linear || x <= 0.0) + return x; + + /* Many of these algorithms are originally defined only for tone mapping to SDR + * (target_headroom_linear = 1). Naively stretching them to a different target + * as is done here doesn't work well for some of them. The main problem that can + * occur is that the tone mapping function ends up boosting some values, i.e. going + * over the y=x line, which is discouraged. See also the num_curves variable. */ + switch (s->algorithm) { + case TONEMAP_LINEAR: { + y = x * s->param * (target_headroom_linear / content_headroom_linear); + break; + } + case TONEMAP_CLIP: { + /* Expect the target display to clip to target_headroom_linear. + * Apply s->param as the linear scale factor before display clipping. */ + return x * s->param; + } + case TONEMAP_GAMMA: { + double u = x / content_headroom_linear; + double v = (u > 0.05) ? pow(u, 1.0 / s->param) + : u * pow(0.05, 1.0 / s->param) / 0.05; + y = target_headroom_linear * v; + break; + } + case TONEMAP_REINHARD: { + double p = s->param; + double k = (1.0 - p) / p; + double v = (x / (x + k)) * ((content_headroom_linear + k) / content_headroom_linear); + y = target_headroom_linear * v; + break; + } + case TONEMAP_HABLE: { + double h_x = hable(x); + double h_peak = hable(content_headroom_linear); + if (h_peak <= 1e-6) + return x; + y = target_headroom_linear * (h_x / h_peak); + break; + } + case TONEMAP_MOBIUS: { + double j_rel = s->param; + double p = content_headroom_linear / target_headroom_linear; + if (p <= 1.0) + return x; + double x_norm = x / target_headroom_linear; + double y_norm = mobius(x_norm, j_rel, p); + y = y_norm * target_headroom_linear; + break; + } +#if CONFIG_LIBPLACEBO + case TONEMAP_LIBPLACEBO_ST2094_40: + case TONEMAP_LIBPLACEBO_ST2094_10: + case TONEMAP_LIBPLACEBO_BT2390: + case TONEMAP_LIBPLACEBO_BT2446A: + case TONEMAP_LIBPLACEBO_SPLINE: { + const struct pl_tone_map_function *pl_func = NULL; + switch (s->algorithm) { + case TONEMAP_LIBPLACEBO_ST2094_40: pl_func = &pl_tone_map_st2094_40; break; + case TONEMAP_LIBPLACEBO_ST2094_10: pl_func = &pl_tone_map_st2094_10; break; + case TONEMAP_LIBPLACEBO_BT2390: pl_func = &pl_tone_map_bt2390; break; + case TONEMAP_LIBPLACEBO_BT2446A: pl_func = &pl_tone_map_bt2446a; break; + case TONEMAP_LIBPLACEBO_SPLINE: pl_func = &pl_tone_map_spline; break; + } + + struct pl_color_space csp; + pl_color_space_from_avframe(&csp, frame); + + struct pl_tone_map_params tm_params = { + .function = pl_func, + .constants = { PL_TONE_MAP_CONSTANTS }, + .input_scaling = PL_HDR_NITS, + .output_scaling = PL_HDR_NITS, + .input_min = 0.0f, + .input_max = (float)(content_headroom_linear * s->ref_white), + .output_min = 0.0f, + .output_max = (float)(target_headroom_linear * s->ref_white), + .hdr = csp.hdr, + }; + + float in_nits = (float)(x * s->ref_white); + float out_nits = pl_tone_map_sample(in_nits, &tm_params); + y = (double)out_nits / s->ref_white; + break; + } +#endif + default: + return x; + } + + /* Enforce tone mapping bounds: y <= target_headroom_linear, y >= 0 */ + y = FFMIN(y, target_headroom_linear); + if (y < 0.0) + y = 0.0; + + return y; +} + +static double eval_log_gain(Smpte2094App5GenContext *s, const AVFrame *frame, + double x, double content_headroom_linear, + double target_headroom_linear) +{ + double epsilon = 1e-5; + double input = FFMAX(x, epsilon); + double tonemapped = eval_tonemap(s, frame, input, content_headroom_linear, target_headroom_linear); + return log2(FFMAX(tonemapped / input, 1e-10)); +} + +static double eval_log_gain_slope(Smpte2094App5GenContext *s, const AVFrame *frame, + double x, double content_headroom_linear, + double target_headroom_linear) +{ + double epsilon = 1e-3; + double x_prev = FFMAX(0.0, x - epsilon); + double x_next = x + epsilon; + double g_next = eval_log_gain(s, frame, x_next, content_headroom_linear, + target_headroom_linear); + double g_prev = eval_log_gain(s, frame, x_prev, content_headroom_linear, + target_headroom_linear); + double slope = (g_next - g_prev) / (x_next - x_prev); + return isnan(slope) ? 0.0 : slope; +} + +/** + * Returns the x in [lo, hi] where the log2 gain crosses zero. + * The caller must ensure that exactly one of the two ends has a positive log2 + * gain. The returned point has a log2 gain <= 0. + */ +static double bisect_gain_crossing(Smpte2094App5GenContext *s, const AVFrame *frame, + double lo, double hi, + double content_headroom_linear, + double target_headroom_linear) +{ + /* In the binary format, the x-coordinates are encoded with a precision of + * 1/1000 so it's not useful to bisect with higher precision. */ + double threshold = 0.5 / 1000.0; + int lo_boosts = eval_log_gain(s, frame, lo, content_headroom_linear, + target_headroom_linear) > 0.0; + + while (hi - lo > threshold) { + double mid = 0.5 * (lo + hi); + int mid_boosts = eval_log_gain(s, frame, mid, content_headroom_linear, + target_headroom_linear) > 0.0; + if (mid_boosts == lo_boosts) + lo = mid; + else + hi = mid; + } + + /* Always return the endpoint with a log2 gain <= 0. */ + return (eval_log_gain(s, frame, lo, content_headroom_linear, + target_headroom_linear) <= 0.0) ? lo : hi; +} + +/** + * Removes the control points where the log2 gain is positive and replaces them + * with two new control points. Returns the size of out_x. + * + * For gain curves that tone map to a lower headroom, the 2094-50 binary format + * enforces that the log2 gain of all control points is <= 0. + * The invalid control points are replaced with the two points that delimit the + * region where the log2 gain is positive, i.e. the points where the gain curve + * crosses the y=0 line. + */ +static int replace_invalid_control_points(Smpte2094App5GenContext *s, const AVFrame *frame, + const double *x_values, int num_points, + double content_headroom_linear, + double target_headroom_linear, + double *out_x, int max_points, + int *had_boosted_points) +{ + int count = 0; + int previous_boosts = 0; + + if (had_boosted_points) + *had_boosted_points = 0; + + for (int i = 0; i < num_points; i++) { + double x = x_values[i]; + int boosts = eval_log_gain(s, frame, x, content_headroom_linear, + target_headroom_linear) > 0.0; + + if (boosts && had_boosted_points) + *had_boosted_points = 1; + + if (i > 0 && boosts != previous_boosts && count < max_points) { + out_x[count++] = bisect_gain_crossing(s, frame, x_values[i - 1], x, + content_headroom_linear, + target_headroom_linear); + } + + /* Always keep the first control point even if its log2 gain is positive, + * to allow interpolating smoothly between this point and the other endpoint + * of the first boosted region. */ + if (i == 0 || !boosts) { + if (count < max_points) + out_x[count++] = x; + } + + previous_boosts = boosts; + } + + /* Make sure that the last point is the same as in the input. */ + if (count > 0 && out_x[count - 1] != x_values[num_points - 1]) { + if (count < max_points) + out_x[count++] = x_values[num_points - 1]; + else + out_x[max_points - 1] = x_values[num_points - 1]; + } + + return count; +} + +/** + * Returns the x coordinate of the knee point when known. Gain curves that have + * a knee have a constant gain value before this point (linear tone mapping curve). + */ +static double get_knee_point(Smpte2094App5GenContext *s, const AVFrame *frame, double content_headroom_linear, double target_headroom_linear) +{ + if (s->algorithm == TONEMAP_MOBIUS) + return s->param * target_headroom_linear; + if (s->algorithm == TONEMAP_LINEAR || s->algorithm == TONEMAP_CLIP) + return content_headroom_linear; + /* Some libplacebo algorithms also likely have a knee point (e.g. ST2094_10) + * but its location is implementation dependent and cannot be known here. */ + return 0.0; +} + +static void compute_gain_curves(Smpte2094App5GenContext *s, const AVFrame *frame, + AVDynamicHDRSmpte2094App5 *app5, + double content_headroom_linear) +{ + /* A lot of tone mapping algorithms are only meant for tone mapping to SDR (log2 headroom 0) + * and do not behave well when naively stretched to a higher headroom, in which case it's + * better to encode a single curve for log2 headroom 0 and let SMPTE ST 2094-50's interpolation + * algorithm do the rest. Some algorithms also produce curves very similar to 2094-50's + * algorithm so encoding extra curves adds little benefit. */ + int num_curves = 1; + switch (s->algorithm) { + /* Use more curves for tone mapping functions that behave well at higher headrooms + * and benefit having more curves. */ + case TONEMAP_MOBIUS: + case TONEMAP_LIBPLACEBO_ST2094_40: + case TONEMAP_LIBPLACEBO_BT2390: + case TONEMAP_LIBPLACEBO_BT2446A: + num_curves = 4; + break; + } + + app5->num_alternate_images = num_curves; + + for (int a = 0; a < num_curves; a++) { + double target_headroom_linear = exp2((a * log2(content_headroom_linear)) / num_curves); + if (target_headroom_linear < 1.0) + target_headroom_linear = 1.0; + + double target_headroom_log2 = log2(target_headroom_linear); + app5->alternate_hdr_headrooms[a] = (uint16_t)round(target_headroom_log2 * 10000.0); + + double start_x = get_knee_point(s, frame, content_headroom_linear, target_headroom_linear); + double control_points_x[MAX_CONTROL_POINTS]; + int num_points = s->num_points; + + if (num_points <= 1 || start_x >= content_headroom_linear) { + /* Encode zero-gain curve up to content_headroom_linear for clip, or uniform attenuation + * for linear tone mapping. + * For clip, the clipping to target_headroom_linear is not encoded; it is expected that the + * output display will clip out of gamut colors. Setting `control_points_x[0] = target_headroom_linear;` + * would encode the clipping curve perfectly for this particular target headroom, but the + * curves interpolated for intermediate headrooms would not be as expected. */ + control_points_x[0] = content_headroom_linear; + num_points = 1; + } else { + /* Space out points using a perceptual power curve (gamma = 2.0). */ + for (int i = 0; i < num_points; i++) { + double t = (double)i / (num_points - 1); + control_points_x[i] = start_x + (content_headroom_linear - start_x) * pow(t, 2.0); + } + control_points_x[0] = start_x; + control_points_x[num_points - 1] = content_headroom_linear; + } + + int had_boosted_points = 0; + double curve_x[MAX_CONTROL_POINTS]; + int num_curve_points = replace_invalid_control_points(s, frame, control_points_x, + num_points, + content_headroom_linear, + target_headroom_linear, + curve_x, MAX_CONTROL_POINTS, + &had_boosted_points); + + /* Use explicit slopes if boosted points were removed/clamped, because + * the slope can differ significantly from the PCHIP slope on those points, + * and using PCHIP could lead to non monotonicity of the tone mapping curve + * in the interpolated regions. */ + app5->gain_curve_use_pchip_slope_flag[a] = !had_boosted_points; + + int out_points = 0; + uint16_t prev_coded_x = 0; + + for (int i = 0; i < num_curve_points; i++) { + double x = curve_x[i]; + double coded_x_d = round(x * 1000.0); + uint16_t coded_x = (uint16_t)av_clipd(coded_x_d, 0, 65535); + + /* Quantization to 1/1000 can cause very close points to round to the + * same integer. Skip duplicates. */ + if (out_points > 0 && coded_x <= prev_coded_x) + continue; + + double log_gain = eval_log_gain(s, frame, x, content_headroom_linear, + target_headroom_linear); + int is_clipped = 0; + /* In the binary format, log2 gains must be in [-6, 0]. */ + if (log_gain > 0.0) { + log_gain = 0.0; + is_clipped = 1; + } + if (log_gain < -6.0) { + log_gain = -6.0; + is_clipped = 1; + } + + double coded_y = round(fabs(log_gain) * 10000.0); + + app5->gain_curve_control_points_x[a][out_points] = coded_x; + app5->gain_curve_control_points_y[a][out_points] = (uint16_t)av_clipd(coded_y, 0, 65535); + + if (had_boosted_points) { + double m = is_clipped ? 0.0 : eval_log_gain_slope(s, frame, x, content_headroom_linear, + target_headroom_linear); + double theta = atan(m) * (36000.0 / M_PI) + 18000.0; + app5->gain_curve_control_points_theta[a][out_points] = (uint16_t)av_clipd(round(theta), 1, 35999); + } else { + app5->gain_curve_control_points_theta[a][out_points] = 0; /* unused */ + } + + prev_coded_x = coded_x; + out_points++; + } + + app5->gain_curve_num_control_points_minus_1[a] = out_points - 1; + + for (int k = 0; k < 6; k++) { + app5->has_component_mixing_coefficient_flag[a][k] = 0; + app5->component_mixing_coefficient[a][k] = 0; + } + + if (s->mixing_type == MIXING_MAXRGB) { + app5->component_mixing_type[a] = 0; + } else if (s->mixing_type == MIXING_RGB) { + app5->component_mixing_type[a] = 1; + } else if (s->mixing_type == MIXING_STANDARD_BALANCED) { + app5->component_mixing_type[a] = 2; + } else if (s->mixing_type == MIXING_BALANCED || s->mixing_type == MIXING_LUMA) { + double yr, yg, yb; + + if (s->gain_space == GAIN_SPACE_BT709) { + /* ITU-R BT.709-6 */ + yr = 0.2126; + yg = 0.7152; + yb = 0.0722; + } else if (s->gain_space == GAIN_SPACE_P3) { + /* SMPTE EG 432-1 (P3-D65) */ + yr = 0.2290; + yg = 0.6917; + yb = 0.0793; + } else { + /* ITU-R BT.2020-2 */ + yr = 0.2627; + yg = 0.6780; + yb = 0.0593; + } + + app5->component_mixing_type[a] = 3; + + if (s->mixing_type == MIXING_BALANCED) { + /* 0.5 Max(R,G,B) + 0.5 Luma */ + app5->has_component_mixing_coefficient_flag[a][0] = 1; + app5->has_component_mixing_coefficient_flag[a][1] = 1; + app5->has_component_mixing_coefficient_flag[a][2] = 1; + app5->has_component_mixing_coefficient_flag[a][3] = 1; + + app5->component_mixing_coefficient[a][0] = (uint16_t)round(0.5 * yr * 50000.0); + app5->component_mixing_coefficient[a][1] = (uint16_t)round(0.5 * yg * 50000.0); + app5->component_mixing_coefficient[a][2] = (uint16_t)round(0.5 * yb * 50000.0); + app5->component_mixing_coefficient[a][3] = (uint16_t)round(0.5 * 50000.0); + } else { + /* Pure Luma (Y) */ + app5->has_component_mixing_coefficient_flag[a][0] = 1; + app5->has_component_mixing_coefficient_flag[a][1] = 1; + app5->has_component_mixing_coefficient_flag[a][2] = 1; + + app5->component_mixing_coefficient[a][0] = (uint16_t)round(yr * 50000.0); + app5->component_mixing_coefficient[a][1] = (uint16_t)round(yg * 50000.0); + app5->component_mixing_coefficient[a][2] = (uint16_t)round(yb * 50000.0); + } + } + } +} + +static int filter_frame(AVFilterLink *inlink, AVFrame *frame) +{ + AVFilterContext *ctx = inlink->dst; + Smpte2094App5GenContext *s = ctx->priv; + AVFilterLink *outlink = ctx->outputs[0]; + + AVDynamicHDRSmpte2094App5 *app5 = av_dynamic_hdr_smpte2094_app5_create_side_data(frame); + if (!app5) { + av_frame_free(&frame); + return AVERROR(ENOMEM); + } + + if (s->mode == 0 && s->has_cached_metadata) { + *app5 = s->cached_metadata; + return ff_filter_frame(outlink, frame); + } + + double peak_nits = s->peak; + if (peak_nits <= 0.0) { + peak_nits = get_signal_peak_nits(s, frame); + } + + double ref_white = s->ref_white; + if (ref_white <= 0.0) + ref_white = 203.0; + + double content_headroom_linear = peak_nits / ref_white; + if (content_headroom_linear < 1.0) + content_headroom_linear = 1.0; + if (content_headroom_linear > 64.0) + content_headroom_linear = 64.0; + + double baseline_headroom_log2 = log2(content_headroom_linear); + + app5->application_version = 0; + app5->minimum_application_version = 0; + app5->has_custom_hdr_reference_white_flag = (ref_white != 203.0); + app5->has_adaptive_tone_map_flag = 1; + app5->hdr_reference_white = (uint16_t)round(ref_white * 5.0); + app5->baseline_hdr_headroom = (uint16_t)round(baseline_headroom_log2 * 10000.0); + app5->use_reference_white_tone_mapping_flag = 0; + app5->gain_application_space_chromaticities_flag = s->gain_space; + app5->has_common_component_mix_params_flag = 1; + app5->has_common_curve_params_flag = 0; + + compute_gain_curves(s, frame, app5, content_headroom_linear); + + if (s->mode == 0) { + s->cached_metadata = *app5; + s->has_cached_metadata = 1; + } + + return ff_filter_frame(outlink, frame); +} + +static const AVFilterPad avfilter_vf_smpte2094app5gen_inputs[] = { + { + .name = "default", + .type = AVMEDIA_TYPE_VIDEO, + .filter_frame = filter_frame, + }, +}; + +const FFFilter ff_vf_smpte2094app5gen = { + .p.name = "smpte2094app5gen", + .p.description = NULL_IF_CONFIG_SMALL("Generate SMPTE ST 2094-50 metadata."), + .p.priv_class = &smpte2094app5gen_class, + .p.flags = AVFILTER_FLAG_METADATA_ONLY, + .priv_size = sizeof(Smpte2094App5GenContext), + .init = init, + FILTER_INPUTS(avfilter_vf_smpte2094app5gen_inputs), + FILTER_OUTPUTS(ff_video_default_filterpad), +}; diff --git a/tests/fate/filter-video.mak b/tests/fate/filter-video.mak index 36e3553d05..e1f1950c3f 100644 --- a/tests/fate/filter-video.mak +++ b/tests/fate/filter-video.mak @@ -912,6 +912,11 @@ FATE_FILTER-$(call ALLYES, TESTSRC2_FILTER SPLIT_FILTER AVGBLUR_FILTER \ FATE_FILTER-$(call FILTERFRAMECRC, TESTSRC SCALE PREMULTIPLY, LAVFI_INDEV) += fate-filter-scale-premultiply fate-filter-scale-premultiply: CMD = framecrc -auto_conversion_filters -lavfi "testsrc,format=rgba,setparams=alpha_mode=premultiplied,format=rgba:alpha_modes=straight" -frames:v 10 +SMPTE2094APP5GEN_DEPS = FFPROBE COLOR_FILTER SMPTE2094APP5GEN_FILTER LAVFI_INDEV +FATE_FILTER_FFPROBE-$(call ALLYES, $(SMPTE2094APP5GEN_DEPS)) += fate-filter-smpte2094app5gen-reinhard fate-filter-smpte2094app5gen-linear +fate-filter-smpte2094app5gen-reinhard: CMD = probeframes -show_entries frame=side_data_list -f lavfi -i "color=c=black:s=64x64:d=0.04,format=yuv420p10le,smpte2094app5gen=algorithm=reinhard:peak=1000:ref_white=406" +fate-filter-smpte2094app5gen-linear: CMD = probeframes -show_entries frame=side_data_list -f lavfi -i "color=c=black:s=64x64:d=0.04,format=yuv420p10le,smpte2094app5gen=algorithm=linear:gain_space=bt709:mixing=balanced:peak=1000" + FATE_SAMPLES_FFPROBE += $(FATE_METADATA_FILTER-yes) FATE_SAMPLES_FFMPEG += $(FATE_FILTER_SAMPLES-yes) FATE_FFPROBE += $(FATE_FILTER_FFPROBE-yes) diff --git a/tests/ref/fate/filter-smpte2094app5gen-linear b/tests/ref/fate/filter-smpte2094app5gen-linear new file mode 100644 index 0000000000..5d37fa9d31 --- /dev/null +++ b/tests/ref/fate/filter-smpte2094app5gen-linear @@ -0,0 +1,31 @@ +[FRAME] +[SIDE_DATA] +side_data_type=HDR Dynamic Metadata SMPTE2094-50 +application_version=0 +minimum_application_version=0 +has_custom_hdr_reference_white_flag=0 +has_adaptive_tone_map_flag=1 +baseline_hdr_headroom=23004 +use_reference_white_tone_mapping_flag=0 +num_alternate_images=1 +gain_application_space_chromaticities_flag=0 +has_common_component_mix_params_flag=1 +has_common_curve_params_flag=0 +alternate_hdr_headroom=0 +component_mixing_type=3 +has_component_mixing_coefficient_flag=1 +component_mixing_coefficient=5315 +has_component_mixing_coefficient_flag=1 +component_mixing_coefficient=17880 +has_component_mixing_coefficient_flag=1 +component_mixing_coefficient=1805 +has_component_mixing_coefficient_flag=1 +component_mixing_coefficient=25000 +has_component_mixing_coefficient_flag=0 +has_component_mixing_coefficient_flag=0 +gain_curve_num_control_points_minus_1=0 +gain_curve_use_pchip_slope_flag=1 +gain_curve_control_point_x=4926 +gain_curve_control_point_y=23004 +[/SIDE_DATA] +[/FRAME] diff --git a/tests/ref/fate/filter-smpte2094app5gen-reinhard b/tests/ref/fate/filter-smpte2094app5gen-reinhard new file mode 100644 index 0000000000..6e002603c0 --- /dev/null +++ b/tests/ref/fate/filter-smpte2094app5gen-reinhard @@ -0,0 +1,44 @@ +[FRAME] +[SIDE_DATA] +side_data_type=HDR Dynamic Metadata SMPTE2094-50 +application_version=0 +minimum_application_version=0 +has_custom_hdr_reference_white_flag=1 +has_adaptive_tone_map_flag=1 +hdr_reference_white=2030 +baseline_hdr_headroom=13004 +use_reference_white_tone_mapping_flag=0 +num_alternate_images=1 +gain_application_space_chromaticities_flag=2 +has_common_component_mix_params_flag=1 +has_common_curve_params_flag=0 +alternate_hdr_headroom=0 +component_mixing_type=0 +gain_curve_num_control_points_minus_1=7 +gain_curve_use_pchip_slope_flag=0 +gain_curve_control_point_x=0 +gain_curve_control_point_x=406 +gain_curve_control_point_x=487 +gain_curve_control_point_x=760 +gain_curve_control_point_x=1095 +gain_curve_control_point_x=1490 +gain_curve_control_point_x=1946 +gain_curve_control_point_x=2463 +gain_curve_control_point_y=0 +gain_curve_control_point_y=3 +gain_curve_control_point_y=804 +gain_curve_control_point_y=3241 +gain_curve_control_point_y=5751 +gain_curve_control_point_y=8245 +gain_curve_control_point_y=10672 +gain_curve_control_point_y=13004 +gain_curve_control_point_theta=18000 +gain_curve_control_point_theta=8854 +gain_curve_control_point_theta=9171 +gain_curve_control_point_theta=10132 +gain_curve_control_point_theta=11089 +gain_curve_control_point_theta=11982 +gain_curve_control_point_theta=12782 +gain_curve_control_point_theta=12677 +[/SIDE_DATA] +[/FRAME] -- 2.52.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
