PR #24114 opened by Aarni Koskela (akx) URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24114 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24114.patch
# Summary of changes This PR reduces the memory usage of the `showspectrumpic` and `showwavespic` filters in their new default operating mode (streaming mode). For both filters, there is a small possibility for slightly less accurate output in streaming mode; the old operating mode is still available as `streaming=never`. ## Why? My particular usecase was rendering a `showspectrumpic` spectrogram of a 90-minute continuous recording, which ended up consuming 2310 MB of memory, which of course then SIGKILLed ffmpeg on the small machine I was running this on. With this PR, that 2310 MB became some 320 MB, depending on the output image size, not the input duration. ## How? In non-streaming mode (the old implementation), both filters buffered all input audio before analyzing it, so memory usage would grow linearly according to input duration. In streaming mode, data is accumulated on the fly instead (see comments in code as to exactly how), so memory usage is bound to some multiple of the output image width. ## Test plan I rendered the spectrograms of a 60-minute DJ set with `always` and `never` streaming modes at 4096x2048. To my human eye, both outputs are very similar (I'd say "they're the same picture", to paraphrase the meme). Beyond Compare's pixel-per-pixel comparison sees differences, as expected. I didn't test performance rigorously with e.g. `hyperfine` (so take the wallclock time column with a pinch of fine salt), but anecdotal stats are as follows: | Run | Max RSS | Wallclock time | |---------|---------|---------| | spectrum-always | 249 MB | 4.63s | | spectrum-never | 1441 MB | 5.84s | | wave-always | 91 MB | 3.25s | | wave-never | 896 MB | 3.70s | ## AI Disclosure The particular implementations here were written by Claude Opus 5 (as noted in commit messages); I read the implementations, and asked GPT-5.6-Sol at High effort to cross-check, too. >From fbe935861fe7c44cde300927d7d88a611fa79a8e Mon Sep 17 00:00:00 2001 From: Aarni Koskela <[email protected]> Date: Sun, 26 Jul 2026 19:29:10 +0300 Subject: [PATCH 1/4] avfilter/avf_showspectrum: bound showspectrumpic memory use This adds a new default mode for showspectrumpic that bounds memory use, at the cost of a small amount of output quality for long inputs. The previous implementation (still available with streaming=never) buffers all input samples until EOF, then computes the output, which means peak memory scales with input duration, possibly until ffmpeg runs out of memory. The new implementation (streaming=auto/always) consumes the input in a streaming fashion; once there is more input than would "fit", spectra are accumulated into a set of columns that span the whole input and processed into the output at EOF. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Signed-off-by: Aarni Koskela <[email protected]> --- libavfilter/avf_showspectrum.c | 417 ++++++++++++++++++++++++++++----- 1 file changed, 354 insertions(+), 63 deletions(-) diff --git a/libavfilter/avf_showspectrum.c b/libavfilter/avf_showspectrum.c index 0345b4458d..662bd5414f 100644 --- a/libavfilter/avf_showspectrum.c +++ b/libavfilter/avf_showspectrum.c @@ -55,6 +55,7 @@ enum DisplayScale { LINEAR, SQRT, CBRT, LOG, FOURTHRT, FIFTHRT, NB_SCALES }; enum ColorMode { CHANNEL, INTENSITY, RAINBOW, MORELAND, NEBULAE, FIRE, FIERY, FRUIT, COOL, MAGMA, GREEN, VIRIDIS, PLASMA, CIVIDIS, TERRAIN, NB_CLMODES }; enum SlideMode { REPLACE, SCROLL, FULLFRAME, RSCROLL, LREPLACE, NB_SLIDES }; enum Orientation { VERTICAL, HORIZONTAL, NB_ORIENTATIONS }; +enum StreamMode { STREAM_AUTO, STREAM_ALWAYS, STREAM_NEVER, NB_STREAM_MODES }; #define DEFAULT_LENGTH 300 @@ -118,6 +119,18 @@ typedef struct ShowSpectrumContext { AVFrame **frames; unsigned int nb_frames; unsigned int frames_size; + + /* showspectrumpic: bounded memory accumulation, see stream_window() */ + int stream_mode; ///< when to stream rather than buffer the input + int streaming; ///< 1 once input buffering has given way to streaming + uint64_t max_buf_samples; ///< input samples to buffer before streaming starts + AVFrame *acc_frame; ///< holds up to one FFT window of input samples + int acc_filled; ///< samples currently held by acc_frame + float **acc_data; ///< nb_display_channels x (nb_acc_cols * bins) magnitude sums + int nb_acc_cols; ///< number of accumulator columns + int acc_col; ///< accumulator column currently being filled + uint64_t acc_col_windows; ///< windows summed into each completed column + uint64_t acc_windows; ///< windows summed into acc_col so far } ShowSpectrumContext; #define OFFSET(x) offsetof(ShowSpectrumContext, x) @@ -357,6 +370,13 @@ static av_cold void uninit(AVFilterContext *ctx) } av_freep(&s->frames); + + av_frame_free(&s->acc_frame); + if (s->acc_data) { + for (i = 0; i < s->nb_display_channels; i++) + av_freep(&s->acc_data[i]); + } + av_freep(&s->acc_data); } static int query_formats(const AVFilterContext *ctx, @@ -1055,6 +1075,16 @@ static int config_output(AVFilterLink *outlink) int i, fft_size, h, w, ret; float overlap; + /* Drop any accumulators before the geometry they were sized after is + * replaced below, while nb_display_channels still matches their layout. */ + av_frame_free(&s->acc_frame); + if (s->acc_data) { + for (i = 0; i < s->nb_display_channels; i++) + av_freep(&s->acc_data[i]); + } + av_freep(&s->acc_data); + s->streaming = 0; + s->old_pts = AV_NOPTS_VALUE; s->dmax = expf(s->limit * M_LN10 / 20.f); s->dmin = expf((s->limit - s->drange) * M_LN10 / 20.f); @@ -1298,6 +1328,11 @@ static int config_output(AVFilterLink *outlink) if (!s->frames) return AVERROR(ENOMEM); + /* Buffering the whole input costs memory proportional to its duration, so + * showspectrumpic only does so while every output column is still worth + * less than one FFT window; past that point it streams instead. */ + s->max_buf_samples = (uint64_t)(s->orientation == VERTICAL ? s->w : s->h) * s->win_size; + return 0; } @@ -1738,11 +1773,305 @@ static const AVOption showspectrumpic_options[] = { { "drange", "set dynamic range in dBFS", OFFSET(drange), AV_OPT_TYPE_FLOAT, {.dbl = 120}, 10, 200, FLAGS }, { "limit", "set upper limit in dBFS", OFFSET(limit), AV_OPT_TYPE_FLOAT, {.dbl = 0}, -100, 100, FLAGS }, { "opacity", "set opacity strength", OFFSET(opacity_factor), AV_OPT_TYPE_FLOAT, {.dbl = 1}, 0, 10, FLAGS }, + { "streaming", "set when to stream instead of buffering the input", OFFSET(stream_mode), AV_OPT_TYPE_INT, {.i64=STREAM_AUTO}, 0, NB_STREAM_MODES-1, FLAGS, .unit = "streaming" }, + { "auto", "stream once the input exceeds one window per column", 0, AV_OPT_TYPE_CONST, {.i64=STREAM_AUTO}, 0, 0, FLAGS, .unit = "streaming" }, + { "always", "always stream, never buffer the whole input", 0, AV_OPT_TYPE_CONST, {.i64=STREAM_ALWAYS}, 0, 0, FLAGS, .unit = "streaming" }, + { "never", "always buffer the whole input", 0, AV_OPT_TYPE_CONST, {.i64=STREAM_NEVER}, 0, 0, FLAGS, .unit = "streaming" }, { NULL } }; AVFILTER_DEFINE_CLASS(showspectrumpic); +/* + * Bounded memory accumulation. + * + * The column geometry depends on the total number of input samples, which is + * not known before EOF, so the input is buffered and rendered in one go by + * showspectrumpic_render_buffered(). That costs memory proportional to the + * input duration, which reaches gigabytes for hour-long inputs, so buffering + * stops once max_buf_samples have been seen and the filter streams instead: + * input is consumed one FFT window at a time and the magnitudes are summed + * into nb_acc_cols accumulator columns of acc_col_windows windows each. + * + * When the accumulator columns run out, adjacent pairs are merged and + * acc_col_windows doubles, so the accumulators keep spanning the whole input + * seen so far at a memory cost that depends only on the output size. Since + * streaming begins with at least one window available per output column, and + * since there are twice as many accumulator columns as output columns, at EOF + * the accumulators are always between one and two times over-resolved, and + * showspectrumpic_render_streamed() can area-average them down to the output + * width without losing horizontal resolution. + */ + +static void flush_accumulator_column(ShowSpectrumContext *s) +{ + const int bins = s->orientation == VERTICAL ? s->h : s->w; + + for (int ch = 0; ch < s->nb_display_channels; ch++) { + memcpy(s->acc_data[ch] + s->acc_col * (size_t)bins, s->magnitudes[ch], + bins * sizeof(**s->magnitudes)); + memset(s->magnitudes[ch], 0, bins * sizeof(**s->magnitudes)); + } +} + +static void fold_accumulator_columns(ShowSpectrumContext *s) +{ + const int bins = s->orientation == VERTICAL ? s->h : s->w; + + for (int ch = 0; ch < s->nb_display_channels; ch++) { + float *data = s->acc_data[ch]; + + for (int col = 0; col < s->nb_acc_cols / 2; col++) { + const float *a = data + (2 * col ) * (size_t)bins; + const float *b = data + (2 * col + 1) * (size_t)bins; + float *dst = data + col * (size_t)bins; + + for (int y = 0; y < bins; y++) + dst[y] = a[y] + b[y]; + } + } + + s->acc_col = s->nb_acc_cols / 2; + s->acc_col_windows *= 2; +} + +static void stream_window(AVFilterContext *ctx) +{ + ShowSpectrumContext *s = ctx->priv; + + ff_filter_execute(ctx, run_channel_fft, s->acc_frame, NULL, s->nb_display_channels); + acalc_magnitudes(s); + + if (++s->acc_windows < s->acc_col_windows) + return; + + flush_accumulator_column(s); + s->acc_windows = 0; + if (++s->acc_col == s->nb_acc_cols) + fold_accumulator_columns(s); +} + +static int stream_samples(AVFilterContext *ctx, AVFrame *in) +{ + ShowSpectrumContext *s = ctx->priv; + int offset = 0; + + while (offset < in->nb_samples) { + const int nb_samples = FFMIN(s->win_size - s->acc_filled, + in->nb_samples - offset); + int ret; + + ret = av_samples_copy(s->acc_frame->extended_data, in->extended_data, + s->acc_filled, offset, nb_samples, + s->nb_display_channels, AV_SAMPLE_FMT_FLTP); + if (ret < 0) + return ret; + + s->acc_filled += nb_samples; + offset += nb_samples; + + if (s->acc_filled == s->win_size) { + s->acc_filled = 0; + stream_window(ctx); + } + } + + return 0; +} + +static int start_streaming(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + ShowSpectrumContext *s = ctx->priv; + const int bins = s->orientation == VERTICAL ? s->h : s->w; + const int sz = s->orientation == VERTICAL ? s->w : s->h; + int ret; + + s->nb_acc_cols = 2 * sz; + s->acc_col_windows = 1; + s->acc_windows = 0; + s->acc_col = 0; + s->acc_filled = 0; + s->hop_size = s->win_size; + + s->acc_frame = ff_get_audio_buffer(inlink, s->win_size); + if (!s->acc_frame) + return AVERROR(ENOMEM); + + s->acc_data = av_calloc(s->nb_display_channels, sizeof(*s->acc_data)); + if (!s->acc_data) + return AVERROR(ENOMEM); + for (int ch = 0; ch < s->nb_display_channels; ch++) { + s->acc_data[ch] = av_calloc(s->nb_acc_cols * (size_t)bins, + sizeof(**s->acc_data)); + if (!s->acc_data[ch]) + return AVERROR(ENOMEM); + memset(s->magnitudes[ch], 0, bins * sizeof(**s->magnitudes)); + } + + s->streaming = 1; + + av_log(ctx, AV_LOG_VERBOSE, "More than %"PRIu64" input samples, " + "switching to streaming mode.\n", s->max_buf_samples); + + /* fold what has been buffered so far into the accumulators */ + for (unsigned i = 0; i < s->nb_frames; i++) { + ret = stream_samples(ctx, s->frames[i]); + av_frame_free(&s->frames[i]); + if (ret < 0) { + for (unsigned j = i + 1; j < s->nb_frames; j++) + av_frame_free(&s->frames[j]); + s->nb_frames = 0; + return ret; + } + } + s->nb_frames = 0; + av_freep(&s->frames); + s->frames_size = 0; + + return 0; +} + +static int showspectrumpic_render_streamed(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + ShowSpectrumContext *s = ctx->priv; + const int bins = s->orientation == VERTICAL ? s->h : s->w; + const int sz = s->orientation == VERTICAL ? s->w : s->h; + double total_windows, windows_per_column; + int nb_cols; + + /* zero pad and use the trailing samples, as the buffered path does */ + if (s->acc_filled > 0) { + av_samples_set_silence(s->acc_frame->extended_data, s->acc_filled, + s->win_size - s->acc_filled, + s->nb_display_channels, AV_SAMPLE_FMT_FLTP); + s->acc_filled = 0; + stream_window(ctx); + } + + nb_cols = s->acc_col; + + /* the trailing partial column takes part with its own, smaller weight */ + if (s->acc_windows > 0) { + flush_accumulator_column(s); + nb_cols++; + } + + total_windows = s->acc_col * (double)s->acc_col_windows + s->acc_windows; + if (total_windows <= 0) + return 0; + windows_per_column = total_windows / sz; + + for (int x = 0; x < sz; x++) { + const double lo = x * windows_per_column; + const double hi = lo + windows_per_column; + int first = av_clip(lo / s->acc_col_windows, 0, nb_cols - 1); + int last = av_clip(hi / s->acc_col_windows, first, nb_cols - 1); + int ret; + + for (int ch = 0; ch < s->nb_display_channels; ch++) + memset(s->magnitudes[ch], 0, bins * sizeof(**s->magnitudes)); + + for (int col = first; col <= last; col++) { + const double start = col * (double)s->acc_col_windows; + const double weight = col == s->acc_col && s->acc_windows > 0 + ? s->acc_windows : s->acc_col_windows; + const double overlap = FFMIN(hi, start + weight) - FFMAX(lo, start); + float scale; + + if (overlap <= 0) + continue; + scale = overlap / (weight * windows_per_column); + + for (int ch = 0; ch < s->nb_display_channels; ch++) { + const float *src = s->acc_data[ch] + col * (size_t)bins; + float *magnitudes = s->magnitudes[ch]; + + for (int y = 0; y < bins; y++) + magnitudes[y] += src[y] * scale; + } + } + + ret = plot_spectrum_column(inlink, NULL); + if (ret < 0) + return ret; + } + + return 0; +} + +static int showspectrumpic_render_buffered(AVFilterLink *inlink) +{ + AVFilterContext *ctx = inlink->dst; + ShowSpectrumContext *s = ctx->priv; + int consumed = 0; + int x = 0, sz = s->orientation == VERTICAL ? s->w : s->h; + unsigned int nb_frame = 0; + int ch, spf, spb; + int src_offset = 0; + AVFrame *fin; + + spf = s->win_size * (s->samples / ((s->win_size * sz) * ceil(s->samples / (float)(s->win_size * sz)))); + spf = FFMAX(1, spf); + s->hop_size = spf; + + spb = (s->samples / (spf * sz)) * spf; + + fin = ff_get_audio_buffer(inlink, spf); + if (!fin) + return AVERROR(ENOMEM); + + while (x < sz) { + int acc_samples = 0; + int dst_offset = 0; + + while (nb_frame < s->nb_frames) { + AVFrame *cur_frame = s->frames[nb_frame]; + int cur_frame_samples = cur_frame->nb_samples; + int nb_samples = 0; + + if (acc_samples < spf) { + nb_samples = FFMIN(spf - acc_samples, cur_frame_samples - src_offset); + acc_samples += nb_samples; + av_samples_copy(fin->extended_data, cur_frame->extended_data, + dst_offset, src_offset, nb_samples, + cur_frame->ch_layout.nb_channels, AV_SAMPLE_FMT_FLTP); + } + + src_offset += nb_samples; + dst_offset += nb_samples; + if (cur_frame_samples <= src_offset) { + av_frame_free(&s->frames[nb_frame]); + nb_frame++; + src_offset = 0; + } + + if (acc_samples == spf) + break; + } + + ff_filter_execute(ctx, run_channel_fft, fin, NULL, s->nb_display_channels); + acalc_magnitudes(s); + + consumed += spf; + if (consumed >= spb) { + int h = s->orientation == VERTICAL ? s->h : s->w; + + scale_magnitudes(s, 1.f / (consumed / spf)); + plot_spectrum_column(inlink, fin); + consumed = 0; + x++; + for (ch = 0; ch < s->nb_display_channels; ch++) + memset(s->magnitudes[ch], 0, h * sizeof(float)); + } + } + + av_frame_free(&fin); + + return 0; +} + static int showspectrumpic_request_frame(AVFilterLink *outlink) { AVFilterContext *ctx = outlink->src; @@ -1752,69 +2081,11 @@ static int showspectrumpic_request_frame(AVFilterLink *outlink) ret = ff_request_frame(inlink); if (ret == AVERROR_EOF && s->outpicref && s->samples > 0) { - int consumed = 0; - int x = 0, sz = s->orientation == VERTICAL ? s->w : s->h; - unsigned int nb_frame = 0; - int ch, spf, spb; - int src_offset = 0; - AVFrame *fin; + ret = s->streaming ? showspectrumpic_render_streamed(inlink) + : showspectrumpic_render_buffered(inlink); + if (ret < 0) + return ret; - spf = s->win_size * (s->samples / ((s->win_size * sz) * ceil(s->samples / (float)(s->win_size * sz)))); - spf = FFMAX(1, spf); - s->hop_size = spf; - - spb = (s->samples / (spf * sz)) * spf; - - fin = ff_get_audio_buffer(inlink, spf); - if (!fin) - return AVERROR(ENOMEM); - - while (x < sz) { - int acc_samples = 0; - int dst_offset = 0; - - while (nb_frame < s->nb_frames) { - AVFrame *cur_frame = s->frames[nb_frame]; - int cur_frame_samples = cur_frame->nb_samples; - int nb_samples = 0; - - if (acc_samples < spf) { - nb_samples = FFMIN(spf - acc_samples, cur_frame_samples - src_offset); - acc_samples += nb_samples; - av_samples_copy(fin->extended_data, cur_frame->extended_data, - dst_offset, src_offset, nb_samples, - cur_frame->ch_layout.nb_channels, AV_SAMPLE_FMT_FLTP); - } - - src_offset += nb_samples; - dst_offset += nb_samples; - if (cur_frame_samples <= src_offset) { - av_frame_free(&s->frames[nb_frame]); - nb_frame++; - src_offset = 0; - } - - if (acc_samples == spf) - break; - } - - ff_filter_execute(ctx, run_channel_fft, fin, NULL, s->nb_display_channels); - acalc_magnitudes(s); - - consumed += spf; - if (consumed >= spb) { - int h = s->orientation == VERTICAL ? s->h : s->w; - - scale_magnitudes(s, 1.f / (consumed / spf)); - plot_spectrum_column(inlink, fin); - consumed = 0; - x++; - for (ch = 0; ch < s->nb_display_channels; ch++) - memset(s->magnitudes[ch], 0, h * sizeof(float)); - } - } - - av_frame_free(&fin); s->outpicref->pts = 0; if (s->legend) @@ -1832,11 +2103,31 @@ static int showspectrumpic_filter_frame(AVFilterLink *inlink, AVFrame *insamples AVFilterContext *ctx = inlink->dst; ShowSpectrumContext *s = ctx->priv; void *ptr; + int ret; + + if (!s->streaming && s->stream_mode != STREAM_NEVER && + (s->stream_mode == STREAM_ALWAYS || + s->samples + insamples->nb_samples > s->max_buf_samples)) { + ret = start_streaming(inlink); + if (ret < 0) { + av_frame_free(&insamples); + return ret; + } + } + + if (s->streaming) { + ret = stream_samples(ctx, insamples); + s->samples += insamples->nb_samples; + av_frame_free(&insamples); + return ret; + } if (s->nb_frames + 1ULL > s->frames_size / sizeof(*(s->frames))) { ptr = av_fast_realloc(s->frames, &s->frames_size, s->frames_size * 2); - if (!ptr) + if (!ptr) { + av_frame_free(&insamples); return AVERROR(ENOMEM); + } s->frames = ptr; } -- 2.52.0 >From 03e09b20ba878abfcded8e56569bf8d915e85141 Mon Sep 17 00:00:00 2001 From: Aarni Koskela <[email protected]> Date: Sun, 26 Jul 2026 19:29:23 +0300 Subject: [PATCH 2/4] avfilter/avf_showwaves: bound showwavespic memory use This adds a new default mode for showwavespic that bounds memory use, at the cost of very minimal (likely imperceptible) differences in the output. The previous implementation (still available with streaming=never) buffers all of the input, then reduces them into the output waveform image at EOF. This means peak memory scales with input duration, possibly until ffmpeg runs out of memory. The new implementation (streaming=auto/always) reduces the input into accumulators as they arrive, bounding memory use to a small multiple of the output image size instead of the input duration. Co-Authored-By: Claude Opus 5 (1M context) <[email protected]> Signed-off-by: Aarni Koskela <[email protected]> --- libavfilter/avf_showwaves.c | 242 +++++++++++++++++++++++++++++++++++- 1 file changed, 241 insertions(+), 1 deletion(-) diff --git a/libavfilter/avf_showwaves.c b/libavfilter/avf_showwaves.c index dda15c5306..1d8520ad47 100644 --- a/libavfilter/avf_showwaves.c +++ b/libavfilter/avf_showwaves.c @@ -66,6 +66,19 @@ enum ShowWavesFilterMode { FILTER_NB, }; +enum ShowWavesStreamMode { + STREAM_AUTO, + STREAM_ALWAYS, + STREAM_NEVER, + STREAM_NB, +}; + +/* Accumulator columns per output column, and the overall ceiling on their + * number. One value per column and channel is cheap, so the accumulator is + * kept far finer than the output. */ +#define ACC_OVERSAMPLE 64 +#define ACC_MAX_COLS (1 << 18) + struct frame_node { AVFrame *frame; struct frame_node *next; @@ -101,6 +114,16 @@ typedef struct ShowWavesContext { struct frame_node *last_frame; int64_t total_samples; int64_t *sum; /* abs sum of the samples per channel */ + + /* single picture: bounded memory accumulation, see stream_sample() */ + int stream_mode; ///< when to stream rather than buffer the input + int streaming; ///< 1 once input buffering has given way to streaming + int64_t max_buf_samples; ///< input samples to buffer before streaming starts + int64_t *acc_data; ///< nb_acc_cols x nb_channels sums or peaks + int nb_acc_cols; ///< number of accumulator columns + int acc_col; ///< accumulator column currently being filled + int64_t acc_col_samples; ///< samples reduced into each completed column + int64_t acc_samples; ///< samples reduced into acc_col so far } ShowWavesContext; #define OFFSET(x) offsetof(ShowWavesContext, x) @@ -151,6 +174,7 @@ static av_cold void uninit(AVFilterContext *ctx) av_freep(&tmp); } av_freep(&showwaves->sum); + av_freep(&showwaves->acc_data); showwaves->last_frame = NULL; } } @@ -573,6 +597,181 @@ inline static int push_frame(AVFilterLink *outlink, int i, int64_t pts) return ret; } +/* + * Bounded memory accumulation. + * + * The number of samples per output column is total_samples / w, which is only + * known at EOF, so the input is buffered and reduced in one go by + * push_single_pic(). That costs memory proportional to the input duration, so + * buffering stops once max_buf_samples have been seen and the samples are + * reduced as they arrive instead, into nb_acc_cols accumulator columns of + * acc_col_samples samples each. + * + * When the accumulator columns run out, adjacent pairs are merged and + * acc_col_samples doubles, so the accumulators keep spanning the whole input + * seen so far at a memory cost that depends only on the output width. At EOF + * push_single_pic_streamed() reduces them the rest of the way down to w + * columns. Merging is exact for both filter modes, since a sum of sums is a + * sum and a max of maxima is a max. + */ + +static void fold_accumulator_columns(ShowWavesContext *showwaves, int nb_channels) +{ + const int peak = showwaves->filter_mode == FILTER_PEAK; + int64_t *data = showwaves->acc_data; + + for (int col = 0; col < showwaves->nb_acc_cols / 2; col++) { + const int64_t *a = data + (2 * col ) * (size_t)nb_channels; + const int64_t *b = data + (2 * col + 1) * (size_t)nb_channels; + int64_t *dst = data + col * (size_t)nb_channels; + + for (int ch = 0; ch < nb_channels; ch++) + dst[ch] = peak ? FFMAX(a[ch], b[ch]) : a[ch] + b[ch]; + } + + showwaves->acc_col = showwaves->nb_acc_cols / 2; + showwaves->acc_col_samples *= 2; +} + +static void stream_sample(ShowWavesContext *showwaves, const int16_t *p, + int nb_channels) +{ + int64_t *acc = showwaves->acc_data + showwaves->acc_col * (size_t)nb_channels; + + if (showwaves->acc_samples == 0) { + for (int ch = 0; ch < nb_channels; ch++) + acc[ch] = abs(p[ch]); + } else if (showwaves->filter_mode == FILTER_PEAK) { + for (int ch = 0; ch < nb_channels; ch++) + acc[ch] = FFMAX(acc[ch], abs(p[ch])); + } else { + for (int ch = 0; ch < nb_channels; ch++) + acc[ch] += abs(p[ch]); + } + + if (++showwaves->acc_samples < showwaves->acc_col_samples) + return; + + showwaves->acc_samples = 0; + if (++showwaves->acc_col == showwaves->nb_acc_cols) + fold_accumulator_columns(showwaves, nb_channels); +} + +static int start_streaming(AVFilterContext *ctx) +{ + ShowWavesContext *showwaves = ctx->priv; + AVFilterLink *inlink = ctx->inputs[0]; + const int nb_channels = inlink->ch_layout.nb_channels; + struct frame_node *node; + + showwaves->acc_col_samples = 1; + showwaves->acc_samples = 0; + showwaves->acc_col = 0; + + showwaves->acc_data = av_calloc(showwaves->nb_acc_cols * (size_t)nb_channels, + sizeof(*showwaves->acc_data)); + if (!showwaves->acc_data) + return AVERROR(ENOMEM); + + showwaves->streaming = 1; + + av_log(ctx, AV_LOG_VERBOSE, "More than %"PRId64" input samples, " + "switching to streaming mode.\n", showwaves->max_buf_samples); + + /* fold what has been buffered so far into the accumulators */ + node = showwaves->audio_frames; + while (node) { + struct frame_node *tmp = node; + const AVFrame *frame = node->frame; + const int16_t *p = (const int16_t *)frame->data[0]; + + for (int i = 0; i < frame->nb_samples; i++) + stream_sample(showwaves, p + i * nb_channels, nb_channels); + + node = node->next; + av_frame_free(&tmp->frame); + av_freep(&tmp); + } + showwaves->audio_frames = NULL; + showwaves->last_frame = NULL; + + return 0; +} + +static int push_single_pic_streamed(AVFilterLink *outlink) +{ + AVFilterContext *ctx = outlink->src; + AVFilterLink *inlink = ctx->inputs[0]; + ShowWavesContext *showwaves = ctx->priv; + AVFrame *out = showwaves->outpicref; + const int nb_channels = inlink->ch_layout.nb_channels; + const int ch_height = showwaves->split_channels ? outlink->h / nb_channels : outlink->h; + const int linesize = out->linesize[0]; + const int pixstep = showwaves->pixstep; + const int peak = showwaves->filter_mode == FILTER_PEAK; + double total_samples, samples_per_column; + int nb_cols = showwaves->acc_col; + + /* the trailing partial column takes part with its own, smaller weight */ + if (showwaves->acc_samples > 0) + nb_cols++; + + total_samples = showwaves->acc_col * (double)showwaves->acc_col_samples + + showwaves->acc_samples; + if (total_samples < outlink->w) { + av_log(ctx, AV_LOG_ERROR, "Too few samples\n"); + return AVERROR(EINVAL); + } + samples_per_column = total_samples / outlink->w; + + av_log(ctx, AV_LOG_DEBUG, "Create frame averaging %.0f samples per column\n", + samples_per_column); + + for (int col = 0; col < outlink->w; col++) { + const double lo = col * samples_per_column; + const double hi = lo + samples_per_column; + int first = av_clip(lo / showwaves->acc_col_samples, 0, nb_cols - 1); + int last = av_clip(hi / showwaves->acc_col_samples, first, nb_cols - 1); + + for (int ch = 0; ch < nb_channels; ch++) { + uint8_t *buf = out->data[0] + col * pixstep; + double value = 0; + int h; + + for (int i = first; i <= last; i++) { + const int64_t acc = showwaves->acc_data[i * (size_t)nb_channels + ch]; + const double width = i == showwaves->acc_col && showwaves->acc_samples > 0 + ? showwaves->acc_samples : showwaves->acc_col_samples; + const double start = i * (double)showwaves->acc_col_samples; + + if (peak) { + /* a peak belongs to one column only, so assign each + * accumulator column to whichever column holds its middle */ + const double middle = start + width / 2; + + if (middle >= lo && middle < hi) + value = FFMAX(value, acc); + } else { + const double overlap = FFMIN(hi, start + width) - FFMAX(lo, start); + + if (overlap > 0) + value += acc * overlap / (width * samples_per_column); + } + } + + if (showwaves->split_channels) + buf += ch * ch_height * linesize; + av_assert0(col < outlink->w); + h = showwaves->get_h(av_clip_int16(lrint(value)), ch_height); + showwaves->draw_sample(buf, ch_height, linesize, + &showwaves->buf_idy[ch], + &showwaves->fg[ch * 4], h); + } + } + + return push_frame(outlink, 0, 0); +} + static int push_single_pic(AVFilterLink *outlink) { AVFilterContext *ctx = outlink->src; @@ -651,7 +850,11 @@ static int request_frame(AVFilterLink *outlink) ret = ff_request_frame(inlink); if (ret == AVERROR_EOF && showwaves->outpicref) { - push_single_pic(outlink); + int pic_ret = showwaves->streaming ? push_single_pic_streamed(outlink) + : push_single_pic(outlink); + + if (pic_ret < 0) + return pic_ret; } return ret; @@ -832,6 +1035,10 @@ static const AVOption showwavespic_options[] = { { "filter", "set filter mode", OFFSET(filter_mode), AV_OPT_TYPE_INT, {.i64 = FILTER_AVERAGE}, 0, FILTER_NB-1, FLAGS, .unit="filter" }, { "average", "use average samples", 0, AV_OPT_TYPE_CONST, {.i64=FILTER_AVERAGE}, .flags=FLAGS, .unit="filter"}, { "peak", "use peak samples", 0, AV_OPT_TYPE_CONST, {.i64=FILTER_PEAK}, .flags=FLAGS, .unit="filter"}, + { "streaming", "set when to stream instead of buffering the input", OFFSET(stream_mode), AV_OPT_TYPE_INT, {.i64=STREAM_AUTO}, 0, STREAM_NB-1, FLAGS, .unit="streaming" }, + { "auto", "stream once the input exceeds the accumulator size", 0, AV_OPT_TYPE_CONST, {.i64=STREAM_AUTO}, .flags=FLAGS, .unit="streaming"}, + { "always", "always stream, never buffer the whole input", 0, AV_OPT_TYPE_CONST, {.i64=STREAM_ALWAYS}, .flags=FLAGS, .unit="streaming"}, + { "never", "always buffer the whole input", 0, AV_OPT_TYPE_CONST, {.i64=STREAM_NEVER}, .flags=FLAGS, .unit="streaming"}, { NULL } }; @@ -843,9 +1050,23 @@ static int showwavespic_config_input(AVFilterLink *inlink) ShowWavesContext *showwaves = ctx->priv; if (showwaves->single_pic) { + int64_t nb_acc_cols; + showwaves->sum = av_calloc(inlink->ch_layout.nb_channels, sizeof(*showwaves->sum)); if (!showwaves->sum) return AVERROR(ENOMEM); + + /* Twice the output width at the very least, so that the final pass + * never has to stretch the accumulators back up to it. */ + nb_acc_cols = FFMIN((int64_t)ACC_OVERSAMPLE * showwaves->w, ACC_MAX_COLS); + nb_acc_cols = FFMAX(nb_acc_cols, 2 * (int64_t)showwaves->w); + if (nb_acc_cols > INT_MAX) + return AVERROR(EINVAL); + showwaves->nb_acc_cols = nb_acc_cols; + + /* Buffering up to one sample per accumulator column loses nothing, so + * inputs that short are still reduced exactly by push_single_pic(). */ + showwaves->max_buf_samples = showwaves->nb_acc_cols; } return 0; @@ -865,6 +1086,25 @@ static int showwavespic_filter_frame(AVFilterLink *inlink, AVFrame *insamples) if (ret < 0) goto end; + if (!showwaves->streaming && showwaves->stream_mode != STREAM_NEVER && + (showwaves->stream_mode == STREAM_ALWAYS || + showwaves->total_samples + insamples->nb_samples > showwaves->max_buf_samples)) { + ret = start_streaming(ctx); + if (ret < 0) + goto end; + } + + if (showwaves->streaming) { + const int nb_channels = inlink->ch_layout.nb_channels; + const int16_t *p = (const int16_t *)insamples->data[0]; + + for (int i = 0; i < insamples->nb_samples; i++) + stream_sample(showwaves, p + i * nb_channels, nb_channels); + + showwaves->total_samples += insamples->nb_samples; + goto end; + } + /* queue the audio frame */ f = av_malloc(sizeof(*f)); if (!f) { -- 2.52.0 >From 1a58550999e6d8318d1c1325e2949b29603bb170 Mon Sep 17 00:00:00 2001 From: Aarni Koskela <[email protected]> Date: Wed, 12 Aug 2026 16:07:21 +0300 Subject: [PATCH 3/4] avfilter: Bump micro version (new parameters in filters) Signed-off-by: Aarni Koskela <[email protected]> --- libavfilter/version.h | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/libavfilter/version.h b/libavfilter/version.h index 48abf6052f..3efe8c6651 100644 --- a/libavfilter/version.h +++ b/libavfilter/version.h @@ -32,7 +32,7 @@ #include "version_major.h" #define LIBAVFILTER_VERSION_MINOR 3 -#define LIBAVFILTER_VERSION_MICRO 101 +#define LIBAVFILTER_VERSION_MICRO 102 #define LIBAVFILTER_VERSION_INT AV_VERSION_INT(LIBAVFILTER_VERSION_MAJOR, \ -- 2.52.0 >From 167b9df55bdc1f3e373d8e3e7047e2407636797f Mon Sep 17 00:00:00 2001 From: Aarni Koskela <[email protected]> Date: Wed, 12 Aug 2026 17:59:50 +0300 Subject: [PATCH 4/4] avfilter: document new streaming modes for showspectrumpic/showwavespic Signed-off-by: Aarni Koskela <[email protected]> --- doc/filters.texi | 56 ++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 56 insertions(+) diff --git a/doc/filters.texi b/doc/filters.texi index 51885fddde..3595999081 100644 --- a/doc/filters.texi +++ b/doc/filters.texi @@ -33888,6 +33888,31 @@ Allowed range is from -100 to 100. @item opacity Set opacity strength when using pixel format output with alpha component. + +@item streaming +Set streaming accumulation behavior. + +It accepts the following values: +@table @samp +@item auto +Buffer the input until it exceeds one FFT window per output column, then stream. +If the input is known to be long or is of unknown duration, @samp{always} may be +a better choice, as switching temporarily holds both the buffered input and the +accumulators. + +@item always +Never buffer the whole input, at the cost of coarser detail for short inputs. +This avoids the temporary memory increase when @samp{auto} switches modes and +has the lowest peak memory for inputs long enough to trigger that switch. + +@item never +Always buffer the whole input. +This preserves the previous fully buffered rendering behavior and avoids the +streaming approximation, at a memory cost proportional to input duration. +This was the default behavior before the streaming option was added. + +@end table +Default value is @samp{auto}. @end table @subsection Examples @@ -34141,6 +34166,37 @@ Use peak samples values for each drawn sample. @end table Default value is @code{average}. + +@item streaming +Set streaming accumulation behavior. + +If accumulators fold, final output-column boundaries may cross accumulator buckets, +slightly approximating averages and peak locations. + +It accepts the following values: +@table @samp +@item auto +Buffer up to one sample per accumulator column, exactly like @samp{never} for +short-enough inputs, then begin accumulating like @samp{always}. +If the input is known to be long or is of unknown duration, @samp{always} may be +a better choice, as switching temporarily holds both the buffered input and the +accumulators. + +@item always +Accumulate from the first sample. +Inputs that do not require accumulator folding retain one accumulator value per sample. +This avoids the temporary memory increase when @samp{auto} switches modes +and has the lowest peak memory for inputs long enough that @samp{auto} would switch. + +@item never +Always buffer the whole input. This preserves the previous fully buffered +rendering behavior and avoids the streaming approximation, at a memory cost +proportional to input duration. +This was the default behavior before the streaming option was added. + +@end table +Default value is @samp{auto}. + @end table @subsection Examples -- 2.52.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
