PR #25028 opened by Graham Vosloo (GrahamV)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25028
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25028.patch
# Summary of changes
prores_ks spends most of its time in find_slice_quant(), which
estimates the size and the error of every slice for each quantiser of
the profile and, if the slice is still too large, for every further
quantiser up to 127. These five commits make the estimate cheaper. The
encoder output is byte-identical after each of them.
1. Multiply with stored reciprocals instead of dividing every
coefficient by its quantiser. The commit message has the argument
why the result is the same for every input; a separate program
confirmed it for every divisor and numerator that can occur.
2. Look up the lengths of the run and level codes in tables for the
small values that make up nearly all of them.
3. Keep a list of the coefficients that are still coded. A coefficient
that is zero for one quantiser is zero for every larger one, and the
passes of a slice use the quantisers in increasing order, so each
pass only visits what the previous one left.
4. Stop a pass above the profile's range as soon as its size exceeds
the budget; of those passes only "does not fit" is needed.
5. Skip quantisers above the profile's range whose size has a lower
bound, taken from a histogram of the levels, that is already over
the budget.
Frames per second on an Apple M1 Max, master -> this series:
```
one thread default threads
nat1080 proxy 14.81 -> 43.96 (2.97x) 112.6 -> 324.3 (2.88x)
nat1080 lt 9.97 -> 23.21 (2.33x) 76.63 -> 178.6 (2.33x)
nat1080 standard 10.76 -> 20.10 (1.87x) 83.04 -> 155.4 (1.87x)
nat1080 hq 8.75 -> 15.33 (1.75x) 68.65 -> 121.0 (1.76x)
nat1080 4444 6.68 -> 11.74 (1.76x) 52.52 -> 92.31 (1.76x)
nat1080 4444 +alpha 4.83 -> 7.17 (1.48x) 37.91 -> 56.44 (1.49x)
nat4k hq 2.46 -> 4.42 (1.80x) 18.95 -> 33.50 (1.77x)
noisy1080 hq 1.92 -> 7.06 (3.68x) 15.19 -> 54.64 (3.60x)
scr1080 proxy 2.94 -> 23.90 (8.13x) 22.59 -> 178.0 (7.88x)
scr1080 hq 3.41 -> 14.00 (4.11x) 26.21 -> 106.6 (4.07x)
busy1080 hq 2.54 -> 11.06 (4.35x) 18.59 -> 83.51 (4.49x)
```
nat is camera footage (the 4k clip is 2560x1600 footage scaled up),
noisy is the same with noise added, scr is a synthetic desktop with
text-like patterns, busy is a Mandelbrot zoom. The commit messages have
the numbers for each step.
The series does not replace the linear scan above the profile's range
by a bisection. I measured one: the size is not monotonic in the
quantiser (it rises somewhere for 62.5% of the slices that go over
budget), and the search picked another quantiser than the scan for 138
of 223212 such slices. Commits 4 and 5 keep the scan's result exactly.
Output checked with the prores FATE tests and with 80 encodes (1080p
and 4K; every profile, with and without alpha; interlaced,
-mbs_per_slice 2, -bits_per_mb 128, -quant_mat proxy, an odd frame
size) that have the same md5 after every commit.
The Vulkan ProRes encoder shares ProresContext, which gains the
reciprocal tables. It does not read them and no shared function
changes its signature, but I could not build it here.
All numbers are from one machine. I have none for x86.
Tested with `make fate` on AArch64 (macOS); the prores FATE tests also
pass in an x86-64 build run under Rosetta 2.
>From da7dc4dee82428eae22db0715df87b5126f59b66 Mon Sep 17 00:00:00 2001
From: Graham Vosloo <[email protected]>
Date: Mon, 12 Oct 2026 08:29:01 -0400
Subject: [PATCH 1/5] avcodec/proresenc_kostya: estimate with reciprocals
instead of divisions
find_slice_quant() estimates the size and the error of every slice for
each quantiser of the profile and, if the slice is still too large, for
every further quantiser up to 127. Each estimate divides every coefficient
by its quantiser and takes the remainder as the error. Store the
reciprocals next to the quantiser tables and multiply instead; a
coefficient below its quantiser needs neither.
The result is the same for every input. Let x be the numerator, d >= 2
the divisor and r = ceil(2^32 / d), so r * d = 2^32 + e with 0 <= e < d,
and write x = q * d + m with 0 <= m < d. Then
x * r / 2^32 = q + (m + x * e / 2^32) / d
and its integer part is q if x * e < 2^32, for which x * d < 2^32 is
enough. The numerators are magnitudes of int16_t AC coefficients, at
most 32768, and of DC coefficients minus 0x4000, at most 49152. The
divisors are matrix entry times quantiser, 2 * 1 to 63 * 127 with the
matrices of the encoder and below 32768 in any case because the tables
are int16_t, so x * d < 2^31. r fits into 32 bits because d >= 2.
A separate program confirmed this for every d from 2 to 32767 and every
x from 0 to 49152.
The encoder output is unchanged: the prores FATE tests pass with their
references, and 80 encodes (1080p and 4K; camera, noisy, screen-like,
Mandelbrot and testsrc2 material; every profile, with and without alpha;
interlaced, -mbs_per_slice 2, -bits_per_mb 128, -quant_mat proxy and an
odd frame size) have the same md5 as before.
Frames per second on an Apple M1 Max, before -> after:
one thread
nat1080 proxy 14.81 -> 23.44 (1.58x)
nat1080 lt 9.97 -> 13.95 (1.40x)
nat1080 standard 10.76 -> 13.57 (1.26x)
nat1080 hq 8.75 -> 10.44 (1.19x)
nat1080 4444 6.68 -> 8.26 (1.24x)
nat1080 4444 +alpha 4.83 -> 5.62 (1.16x)
nat1080 4444xq 6.30 -> 7.68 (1.22x)
scr1080 proxy 2.94 -> 4.74 (1.61x)
scr1080 hq 3.41 -> 4.33 (1.27x)
scr1080 4444xq 9.89 -> 13.67 (1.38x)
nat4k hq 2.46 -> 3.01 (1.22x)
noisy1080 hq 1.92 -> 2.16 (1.13x)
busy1080 hq 2.54 -> 2.89 (1.14x)
ts1080 hq 14.55 -> 21.20 (1.46x)
default threads
nat1080 proxy 112.6 -> 179.6 (1.60x)
nat1080 lt 76.63 -> 107.9 (1.41x)
nat1080 standard 83.04 -> 105.5 (1.27x)
nat1080 hq 68.65 -> 82.25 (1.20x)
nat1080 4444 52.52 -> 65.61 (1.25x)
nat1080 4444 +alpha 37.91 -> 44.30 (1.17x)
nat1080 4444xq 49.00 -> 60.70 (1.24x)
scr1080 proxy 22.59 -> 37.15 (1.64x)
scr1080 hq 26.21 -> 33.64 (1.28x)
scr1080 4444xq 74.44 -> 103.0 (1.38x)
nat4k hq 18.95 -> 23.21 (1.22x)
noisy1080 hq 15.19 -> 17.18 (1.13x)
busy1080 hq 18.59 -> 21.54 (1.16x)
ts1080 hq 109.1 -> 160.4 (1.47x)
nat: camera footage (1920x1080; for 4k 2560x1600 footage scaled to
3840x2160), scr: synthetic desktop with text-like patterns, noisy: nat
with noise added, busy: Mandelbrot zoom, ts: testsrc2. yuv422p10 input,
yuv444p10 for the 4444 profiles, yuva444p10 for +alpha. 12 frames with
one thread (6 at 4k), 120 (48) with the default 10 threads; best of 3
interleaved runs.
Assisted-by: Claude Opus 5.5
Signed-off-by: Graham Vosloo <[email protected]>
---
libavcodec/proresenc_kostya.c | 57 ++++++++++++++++++++--------
libavcodec/proresenc_kostya_common.c | 2 +
libavcodec/proresenc_kostya_common.h | 8 ++++
3 files changed, 51 insertions(+), 16 deletions(-)
diff --git a/libavcodec/proresenc_kostya.c b/libavcodec/proresenc_kostya.c
index 847a47bd2c..d294b244e1 100644
--- a/libavcodec/proresenc_kostya.c
+++ b/libavcodec/proresenc_kostya.c
@@ -50,6 +50,8 @@ typedef struct ProresThreadData {
DECLARE_ALIGNED(16, uint16_t, emu_buf)[16 * 16];
int16_t custom_q[64];
int16_t custom_chroma_q[64];
+ uint32_t custom_recip[64];
+ uint32_t custom_chroma_recip[64];
struct TrellisNode *nodes;
} ProresThreadData;
@@ -421,11 +423,12 @@ static inline int estimate_vlc(unsigned codebook, int val)
}
static int estimate_dcs(int *error, int16_t *blocks, int blocks_per_slice,
- int scale)
+ int scale, unsigned recip)
{
int i;
int codebook = 5, code, dc, prev_dc, delta, sign, new_sign;
int bits;
+ unsigned abs_dc;
prev_dc = (blocks[0] - 0x4000) / scale;
bits = estimate_vlc(FIRST_DC_CB, MAKE_CODE(prev_dc));
@@ -434,8 +437,12 @@ static int estimate_dcs(int *error, int16_t *blocks, int
blocks_per_slice,
*error += FFABS(blocks[0] - 0x4000) % scale;
for (i = 1; i < blocks_per_slice; i++, blocks += 64) {
- dc = (blocks[0] - 0x4000) / scale;
- *error += FFABS(blocks[0] - 0x4000) % scale;
+ dc = blocks[0] - 0x4000;
+ new_sign = GET_SIGN(dc);
+ abs_dc = FFABS(dc);
+ dc = (uint64_t)abs_dc * recip >> 32;
+ *error += abs_dc - dc * scale;
+ dc = (dc ^ new_sign) - new_sign;
delta = dc - prev_dc;
new_sign = GET_SIGN(delta);
delta = (delta ^ sign) - sign;
@@ -450,12 +457,13 @@ static int estimate_dcs(int *error, int16_t *blocks, int
blocks_per_slice,
}
static int estimate_acs(int *error, int16_t *blocks, int blocks_per_slice,
- const uint8_t *scan, const int16_t *qmat)
+ const uint8_t *scan, const int16_t *qmat,
+ const uint32_t *qrecip)
{
int idx, i;
int prev_run = 4;
int prev_level = 2;
- int run, level;
+ int run;
int max_coeffs, abs_level;
int bits = 0;
@@ -463,11 +471,15 @@ static int estimate_acs(int *error, int16_t *blocks, int
blocks_per_slice,
run = 0;
for (i = 1; i < 64; i++) {
+ const unsigned quant = qmat[scan[i]];
+ const unsigned recip = qrecip[scan[i]];
+
for (idx = scan[i]; idx < max_coeffs; idx += 64) {
- level = blocks[idx] / qmat[scan[i]];
- *error += FFABS(blocks[idx]) % qmat[scan[i]];
- if (level) {
- abs_level = FFABS(level);
+ const unsigned coeff = FFABS(blocks[idx]);
+
+ if (coeff >= quant) {
+ abs_level = (uint64_t)coeff * recip >> 32;
+ *error += coeff - abs_level * quant;
bits += estimate_vlc(ff_prores_run_to_cb[prev_run], run);
bits += estimate_vlc(ff_prores_level_to_cb[prev_level],
abs_level - 1) + 1;
@@ -475,6 +487,7 @@ static int estimate_acs(int *error, int16_t *blocks, int
blocks_per_slice,
prev_level = FFMIN(abs_level, 9);
run = 0;
} else {
+ *error += coeff;
run++;
}
}
@@ -487,15 +500,17 @@ static int estimate_slice_plane(ProresContext *ctx, int
*error, int plane,
const uint16_t *src, ptrdiff_t linesize,
int mbs_per_slice,
int blocks_per_mb,
- const int16_t *qmat, ProresThreadData *td)
+ const int16_t *qmat, const uint32_t *qrecip,
+ ProresThreadData *td)
{
int blocks_per_slice;
int bits;
blocks_per_slice = mbs_per_slice * blocks_per_mb;
- bits = estimate_dcs(error, td->blocks[plane], blocks_per_slice, qmat[0]);
- bits += estimate_acs(error, td->blocks[plane], blocks_per_slice,
ctx->scantable, qmat);
+ bits = estimate_dcs(error, td->blocks[plane], blocks_per_slice, qmat[0],
qrecip[0]);
+ bits += estimate_acs(error, td->blocks[plane], blocks_per_slice,
ctx->scantable,
+ qmat, qrecip);
return FFALIGN(bits, 8);
}
@@ -574,6 +589,8 @@ static int find_slice_quant(AVCodecContext *avctx,
int overquant;
uint16_t *qmat;
uint16_t *qmat_chroma;
+ const uint32_t *qrecip;
+ const uint32_t *qrecip_chroma;
int linesize[4], line_add, picture_height;
int alpha_bits = 0;
@@ -633,13 +650,15 @@ static int find_slice_quant(AVCodecContext *avctx,
src, linesize[0],
mbs_per_slice,
num_cblocks[0],
- ctx->quants[q], td); /* estimate luma
plane */
+ ctx->quants[q], ctx->quants_recip[q],
+ td); /* estimate luma plane */
for (i = 1; i < ctx->num_planes - !!ctx->alpha_bits; i++) { /*
estimate chroma plane */
bits += estimate_slice_plane(ctx, &error, i,
src, linesize[i],
mbs_per_slice,
num_cblocks[i],
- ctx->quants_chroma[q], td);
+ ctx->quants_chroma[q],
+ ctx->quants_chroma_recip[q], td);
}
if (bits > 65000 * 8)
error = SCORE_LIMIT;
@@ -658,25 +677,31 @@ static int find_slice_quant(AVCodecContext *avctx,
if (q < MAX_STORED_Q) {
qmat = ctx->quants[q];
qmat_chroma = ctx->quants_chroma[q];
+ qrecip = ctx->quants_recip[q];
+ qrecip_chroma = ctx->quants_chroma_recip[q];
} else {
qmat = td->custom_q;
qmat_chroma = td->custom_chroma_q;
+ qrecip = td->custom_recip;
+ qrecip_chroma = td->custom_chroma_recip;
for (i = 0; i < 64; i++) {
qmat[i] = ctx->quant_mat[i] * q;
qmat_chroma[i] = ctx->quant_chroma_mat[i] * q;
+ td->custom_recip[i] = PRORES_QUANT_RECIP(qmat[i]);
+ td->custom_chroma_recip[i] =
PRORES_QUANT_RECIP(qmat_chroma[i]);
}
}
bits += estimate_slice_plane(ctx, &error, 0,
src, linesize[0],
mbs_per_slice,
num_cblocks[0],
- qmat, td);/* estimate luma plane */
+ qmat, qrecip, td);/* estimate luma
plane */
for (i = 1; i < ctx->num_planes - !!ctx->alpha_bits; i++) { /*
estimate chroma plane */
bits += estimate_slice_plane(ctx, &error, i,
src, linesize[i],
mbs_per_slice,
num_cblocks[i],
- qmat_chroma, td);
+ qmat_chroma, qrecip_chroma, td);
}
if (bits <= ctx->bits_per_mb * mbs_per_slice)
break;
diff --git a/libavcodec/proresenc_kostya_common.c
b/libavcodec/proresenc_kostya_common.c
index 558637d144..a833903866 100644
--- a/libavcodec/proresenc_kostya_common.c
+++ b/libavcodec/proresenc_kostya_common.c
@@ -260,6 +260,8 @@ av_cold int ff_prores_kostya_encode_init(AVCodecContext
*avctx, ProresContext *c
for (j = 0; j < 64; j++) {
ctx->quants[i][j] = ctx->quant_mat[j] * i;
ctx->quants_chroma[i][j] = ctx->quant_chroma_mat[j] * i;
+ ctx->quants_recip[i][j] =
PRORES_QUANT_RECIP(ctx->quants[i][j]);
+ ctx->quants_chroma_recip[i][j] =
PRORES_QUANT_RECIP(ctx->quants_chroma[i][j]);
}
}
} else {
diff --git a/libavcodec/proresenc_kostya_common.h
b/libavcodec/proresenc_kostya_common.h
index ed5355db0f..45972ac528 100644
--- a/libavcodec/proresenc_kostya_common.h
+++ b/libavcodec/proresenc_kostya_common.h
@@ -45,6 +45,12 @@ FF_VISIBILITY_PUSH_HIDDEN
#define MAX_STORED_Q 16
+/**
+ * Reciprocal of a quantiser: x / d == (uint64_t)x * PRORES_QUANT_RECIP(d) >>
32
+ * for 2 <= d and x * d < 2^32.
+ */
+#define PRORES_QUANT_RECIP(d) (UINT32_MAX / (d) + 1)
+
enum {
PRORES_PROFILE_AUTO = -1,
PRORES_PROFILE_PROXY = 0,
@@ -84,6 +90,8 @@ typedef struct ProresContext {
DECLARE_ALIGNED(16, uint16_t, emu_buf)[16*16];
int16_t quants[MAX_STORED_Q][64];
int16_t quants_chroma[MAX_STORED_Q][64];
+ uint32_t quants_recip[MAX_STORED_Q][64];
+ uint32_t quants_chroma_recip[MAX_STORED_Q][64];
int16_t custom_q[64];
int16_t custom_chroma_q[64];
const uint8_t *quant_mat;
--
2.52.0
>From 18c8114f1c64993eaeddd546a34146f41d89a872 Mon Sep 17 00:00:00 2001
From: Graham Vosloo <[email protected]>
Date: Mon, 12 Oct 2026 09:41:49 -0400
Subject: [PATCH 2/5] avcodec/proresenc_kostya: look up the code lengths in the
estimate
estimate_vlc() derives the length of a run or level code from the
codebook descriptor with a data dependent branch between the Rice and
the exp-Golomb part. In the estimate nearly all runs and levels are
small, so tabulate their lengths for each of the 16 run and 10 level
codebook selectors and keep estimate_vlc() for the rest. The tables are
filled from estimate_vlc() itself; the output is unchanged (prores FATE
tests, and the same md5 for the 80 encodes of the previous commit).
The level table has 256 entries per selector because the 4444 profiles
reach high levels on synthetic material: with 64 entries the screen-like
clip below was 2 to 5% slower than before this commit with those
profiles, instead of faster.
Frames per second on an Apple M1 Max, master -> previous commit -> this
commit:
one thread
nat1080 proxy 14.81 -> 23.44 -> 25.53 (1.72x)
nat1080 lt 9.97 -> 13.95 -> 15.87 (1.59x)
nat1080 standard 10.76 -> 13.57 -> 15.58 (1.45x)
nat1080 hq 8.75 -> 10.44 -> 12.10 (1.38x)
nat1080 4444 6.68 -> 8.26 -> 9.46 (1.42x)
nat1080 4444 +alpha 4.83 -> 5.62 -> 6.33 (1.31x)
nat1080 4444xq 6.30 -> 7.68 -> 8.71 (1.38x)
scr1080 proxy 2.94 -> 4.74 -> 5.70 (1.94x)
scr1080 hq 3.41 -> 4.33 -> 6.12 (1.79x)
scr1080 4444xq 9.89 -> 13.67 -> 15.83 (1.60x)
nat4k hq 2.46 -> 3.01 -> 3.47 (1.41x)
noisy1080 hq 1.92 -> 2.16 -> 2.85 (1.48x)
busy1080 hq 2.54 -> 2.89 -> 4.07 (1.60x)
ts1080 hq 14.55 -> 21.20 -> 24.44 (1.68x)
default threads
nat1080 proxy 112.6 -> 179.6 -> 194.8 (1.73x)
nat1080 lt 76.63 -> 107.9 -> 125.1 (1.63x)
nat1080 standard 83.04 -> 105.5 -> 122.8 (1.48x)
nat1080 hq 68.65 -> 82.25 -> 95.77 (1.40x)
nat1080 4444 52.52 -> 65.61 -> 74.67 (1.42x)
nat1080 4444 +alpha 37.91 -> 44.30 -> 50.36 (1.33x)
nat1080 4444xq 49.00 -> 60.70 -> 69.97 (1.43x)
scr1080 proxy 22.59 -> 37.15 -> 44.23 (1.96x)
scr1080 hq 26.21 -> 33.64 -> 47.30 (1.80x)
scr1080 4444xq 74.44 -> 103.0 -> 120.2 (1.62x)
nat4k hq 18.95 -> 23.21 -> 26.73 (1.41x)
noisy1080 hq 15.19 -> 17.18 -> 22.44 (1.48x)
busy1080 hq 18.59 -> 21.54 -> 29.93 (1.61x)
ts1080 hq 109.1 -> 160.4 -> 184.1 (1.69x)
nat: camera footage (1920x1080; for 4k 2560x1600 footage scaled to
3840x2160), scr: synthetic desktop with text-like patterns, noisy: nat
with noise added, busy: Mandelbrot zoom, ts: testsrc2. yuv422p10 input,
yuv444p10 for the 4444 profiles, yuva444p10 for +alpha. 12 frames with
one thread (6 at 4k), 120 (48) with the default 10 threads; best of 3
interleaved runs.
Assisted-by: Claude Opus 5.5
Signed-off-by: Graham Vosloo <[email protected]>
---
libavcodec/proresenc_kostya.c | 40 ++++++++++++++++++++++++++++++++---
1 file changed, 37 insertions(+), 3 deletions(-)
diff --git a/libavcodec/proresenc_kostya.c b/libavcodec/proresenc_kostya.c
index d294b244e1..c1abc714c2 100644
--- a/libavcodec/proresenc_kostya.c
+++ b/libavcodec/proresenc_kostya.c
@@ -25,6 +25,7 @@
#include "libavutil/mem_internal.h"
#include "libavutil/opt.h"
#include "libavutil/pixdesc.h"
+#include "libavutil/thread.h"
#include "avcodec.h"
#include "codec_internal.h"
#include "encode.h"
@@ -38,6 +39,9 @@
#define TRELLIS_WIDTH 16
#define SCORE_LIMIT INT_MAX / 2
+#define RUN_BITS_TAB_SIZE 64
+#define LEVEL_BITS_TAB_SIZE 256
+
struct TrellisNode {
int prev_node;
int quant;
@@ -422,6 +426,34 @@ static inline int estimate_vlc(unsigned codebook, int val)
}
}
+/* code lengths of small runs and levels for every codebook selector */
+static uint8_t run_bits_tab[16][RUN_BITS_TAB_SIZE];
+static uint8_t level_bits_tab[10][LEVEL_BITS_TAB_SIZE];
+
+static av_cold void init_bits_tabs(void)
+{
+ for (int i = 0; i < 16; i++)
+ for (int j = 0; j < RUN_BITS_TAB_SIZE; j++)
+ run_bits_tab[i][j] = estimate_vlc(ff_prores_run_to_cb[i], j);
+ for (int i = 0; i < 10; i++)
+ for (int j = 0; j < LEVEL_BITS_TAB_SIZE; j++)
+ level_bits_tab[i][j] = estimate_vlc(ff_prores_level_to_cb[i], j);
+}
+
+static av_always_inline int estimate_run(int prev_run, int run)
+{
+ if (run < RUN_BITS_TAB_SIZE)
+ return run_bits_tab[prev_run][run];
+ return estimate_vlc(ff_prores_run_to_cb[prev_run], run);
+}
+
+static av_always_inline int estimate_level(int prev_level, int level)
+{
+ if (level < LEVEL_BITS_TAB_SIZE)
+ return level_bits_tab[prev_level][level];
+ return estimate_vlc(ff_prores_level_to_cb[prev_level], level);
+}
+
static int estimate_dcs(int *error, int16_t *blocks, int blocks_per_slice,
int scale, unsigned recip)
{
@@ -480,9 +512,8 @@ static int estimate_acs(int *error, int16_t *blocks, int
blocks_per_slice,
if (coeff >= quant) {
abs_level = (uint64_t)coeff * recip >> 32;
*error += coeff - abs_level * quant;
- bits += estimate_vlc(ff_prores_run_to_cb[prev_run], run);
- bits += estimate_vlc(ff_prores_level_to_cb[prev_level],
- abs_level - 1) + 1;
+ bits += estimate_run(prev_run, run);
+ bits += estimate_level(prev_level, abs_level - 1) + 1;
prev_run = FFMIN(run, 15);
prev_level = FFMIN(abs_level, 9);
run = 0;
@@ -960,6 +991,7 @@ static void prores_fdct(FDCTDSPContext *fdsp, const
uint16_t *src,
static av_cold int encode_init(AVCodecContext *avctx)
{
+ static AVOnce init_static_once = AV_ONCE_INIT;
ProresContext *ctx = avctx->priv_data;
int err = 0, i, j, min_quant, max_quant;
@@ -974,6 +1006,8 @@ static av_cold int encode_init(AVCodecContext *avctx)
min_quant = ctx->profile_info->min_quant;
max_quant = ctx->profile_info->max_quant;
+ ff_thread_once(&init_static_once, init_bits_tabs);
+
ctx->slice_q = av_malloc_array(ctx->slices_per_picture,
sizeof(*ctx->slice_q));
if (!ctx->slice_q)
return AVERROR(ENOMEM);
--
2.52.0
>From 2bc8945540d2484975904d5cd0c34310378eeb7e Mon Sep 17 00:00:00 2001
From: Graham Vosloo <[email protected]>
Date: Mon, 12 Oct 2026 09:45:08 -0400
Subject: [PATCH 3/5] avcodec/proresenc_kostya: estimate only the coefficients
that are still coded
Every estimate pass walks over all AC coefficients of the slice, with a
stride of 64 because the coefficients are coded position by position
across the blocks, although most of them are quantised to zero. A
coefficient that is zero for one quantiser is zero for every larger one,
and the passes of a slice use the quantisers in increasing order.
So collect the coefficients that the smallest quantiser of the profile
does not zero once per slice, in coding order and with their position,
and let each pass drop the ones it zeroes. A run is the distance between
two positions. The error of a zeroed coefficient is its magnitude, so
the error of a pass is the sum of all magnitudes, also computed once,
minus what the coded levels represent.
The output is unchanged (prores FATE tests, and the same md5 for the 80
encodes of the first commit of this series). The list takes 36 kB per
slice thread.
Frames per second on an Apple M1 Max, master -> previous commit -> this
commit:
one thread
nat1080 proxy 14.81 -> 25.53 -> 43.64 (2.95x)
nat1080 lt 9.97 -> 15.87 -> 23.44 (2.35x)
nat1080 standard 10.76 -> 15.58 -> 20.30 (1.89x)
nat1080 hq 8.75 -> 12.10 -> 15.52 (1.77x)
nat1080 4444 6.68 -> 9.46 -> 11.85 (1.77x)
nat1080 4444 +alpha 4.83 -> 6.33 -> 7.25 (1.50x)
nat1080 4444xq 6.30 -> 8.71 -> 10.70 (1.70x)
scr1080 proxy 2.94 -> 5.70 -> 13.23 (4.50x)
scr1080 hq 3.41 -> 6.12 -> 8.36 (2.45x)
scr1080 4444xq 9.89 -> 15.83 -> 21.20 (2.14x)
nat4k hq 2.46 -> 3.47 -> 4.46 (1.81x)
noisy1080 hq 1.92 -> 2.85 -> 5.01 (2.61x)
busy1080 hq 2.54 -> 4.07 -> 5.69 (2.24x)
ts1080 hq 14.55 -> 24.44 -> 39.34 (2.70x)
default threads
nat1080 proxy 112.6 -> 194.8 -> 320.9 (2.85x)
nat1080 lt 76.63 -> 125.1 -> 182.4 (2.38x)
nat1080 standard 83.04 -> 122.8 -> 159.4 (1.92x)
nat1080 hq 68.65 -> 95.77 -> 120.2 (1.75x)
nat1080 4444 52.52 -> 74.67 -> 91.39 (1.74x)
nat1080 4444 +alpha 37.91 -> 50.36 -> 56.82 (1.50x)
nat1080 4444xq 49.00 -> 69.97 -> 84.33 (1.72x)
scr1080 proxy 22.59 -> 44.23 -> 100.9 (4.47x)
scr1080 hq 26.21 -> 47.30 -> 64.21 (2.45x)
scr1080 4444xq 74.44 -> 120.2 -> 160.0 (2.15x)
nat4k hq 18.95 -> 26.73 -> 33.87 (1.79x)
noisy1080 hq 15.19 -> 22.44 -> 39.05 (2.57x)
busy1080 hq 18.59 -> 29.93 -> 42.02 (2.26x)
ts1080 hq 109.1 -> 184.1 -> 293.4 (2.69x)
nat: camera footage (1920x1080; for 4k 2560x1600 footage scaled to
3840x2160), scr: synthetic desktop with text-like patterns, noisy: nat
with noise added, busy: Mandelbrot zoom, ts: testsrc2. yuv422p10 input,
yuv444p10 for the 4444 profiles, yuva444p10 for +alpha. 12 frames with
one thread (6 at 4k), 120 (48) with the default 10 threads; best of 3
interleaved runs.
Assisted-by: Claude Opus 5.5
Signed-off-by: Graham Vosloo <[email protected]>
---
libavcodec/proresenc_kostya.c | 102 ++++++++++++++++++++++++----------
1 file changed, 74 insertions(+), 28 deletions(-)
diff --git a/libavcodec/proresenc_kostya.c b/libavcodec/proresenc_kostya.c
index c1abc714c2..2e457dd76d 100644
--- a/libavcodec/proresenc_kostya.c
+++ b/libavcodec/proresenc_kostya.c
@@ -49,6 +49,13 @@ struct TrellisNode {
int score;
};
+/* a coefficient that the smallest quantiser of the profile does not zero */
+typedef struct ProresCoeff {
+ uint16_t pos; ///< position in coding order
+ uint16_t value; ///< absolute value
+ uint8_t idx; ///< index into the quantisation matrix
+} ProresCoeff;
+
typedef struct ProresThreadData {
DECLARE_ALIGNED(16, int16_t, blocks)[MAX_PLANES][64 * 4 *
MAX_MBS_PER_SLICE];
DECLARE_ALIGNED(16, uint16_t, emu_buf)[16 * 16];
@@ -56,6 +63,9 @@ typedef struct ProresThreadData {
int16_t custom_chroma_q[64];
uint32_t custom_recip[64];
uint32_t custom_chroma_recip[64];
+ ProresCoeff coeffs[MAX_PLANES - 1][63 * 4 * MAX_MBS_PER_SLICE];
+ int nb_coeffs[MAX_PLANES - 1];
+ int coeffs_sum[MAX_PLANES - 1];
struct TrellisNode *nodes;
} ProresThreadData;
@@ -488,41 +498,71 @@ static int estimate_dcs(int *error, int16_t *blocks, int
blocks_per_slice,
return bits;
}
-static int estimate_acs(int *error, int16_t *blocks, int blocks_per_slice,
- const uint8_t *scan, const int16_t *qmat,
- const uint32_t *qrecip)
+/**
+ * Collect the AC coefficients that are not quantised to zero by qmat,
+ * in coding order.
+ *
+ * @return sum of the absolute values of all AC coefficients
+ */
+static int collect_acs(ProresCoeff *coeffs, int *nb_coeffs,
+ const int16_t *blocks, int blocks_per_slice,
+ const uint8_t *scan, const int16_t *qmat)
{
int idx, i;
- int prev_run = 4;
- int prev_level = 2;
- int run;
- int max_coeffs, abs_level;
- int bits = 0;
-
- max_coeffs = blocks_per_slice << 6;
- run = 0;
+ int max_coeffs = blocks_per_slice << 6;
+ int pos = 0, nb = 0, sum = 0;
for (i = 1; i < 64; i++) {
const unsigned quant = qmat[scan[i]];
- const unsigned recip = qrecip[scan[i]];
- for (idx = scan[i]; idx < max_coeffs; idx += 64) {
- const unsigned coeff = FFABS(blocks[idx]);
+ for (idx = scan[i]; idx < max_coeffs; idx += 64, pos++) {
+ const unsigned value = FFABS(blocks[idx]);
- if (coeff >= quant) {
- abs_level = (uint64_t)coeff * recip >> 32;
- *error += coeff - abs_level * quant;
- bits += estimate_run(prev_run, run);
- bits += estimate_level(prev_level, abs_level - 1) + 1;
- prev_run = FFMIN(run, 15);
- prev_level = FFMIN(abs_level, 9);
- run = 0;
- } else {
- *error += coeff;
- run++;
- }
+ sum += value;
+ if (value >= quant)
+ coeffs[nb++] = (ProresCoeff){ pos, value, scan[i] };
}
}
+ *nb_coeffs = nb;
+
+ return sum;
+}
+
+/**
+ * Estimate the AC coefficients from the list made by collect_acs().
+ *
+ * Coefficients that are quantised to zero are removed from the list, which
+ * is why this must be called with quantisers in non-decreasing order.
+ */
+static int estimate_acs(int *error, ProresCoeff *coeffs, int *nb_coeffs,
+ int coeffs_sum, const int16_t *qmat,
+ const uint32_t *qrecip)
+{
+ int i, j = 0;
+ int prev_run = 4;
+ int prev_level = 2;
+ int prev_pos = -1;
+ int run, abs_level;
+ int bits = 0;
+
+ for (i = 0; i < *nb_coeffs; i++) {
+ const unsigned value = coeffs[i].value;
+ const unsigned quant = qmat[coeffs[i].idx];
+
+ if (value >= quant) {
+ abs_level = (uint64_t)value * qrecip[coeffs[i].idx] >> 32;
+ coeffs_sum -= abs_level * quant;
+ run = coeffs[i].pos - prev_pos - 1;
+ bits += estimate_run(prev_run, run);
+ bits += estimate_level(prev_level, abs_level - 1) + 1;
+ prev_run = FFMIN(run, 15);
+ prev_level = FFMIN(abs_level, 9);
+ prev_pos = coeffs[i].pos;
+ coeffs[j++] = coeffs[i];
+ }
+ }
+ *nb_coeffs = j;
+ *error += coeffs_sum;
return bits;
}
@@ -540,8 +580,8 @@ static int estimate_slice_plane(ProresContext *ctx, int
*error, int plane,
blocks_per_slice = mbs_per_slice * blocks_per_mb;
bits = estimate_dcs(error, td->blocks[plane], blocks_per_slice, qmat[0],
qrecip[0]);
- bits += estimate_acs(error, td->blocks[plane], blocks_per_slice,
ctx->scantable,
- qmat, qrecip);
+ bits += estimate_acs(error, td->coeffs[plane], &td->nb_coeffs[plane],
+ td->coeffs_sum[plane], qmat, qrecip);
return FFALIGN(bits, 8);
}
@@ -658,6 +698,12 @@ static int find_slice_quant(AVCodecContext *avctx,
pwidth, picture_height,
td->blocks[i], td->emu_buf,
mbs_per_slice, num_cblocks[i], is_chroma[i]);
+ td->coeffs_sum[i] = collect_acs(td->coeffs[i], &td->nb_coeffs[i],
+ td->blocks[i],
+ mbs_per_slice * num_cblocks[i],
+ ctx->scantable,
+ i ? ctx->quants_chroma[min_quant]
+ : ctx->quants[min_quant]);
} else {
get_alpha_data(ctx, src, linesize[i], xp, yp,
pwidth, picture_height,
--
2.52.0
>From eeea0d6ca51f5ff0140160377762e95ed57b5e3f Mon Sep 17 00:00:00 2001
From: Graham Vosloo <[email protected]>
Date: Mon, 12 Oct 2026 09:50:45 -0400
Subject: [PATCH 4/5] avcodec/proresenc_kostya: stop estimating a quantiser
that is over budget
When a slice does not fit with the largest quantiser of the profile,
find_slice_quant() tries the following quantisers one by one and takes
the first that fits. Of all the earlier ones it only needs to know that
they do not fit, and the size estimate only grows while it is computed,
so stop as soon as it exceeds the budget. Only quantiser 127 is always
estimated in full, because it is used with its size and error when
nothing fits.
A pass that stops has removed the coefficients it zeroed from the first
part of the list only; the rest is kept as it is, which later passes
handle because they test every coefficient anyway.
The output is unchanged (prores FATE tests, and the same md5 for the 80
encodes of the first commit of this series). Counted over 87 encodes of
the clips below, 38% of the slices did not fit with the largest
quantiser of their profile and needed 33 such passes on average, all
but the last of which now stop early. Slices that fit gain nothing; the
camera footage is about 1% slower than with the previous commit, which
is the price of one more comparison per coded coefficient.
Frames per second on an Apple M1 Max, master -> previous commit -> this
commit:
one thread
nat1080 proxy 14.81 -> 43.64 -> 43.80 (2.96x)
nat1080 lt 9.97 -> 23.44 -> 23.12 (2.32x)
nat1080 standard 10.76 -> 20.30 -> 20.07 (1.87x)
nat1080 hq 8.75 -> 15.52 -> 15.35 (1.75x)
nat1080 4444 6.68 -> 11.85 -> 11.71 (1.75x)
nat1080 4444 +alpha 4.83 -> 7.25 -> 7.16 (1.48x)
nat1080 4444xq 6.30 -> 10.70 -> 10.58 (1.68x)
scr1080 proxy 2.94 -> 13.23 -> 18.60 (6.33x)
scr1080 hq 3.41 -> 8.36 -> 10.06 (2.95x)
scr1080 4444xq 9.89 -> 21.20 -> 21.28 (2.15x)
nat4k hq 2.46 -> 4.46 -> 4.41 (1.79x)
noisy1080 hq 1.92 -> 5.01 -> 5.78 (3.01x)
busy1080 hq 2.54 -> 5.69 -> 7.68 (3.02x)
ts1080 hq 14.55 -> 39.34 -> 41.24 (2.83x)
default threads
nat1080 proxy 112.6 -> 320.9 -> 324.3 (2.88x)
nat1080 lt 76.63 -> 182.4 -> 179.4 (2.34x)
nat1080 standard 83.04 -> 159.4 -> 156.4 (1.88x)
nat1080 hq 68.65 -> 120.2 -> 121.3 (1.77x)
nat1080 4444 52.52 -> 91.39 -> 91.74 (1.75x)
nat1080 4444 +alpha 37.91 -> 56.82 -> 56.55 (1.49x)
nat1080 4444xq 49.00 -> 84.33 -> 83.57 (1.71x)
scr1080 proxy 22.59 -> 100.9 -> 140.2 (6.21x)
scr1080 hq 26.21 -> 64.21 -> 77.12 (2.94x)
scr1080 4444xq 74.44 -> 160.0 -> 161.1 (2.16x)
nat4k hq 18.95 -> 33.87 -> 33.68 (1.78x)
noisy1080 hq 15.19 -> 39.05 -> 44.98 (2.96x)
busy1080 hq 18.59 -> 42.02 -> 56.66 (3.05x)
ts1080 hq 109.1 -> 293.4 -> 303.8 (2.78x)
nat: camera footage (1920x1080; for 4k 2560x1600 footage scaled to
3840x2160), scr: synthetic desktop with text-like patterns, noisy: nat
with noise added, busy: Mandelbrot zoom, ts: testsrc2. yuv422p10 input,
yuv444p10 for the 4444 profiles, yuva444p10 for +alpha. 12 frames with
one thread (6 at 4k), 120 (48) with the default 10 threads; best of 3
interleaved runs.
Assisted-by: Claude Opus 5.5
Signed-off-by: Graham Vosloo <[email protected]>
---
libavcodec/proresenc_kostya.c | 33 +++++++++++++++++++++++++--------
1 file changed, 25 insertions(+), 8 deletions(-)
diff --git a/libavcodec/proresenc_kostya.c b/libavcodec/proresenc_kostya.c
index 2e457dd76d..d07411bf61 100644
--- a/libavcodec/proresenc_kostya.c
+++ b/libavcodec/proresenc_kostya.c
@@ -533,10 +533,11 @@ static int collect_acs(ProresCoeff *coeffs, int
*nb_coeffs,
*
* Coefficients that are quantised to zero are removed from the list, which
* is why this must be called with quantisers in non-decreasing order.
+ * The estimate stops once it exceeds max_bits; the error is not valid then.
*/
static int estimate_acs(int *error, ProresCoeff *coeffs, int *nb_coeffs,
int coeffs_sum, const int16_t *qmat,
- const uint32_t *qrecip)
+ const uint32_t *qrecip, int max_bits)
{
int i, j = 0;
int prev_run = 4;
@@ -559,9 +560,16 @@ static int estimate_acs(int *error, ProresCoeff *coeffs,
int *nb_coeffs,
prev_level = FFMIN(abs_level, 9);
prev_pos = coeffs[i].pos;
coeffs[j++] = coeffs[i];
+ if (bits > max_bits) {
+ i++;
+ break;
+ }
}
}
- *nb_coeffs = j;
+ /* keep the coefficients that were not looked at */
+ if (i < *nb_coeffs)
+ memmove(coeffs + j, coeffs + i, (*nb_coeffs - i) * sizeof(*coeffs));
+ *nb_coeffs += j - i;
*error += coeffs_sum;
return bits;
@@ -572,7 +580,7 @@ static int estimate_slice_plane(ProresContext *ctx, int
*error, int plane,
int mbs_per_slice,
int blocks_per_mb,
const int16_t *qmat, const uint32_t *qrecip,
- ProresThreadData *td)
+ int max_bits, ProresThreadData *td)
{
int blocks_per_slice;
int bits;
@@ -581,7 +589,7 @@ static int estimate_slice_plane(ProresContext *ctx, int
*error, int plane,
bits = estimate_dcs(error, td->blocks[plane], blocks_per_slice, qmat[0],
qrecip[0]);
bits += estimate_acs(error, td->coeffs[plane], &td->nb_coeffs[plane],
- td->coeffs_sum[plane], qmat, qrecip);
+ td->coeffs_sum[plane], qmat, qrecip, max_bits - bits);
return FFALIGN(bits, 8);
}
@@ -728,14 +736,15 @@ static int find_slice_quant(AVCodecContext *avctx,
mbs_per_slice,
num_cblocks[0],
ctx->quants[q], ctx->quants_recip[q],
- td); /* estimate luma plane */
+ INT_MAX, td); /* estimate luma plane */
for (i = 1; i < ctx->num_planes - !!ctx->alpha_bits; i++) { /*
estimate chroma plane */
bits += estimate_slice_plane(ctx, &error, i,
src, linesize[i],
mbs_per_slice,
num_cblocks[i],
ctx->quants_chroma[q],
- ctx->quants_chroma_recip[q], td);
+ ctx->quants_chroma_recip[q],
+ INT_MAX, td);
}
if (bits > 65000 * 8)
error = SCORE_LIMIT;
@@ -749,6 +758,10 @@ static int find_slice_quant(AVCodecContext *avctx,
overquant = max_quant;
} else {
for (q = max_quant + 1; q < 128; q++) {
+ /* the last quantiser is used even if it does not fit */
+ const int max_bits = q < 127 ? ctx->bits_per_mb * mbs_per_slice
+ : INT_MAX;
+
bits = alpha_bits;
error = 0;
if (q < MAX_STORED_Q) {
@@ -772,13 +785,17 @@ static int find_slice_quant(AVCodecContext *avctx,
src, linesize[0],
mbs_per_slice,
num_cblocks[0],
- qmat, qrecip, td);/* estimate luma
plane */
+ qmat, qrecip, max_bits - bits,
+ td);/* estimate luma plane */
for (i = 1; i < ctx->num_planes - !!ctx->alpha_bits; i++) { /*
estimate chroma plane */
+ if (bits > max_bits)
+ break;
bits += estimate_slice_plane(ctx, &error, i,
src, linesize[i],
mbs_per_slice,
num_cblocks[i],
- qmat_chroma, qrecip_chroma, td);
+ qmat_chroma, qrecip_chroma,
+ max_bits - bits, td);
}
if (bits <= ctx->bits_per_mb * mbs_per_slice)
break;
--
2.52.0
>From 8043dde5b9330816d1894b501fccd0ef979d820c Mon Sep 17 00:00:00 2001
From: Graham Vosloo <[email protected]>
Date: Mon, 12 Oct 2026 09:57:08 -0400
Subject: [PATCH 5/5] avcodec/proresenc_kostya: skip quantisers that cannot fit
The scan for the first quantiser above the profile's range with which a
slice fits still starts an estimate for every candidate. Most candidates
can be ruled out without one: each DC coefficient takes at least one
bit, and each coded AC coefficient at least one bit for its run, one for
its sign and 1, 2, 3, 3, 4, 4, 5 bits for a level of 1, 2, ... 7 or
more in whichever codebook is selected (the shortest code of each level
over the ten level codebooks; no run code is shorter than one bit).
A coefficient has a level of at least l with quantiser q if its level
with quantiser 1 is at least l * q, so a histogram of the levels at
quantiser 1 gives this lower bound for every q with five look-ups. If
the bound exceeds the budget, the estimate would too, and the linear
scan would have moved on to the next quantiser. The quantiser that is
found, and with it the output, is therefore the same as before (prores
FATE tests, and the same md5 for the 80 encodes of the first commit of
this series).
The bound is about half of the real size, so the histogram is only made
for slices that are more than twice over budget with the last quantiser
of the profile; for the others it cost more than it saved (about 2% for
proxy on camera footage). To have the reciprocals for quantiser 1, fill
the tables from 1 instead of from the first quantiser of the profile.
A search that assumes the size to fall monotonically with the quantiser
(doubling the step, then bisecting) was measured as an alternative. It
is not equivalent: over 87 encodes of the clips below, 223212 slices
were over budget, the size rose somewhere between two consecutive
quantisers for 139562 of them, and the search chose another quantiser
than the linear scan for 138 slices, changing the output of four of 48
1080p encodes. On top of this commit it was 1.25 to 1.3 times faster on
the screen-like and Mandelbrot clips, 13% slower on the noisy one and
no faster on camera footage, so it is not included.
Frames per second on an Apple M1 Max, master -> previous commit -> this
commit:
one thread
nat1080 proxy 14.81 -> 43.80 -> 43.96 (2.97x)
nat1080 lt 9.97 -> 23.12 -> 23.21 (2.33x)
nat1080 standard 10.76 -> 20.07 -> 20.10 (1.87x)
nat1080 hq 8.75 -> 15.35 -> 15.33 (1.75x)
nat1080 4444 6.68 -> 11.71 -> 11.74 (1.76x)
nat1080 4444 +alpha 4.83 -> 7.16 -> 7.17 (1.48x)
nat1080 4444xq 6.30 -> 10.58 -> 10.58 (1.68x)
scr1080 proxy 2.94 -> 18.60 -> 23.90 (8.13x)
scr1080 hq 3.41 -> 10.06 -> 14.00 (4.11x)
scr1080 4444xq 9.89 -> 21.28 -> 21.35 (2.16x)
nat4k hq 2.46 -> 4.41 -> 4.42 (1.80x)
noisy1080 hq 1.92 -> 5.78 -> 7.06 (3.68x)
busy1080 hq 2.54 -> 7.68 -> 11.06 (4.35x)
ts1080 hq 14.55 -> 41.24 -> 42.70 (2.93x)
default threads
nat1080 proxy 112.6 -> 324.3 -> 324.3 (2.88x)
nat1080 lt 76.63 -> 179.4 -> 178.6 (2.33x)
nat1080 standard 83.04 -> 156.4 -> 155.4 (1.87x)
nat1080 hq 68.65 -> 121.3 -> 121.0 (1.76x)
nat1080 4444 52.52 -> 91.74 -> 92.31 (1.76x)
nat1080 4444 +alpha 37.91 -> 56.55 -> 56.44 (1.49x)
nat1080 4444xq 49.00 -> 83.57 -> 83.39 (1.70x)
scr1080 proxy 22.59 -> 140.2 -> 178.0 (7.88x)
scr1080 hq 26.21 -> 77.12 -> 106.6 (4.07x)
scr1080 4444xq 74.44 -> 161.1 -> 160.6 (2.16x)
nat4k hq 18.95 -> 33.68 -> 33.50 (1.77x)
noisy1080 hq 15.19 -> 44.98 -> 54.64 (3.60x)
busy1080 hq 18.59 -> 56.66 -> 83.51 (4.49x)
ts1080 hq 109.1 -> 303.8 -> 321.7 (2.95x)
nat: camera footage (1920x1080; for 4k 2560x1600 footage scaled to
3840x2160), scr: synthetic desktop with text-like patterns, noisy: nat
with noise added, busy: Mandelbrot zoom, ts: testsrc2. yuv422p10 input,
yuv444p10 for the 4444 profiles, yuva444p10 for +alpha. 12 frames with
one thread (6 at 4k), 120 (48) with the default 10 threads; best of 3
interleaved runs.
Assisted-by: Claude Opus 5.5
Signed-off-by: Graham Vosloo <[email protected]>
---
libavcodec/proresenc_kostya.c | 45 ++++++++++++++++++++++++++++
libavcodec/proresenc_kostya_common.c | 5 ++--
2 files changed, 47 insertions(+), 3 deletions(-)
diff --git a/libavcodec/proresenc_kostya.c b/libavcodec/proresenc_kostya.c
index d07411bf61..8de8e6d27e 100644
--- a/libavcodec/proresenc_kostya.c
+++ b/libavcodec/proresenc_kostya.c
@@ -42,6 +42,8 @@
#define RUN_BITS_TAB_SIZE 64
#define LEVEL_BITS_TAB_SIZE 256
+#define LEVEL_HIST_SIZE 1024 // more than 7 times the largest quantiser
+
struct TrellisNode {
int prev_node;
int quant;
@@ -66,6 +68,7 @@ typedef struct ProresThreadData {
ProresCoeff coeffs[MAX_PLANES - 1][63 * 4 * MAX_MBS_PER_SLICE];
int nb_coeffs[MAX_PLANES - 1];
int coeffs_sum[MAX_PLANES - 1];
+ int level_hist[LEVEL_HIST_SIZE];
struct TrellisNode *nodes;
} ProresThreadData;
@@ -575,6 +578,21 @@ static int estimate_acs(int *error, ProresCoeff *coeffs,
int *nb_coeffs,
return bits;
}
+/**
+ * Add the levels that the listed coefficients have at quantiser 1 to hist.
+ * Levels below min_level are left out.
+ */
+static void hist_levels(int *hist, const ProresCoeff *coeffs, int nb_coeffs,
+ const uint32_t *qrecip, int min_level)
+{
+ for (int i = 0; i < nb_coeffs; i++) {
+ unsigned level = (uint64_t)coeffs[i].value * qrecip[coeffs[i].idx] >>
32;
+
+ if (level >= min_level)
+ hist[FFMIN(level, LEVEL_HIST_SIZE - 1)]++;
+ }
+}
+
static int estimate_slice_plane(ProresContext *ctx, int *error, int plane,
const uint16_t *src, ptrdiff_t linesize,
int mbs_per_slice,
@@ -666,6 +684,8 @@ static int find_slice_quant(AVCodecContext *avctx,
int mbs, prev, cur, new_score;
int slice_bits[TRELLIS_WIDTH], slice_score[TRELLIS_WIDTH];
int overquant;
+ int *hist = td->level_hist;
+ int num_blocks = 0, use_hist = 0;
uint16_t *qmat;
uint16_t *qmat_chroma;
const uint32_t *qrecip;
@@ -757,11 +777,36 @@ static int find_slice_quant(AVCodecContext *avctx,
slice_score[max_quant + 1] = slice_score[max_quant] + 1;
overquant = max_quant;
} else {
+ /* The bound below is too weak to help a slice that almost fits. */
+ if (slice_bits[max_quant] > 2 * ctx->bits_per_mb * mbs_per_slice) {
+ /* hist[l]: number of coefficients with a level of at least l at
+ * quantiser 1; hist[l * q] of them have at least l at quantiser q
*/
+ memset(hist, 0, sizeof(td->level_hist));
+ for (i = 0; i < ctx->num_planes - !!ctx->alpha_bits; i++) {
+ hist_levels(hist, td->coeffs[i], td->nb_coeffs[i],
+ i ? ctx->quants_chroma_recip[1] :
ctx->quants_recip[1],
+ max_quant + 1);
+ num_blocks += mbs_per_slice * num_cblocks[i];
+ }
+ for (i = LEVEL_HIST_SIZE - 2; i > max_quant; i--)
+ hist[i] += hist[i + 1];
+ use_hist = 1;
+ }
+
for (q = max_quant + 1; q < 128; q++) {
/* the last quantiser is used even if it does not fit */
const int max_bits = q < 127 ? ctx->bits_per_mb * mbs_per_slice
: INT_MAX;
+ /* Skip quantisers that cannot fit: every DC coefficient takes
+ * at least 1 bit and every coded AC coefficient at least 1 bit
+ * for the run, 1 for the sign and 1, 2, 3, 3, 4, 4, 5 for a
+ * level of 1 to 7 or more, whichever codebook is in use. */
+ if (use_hist &&
+ alpha_bits + num_blocks + 3 * hist[q] + hist[2 * q] +
+ hist[3 * q] + hist[5 * q] + hist[7 * q] > max_bits)
+ continue;
+
bits = alpha_bits;
error = 0;
if (q < MAX_STORED_Q) {
diff --git a/libavcodec/proresenc_kostya_common.c
b/libavcodec/proresenc_kostya_common.c
index a833903866..245d02715f 100644
--- a/libavcodec/proresenc_kostya_common.c
+++ b/libavcodec/proresenc_kostya_common.c
@@ -168,7 +168,7 @@ static const prores_profile prores_profile_info[6] = {
av_cold int ff_prores_kostya_encode_init(AVCodecContext *avctx, ProresContext
*ctx,
enum AVPixelFormat pix_fmt)
{
- int mps, i, j, min_quant;
+ int mps, i, j;
int interlaced = !!(avctx->flags & AV_CODEC_FLAG_INTERLACED_DCT);
avctx->bits_per_raw_sample = 10;
@@ -255,8 +255,7 @@ av_cold int ff_prores_kostya_encode_init(AVCodecContext
*avctx, ProresContext *c
return AVERROR_INVALIDDATA;
}
- min_quant = ctx->profile_info->min_quant;
- for (i = min_quant; i < MAX_STORED_Q; i++) {
+ for (i = 1; i < MAX_STORED_Q; i++) {
for (j = 0; j < 64; j++) {
ctx->quants[i][j] = ctx->quant_mat[j] * i;
ctx->quants_chroma[i][j] = ctx->quant_chroma_mat[j] * i;
--
2.52.0
_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]
