PR #25869 opened by Wesley Maa (WT-MM)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25869
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25869.patch
`ff_vc1_inv_trans_4x4_neon` and `ff_vc1_inv_trans_8x4_neon` overflow a 16-bit
lane in their second (4-point) pass, so bright pixels of in-range VC-1
residuals come out black. This patch fixes the overflow and extends checkasm
to cover the range that hid it.
## Explanation
The 4-point pass computes
t1 = 17 * src[0] + 17 * src[16]
t2 = 17 * src[0] - 17 * src[16]
in 16-bit lanes (`mul`, `mul`, `add`, `sub`) and then folds them in with
`ssra #1`. The add/mul wrap is harmless as long as the final result fits, but
the arithmetic shift is not: once `|src[0] ± src[16]| > 1927`, `17 * (src[0] ±
src[16])` no longer fits in an `int16`, the shift brings in the wrong sign bit
and the output flips sign. For example `src[0] = 1823, src[16] = -110` gives
`t2 = 32861`: the C code outputs `256`, clipped to `255`, while NEON wraps `t2`
to `-32675` and the residual saturates to `0`.
These values are well within the VC-1 limits for intermediate (E, ±4096) and
final (R, ±512) values and occur in real streams: on a real-world WMV3 Main
profile file, bright 4x4 and 8x4 inter blocks over a dark prediction decode as
isolated black pixels, and the error propagates through the following P frames
until the next keyframe.
The fix computes `a = src[0] ± src[16]` first, which cannot overflow, and
derives `(17 * a) >> 1` as `8 * a + (a >> 1)` with `shl #3` + `ssra #1`. This is
exact because `16 * a` is even, and the rest of the stage fits in 16 bits for
the whole spec range. The instruction count is unchanged (the two `neg` go
away) and `checkasm --bench` shows no difference.
The 4-point first passes (rows of 4x4 and 4x8) have the same shape, but an
overflow there implies an intermediate value outside the spec range, so they
are left as is.
## Reproducing
checkasm's random coefficients are attenuated by 4 and never reach this
range. The second patch also runs each adding inverse transform on a large
in-range DC block of either sign; with only that patch applied:
$ make checkasm && tests/checkasm/checkasm --test=vc1dsp
FAILURE: vc1dsp.vc1_inv_trans_8x4_neon (tests/checkasm/vc1dsp.c:338)
FAILURE: vc1dsp.vc1_inv_trans_4x4_neon (tests/checkasm/vc1dsp.c:338)
- vc1dsp.inv_trans [FAILED]
checkasm: 2 of 27 tests failed
With both patches all 27 vc1dsp tests pass. The WMV3 file mentioned above
decodes bit-exact against `-cpuflags 0` for all 150,256 frames (10 frames
differed before).
Tested on an Apple M3 Max with checkasm, `fate-vc1` and the wmv3 FATE tests,
also with AddressSanitizer. The 32-bit `arm/vc1dsp_neon.S` has the same
structure but was not tested.
## References
- SMPTE ST 421:2013 (inverse transform and its range limits):
https://pub.smpte.org/doc/st421/20131008-pub/
From 9bb11dcd6e949e20cea0dadaf045d248228ade44 Mon Sep 17 00:00:00 2001
From: Wesley Maa <[email protected]>
Date: Sun, 8 Nov 2026 11:15:57 -0500
Subject: [PATCH 1/2] aarch64/vc1dsp: fix 16-bit overflow in the 4-point stage
of inv_trans_4x4/8x4
The vertical (second, 4-point) pass of ff_vc1_inv_trans_4x4_neon and
ff_vc1_inv_trans_8x4_neon formed
t1 = 17 * src[0] + 17 * src[16]
t2 = 17 * src[0] - 17 * src[16]
in 16-bit lanes (mul, mul, add, sub) and then folded them into
t3/2 and t4/2 with "ssra #1". Additions and multiplications wrap
harmlessly as long as the final result fits, but the arithmetic shift
does not: once |src[0] +- src[16]| exceeds 1927, 17 * (src[0] +- src[16])
no longer fits in int16, the shift brings in the wrong sign bit and the
output flips sign. For example a column of first-pass values
src[0] = 1823, src[16] = -110 gives t2 = 32861; the C code computes
(t2 + t4 + 64) >> 7 = 256 and clips it to 255, while the NEON code
wraps t2 to -32675 and the residual saturates to 0.
Such values are well within the ranges the VC-1 spec allows for the
intermediate (E, up to +-4096) and final (R, up to +-512) values, and
they do occur in real streams: decoding a real-world WMV3 Main profile
file, bright 4x4 and 8x4 inter blocks (residual around +256 over a
dark prediction) come out as isolated black pixels, and the error then
propagates through the following P frames until the next key frame.
Compute a = src[0] +- src[16] first, which cannot overflow, and derive
t >> 1 = (17 * a) >> 1 = 8 * a + (a >> 1) with shl #3 + ssra #1. This is
exact because 16 * a is even. Everything after that is modular
add/sub, and the value that is finally shifted, (t1 +- t3) >> 1, is
about 64 * R and fits in 16 bits for every R in [-512, 511], so the
stage now matches the C code for the whole spec range. The instruction
count is unchanged (the two neg are no longer needed) and checkasm
--bench shows no difference.
The 4-point first passes (rows of 4x4 and 4x8) have the same shape, but
there an overflowing t1 implies max(t1 + t3, t1 - t3) >= 32768, i.e. an
intermediate value E >= 4096 outside the spec range, so they are left
untouched.
Signed-off-by: Wesley Maa <[email protected]>
---
libavcodec/aarch64/vc1dsp_neon.S | 40 ++++++++++++++++----------------
1 file changed, 20 insertions(+), 20 deletions(-)
diff --git a/libavcodec/aarch64/vc1dsp_neon.S b/libavcodec/aarch64/vc1dsp_neon.S
index d8e77880266..a93df5dc669 100644
--- a/libavcodec/aarch64/vc1dsp_neon.S
+++ b/libavcodec/aarch64/vc1dsp_neon.S
@@ -278,18 +278,18 @@ function ff_vc1_inv_trans_8x4_neon, export=1
mul v3.8h, v4.8h, v0.h[5] //
22/2 * src[24]
trn1 v2.4s, v6.4s, v2.4s
mul v4.8h, v4.8h, v0.h[4] //
10/2 * src[24]
- mul v6.8h, v7.8h, v0.h[6] // 17 * src[0]
- mul v1.8h, v1.8h, v0.h[6] //
17 * src[16]
+ add v6.8h, v7.8h, v1.8h // src[0]
+ src[16]
+ sub v1.8h, v7.8h, v1.8h // src[0]
- src[16]
mls v3.8h, v2.8h, v0.h[4] // t4/2 = -
10/2 * src[8] + 22/2 * src[24]
mla v4.8h, v2.8h, v0.h[5] // t3/2 =
22/2 * src[8] + 10/2 * src[24]
- add v0.8h, v6.8h, v1.8h // t1 = 17 * src[0]
+ 17 * src[16]
- sub v1.8h, v6.8h, v1.8h // t2 = 17 * src[0]
- 17 * src[16]
- neg v2.8h, v3.8h // -t4/2
- neg v6.8h, v4.8h // -t3/2
- ssra v4.8h, v0.8h, #1 // (t1 + t3) >> 1
- ssra v2.8h, v1.8h, #1 // (t2 - t4) >> 1
- ssra v3.8h, v1.8h, #1 // (t2 + t4) >> 1
- ssra v6.8h, v0.8h, #1 // (t1 - t3) >> 1
+ shl v0.8h, v6.8h, #3 // 8 * (src[0]
+ src[16])
+ shl v7.8h, v1.8h, #3 // 8 * (src[0]
- src[16])
+ ssra v0.8h, v6.8h, #1 // t1 >> 1
+ ssra v7.8h, v1.8h, #1 // t2 >> 1
+ sub v6.8h, v0.8h, v4.8h // (t1 - t3) >> 1
+ add v4.8h, v4.8h, v0.8h // (t1 + t3) >> 1
+ sub v2.8h, v7.8h, v3.8h // (t2 - t4) >> 1
+ add v3.8h, v3.8h, v7.8h // (t2 + t4) >> 1
srshr v0.8h, v4.8h, #6 // (t1 + t3 + 64) >> 7
srshr v1.8h, v2.8h, #6 // (t2 - t4 + 64) >> 7
srshr v2.8h, v3.8h, #6 // (t2 + t4 + 64) >> 7
@@ -505,18 +505,18 @@ function ff_vc1_inv_trans_4x4_neon, export=1
trn2 v1.2s, v1.2s, v2.2s // 20 21 22 23
mul v2.4h, v4.4h, v0.h[1] //
22/2 * src[24]
mul v4.4h, v4.4h, v0.h[0] //
10/2 * src[24]
- mul v7.4h, v16.4h, v0.h[2] // 17 * src[0]
- mul v1.4h, v1.4h, v0.h[2] //
17 * src[16]
+ add v7.4h, v16.4h, v1.4h // src[0]
+ src[16]
+ sub v1.4h, v16.4h, v1.4h // src[0]
- src[16]
mls v2.4h, v3.4h, v0.h[0] // t4/2 = -
10/2 * src[8] + 22/2 * src[24]
mla v4.4h, v3.4h, v0.h[1] // t3/2 =
22/2 * src[8] + 10/2 * src[24]
- add v0.4h, v7.4h, v1.4h // t1 = 17 * src[0]
+ 17 * src[16]
- sub v1.4h, v7.4h, v1.4h // t2 = 17 * src[0]
- 17 * src[16]
- neg v3.4h, v2.4h // -t4/2
- neg v7.4h, v4.4h // -t3/2
- ssra v4.4h, v0.4h, #1 // (t1 + t3) >> 1
- ssra v3.4h, v1.4h, #1 // (t2 - t4) >> 1
- ssra v2.4h, v1.4h, #1 // (t2 + t4) >> 1
- ssra v7.4h, v0.4h, #1 // (t1 - t3) >> 1
+ shl v0.4h, v7.4h, #3 // 8 * (src[0]
+ src[16])
+ shl v16.4h, v1.4h, #3 // 8 * (src[0]
- src[16])
+ ssra v0.4h, v7.4h, #1 // t1 >> 1
+ ssra v16.4h, v1.4h, #1 // t2 >> 1
+ sub v7.4h, v0.4h, v4.4h // (t1 - t3) >> 1
+ add v4.4h, v4.4h, v0.4h // (t1 + t3) >> 1
+ sub v3.4h, v16.4h, v2.4h // (t2 - t4) >> 1
+ add v2.4h, v2.4h, v16.4h // (t2 + t4) >> 1
trn1 v0.2d, v4.2d, v3.2d
trn1 v1.2d, v2.2d, v7.2d
srshr v0.8h, v0.8h, #6 // (t1 + t3 + 64) >> 7, (t2 -
t4 + 64) >> 7
--
2.54.0
From ea539f2bdf7028587a92817d9338557b0a29d51e Mon Sep 17 00:00:00 2001
From: Wesley Maa <[email protected]>
Date: Sun, 8 Nov 2026 11:16:07 -0500
Subject: [PATCH 2/2] checkasm/vc1dsp: test inverse transforms with a large
in-range DC
The random coefficients used for the adding inverse transforms are
attenuated by a factor of 4 so that implementations limited to 16-bit
intermediates still pass, which means the upper part of the range the
VC-1 spec allows is never exercised. This hid a sign flip in the
aarch64 vc1_inv_trans_4x4 and vc1_inv_trans_8x4 functions, where
17 * (src[0] +- src[16]) overflowed 16 bits for valid input.
In addition to the random input, run each function on a DC-only block
of either sign whose output lies just outside the 8-bit range (about
+-280). The DC is chosen per transform size so that all intermediate
and final values stay within the spec limits.
Without the preceding aarch64 fix this reports
vc1dsp.vc1_inv_trans_4x4_neon and vc1dsp.vc1_inv_trans_8x4_neon as
failing.
Signed-off-by: Wesley Maa <[email protected]>
---
tests/checkasm/vc1dsp.c | 15 +++++++++++++++
1 file changed, 15 insertions(+)
diff --git a/tests/checkasm/vc1dsp.c b/tests/checkasm/vc1dsp.c
index 52049a4aa35..746d76957d0 100644
--- a/tests/checkasm/vc1dsp.c
+++ b/tests/checkasm/vc1dsp.c
@@ -323,6 +323,21 @@ static void check_inv_trans_adding(void)
call_new(inv_trans_out1 + 24 + 8, 24, inv_trans_in1);
if (memcmp(inv_trans_out0, inv_trans_out1, 10 * 24))
fail();
+
+ for (int sign = -1; sign <= 1; sign += 2) {
+ int dc = tests[k].width == 8 && tests[k].height == 8 ? 2000 :
+ tests[k].width == 4 && tests[k].height == 4 ? 1000 :
+ 1400;
+ memset(inv_trans_in0, 0, 8 * 8 * sizeof (int16_t));
+ memset(inv_trans_in1, 0, 8 * 8 * sizeof (int16_t));
+ inv_trans_in0[0] = inv_trans_in1[0] = sign * dc;
+ RANDOMIZE_BUFFER8(inv_trans_out, 10 * 24);
+ call_ref(inv_trans_out0 + 24 + 8, 24, inv_trans_in0);
+ call_new(inv_trans_out1 + 24 + 8, 24, inv_trans_in1);
+ if (memcmp(inv_trans_out0, inv_trans_out1, 10 * 24))
+ fail();
+ }
+
bench_new(inv_trans_out1 + 24 + 8, 24, inv_trans_in1);
av_free(coeffs);
}
--
2.54.0
_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]