PR #25869 opened by Wesley Maa (WT-MM)
URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25869
Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/25869.patch

`ff_vc1_inv_trans_4x4_neon` and `ff_vc1_inv_trans_8x4_neon` overflow a 16-bit
lane in their second (4-point) pass, so bright pixels of in-range VC-1
residuals come out black. This patch fixes the overflow and extends checkasm
to cover the range that hid it.

## Explanation

The 4-point pass computes

    t1 = 17 * src[0] + 17 * src[16]
    t2 = 17 * src[0] - 17 * src[16]

in 16-bit lanes (`mul`, `mul`, `add`, `sub`) and then folds them in with
`ssra #1`. The add/mul wrap is harmless as long as the final result fits, but
the arithmetic shift is not: once `|src[0] ± src[16]| > 1927`, `17 * (src[0] ±
src[16])` no longer fits in an `int16`, the shift brings in the wrong sign bit
and the output flips sign. For example `src[0] = 1823, src[16] = -110` gives
`t2 = 32861`: the C code outputs `256`, clipped to `255`, while NEON wraps `t2`
to `-32675` and the residual saturates to `0`.

These values are well within the VC-1 limits for intermediate (E, ±4096) and
final (R, ±512) values and occur in real streams: on a real-world WMV3 Main
profile file, bright 4x4 and 8x4 inter blocks over a dark prediction decode as
isolated black pixels, and the error propagates through the following P frames
until the next keyframe.

The fix computes `a = src[0] ± src[16]` first, which cannot overflow, and
derives `(17 * a) >> 1` as `8 * a + (a >> 1)` with `shl #3` + `ssra #1`. This is
exact because `16 * a` is even, and the rest of the stage fits in 16 bits for
the whole spec range. The instruction count is unchanged (the two `neg` go
away) and `checkasm --bench` shows no difference.

The 4-point first passes (rows of 4x4 and 4x8) have the same shape, but an
overflow there implies an intermediate value outside the spec range, so they
are left as is.

## Reproducing

checkasm's random coefficients are attenuated by 4 and never reach this
range. The second patch also runs each adding inverse transform on a large
in-range DC block of either sign; with only that patch applied:

    $ make checkasm && tests/checkasm/checkasm --test=vc1dsp
    FAILURE: vc1dsp.vc1_inv_trans_8x4_neon (tests/checkasm/vc1dsp.c:338)
    FAILURE: vc1dsp.vc1_inv_trans_4x4_neon (tests/checkasm/vc1dsp.c:338)
     - vc1dsp.inv_trans       [FAILED]
    checkasm: 2 of 27 tests failed

With both patches all 27 vc1dsp tests pass. The WMV3 file mentioned above
decodes bit-exact against `-cpuflags 0` for all 150,256 frames (10 frames
differed before).

Tested on an Apple M3 Max with checkasm, `fate-vc1` and the wmv3 FATE tests,
also with AddressSanitizer. The 32-bit `arm/vc1dsp_neon.S` has the same
structure but was not tested.

## References

- SMPTE ST 421:2013 (inverse transform and its range limits): 
https://pub.smpte.org/doc/st421/20131008-pub/



From 9bb11dcd6e949e20cea0dadaf045d248228ade44 Mon Sep 17 00:00:00 2001
From: Wesley Maa <[email protected]>
Date: Sun, 8 Nov 2026 11:15:57 -0500
Subject: [PATCH 1/2] aarch64/vc1dsp: fix 16-bit overflow in the 4-point stage
 of inv_trans_4x4/8x4

The vertical (second, 4-point) pass of ff_vc1_inv_trans_4x4_neon and
ff_vc1_inv_trans_8x4_neon formed

    t1 = 17 * src[0] + 17 * src[16]
    t2 = 17 * src[0] - 17 * src[16]

in 16-bit lanes (mul, mul, add, sub) and then folded them into
t3/2 and t4/2 with "ssra #1". Additions and multiplications wrap
harmlessly as long as the final result fits, but the arithmetic shift
does not: once |src[0] +- src[16]| exceeds 1927, 17 * (src[0] +- src[16])
no longer fits in int16, the shift brings in the wrong sign bit and the
output flips sign. For example a column of first-pass values
src[0] = 1823, src[16] = -110 gives t2 = 32861; the C code computes
(t2 + t4 + 64) >> 7 = 256 and clips it to 255, while the NEON code
wraps t2 to -32675 and the residual saturates to 0.

Such values are well within the ranges the VC-1 spec allows for the
intermediate (E, up to +-4096) and final (R, up to +-512) values, and
they do occur in real streams: decoding a real-world WMV3 Main profile
file, bright 4x4 and 8x4 inter blocks (residual around +256 over a
dark prediction) come out as isolated black pixels, and the error then
propagates through the following P frames until the next key frame.

Compute a = src[0] +- src[16] first, which cannot overflow, and derive
t >> 1 = (17 * a) >> 1 = 8 * a + (a >> 1) with shl #3 + ssra #1. This is
exact because 16 * a is even. Everything after that is modular
add/sub, and the value that is finally shifted, (t1 +- t3) >> 1, is
about 64 * R and fits in 16 bits for every R in [-512, 511], so the
stage now matches the C code for the whole spec range. The instruction
count is unchanged (the two neg are no longer needed) and checkasm
--bench shows no difference.

The 4-point first passes (rows of 4x4 and 4x8) have the same shape, but
there an overflowing t1 implies max(t1 + t3, t1 - t3) >= 32768, i.e. an
intermediate value E >= 4096 outside the spec range, so they are left
untouched.

Signed-off-by: Wesley Maa <[email protected]>
---
 libavcodec/aarch64/vc1dsp_neon.S | 40 ++++++++++++++++----------------
 1 file changed, 20 insertions(+), 20 deletions(-)

diff --git a/libavcodec/aarch64/vc1dsp_neon.S b/libavcodec/aarch64/vc1dsp_neon.S
index d8e77880266..a93df5dc669 100644
--- a/libavcodec/aarch64/vc1dsp_neon.S
+++ b/libavcodec/aarch64/vc1dsp_neon.S
@@ -278,18 +278,18 @@ function ff_vc1_inv_trans_8x4_neon, export=1
         mul             v3.8h, v4.8h, v0.h[5]   //                             
                              22/2 * src[24]
         trn1            v2.4s, v6.4s, v2.4s
         mul             v4.8h, v4.8h, v0.h[4]   //                             
                              10/2 * src[24]
-        mul             v6.8h, v7.8h, v0.h[6]   //            17 * src[0]
-        mul             v1.8h, v1.8h, v0.h[6]   //                             
               17 * src[16]
+        add             v6.8h, v7.8h, v1.8h     //                 src[0]      
           +        src[16]
+        sub             v1.8h, v7.8h, v1.8h     //                 src[0]      
           -        src[16]
         mls             v3.8h, v2.8h, v0.h[4]   //  t4/2 =                - 
10/2 * src[8]                  + 22/2 * src[24]
         mla             v4.8h, v2.8h, v0.h[5]   //  t3/2 =                  
22/2 * src[8]                  + 10/2 * src[24]
-        add             v0.8h, v6.8h, v1.8h     //   t1  =    17 * src[0]      
           +   17 * src[16]
-        sub             v1.8h, v6.8h, v1.8h     //   t2  =    17 * src[0]      
           -   17 * src[16]
-        neg             v2.8h, v3.8h            // -t4/2
-        neg             v6.8h, v4.8h            // -t3/2
-        ssra            v4.8h, v0.8h, #1        // (t1 + t3) >> 1
-        ssra            v2.8h, v1.8h, #1        // (t2 - t4) >> 1
-        ssra            v3.8h, v1.8h, #1        // (t2 + t4) >> 1
-        ssra            v6.8h, v0.8h, #1        // (t1 - t3) >> 1
+        shl             v0.8h, v6.8h, #3        //            8 * (src[0]      
           +        src[16])
+        shl             v7.8h, v1.8h, #3        //            8 * (src[0]      
           -        src[16])
+        ssra            v0.8h, v6.8h, #1        //  t1 >> 1
+        ssra            v7.8h, v1.8h, #1        //  t2 >> 1
+        sub             v6.8h, v0.8h, v4.8h     // (t1 - t3) >> 1
+        add             v4.8h, v4.8h, v0.8h     // (t1 + t3) >> 1
+        sub             v2.8h, v7.8h, v3.8h     // (t2 - t4) >> 1
+        add             v3.8h, v3.8h, v7.8h     // (t2 + t4) >> 1
         srshr           v0.8h, v4.8h, #6        // (t1 + t3 + 64) >> 7
         srshr           v1.8h, v2.8h, #6        // (t2 - t4 + 64) >> 7
         srshr           v2.8h, v3.8h, #6        // (t2 + t4 + 64) >> 7
@@ -505,18 +505,18 @@ function ff_vc1_inv_trans_4x4_neon, export=1
         trn2            v1.2s, v1.2s, v2.2s     // 20 21 22 23
         mul             v2.4h, v4.4h, v0.h[1]   //                             
                              22/2 * src[24]
         mul             v4.4h, v4.4h, v0.h[0]   //                             
                              10/2 * src[24]
-        mul             v7.4h, v16.4h, v0.h[2]  //            17 * src[0]
-        mul             v1.4h, v1.4h, v0.h[2]   //                             
               17 * src[16]
+        add             v7.4h, v16.4h, v1.4h    //                 src[0]      
           +        src[16]
+        sub             v1.4h, v16.4h, v1.4h    //                 src[0]      
           -        src[16]
         mls             v2.4h, v3.4h, v0.h[0]   //  t4/2 =                - 
10/2 * src[8]                  + 22/2 * src[24]
         mla             v4.4h, v3.4h, v0.h[1]   //  t3/2 =                  
22/2 * src[8]                  + 10/2 * src[24]
-        add             v0.4h, v7.4h, v1.4h     //   t1  =    17 * src[0]      
           +   17 * src[16]
-        sub             v1.4h, v7.4h, v1.4h     //   t2  =    17 * src[0]      
           -   17 * src[16]
-        neg             v3.4h, v2.4h            // -t4/2
-        neg             v7.4h, v4.4h            // -t3/2
-        ssra            v4.4h, v0.4h, #1        // (t1 + t3) >> 1
-        ssra            v3.4h, v1.4h, #1        // (t2 - t4) >> 1
-        ssra            v2.4h, v1.4h, #1        // (t2 + t4) >> 1
-        ssra            v7.4h, v0.4h, #1        // (t1 - t3) >> 1
+        shl             v0.4h, v7.4h, #3        //            8 * (src[0]      
           +        src[16])
+        shl             v16.4h, v1.4h, #3       //            8 * (src[0]      
           -        src[16])
+        ssra            v0.4h, v7.4h, #1        //  t1 >> 1
+        ssra            v16.4h, v1.4h, #1       //  t2 >> 1
+        sub             v7.4h, v0.4h, v4.4h     // (t1 - t3) >> 1
+        add             v4.4h, v4.4h, v0.4h     // (t1 + t3) >> 1
+        sub             v3.4h, v16.4h, v2.4h    // (t2 - t4) >> 1
+        add             v2.4h, v2.4h, v16.4h    // (t2 + t4) >> 1
         trn1            v0.2d, v4.2d, v3.2d
         trn1            v1.2d, v2.2d, v7.2d
         srshr           v0.8h, v0.8h, #6        // (t1 + t3 + 64) >> 7, (t2 - 
t4 + 64) >> 7
-- 
2.54.0


From ea539f2bdf7028587a92817d9338557b0a29d51e Mon Sep 17 00:00:00 2001
From: Wesley Maa <[email protected]>
Date: Sun, 8 Nov 2026 11:16:07 -0500
Subject: [PATCH 2/2] checkasm/vc1dsp: test inverse transforms with a large
 in-range DC

The random coefficients used for the adding inverse transforms are
attenuated by a factor of 4 so that implementations limited to 16-bit
intermediates still pass, which means the upper part of the range the
VC-1 spec allows is never exercised. This hid a sign flip in the
aarch64 vc1_inv_trans_4x4 and vc1_inv_trans_8x4 functions, where
17 * (src[0] +- src[16]) overflowed 16 bits for valid input.

In addition to the random input, run each function on a DC-only block
of either sign whose output lies just outside the 8-bit range (about
+-280). The DC is chosen per transform size so that all intermediate
and final values stay within the spec limits.

Without the preceding aarch64 fix this reports
vc1dsp.vc1_inv_trans_4x4_neon and vc1dsp.vc1_inv_trans_8x4_neon as
failing.

Signed-off-by: Wesley Maa <[email protected]>
---
 tests/checkasm/vc1dsp.c | 15 +++++++++++++++
 1 file changed, 15 insertions(+)

diff --git a/tests/checkasm/vc1dsp.c b/tests/checkasm/vc1dsp.c
index 52049a4aa35..746d76957d0 100644
--- a/tests/checkasm/vc1dsp.c
+++ b/tests/checkasm/vc1dsp.c
@@ -323,6 +323,21 @@ static void check_inv_trans_adding(void)
             call_new(inv_trans_out1 + 24 + 8, 24, inv_trans_in1);
             if (memcmp(inv_trans_out0, inv_trans_out1, 10 * 24))
                 fail();
+
+            for (int sign = -1; sign <= 1; sign += 2) {
+                int dc = tests[k].width == 8 && tests[k].height == 8 ? 2000 :
+                         tests[k].width == 4 && tests[k].height == 4 ? 1000 :
+                                                                       1400;
+                memset(inv_trans_in0, 0, 8 * 8 * sizeof (int16_t));
+                memset(inv_trans_in1, 0, 8 * 8 * sizeof (int16_t));
+                inv_trans_in0[0] = inv_trans_in1[0] = sign * dc;
+                RANDOMIZE_BUFFER8(inv_trans_out, 10 * 24);
+                call_ref(inv_trans_out0 + 24 + 8, 24, inv_trans_in0);
+                call_new(inv_trans_out1 + 24 + 8, 24, inv_trans_in1);
+                if (memcmp(inv_trans_out0, inv_trans_out1, 10 * 24))
+                    fail();
+            }
+
             bench_new(inv_trans_out1 + 24 + 8, 24, inv_trans_in1);
             av_free(coeffs);
         }
-- 
2.54.0

_______________________________________________
ffmpeg-devel mailing list -- [email protected]
To unsubscribe send an email to [email protected]

Reply via email to