PR #24959 opened by Aaditya Agarwal (aaditya8979) URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24959 Patch URL: https://code.ffmpeg.org/FFmpeg/FFmpeg/pulls/24959.patch
This patch introduces high-bit-depth (10-bit and 12-bit) NEON optimizations for the SAO band filter, completely eliminating the scalar C fallback for HDR HEVC and VVC streams on AArch64. ### The Problem While the SAO *edge* filter has comprehensive NEON coverage for 10-bit and 12-bit depths, the SAO *band* filter was entirely missing HBD implementations (`ff_h26x_sao_band_filter_16x16_10_neon`, etc). Consequently, whenever a 10-bit or 12-bit HDR stream hit a band filter CTU, the pipeline forcefully fell back to the scalar C `sao_band_filter` function, severely bottlenecking decoding performance. ### Architectural Solutions The 8-bit NEON band filter relies on a single `tbl` instruction to look up offset values. Porting this to 10/12-bit presents two distinct architectural challenges on AArch64, which this patch solves: **1. Dual-Table Splitting for 16-bit Lookups** The offset table is 32 × `int16_t` (64 bytes). The AArch64 `tbl` instruction strictly operates on 8-bit bytes, meaning it cannot natively retrieve a 16-bit offset using an index vector. * **Solution:** We construct the 64-byte `int16_t` table on the stack and load it into four Q registers. We then use `uzp1` and `uzp2` to split the table into separate Low-Byte (`v16`/`v17`) and High-Byte (`v18`/`v19`) halves. * By performing two parallel 8-bit `tbl` lookups using a narrowed band index, we retrieve the LSB and MSB independently. A final `zip1` instruction perfectly reconstructs the `int16_t` offsets in the vector accumulator. **2. Dynamic Variable Shift Resolution** The band index calculation requires a right-shift by 5 for 10-bit content (`src[x] >> (10-5)`) and a right-shift by 7 for 12-bit content (`src[x] >> (12-5)`). Creating separate assembly bodies for 10-bit and 12-bit would result in massive code duplication. * **Solution:** We store the negated shift amount (`-5` or `-7`) in `v20.8h`. The `ushl` instruction natively performs an unsigned right-shift when fed a negative shift vector. This allows the 10-bit and 12-bit paths to share the exact same execution body, differing only by two instructions in the prologue. ### Cross-Codec Impact Because these functions use the shared `h26x` prefix and are wired into `hevcdsp_init_aarch64.c`, this single patch directly accelerates the SAO band filter for **both HEVC and VVC** codecs. ### Benchmarks Tested on Apple M4 (`checkasm --test=hevc_sao --bench`). All correctness tests pass. ```text Function C (ns) NEON (ns) Speedup -------------------------------------------------------- hevc_sao_band_8_10 113.5 3.8 29.88x hevc_sao_band_8_12 118.5 4.5 26.15x hevc_sao_band_16_10 410.3 15.7 26.13x hevc_sao_band_16_12 417.4 15.8 26.28x hevc_sao_band_32_10 1569.9 63.1 24.82x hevc_sao_band_32_12 1616.3 58.6 27.54x hevc_sao_band_48_10 3630.3 142.3 25.44x hevc_sao_band_48_12 3751.8 139.9 26.68x hevc_sao_band_64_10 6422.0 253.2 25.30x hevc_sao_band_64_12 6653.3 243.7 27.11x Yields a sustained 25× to 30× throughput increase across all block widths for High Bit Depth content. From 8a0a9fe51f064b69c51191709289f419f15175bf Mon Sep 17 00:00:00 2001 From: Aaditya Agarwal <[email protected]> Date: Fri, 9 Oct 2026 02:09:26 +0530 Subject: [PATCH] aarch64/hevc: add NEON optimizations for 10/12-bit SAO band filter MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The SAO band filter had no high-bit-depth NEON implementation on AArch64; 10-bit and 12-bit streams fell back entirely to scalar C. This patch adds ff_h26x_sao_band_filter_{8x8,16x16}_{10,12}_neon covering both HEVC and VVC (the h26x prefix is shared between the two codecs). Dual-table splitting to work around the 8-bit tbl limitation: The offset table is 32 × int16_t (64 bytes). AArch64 tbl only handles 8-bit table elements, so we cannot look up a 16-bit offset directly. Instead, the table is built on the stack as int16_t, loaded into four Q registers (entries 0-15 in q0/q1, entries 16-31 in q2/q3), then split into separate LSB and MSB register pairs via uzp1/uzp2: uzp1 v16/v17: low bytes of entries [0-15] / [16-31] uzp2 v18/v19: high bytes of entries [0-15] / [16-31] Two parallel tbl lookups on the narrowed 8-bit band index retrieve the LSB and MSB bytes independently; zip1 reconstructs the int16_t offsets in a single instruction before the signed add and clamp. Variable shift without instruction duplication: The band index is src[x] >> (bit_depth - 5): shift 5 for 10-bit, shift 7 for 12-bit. Rather than duplicating the entire pixel loop, the shift amount is broadcast into v20.8h as a negative value and applied with ushl (USHL with a negative element performs an unsigned right shift of that magnitude), making the 10/12-bit paths share a single code body with only a two-instruction difference in the prologue. Cross-codec impact: Because the functions carry the h26x prefix and are registered in hevcdsp_init_aarch64.c, this patch directly accelerates the SAO band filter for all 10-bit and 12-bit HEVC and VVC streams — covering every HDR decode path on AArch64 that has SAO enabled. Benchmarked on Apple M4 (checkasm --test=hevc_sao --bench): Function C (ns) NEON (ns) Speedup -------------------------------------------------------- hevc_sao_band_8_10 113.5 3.8 29.88x hevc_sao_band_8_12 118.5 4.5 26.15x hevc_sao_band_16_10 410.3 15.7 26.13x hevc_sao_band_16_12 417.4 15.8 26.28x hevc_sao_band_32_10 1569.9 63.1 24.82x hevc_sao_band_32_12 1616.3 58.6 27.54x hevc_sao_band_48_10 3630.3 142.3 25.44x hevc_sao_band_48_12 3751.8 139.9 26.68x hevc_sao_band_64_10 6422.0 253.2 25.30x hevc_sao_band_64_12 6653.3 243.7 27.11x All checkasm hevc_sao tests pass. --- libavcodec/aarch64/h26x/dsp.h | 16 +++ libavcodec/aarch64/h26x/sao_neon.S | 158 ++++++++++++++++++++++ libavcodec/aarch64/hevcdsp_init_aarch64.c | 10 ++ 3 files changed, 184 insertions(+) diff --git a/libavcodec/aarch64/h26x/dsp.h b/libavcodec/aarch64/h26x/dsp.h index 2b8eae130a..70dec558f9 100644 --- a/libavcodec/aarch64/h26x/dsp.h +++ b/libavcodec/aarch64/h26x/dsp.h @@ -32,6 +32,22 @@ void ff_h26x_sao_band_filter_16x16_8_neon(uint8_t *_dst, const uint8_t *_src, ptrdiff_t stride_dst, ptrdiff_t stride_src, const int16_t *sao_offset_val, int sao_left_class, int width, int height); +void ff_h26x_sao_band_filter_8x8_10_neon(uint8_t *_dst, const uint8_t *_src, + ptrdiff_t stride_dst, ptrdiff_t stride_src, + const int16_t *sao_offset_val, int sao_left_class, + int width, int height); +void ff_h26x_sao_band_filter_16x16_10_neon(uint8_t *_dst, const uint8_t *_src, + ptrdiff_t stride_dst, ptrdiff_t stride_src, + const int16_t *sao_offset_val, int sao_left_class, + int width, int height); +void ff_h26x_sao_band_filter_8x8_12_neon(uint8_t *_dst, const uint8_t *_src, + ptrdiff_t stride_dst, ptrdiff_t stride_src, + const int16_t *sao_offset_val, int sao_left_class, + int width, int height); +void ff_h26x_sao_band_filter_16x16_12_neon(uint8_t *_dst, const uint8_t *_src, + ptrdiff_t stride_dst, ptrdiff_t stride_src, + const int16_t *sao_offset_val, int sao_left_class, + int width, int height); #define SAO_EDGE_FILTER_PROTO(codec, size, depth) \ void ff_##codec##_sao_edge_filter_##size##_##depth##_neon( \ uint8_t *dst, const uint8_t *src, ptrdiff_t stride_dst, \ diff --git a/libavcodec/aarch64/h26x/sao_neon.S b/libavcodec/aarch64/h26x/sao_neon.S index 083f0eb89d..a1431729d4 100644 --- a/libavcodec/aarch64/h26x/sao_neon.S +++ b/libavcodec/aarch64/h26x/sao_neon.S @@ -100,6 +100,164 @@ function ff_h26x_sao_band_filter_16x16_8_neon, export=1 ret endfunc +// void ff_h26x_sao_band_filter_{8x8,16x16}_{10,12}_neon( +// uint8_t *_dst, const uint8_t *_src, +// ptrdiff_t stride_dst, ptrdiff_t stride_src, +// const int16_t *sao_offset_val, int sao_left_class, +// int width, int height) +// +// High-bit-depth SAO band filter. The 8-bit version uses a single tbl +// because index and table element are both one byte. For 10/12-bit the +// table is 32 × int16_t (64 bytes) and tbl only accepts 8-bit elements, +// so we split via uzp1/uzp2 into lo-byte (v16,v17) and hi-byte (v18,v19) +// halves, run two parallel tbl lookups on the narrowed 8-bit index, then +// zip1 the results back into int16_t offsets. +// +// Band index = src[x] >> (bit_depth − 5): 10-bit → 5 12-bit → 7 +// We negate the shift and store it in v20; ushl with a negative element +// performs an unsigned right shift of that magnitude. +// +// v16/v17: lo bytes of band table [0..15] / [16..31] +// v18/v19: hi bytes of band table [0..15] / [16..31] +// v20.8h : negated shift (−5 or −7) +// v30.8h : 0 (clamp floor) +// v31.8h : 1023 or 4095 (clamp ceiling) + +// ── 8x8 entry points ────────────────────────────────────────────────────── +function ff_h26x_sao_band_filter_8x8_12_neon, export=1 + mvni v31.8h, #0xf0, lsl #8 // max = 4095 + movi v20.8h, #7 // shift = 12 - 5 + b .Lsao_band_8x8_16bit +endfunc + +function ff_h26x_sao_band_filter_8x8_10_neon, export=1 + mvni v31.8h, #0xfc, lsl #8 // max = 1023 + movi v20.8h, #5 // shift = 10 - 5 +.Lsao_band_8x8_16bit: + neg v20.8h, v20.8h // negate: ushl with neg = right-shift + + // Build 32-entry int16_t lookup table (64 bytes, zero-initialised) + sub sp, sp, #64 + movi v0.16b, #0 + stp q0, q0, [sp] + stp q0, q0, [sp, #32] + + mov w8, #4 +0: ldrsh x9, [x4, x8, lsl #1] // sao_offset_val[k+1] + subs w8, w8, #1 + add w10, w8, w5 // k + sao_left_class + and w10, w10, #0x1f // mod 32 + lsl x10, x10, #1 // byte offset (int16) + strh w9, [sp, x10] + bne 0b + + // Split table into lo/hi byte halves for dual tbl + ldp q0, q1, [sp] // entries 0-15 (int16) + ldp q2, q3, [sp, #32] // entries 16-31 (int16) + add sp, sp, #64 + + uzp1 v16.16b, v0.16b, v1.16b // lo bytes: entries 0-15 + uzp1 v17.16b, v2.16b, v3.16b // lo bytes: entries 16-31 + uzp2 v18.16b, v0.16b, v1.16b // hi bytes: entries 0-15 + uzp2 v19.16b, v2.16b, v3.16b // hi bytes: entries 16-31 + + movi v30.8h, #0 + +1: ld1 {v0.8h}, [x1], x3 // 8 × uint16_t + subs w7, w7, #1 + ushl v1.8h, v0.8h, v20.8h // right-shift by 5 or 7 + xtn v2.8b, v1.8h // narrow 16-bit index to 8-bit + tbl v3.8b, {v16.16b, v17.16b}, v2.8b // lo offset bytes + tbl v4.8b, {v18.16b, v19.16b}, v2.8b // hi offset bytes + zip1 v3.16b, v3.16b, v4.16b // reconstruct int16_t offsets + add v0.8h, v0.8h, v3.8h // src[x] + offset + smax v0.8h, v0.8h, v30.8h // clamp ≥ 0 + smin v0.8h, v0.8h, v31.8h // clamp ≤ max_val + st1 {v0.8h}, [x0], x2 + bne 1b + ret +endfunc + +// ── 16x16 entry points ──────────────────────────────────────────────────── +function ff_h26x_sao_band_filter_16x16_12_neon, export=1 + mvni v31.8h, #0xf0, lsl #8 // max = 4095 + movi v20.8h, #7 + b .Lsao_band_16x16_16bit +endfunc + +function ff_h26x_sao_band_filter_16x16_10_neon, export=1 + mvni v31.8h, #0xfc, lsl #8 // max = 1023 + movi v20.8h, #5 +.Lsao_band_16x16_16bit: + neg v20.8h, v20.8h + + // Build table (identical to 8x8 path) + sub sp, sp, #64 + movi v0.16b, #0 + stp q0, q0, [sp] + stp q0, q0, [sp, #32] + + mov w8, #4 +0: ldrsh x9, [x4, x8, lsl #1] + subs w8, w8, #1 + add w10, w8, w5 + and w10, w10, #0x1f + lsl x10, x10, #1 + strh w9, [sp, x10] + bne 0b + + ldp q0, q1, [sp] + ldp q2, q3, [sp, #32] + add sp, sp, #64 + + uzp1 v16.16b, v0.16b, v1.16b + uzp1 v17.16b, v2.16b, v3.16b + uzp2 v18.16b, v0.16b, v1.16b + uzp2 v19.16b, v2.16b, v3.16b + + movi v30.8h, #0 + + // Round width to 16-pixel (32-byte) boundary; adjust stride remainders + add w6, w6, #15 + bic w6, w6, #15 + lsl w6, w6, #1 // width in bytes + sub x2, x2, x6 // dst stride remainder + sub x3, x3, x6 // src stride remainder + +1: mov w8, w6 // bytes remaining this row +2: ld1 {v0.8h, v1.8h}, [x1], #32 // 16 × uint16_t + + // First 8 pixels (v0) + ushl v2.8h, v0.8h, v20.8h + xtn v4.8b, v2.8h + tbl v5.8b, {v16.16b, v17.16b}, v4.8b + tbl v6.8b, {v18.16b, v19.16b}, v4.8b + zip1 v5.16b, v5.16b, v6.16b + add v0.8h, v0.8h, v5.8h + smax v0.8h, v0.8h, v30.8h + smin v0.8h, v0.8h, v31.8h + + // Next 8 pixels (v1) + ushl v3.8h, v1.8h, v20.8h + xtn v4.8b, v3.8h + tbl v5.8b, {v16.16b, v17.16b}, v4.8b + tbl v6.8b, {v18.16b, v19.16b}, v4.8b + zip1 v5.16b, v5.16b, v6.16b + add v1.8h, v1.8h, v5.8h + smax v1.8h, v1.8h, v30.8h + smin v1.8h, v1.8h, v31.8h + + subs w8, w8, #32 + st1 {v0.8h, v1.8h}, [x0], #32 + bne 2b + + subs w7, w7, #1 + add x0, x0, x2 + add x1, x1, x3 + bne 1b + ret +endfunc + .Lhevc_sao_edge_pos_8: .word 1 // horizontal .word HEVC_SAO_STRIDE // vertical diff --git a/libavcodec/aarch64/hevcdsp_init_aarch64.c b/libavcodec/aarch64/hevcdsp_init_aarch64.c index 12f8d93803..5c4451cd3a 100644 --- a/libavcodec/aarch64/hevcdsp_init_aarch64.c +++ b/libavcodec/aarch64/hevcdsp_init_aarch64.c @@ -343,6 +343,11 @@ av_cold void ff_hevc_dsp_init_aarch64(HEVCDSPContext *c, const int bit_depth) c->sao_edge_filter[2] = c->sao_edge_filter[3] = c->sao_edge_filter[4] = ff_hevc_sao_edge_filter_16x16_10_neon; + c->sao_band_filter[0] = ff_h26x_sao_band_filter_8x8_10_neon; + c->sao_band_filter[1] = + c->sao_band_filter[2] = + c->sao_band_filter[3] = + c->sao_band_filter[4] = ff_h26x_sao_band_filter_16x16_10_neon; } if (bit_depth == 12) { c->hevc_h_loop_filter_luma = ff_hevc_h_loop_filter_luma_12_neon; @@ -363,5 +368,10 @@ av_cold void ff_hevc_dsp_init_aarch64(HEVCDSPContext *c, const int bit_depth) c->sao_edge_filter[2] = c->sao_edge_filter[3] = c->sao_edge_filter[4] = ff_hevc_sao_edge_filter_16x16_12_neon; + c->sao_band_filter[0] = ff_h26x_sao_band_filter_8x8_12_neon; + c->sao_band_filter[1] = + c->sao_band_filter[2] = + c->sao_band_filter[3] = + c->sao_band_filter[4] = ff_h26x_sao_band_filter_16x16_12_neon; } } -- 2.52.0 _______________________________________________ ffmpeg-devel mailing list -- [email protected] To unsubscribe send an email to [email protected]
