yihua opened a new pull request, #11910: URL: https://github.com/apache/arrow-rs/pull/11910
# Which issue does this PR close? - Closes #11908 Stacked on #11909 (the benchmarks), so this shows both commits until that merges. # Rationale for this change Converting `FIXED_LEN_BYTE_ARRAY` decimals to native integers dominates reading them, because `sign_extend_be` copies a runtime length and compiles to a `memcpy` call per value. # What changes are included in this PR? `consume_batch` matches the column's byte width once per batch and converts with a loop specialized to it (`as_chunks::<W>()` plus a new `sign_extend_be_fixed`), so each copy is a constant-size move. Every width each decimal type accepts is covered, which is 60 small monomorphized loops, and the existing path stays as a fallback. # Are these changes tested? `test_read_decimal_of_every_width` round-trips the minimal width for every precision from 1 to 76, with negatives, boundaries and nulls, as each decimal type that can hold it. `arrow_reader` benchmarks against `main` with the benchmarks from #11909, Apple M3 Max, mean of two alternating runs: | Benchmark (mandatory, no NULLs unless noted) | main | this PR | change | | --- | --- | --- | --- | | FLBA(16) as Decimal128, plain | 281 µs | 65 µs | -77% | | FLBA(16) as Decimal128, plain, half NULLs | 421 µs | 210 µs | -50% | | FLBA(16) as Decimal128, byte_stream_split | 640 µs | 420 µs | -34% | | FLBA(7) as Decimal128, plain | 658 µs | 69 µs | -89% | | FLBA(7) as Decimal128, plain, half NULLs | 789 µs | 265 µs | -66% | | FLBA(7) as Decimal64, plain | 352 µs | 55 µs | -84% | | BYTE_ARRAY as Decimal128, plain (unchanged path) | 821 µs | 849 µs | noise | # Are there any significant user-facing changes? No. AI usage: written with an AI coding assistant under my direction; I reviewed every line and ran the tests and benchmarks. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
