yihua opened a new issue, #11908: URL: https://github.com/apache/arrow-rs/issues/11908
### Is your feature request related to a problem or challenge? Reading `FIXED_LEN_BYTE_ARRAY` decimals is several times slower than reading the same values stored as `INT64`, and most of the time goes into converting each value in `FixedLenByteArrayReader::consume_batch`: `sign_extend_be` copies a length known only at run time, so every value pays for a `memcpy` call. This layout is common, since parquet-mr based writers (Spark's legacy format, Hive, parquet-avro) store every decimal this way, and all writers use it above precision 18. ### Describe the solution you'd like All values in a column have the same width, so the reader could pick a conversion loop specialized to that width once per batch, making the per-value copy a constant-size move. ### Describe alternatives you've considered Converting while decoding, as the `TODO` in `consume_batch` suggests, would also skip the intermediate `FixedSizeBinaryArray`, but it is a larger change. ### Additional context The dictionary path in `ValueDecoder::read` has a similar per-value copy of a runtime length, which could be a follow-up. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
