Rich-T-kid opened a new issue, #12053: URL: https://github.com/apache/arrow-rs/issues/12053
### Is your feature request related to a problem or challenge? #12043 speeds up row encoding for oversized sparse dictionaries by gathering one value per input row before encoding. `gather_dictionary_values` currently calls `MutableArrayData::try_extend` once per key. That cost is acceptable for the high-cardinality case targeted by the PR, but it limits the fast path and moves the crossover higher than necessary. ### Describe the solution you would like Benchmark and optimize the row-aligned dictionary gather so it avoids one dynamic `try_extend` call per key where possible. Preserve sliced keys, nulls, arbitrary key widths, and the generic values-array fallback. Use the `string_dictionary_sparse_keys` row-format cases across dictionary sizes to verify the crossover and guard against regressions. ### Describe alternatives you have considered Potential approaches include using `take`, specializing common byte-array value types, or batching contiguous key ranges while keeping `MutableArrayData` as the generic fallback. ### Additional context Follow-up from #12043: https://github.com/apache/arrow-rs/pull/12043#discussion_r4456329171 Current benchmark results: https://github.com/apache/arrow-rs/pull/12043#issuecomment-6606029732 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
