Rich-T-kid opened a new issue, #26848: URL: https://github.com/apache/datafusion/issues/26848
### Is your feature request related to a problem or challenge? Related to #26780. `create_hashes` hashes each dictionary value once and then scatters those hashes through the keys. For nullable dictionaries, the scatter currently decodes an `Option` and branches for every row. Once dictionary value hashes are reused, this O(rows) work becomes a larger share of the hashing cost. On the existing 8,192-row `with_hashes` benchmark with 3% null keys, nullable dictionary scatter takes about 5.9 us for one column and 17.0 us for three columns on an Apple M4 Max. The no-null cases take about 2.0 us and 6.1 us respectively. ### Describe the solution you would like Process key validity in 64-row words: - reuse the current dense loop for all-valid words - pipeline small groups of dictionary hash loads for mostly-valid words - walk only set bits for sparse words - leave null output slots unchanged, including when dictionary values themselves are null The benchmark should cover single and composite keys, several dictionary key widths, null densities, slices, and 8/64-row boundaries. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
