lxc512157407 commented on issue #24822: URL: https://github.com/apache/datafusion/issues/24822#issuecomment-5535114049
Hi @Rich-T-kid, thanks for taking a look. Yes — I have a WIP branch: https://github.com/lxc512157407/datafusion/tree/fix/dictionary-regression What it currently addresses is a **hash-layer** regression on the dictionary path: generalizing `hash_utils` over `BuildHasher` split the `hash_dictionary` hot loop into separate monomorphizations (RandomState path vs original), costing ~15% on `dictionary_utf8_int32`. The fix separates "hash the dictionary values" (per-hasher) from "scatter hashes to the key codes" (shared, non-generic hot loop) — commit `3a3bfc57`. To set expectations honestly: that is one component of the tax in this issue, not all of it. The 27–37% numbers here point at the aggregation path itself — group keys appear to be upcast/normalized per batch rather than grouping on the codes directly (for single-batch aggregation the dictionary codes are already perfect group ids). So the branch should remove part of the gap, and the per-batch normalization is the bigger remaining suspect. The branch is from April and is ~1500 commits behind current main; I'll rebase it and re-run the Q2/Q19/Q20 numbers from the issue on top. Happy to share the standalone bench (same batches, only the group-key column encoding differs) if you want to poke at it yourself — and glad to collaborate on the group-by-on-codes direction. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
