Rich-T-kid commented on issue #24822: URL: https://github.com/apache/datafusion/issues/24822#issuecomment-5542336472
> Re-measured on current `main` (`9fc7a4d`, 2026-09-04) with a self-contained repro — the shape of the tax has changed since the original report, and one part of it is already fixed upstream: > > **Method**: identical in-memory tables where only the group-key encoding differs (`Utf8View` vs `Dictionary(Int32, Utf8View)`, produced by `cast`-ing the exact same batches); same `GROUP BY` + `COUNT(*)` + `SUM` on both; single target partition; 5 timed rounds after 1 warmup, median. Repro (buildable example, run with `cargo run --release -p datafusion-examples --example dict_group_tax`): https://github.com/lxc512157407/datafusion/tree/bench/dict-group-tax > > case (key shape) Utf8View ms Dictionary ms slowdown > q2-like: 2 keys, string (150 distinct, short) + int, 6M rows 38.7 39.8 **+3%** > q19-like: 1 key, string (7 distinct), 6M rows 25.0 29.3 **+17%** > q20-like: 1 key, long string (100k distinct, >12 bytes), 2M rows 32.4 57.2 **+77%** > Observations: > > 1. **The Q2-style multi-key tax from the original report is gone on main** (+3%, within noise). For context, main now contains a dictionary hashing path (introduced via [fix(proto): preserve CSV/JSON scan options on roundtrip #24233](https://github.com/apache/datafusion/pull/24233), see `hash_dictionary_scatter` in `datafusion/common/src/hash_utils.rs`) that hashes each dictionary value once and scatters via const-generic monomorphic loops — same shape as the fix we prototyped in the branch linked earlier in this thread. That work looks absorbed. > 2. **The remaining tax concentrates on high-cardinality long-string dictionary keys** (q20-like, +77%): when the dictionary has ~100k distinct values the once-per-value hashing no longer amortizes, and the group-by additionally pays for materializing dictionary-encoded group keys. Note the view variant here has long strings heap-escaped (>12 bytes) — so this is not an inline-view advantage; the dictionary column carries strictly less data and is still slower. > 3. Low-cardinality single-key (q19-like) retains a modest +17%. > > So the residual gap is narrower than the original 27–37% report but is still real for high-cardinality string group keys. Happy to help dig into the aggregation-side kernel if there's interest — the repro branch above can be the baseline. this is more inline with what I was expecting. the most concerning case is q19 where the the cardinality is very low I'd expect dictionary arrays to be much more performant. I'd like to give a more in depth response to this in a bit but i'm curious, could you try running these benchmarks again with `dict<_,utf8>` instead of `dict<_,utf8View>` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
