yinli-systems commented on issue #24117: URL: https://github.com/apache/datafusion/issues/24117#issuecomment-5671192732
Benchmark evidence update from the first local prototype (not a proposed universal threshold yet). **Method** - DataFusion `c149764`, release/LTO build, Apple M5 Max, Rust 1.97 - `GROUP BY key, COUNT(*)`, 10M identical logical rows, 8,192-row input batches - median of iterations 2-6 after two warmups - timer includes planning + execution; input construction is excluded - dictionary cases cast the final group key back to `Utf8`, so the comparison includes preserving the user-visible output type - compared plain `Utf8`, pre-encoded dictionaries with a shared values array, pre-encoded dictionaries with independent per-batch values arrays, and a runtime `arrow_cast` from `Utf8` Single target partition results (milliseconds; lower is better): | value bytes | NDV | plain | dict/shared | dict/per-batch | runtime cast | |---:|---:|---:|---:|---:|---:| | 8 | 10 | 48.87 | 35.43 (-27.5%) | 35.49 (-27.4%) | 94.82 (+94.0%) | | 8 | 100 | 47.85 | 38.87 (-18.8%) | 37.95 (-20.7%) | 98.37 (+105.6%) | | 8 | 1,000 | 42.66 | 44.88 (+5.2%) | 45.33 (+6.3%) | 111.74 (+161.9%) | | 64 | 10 | 59.09 | 35.64 (-39.7%) | 36.44 (-38.3%) | 114.25 (+93.4%) | | 64 | 100 | 57.09 | 37.63 (-34.1%) | 38.41 (-32.7%) | 119.16 (+108.7%) | | 64 | 1,000 | 59.36 | 49.04 (-17.4%) | 48.17 (-18.9%) | 132.70 (+123.6%) | | 64 | 10,000 | 64.36 | 115.09 (+78.8%) | 118.20 (+83.6%) | 401.75 (+524.2%) | A 4-target-partition check at NDV=100 kept the result: dictionary was ~19.5% faster for 8-byte values and ~36.8% faster for 64-byte values. At 1M rows / NDV=100 / 8-byte values, it remained ~21.6% faster. Shared vs per-batch values did not materially change these low-NDV cases; I still expect reuse to matter as dictionary cardinality grows, consistent with #25185. **What this rules out** - A generic physical-plan `Utf8 -> Dictionary` runtime cast is not viable here, even at NDV=10. It loses by roughly 2x and gets much worse as NDV grows. - NDV ratio alone is not enough. At the same 1,000 / 10M ratio, short strings lose while 64-byte strings win. - Absolute NDV alone is not enough either; row count and value width affect the crossover. This supports a scan-native-only first policy: use dictionary materialization only when the source can supply it without a generic runtime build; fail closed when NDV/row count/byte-width evidence is missing; preserve the original query output type; and keep the rule opt-in until it has cross-machine / Parquet results. I now have a single-file benchmark change that adds explicit input encoding, absolute NDV, value length, input batch size, shared/per-batch values, and output normalization while preserving the existing default dictionary benchmark behavior. Its four focused tests and default-feature clippy pass locally. If this benchmark slice is useful as a prerequisite PR, I can submit it independently while we settle the selective scan/planning hook for #24117. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
