Rich-T-kid opened a new pull request, #27479:
URL: https://github.com/apache/datafusion/pull/27479

   ## Which issue does this PR close?
   
   - N/A. This performance opportunity was found while profiling ClickBench Q5.
   
   ## Rationale for this change
   
   The inner aggregate for `COUNT(DISTINCT dictionary_column)` has no 
accumulators,
   but it currently interns one group key per input row. Repeated dictionary 
keys
   identify the same batch-local value, so this repeats group lookup work.
   
   On ClickBench Q5 (`COUNT(DISTINCT SearchPhrase)`), a temporary trace recorded
   12,312 eligible batches and reduced 99,997,497 input rows to 9,544,369 keys
   before group interning, a 90.46% reduction.
   
   A controlled local B-C-C-B comparison against `c177f5704`, with nine 
iterations
   per leg, measured:
   
   | Leg | Median |
   |---|---:|
   | Base 1 | 611.97 ms |
   | Candidate 1 | 557.78 ms |
   | Candidate 2 | 551.47 ms |
   | Base 2 | 558.45 ms |
   
   The order-balanced median geomean improved by 5.13%. Excluding each leg's 
first
   iteration improved the median geomean by 5.62% and the mean by 6.01%. 
Candidate
   was faster in both pairings, and fair-pool peak was effectively unchanged
   (-0.01%). A separate earlier clean run measured a 15.75% improvement.
   
   ## What changes are included in this PR?
   
   For a zero-accumulator aggregate with exactly one dictionary group column, 
this
   reduces the input to one row per occupied dictionary key plus one null key.
   Normal group interning still resolves duplicate logical values, null 
dictionary
   values, and null keys.
   
   The existing path is retained unless dictionary metadata guarantees at least 
a
   25% reduction in rows.
   
   ## What is the testing strategy for this PR?
   
   - Unit tests cover the production partial DISTINCT path, the 25% fallback
     boundary, unsigned dictionary keys, duplicate logical values, null keys, 
and
     null dictionary values.
   - `cargo xtask ci step test hash-collisions`
   - Full local pre-push hook: formatting, workspace-wide Clippy with 
integration
     features, and typos.
   - ClickBench Q5 correctness check comparing the dictionary expression with a
     materialized plain-string expression.
   - Hosted TPC-H, TPC-DS, and dictionary-enabled ClickBench runs will be linked
     after completion.
   
   ## Are there any user-facing changes?
   
   No API or behavior changes. Eligible DISTINCT aggregates do less group-key 
work.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to