jayzhan211 commented on PR #25724:
URL: https://github.com/apache/datafusion/pull/25724#issuecomment-6887611473

   I'm turning this PR back to draft: with buckets on, it makes some 
aggregations faster and others slower, and I don't see a fix yet that doesn't 
add more tuned limits.
   
   On the benchmark runner (same commit, threshold 0 vs 262144):
   
   - Faster: ClickBench Q32 1.36x, Q15 1.23x, Q4 1.17x 
([results](https://github.com/apache/datafusion/pull/25724#issuecomment-6887378076));
 suite total 1.7%. TPC-H SF10 unchanged.
   - Slower: `clickbench_extended` Q15 (`covar_samp` grouped by `UserID`) 
1.11-1.13x, and that suite's peak memory goes from 12.8 to 14.1 GiB 
([results](https://github.com/apache/datafusion/pull/25724#issuecomment-6887429476)).
 My laptop reproduces it at 1.22x.
   
   My laptop also shows costs the description understates:
   
   - Peak memory up to 1.9x, not 1.13x, when the partial aggregation is skipped 
and groups repeat (4M groups, 8 aggregates): rows wait in buckets until they 
are compacted.
   - A query that ends just above the threshold pays for the move and gains 
nothing: ClickBench Q30 on half the data ends at about 264k groups per 
partition and runs 1.09x slower.
   - Under a 2 GB memory limit, the 8-aggregate query above runs 2.2x slower.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to