jayzhan211 commented on PR #25724: URL: https://github.com/apache/datafusion/pull/25724#issuecomment-6381191489
@Rachelint I'll take the Partial flush out of this PR and leave it to #26124 / #26116, so this PR only buckets the Final stage. On compaction, I'd rather keep it. Without it, a bucket holds every row it receives until the end. When the Partial stage passes rows through mostly unreduced (skipped, or few rows per group), Final memory then grows with the input rows instead of the groups. Before I added compaction, peak RSS was 2.3x of main on 20M random integer keys and 2.26x on TPC-H SF10 `GROUP BY l_partkey, l_suppkey`. With compaction it was 1.13x and 1.40x. It does cost something on Q32 (0.34x → 0.42x of main, still faster). But when the table being split kept more than 80% of its rows as groups, as in Q32, buckets already wait four times longer before their first compaction. If you have a case where it still hurts, could you share it? I'd rather tune the wait than drop it. I answered the #24704 question on the issue itself. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
