mkleen commented on PR #26000: URL: https://github.com/apache/datafusion/pull/26000#issuecomment-6943133242
@alamb fair question. The earlier numbers were for an older version. I re-ran the current pr using the #24929 method (target_partitions = 1, 4M rows, ~1M distinct x, smallest -m that finishes with -d 0 --mem-pool-type greedy). ``` SELECT g, sum(DISTINCT x) FROM t GROUP BY g, with single_distinct_prefer_native off and on: ┌─────────┬──────────┬─────────┬──────────────┬─────────────┬───────────────┬──────────────┐ │ Groups │ Time off │ Time on │ Peak RSS off │ Peak RSS on │ Min limit off │ Min limit on │ ├─────────┼──────────┼─────────┼──────────────┼─────────────┼───────────────┼──────────────┤ │ 2,000 │ 0.124 s │ 0.245 s │ 539 MiB │ 306 MiB │ 192M │ 80M │ ├─────────┼──────────┼─────────┼──────────────┼─────────────┼───────────────┼──────────────┤ │ 500,000 │ 0.157 s │ 0.405 s │ 538 MiB │ 425 MiB │ 192M │ 240M │ └─────────┴──────────┴─────────┴──────────────┴─────────────┴───────────────┴──────────────┘ ``` In the current state, we are trading performance for memory efficiency when we have few large groups and no memory limit. At the same time it hurts with many small groups. So it's complicated and as @asolimando suggested in https://github.com/apache/datafusion/pull/26000#issuecomment-6265078225 this might be a cost-based-optimization (CBO) usecase. The current knob behaviour is tricky to figure out correctly. Anyway, your suggestion to makes things faster makes sense to me. So to to improve the performance of `sum(DISTINCT x)` maybe we should have a native distinct GroupsAccumulator for sum? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
