cloud-fan commented on PR #59823: URL: https://github.com/apache/spark/pull/59823#issuecomment-6456209504
Thanks for the review. The benchmark results are in 6a5684c. The description now lists the spill-check cases with their results, says that the default keeps aggregating recurring keys only when the map holds about half of a task's keys or more, and covers the new tests. On dominant group keys: yes, the default is meant to pass such inputs through when the map fills. The alternative is spilling most of the high-cardinality tail every epoch, which is why the "hot keys" cases are about 1.3x faster in total work. The cost you describe is a different one: a single reducer that receives the hot key's later rows one by one. The benchmark runs on `local[1]`, so it measures total work, and no case there can show that tail, even with more reducers. So 40163fb adds to the tuning guide that when one or two keys, such as a NULL or default value, hold a large share of the rows, setting `minCompactionToSpill` to `minCompaction` or below keeps such inputs aggregating through spills. Keeping a few heavy keys aggregating during pass-through would avoid the trade-off, but it adds a probe for every passed-through row, so I added it to SPARK-60601, where it can be measured. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
