wangyum commented on PR #59027: URL: https://github.com/apache/spark/pull/59027#issuecomment-6544954116
Ran the staggered-overlap cases at ~250x scale (build ~200K-1M ranges, probe ~2.5M-100M rows) as plain Spark SQL on a real distributed cluster, flag on vs off: - staggered 5% overlap (long): 2.4 min vs 2.4 min - staggered 90% overlap (long): 29 min vs 31 min (range join faster) - staggered 5% overlap (double): 2.7 min vs 2.3 min - staggered 90% overlap (double): 31 min vs 30 min The microbenchmark's 0.5X-0.6X/0.7X-0.8X ratios are measured in a single JVM process, where the join's own CPU cost is the whole story; at distributed scale, shuffle/I/O/task-scheduling overhead dominates and the slowdown is within noise. So documentation alone looks sufficient here -- I've reworded the first trade-off in the config doc to describe which key families are more exposed without citing fixed ratios, since those numbers don't hold once the workload runs on a real cluster and risk reading as a stronger guarantee than they are. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
