LuciferYang commented on PR #58424: URL: https://github.com/apache/spark/pull/58424#issuecomment-5548572378
The rewrite has no cost gate: it fires on plan shape alone. A small dimension table's self-join currently plans as a broadcast hash join with no shuffle exchange; rewriting it to COUNT(DISTINCT) adds a hashpartitioning(equiKeys) exchange plus a two-phase aggregate per qualifying subquery, which the Q95-style high-multiplicity win does not offset. With the flag enabled globally, queries over small tables regress across the board. Once the shape and self-join guards pass, skip the rewrite when sjLeft.stats.sizeInBytes is at or below autoBroadcastJoinThreshold. Without ANALYZE that number is the raw file size, the same one JoinSelection's static broadcast check uses, and both sides are the same relation so one check suffices. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
