cloud-fan commented on code in PR #58404:
URL: https://github.com/apache/spark/pull/58404#discussion_r3897664546
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/PushDownLeftSemiAntiJoin.scala:
##########
@@ -75,7 +74,11 @@ object PushDownLeftSemiAntiJoin extends Rule[LogicalPlan]
val makeJoinCondition = (predicates: Seq[Expression]) => {
replaceAlias(predicates.reduce(And), aliasMap)
}
- pushDownJoin(join, canPushDownPredicate, makeJoinCondition)
+ pushDownJoin(
+ join,
+ canPushDownPredicate,
+ makeJoinCondition,
+ canPlanAsBroadcastHashJoin(_, conf))
Review Comment:
Addressed in the current revision. Aggregate pushdown now keeps the original
`canPlanAsBroadcastHashJoin(join, conf)` guard and also validates `pushedJoin`.
The new CBO stats test covers the case where the original join has a
`BuildLeft` fallback that must block pushdown.
##########
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/optimizer/joins.scala:
##########
@@ -289,6 +291,20 @@ case object BuildLeft extends BuildSide
trait JoinSelectionHelper extends Logging {
+ // Keep this synchronized with BroadcastExchangeExec's non-Long hashed
relation limit.
+ private val maxBroadcastHashRows = (BytesToBytesMap.MAX_CAPACITY /
1.5).toLong
+
+ private def canBuildNullAwareAntiJoinHashRelation(
+ rightKeys: Seq[Expression],
+ right: LogicalPlan,
+ conf: SQLConf): Boolean = {
+ val usesLongHashedRelation =
+ rightKeys.length == 1 &&
rightKeys.head.dataType.isInstanceOf[IntegralType]
+ usesLongHashedRelation ||
right.stats.rowCount.fold(canBroadcastBySize(right, conf)) {
+ _ < maxBroadcastHashRows
+ }
Review Comment:
Addressed in the current revision. The row-count-based expansion has been
removed: the specialized hash path now requires the right side to satisfy
`canBroadcastBySize`, while threshold-zero and over-threshold cases retain the
identity-broadcast nested-loop fallback.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
