andygrove commented on PR #2199: URL: https://github.com/apache/datafusion-ballista/pull/2199#issuecomment-5099646635
For reviewers, it may be useful to compare this with how Spark handles the same problem. Spark plans single column `NOT IN` as a null-aware anti join (SPARK-32290). It broadcasts the subquery side as a special `HashedRelation` that carries two flags computed while building the broadcast: whether the subquery is empty and whether it contains a NULL key. Each streamed partition of the outer table then applies purely local logic. The subquery is empty, so emit the row. The subquery has a NULL, or my key is NULL, or my key matches, so drop the row. No cross partition coordination is needed and the outer table stays fully distributed. This PR reaches the same place but at the plan level instead of inside the join operator. The one row `count(*)` / `count(b)` aggregate captures exactly the two global facts Spark bakes into its broadcast relation, and the plain anti join takes the place of Spark's special null-aware probe logic. The filter over the cross join is the same local decision Spark makes per streamed row. A few practical differences: - Spark makes one pass over the subquery, since the flags are computed while building the broadcast. This rewrite scans the subquery twice, once for the anti join and once for the aggregate. Both scans are cheap relative to the join itself and both distribute. - Spark's null-aware anti join requires the subquery to be broadcastable and falls back to `BroadcastNestedLoopJoin` otherwise. The rewritten plan here has no broadcast requirement at all, since the anti join can shuffle on the key, so large subqueries are fine. - Both approaches are limited to single column `NOT IN`. Multi column predicates fall through to the existing paths. The closest structural match to Spark would be a null-aware join that builds on the subquery side, which is the upstream DataFusion improvement tracked as option 1 in #2198. If that lands, the special casing here can eventually go away and the swap optimizations would produce Spark's shape natively. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
