WangGuangxin opened a new pull request, #12922:
URL: https://github.com/apache/gluten/pull/12922
## What changes are proposed in this pull request?
This PR adds Velox support for `BroadcastNestedLoopJoinExec` in `FULL
OUTER JOIN` cases by rewriting the Spark physical plan into a composition that
Velox already supports, instead of falling back the original
full outer BNLJ.
The rewrite transforms:
- `BroadcastNestedLoopJoinExec(..., FullOuter, ...)`
into:
- one outer BNLJ branch that produces matched rows plus unmatched
streamed-side rows
- one outer BNLJ branch with a synthetic non-null marker column to
identify unmatched broadcast-side rows
- a `Union` to combine both branches
This avoids introducing native Velox full outer nested-loop join support
while preserving full outer join semantics, including null-sensitive cases.
In addition, this PR adds a conservative size-based guard for the rewrite:
- the rewrite is applied only when both logical join sides have known
statistics
- both sides must be at or below
`spark.gluten.sql.columnar.broadcastNestedLoopJoin.fullOuterRewriteThreshold`
Otherwise, Gluten keeps the original behavior and lets the plan fall back
instead of forcing the rewrite on larger joins.
## Why are the changes needed?
Velox does not support Spark's `FULL OUTER` broadcast nested loop join
end-to-end today, so Gluten falls back for these queries.
A direct native implementation is not a good short-term option because:
- the existing Velox/Gluten BNLJ pipeline is built around a broadcast
build side
- full outer semantics require emitting unmatched rows from both sides
- handling unmatched broadcast-side rows natively would need extra
protocol/state beyond the current execution model
A rewrite-based solution is lower risk because it reuses operators already
supported by Velox.
However, rewriting every full outer BNLJ into two joins can be more
expensive than a single Spark row-based fallback join. To keep the feature
safe, this PR only enables the rewrite for small, stats-known joins
and preserves fallback for larger or unknown-size cases.
## How was this patch tested?
More UT
## Config
A new config is introduced:
-
`spark.gluten.sql.columnar.broadcastNestedLoopJoin.fullOuterRewriteThreshold`
It controls the maximum per-side logical plan size for enabling the full
outer BNLJ rewrite.
## Was this patch authored or co-authored using generative AI tooling?
Yes. Generated-by: GPT-5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
