LukaZdravic opened a new pull request, #59134:
URL: https://github.com/apache/spark/pull/59134
### What changes were proposed in this pull request?
SQL ASOF JOIN rebuilds each STRUCT operand of `MATCH_CONDITION` from its
fields, so both sides line up by position. The rebuilt struct is never NULL.
This PR keeps it NULL when the original struct is NULL, as
`If(IsNull(original), NULL, rebuilt)`, at every nesting level. Structs that can
never be NULL, like `(a, b)` tuples, are unchanged.
### Why are the changes needed?
A NULL operand must never match, but a NULL struct did:
```sql
SET spark.sql.join.asofJoin.enabled=true;
SELECT r.tag
FROM VALUES (CAST(NULL AS STRUCT<a: INT>), 'l') AS t(k, tag)
ASOF JOIN VALUES (CAST(NULL AS STRUCT<a: INT>), 'rnull'), (named_struct('a',
1), 'r1') AS r(k, tag)
MATCH_CONDITION (t.k >= r.k);
```
- Before: `rnull`. The NULL struct was rebuilt as `{a: NULL}`, which
compares as equal to another rebuilt NULL struct.
- After: no row, the same as a plain `WHERE t.k >= r.k`.
Nested structs had a second problem: `{e: NULL}` and `{e: {a: NULL}}` were
rebuilt as the same value, so `{e: {a: NULL}} > {e: NULL}` found no match.
### Does this PR introduce _any_ user-facing change?
No. SQL ASOF JOIN is not in a released version yet.
### How was this patch tested?
- `ResolveAsOfJoinSuite`: a nullable struct operand gets the NULL guard, and
a non-nullable one does not.
- `AsOfJoinSortMergeSQLSuite`: flat and nested NULL structs, with ANSI on
and off. It covers NULL on both sides, on the left only, on the right only,
LEFT ASOF JOIN, and the nested `{e: {a: NULL}} > {e: NULL}` case.
Without the fix, all three tests fail. The existing ASOF suites and the
`join-asof-*.sql` golden files pass, and the golden outputs do not change.
The forward form of the nested case (`<`) also needs SPARK-59829, because
forward joins still read the distance on master.
So waiting for https://github.com/apache/spark/pull/59102 to merge before
adding that one test also.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
