LukaZdravic opened a new pull request, #58827: URL: https://github.com/apache/spark/pull/58827
### What changes were proposed in this pull request? Inside an ASOF JOIN `MATCH_CONDITION`, comparing two `ARRAY` operands required identical element types, so `array<int> >= array<bigint>` failed with `ASOF_JOIN_MATCH_CONDITION_INVALID_TYPE`. The same comparison outside ASOF works, because Spark coerces the element types. This PR makes `MATCH_CONDITION` array operands follow the same element-type rules as the comparison operator: - **Type check:** array elements now reuse the top-level operand rule (`MatchConditionTypes.areOperandsCompatible`), so coercible element types are accepted (via `TypeCoercion.findWiderTypeForTwo`), not only identical ones. Positional structs whose fields match by position but not name are still supported. - **Ordering:** the element-wise distance is built as a `zip_with`. Its lambda variables are created already-resolved, so `ResolveLambdaVariables` never re-types them; feeding two arrays of different element types would read one side through the wrong type. The fix widens both arrays to their common element type and casts them before building the `zip_with`, so both inputs and the lambda variables share one element type. The comparison (`asOfCondition`) is coerced by the analyzer's existing type-coercion pass, exactly like the plain comparison operator (the analyzed plan shows `cast(a as array<bigint>) >= a`). The single-pass resolver already defers array (lambda) ordering to the fixed-point analyzer, so one fix covers both analyzers. ### Why are the changes needed? A valid array as-of condition that is a legal comparison could not be written. `array<int>` vs `array<bigint>` (and other coercible element types) failed analysis even though the equivalent `>=` comparison succeeds. Reproduces with ANSI on and off. Before: ```sql SELECT * FROM (VALUES (array(1))) l(a) ASOF JOIN (VALUES (array(CAST(1 AS BIGINT)))) r(b) MATCH_CONDITION (l.a >= r.b); -- [ASOF_JOIN_MATCH_CONDITION_INVALID_TYPE] ... ``` After: the join runs and matches element-wise, like the comparison operator. ### Does this PR introduce _any_ user-facing change? Yes. ASOF JOIN `MATCH_CONDITION` now accepts `ARRAY` operands whose element types differ but are coercible (for example `array<int>` vs `array<bigint>`), matching the comparison operator. Previously these failed with `ASOF_JOIN_MATCH_CONDITION_INVALID_TYPE`. This also aligns array operands with the scalar path: `array<int>` vs `array<string>` is now accepted (string promotion), as scalar `int >= string` already was. Operands that were already valid are unchanged. ### How was this patch tested? - Unit tests in `AsOfJoinMatchConditionTypesSuite` for coercible and nested-coercible array elements, plus a negative case (`array<int>` vs `array<binary>`). - Analysis test in `AsOfJoinSQLSuite` asserting the join and its order expression resolve for `array<int>` vs `array<bigint>`. - Execution test in `AsOfJoinSortMergeSQLSuite` (ANSI enabled and disabled) verifying the correct match. - Golden case added in `join-asof-datatypes.sql`; `results` and `analyzer-results` regenerated. - Ran the three suites and the regenerated golden file locally: all pass. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
