richardc-db opened a new pull request, #59244:
URL: https://github.com/apache/spark/pull/59244
### What changes were proposed in this pull request?
Under ANSI mode, `GetArrayItemUtil.computeNullabilityFromArray` (shared by
`ElementAt` and `GetArrayItem`) now also considers `ordinal.nullable` for a
non-foldable index, so `element_at(arr, idx)` and `arr[idx]` are nullable
whenever the index can be null.
With a non-nullable index the result keeps the existing non-nullable type,
so plans that do not involve a nullable index are unchanged.
### Why are the changes needed?
Both expressions are null-intolerant and return NULL for a NULL index, but
under ANSI they reported `nullable = false` whenever the array and its elements
were non-nullable (e.g. `array('a', 'b')`).
The optimizer trusts that flag: `NullPropagation` truncates a `Coalesce`
after its first non-nullable child, so a fallback such as
`coalesce(element_at(arr, idx), 'z')` is removed. The null then lands in a
column the plan treats as non-nullable, so the row carries no null marker and
readers see whatever is in the slot: the query silently returns a wrong element
(e.g. `'b'` instead of `'z'`) or fails with `INVALID_ARRAY_INDEX`.
### Does this PR introduce _any_ user-facing change?
Yes, a correctness fix. With `spark.sql.ansi.enabled=true`, `SELECT
coalesce(element_at(array('a', 'b'), CAST(nullif(id, 0) AS INT)), 'z') FROM
range(2)` previously returned `b, a`; it now returns `z, a`. `SELECT
coalesce(array('a', 'b')[CAST(nullif(id, 0) AS INT)], 'z') FROM range(2)`
previously failed with `INVALID_ARRAY_INDEX` (index -1); it now returns `z, b`.
`element_at` / `arr[idx]` with a nullable index now report a nullable result
type under ANSI, matching non-ANSI mode. Affects released versions since Spark
3.1 (SPARK-33386).
### How was this patch tested?
New assertions in `CollectionExpressionsSuite` (`ElementAt`) and
`ComplexTypeSuite` (`GetArrayItem`): a non-null array with a nullable index is
nullable, and with a non-nullable index stays non-nullable under ANSI.
New end-to-end test in `DataFrameFunctionsSuite` checking that
`coalesce(element_at(...), 'z')` and `coalesce(arr[...], 'z')` return `'z'` for
a null index under ANSI. All three tests fail without the fix and pass with it
(`build/sbt 'catalyst/testOnly *ComplexTypeSuite *CollectionExpressionsSuite'`,
`build/sbt 'sql/testOnly *DataFrameFunctionsSuite -- -z SPARK-59992'`).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
