dongjoon-hyun opened a new pull request, #59963:
URL: https://github.com/apache/spark/pull/59963
### What changes were proposed in this pull request?
This PR aims to fix the generated code of `ElementAt` and `GetArrayItem` to
assign `isNull` only when the expression is nullable.
`BinaryExpression.nullSafeCodeGen` declares `ev.isNull` only for a nullable
expression. When the child is `CreateArray`, or `GetArrayStructFields` over it,
and the index is a literal, `ElementAt` and `GetArrayItem` can be non-nullable
because:
- the index selects an element that cannot be NULL,
- the index is out of bounds under ANSI mode, which raises an error, or
- the index of `element_at` is 0, which raises an error.
However, `doGenCode` generates `isNull = true` regardless of `nullable` when
the array can contain NULL, and in the non-ANSI out-of-bounds branch. So, the
generated Java code assigns to an undeclared variable and fails to compile.
```java
UTF8String filter_value_1 = null;
int filter_elementAtIndex_0 =
org.apache.spark.sql.catalyst.expressions.ArrayExpressionUtils.resolveArrayIndex(
filter_arrayData_0.numElements(), (int) 5,
((org.apache.spark.sql.catalyst.trees.SQLQueryContext) references[1] /* errCtx
*/));
if (filter_arrayData_0.isNullAt(filter_elementAtIndex_0)) {
filter_isNull_1 = true; // This variable is not declared.
} else {
filter_value_1 = filter_arrayData_0.getUTF8String(filter_elementAtIndex_0);
}
```
The nullability is correct and this PR does not change it. When the
expression is not nullable, the removed assignments are unreachable.
### Why are the changes needed?
The code generation has failed since Apache Spark
[3.0.0](https://github.com/apache/spark/releases/tag/v3.0.0) (2020-06-06) for a
selected element that cannot be NULL (SPARK-26965), and since Apache Spark
[3.1.1](https://github.com/apache/spark/releases/tag/v3.1.1) (2021-02-22) for
an out-of-bounds index under ANSI mode (SPARK-33386).
Most operators fall back to the interpreted evaluation, which disables
whole-stage codegen and logs a compilation error per task. The ordering of
`TakeOrderedAndProjectExec` and the predicates of `MergeRowsExec` have no such
fallback, so `ORDER BY ... LIMIT` and `MERGE INTO` queries fail.
### Does this PR introduce _any_ user-facing change?
Yes. Queries that failed with `CompileException` now succeed. For example:
```sql
SELECT id FROM range(3) ORDER BY element_at(array(id, nullif(id, 0)), 1)
DESC LIMIT 1;
```
**BEFORE** (Apache Spark
[4.3.0](https://github.com/apache/spark/releases/tag/v4.3.0) (2026-10-27))
```
org.codehaus.commons.compiler.CompileException: File 'generated.java', Line
51, Column 1: Expression "isNull_0" is not an rvalue
```
**AFTER**
```
2
```
Queries that used the fallback return the same results, without the
compilation errors in the logs.
### How was this patch tested?
Pass the CIs with the newly added test cases. Each of them fails without
this fix.
```
$ build/sbt 'catalyst/testOnly *CollectionExpressionsSuite *ComplexTypeSuite
-- -z "generates code when the result is not nullable"'
$ build/sbt 'sql/testOnly *DataFrameFunctionsSuite -- -z "compile when the
result is not nullable"'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
