LuciferYang opened a new pull request, #60159:
URL: https://github.com/apache/spark/pull/60159
### What changes were proposed in this pull request?
This is a backport of #58881 to `branch-4.1`, the same change as #60121 for
`branch-4.3`.
`ExpandExec.doConsume` hands each output column that varies across its
projections to the consumer as `JavaCode.global` / `JavaCode.isNullGlobal`
instead of `JavaCode.variable` / `JavaCode.isNullVariable`. The value lives in
mutable state, a field or a slot of a compacted array such as
`expand_mutableStateArray_0[1]`, not in a local variable. See #58881 for the
full description.
Differences from the `master` commit:
- `branch-4.1` does not have SPARK-33301 or SPARK-59295, so there is no
`CompactedSlot` workaround to remove, no
`spark.sql.codegen.wholeStage.splitExpressions` conf, and no `collectInputs`.
This backport only changes `ExpandExec`.
- The testing check on method parameters (`assertMethodParameters`,
`checkParameterNames`) and its tests in `CollectInputsSuite` and
`WithExpressionEvalSuite` are not backported.
### Why are the changes needed?
A consumer that moves code into a method passes the local variables the code
reads as parameters. Taken for a local variable, the Expand's slot becomes a
parameter named `expand_mutableStateArray_0[1]`, which does not compile:
```sql
CREATE OR REPLACE TEMP VIEW t AS
SELECT concat('a', CAST(id AS STRING)) AS a, concat('b', CAST(id AS
STRING)) AS b FROM range(10);
SELECT c,
concat(upper(v), lower(v), reverse(v), trim(v), ltrim(v), rtrim(v),
initcap(v), repeat(v, 2), lpad(v, 10, 'x'), rpad(v, 10, 'y')) AS x,
concat(concat(upper(v), lower(v), reverse(v), trim(v), ltrim(v), rtrim(v),
initcap(v), repeat(v, 2), lpad(v, 10, 'x'), rpad(v, 10, 'y')), 'z') AS y
FROM t UNPIVOT (v FOR c IN (a, b));
```
Janino fails with `')' expected instead of '['`. The stage falls back to
running without whole-stage codegen, or the query fails with
`spark.sql.codegen.fallback=false`.
### Does this PR introduce _any_ user-facing change?
Yes. Such a stage now compiles under whole-stage codegen.
### How was this patch tested?
The `WholeStageCodegenSuite` tests from #58881, without the loop over
`spark.sql.codegen.wholeStage.splitExpressions` and without the case that
builds the methods from local variables. For the subexpression method (the
query above), the switch cases of the Expand that `RewriteDistinctAggregates`
adds, and the split aggregate functions, the method calls `lpad` on the slot,
the code compiles, and the rows match those computed without whole-stage
codegen. Without the change to `ExpandExec`, each of the three fails.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
