[
https://issues.apache.org/jira/browse/SPARK-60021?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18124673#comment-18124673
]
Max Gekk commented on SPARK-60021:
----------------------------------
SPARK-33301 (https://github.com/apache/spark/pull/59225) adds a workaround for
this on master: a slot of a compacted mutable state array counts as a field
when code moves into a method (CompactedSlot, CodegenContext.isCompactedSlot,
the slot clause in collectInputs and the slot cases in CollectInputsSuite).
Once the ExpandExec outputs are labelled as globals, the workaround is
redundant: if it is on master by then, please remove it in the same change.
ExpandExec looked like the only producer that hands out such a slot as a local
variable (JavaCode.variable/isNullVariable built from an addMutableState name),
which is worth re-checking.
> Label ExpandExec's outputs held in mutable state as global fields in codegen
> ----------------------------------------------------------------------------
>
> Key: SPARK-60021
> URL: https://issues.apache.org/jira/browse/SPARK-60021
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.2.0, 4.3.0, 4.1.3
> Reporter: Max Gekk
> Priority: Major
>
> Since SPARK-35329, ExpandExec keeps its outputs in mutable state fields
> (addMutableState), which may be compacted into an array slot such as
> expand_mutableStateArray_0[k] for non-primitive types. But it labels them as
> local variables, JavaCode.isNullVariable(isNull) and JavaCode.variable(value,
> ...) in ExpandExec.doConsume, while every other producer that keeps its
> outputs in fields labels them isNullGlobal/global.
> As a result, CodegenContext.getLocalInputVariableValues takes such a slot as
> a local input, and a method split out of the stage (e.g. a subexpression
> elimination method) declares it as a parameter. No Java parameter can be
> named "array[i]", so the generated code fails to compile and the stage falls
> back to the non-codegen path (or fails when the fallback is off). For
> example, with Janino:
> ')' expected instead of '['
> Example query:
> CREATE TEMP VIEW t AS SELECT concat('a', CAST(id AS STRING)) AS a,
> concat('b', CAST(id AS STRING)) AS b FROM range(10);
> SELECT c,
> concat(upper(v), lower(v), reverse(v), trim(v), ltrim(v), rtrim(v),
> initcap(v), repeat(v, 2), lpad(v, 10, 'x'), rpad(v, 10, 'y')) AS x,
> concat(concat(upper(v), lower(v), reverse(v), trim(v), ltrim(v),
> rtrim(v), initcap(v), repeat(v, 2), lpad(v, 10, 'x'), rpad(v, 10, 'y')), 'z')
> AS y
> FROM t UNPIVOT (v FOR c IN (a, b));
> Released 4.1.3, 4.2.0 and 4.3.0 fail the same way.
> Proposed fix: label ExpandExec's outputs as JavaCode.isNullGlobal /
> JavaCode.global, so methods split out of the stage read them as fields. This
> came up in the review of https://github.com/apache/spark/pull/59225
> (SPARK-33301), which currently works around it with a compacted-slot rule;
> once this is fixed, that PR can drop the workaround.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
