Max Gekk created SPARK-60021:
--------------------------------
Summary: Label ExpandExec's outputs held in mutable state as
global fields in codegen
Key: SPARK-60021
URL: https://issues.apache.org/jira/browse/SPARK-60021
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Max Gekk
Since SPARK-35329, ExpandExec keeps its outputs in mutable state fields
(addMutableState), which may be compacted into an array slot such as
expand_mutableStateArray_0[k] for non-primitive types. But it labels them as
local variables, JavaCode.isNullVariable(isNull) and JavaCode.variable(value,
...) in ExpandExec.doConsume, while every other producer that keeps its outputs
in fields labels them isNullGlobal/global.
As a result, CodegenContext.getLocalInputVariableValues takes such a slot as a
local input, and a method split out of the stage (e.g. a subexpression
elimination method) declares it as a parameter. No Java parameter can be named
"array[i]", so the generated code fails to compile and the stage falls back to
the non-codegen path (or fails when the fallback is off). For example, with
Janino:
')' expected instead of '['
Example query:
CREATE TEMP VIEW t AS SELECT concat('a', CAST(id AS STRING)) AS a,
concat('b', CAST(id AS STRING)) AS b FROM range(10);
SELECT c,
concat(upper(v), lower(v), reverse(v), trim(v), ltrim(v), rtrim(v),
initcap(v), repeat(v, 2), lpad(v, 10, 'x'), rpad(v, 10, 'y')) AS x,
concat(concat(upper(v), lower(v), reverse(v), trim(v), ltrim(v), rtrim(v),
initcap(v), repeat(v, 2), lpad(v, 10, 'x'), rpad(v, 10, 'y')), 'z') AS y
FROM t UNPIVOT (v FOR c IN (a, b));
Released 4.1.3, 4.2.0 and 4.3.0 fail the same way.
Proposed fix: label ExpandExec's outputs as JavaCode.isNullGlobal /
JavaCode.global, so methods split out of the stage read them as fields. This
came up in the review of https://github.com/apache/spark/pull/59225
(SPARK-33301), which currently works around it with a compacted-slot rule; once
this is fixed, that PR can drop the workaround.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
