[
https://issues.apache.org/jira/browse/SPARK-59954?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Dongjoon Hyun updated SPARK-59954:
----------------------------------
Affects Version/s: 4.4.0
(was: 5.0.0)
> DECLARE VARIABLE fails with INTERNAL_ERROR when the default value applies
> NULLIF or BETWEEN to a non-foldable operand
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-59954
> URL: https://issues.apache.org/jira/browse/SPARK-59954
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.4.0
> Reporter: Yang Jie
> Assignee: Yang Jie
> Priority: Major
> Labels: pull-request-available
>
> A DECLARE VARIABLE whose default value applies NULLIF or BETWEEN to a
> non-foldable operand fails at planning:
> {code:sql}
> DECLARE OR REPLACE VARIABLE v1 DOUBLE DEFAULT nullif(rand(), 0.5);
> DECLARE OR REPLACE VARIABLE v2 BOOLEAN DEFAULT rand() BETWEEN 0.1 AND 0.9;
> {code}
> {noformat}
> [INTERNAL_ERROR] The Spark SQL phase planning failed with an internal error.
> ... SQLSTATE: XX000
> Caused by: java.lang.AssertionError: assertion failed: No plan for
> CreateVariable default(if ((_common_expr_0#0 = 0.5)) null else
> _common_expr_0#0) ...
> +- Project [rand(...) AS _common_expr_0#0]
> +- ResolvedIdentifier ..., session.v1
> {noformat}
> A deterministic Scala UDF in place of rand() fails the same way. nullif(1, 3)
> runs, since its operand is foldable and the With is inlined.
> NULLIF and BETWEEN become a With that references its operand twice.
> RewriteWithExpression pre-evaluates such a definition in a Project over the
> first child whose output covers the definition's references. The children of
> CreateVariable are the ResolvedIdentifier of each variable name, and a
> default value references no column, so the definition is put in a Project
> over the first ResolvedIdentifier. V2CommandStrategy only plans a
> CreateVariable whose children are all ResolvedIdentifier, so the planner
> finds no plan for it.
> Reproduced on master and on branch-4.2.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
