Max Gekk created SPARK-60377:
--------------------------------
Summary: FilterExec hoists an IsNotNull check on a non-attribute
expression ahead of an earlier conjunct under whole-stage codegen
Key: SPARK-60377
URL: https://issues.apache.org/jira/browse/SPARK-60377
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Max Gekk
Since SPARK-51356, FilterExec under whole-stage codegen generates an IsNotNull
check on a non-attribute null-intolerant expression just before the first other
conjunct that contains the expression, in both
GeneratePredicateHelper.generatePredicateCode and the CSE branch of
FilterExec.doConsume. It does so even when the check is written after that
conjunct in the condition. If the conjunct is FALSE, And.eval never evaluates
the later check in interpreted mode, but the generated code evaluates it first,
so an error the check raises on the row is raised under codegen only.
Reproduction with ANSI on and Parquet filter pushdown off:
{code:sql}
CREATE TABLE t USING parquet AS SELECT * FROM VALUES (0, 1, 0, 1, 'x') AS v(a,
b, y, x, s);
SELECT a FROM t
WHERE NOT (a > 0 AND b > 0) AND coalesce(b, cast(s AS INT)) < 0 AND cast(s AS
INT) IS NOT NULL;
{code}
The first conjunct is TRUE and the second is FALSE, so the query returns no
rows with spark.sql.codegen.wholeStage=false. With whole-stage codegen it
raises CAST_INVALID_INPUT, because the check cast(s AS INT) IS NOT NULL is
generated before the second conjunct. This happens regardless of
spark.sql.legacy.filterShortCircuitOnNullConjunct.
SPARK-60315 (PR #59573) restricts the hoisting once a conjunct that can be NULL
has been generated, since the NULL-conjunct flag made the hoisted check
reachable on more rows. This ticket is for the remaining case: the nested
IsNotNull check should be hoisted only when it is written before the conjunct
in the condition, i.e. only the checks with an index below
precedingNotNulls(k), so that the generated code follows the evaluation order
of And.eval. The hoisting is then no longer needed for a check written after
the conjunct, which the leftover checks at the end of the condition cover.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
