Max Gekk created SPARK-61459:
--------------------------------
Summary: GeneratePredicate skips FilterExec's CSE gate: a filter
raises only with whole-stage codegen off
Key: SPARK-61459
URL: https://issues.apache.org/jira/browse/SPARK-61459
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 5.0.0
Reporter: Max Gekk
With whole-stage codegen off, FilterExec evaluates its condition through
Predicate.create (FilterEvaluatorFactory), which under the default codegen
factory mode builds a GeneratePredicate with subexpression elimination. Since
SPARK-60332, GeneratePredicate evaluates each common subexpression of the
top-level conjuncts just before the first conjunct whose tree contains it
(CodegenContext.generateConjunctsWithSubexprCodes / firstReferencingExpression).
With whole-stage codegen on, FilterExec.doConsume takes its subexpression
elimination branch (SPARK-56032) only when
spark.sql.subexpressionElimination.enabled and
spark.sql.subexpressionElimination.filterExec.enabled are on AND some common
subexpression is not CollapseProject.isCheap (basicPhysicalOperators.scala, the
condition before ctx.subexpressionEliminationForWholeStageCodegen). Otherwise
it takes the non-CSE path, where every conjunct short-circuits inside itself.
GeneratePredicate applies neither the isCheap test nor
spark.sql.subexpressionElimination.filterExec.enabled. So when a common
subexpression's first reference is in a conditionally evaluated part of its
first conjunct (here the right side of an OR), codegen off evaluates it on a
row where the OR short-circuits, while codegen on does not. Under ANSI mode an
error it raises fails the query only with whole-stage codegen off.
Reproducer (ANSI mode on; REPARTITION(1) keeps both rows in one Parquet file so
that row-group pruning on b < 0 cannot skip the first row):
{code:sql}
SET spark.sql.ansi.enabled=true;
CREATE TABLE t (s STRING, b INT, arr ARRAY<INT>) USING parquet;
INSERT INTO t SELECT /*+ REPARTITION(1) */ * FROM VALUES
('x', 1, array(1, 2)), ('y', -1, array(1, 2, 3, 4, 5, 6)) AS d(s, b, arr);
SET spark.sql.codegen.wholeStage=true;
SELECT * FROM t WHERE (s = 'x' OR arr[5] > 0) AND b < 0 AND arr[5] < 10;
-- y -1 [1,2,3,4,5,6]
SET spark.sql.codegen.wholeStage=false;
SELECT * FROM t WHERE (s = 'x' OR arr[5] > 0) AND b < 0 AND arr[5] < 10;
-- [INVALID_ARRAY_INDEX] The index 5 is out of bounds. The array has 2
elements. ... SQLSTATE: 22003
{code}
On the first row the OR is TRUE at s = 'x' and b < 0 is FALSE, so And.eval
never reaches arr[5]. arr[5] (GetArrayItem over a column and a literal) is
common to the OR conjunct and the last conjunct; it is an ExtractValue over
cheap children, so whole-stage codegen does no elimination, while
GeneratePredicate evaluates it before the OR conjunct.
Observed on master at 183ccbe75d4 (ANSI on, same data; the arr[5] rows were run
both on the table and on a temp view of the same rows, the arr[b + 6] rows on
the temp view only):
||query||wholeStage||filterExec.enabled||subexpressionElimination.enabled||codegen.factoryMode||result||
|cheap arr[5]|true|true|true|FALLBACK|1 row|
|cheap arr[5]|true|false|true|FALLBACK|1 row|
|cheap arr[5]|false|true|true|FALLBACK|INVALID_ARRAY_INDEX|
|cheap arr[5]|false|false|true|FALLBACK|INVALID_ARRAY_INDEX|
|cheap arr[5]|false|true|false|FALLBACK|1 row|
|cheap arr[5]|false|true|true|NO_CODEGEN|1 row|
|non-cheap arr[b + 6]|true|true|true|FALLBACK|INVALID_ARRAY_INDEX|
|non-cheap arr[b + 6]|true|false|true|FALLBACK|1 row|
|non-cheap arr[b + 6]|false|true|true|FALLBACK|INVALID_ARRAY_INDEX|
|non-cheap arr[b + 6]|false|false|true|FALLBACK|INVALID_ARRAY_INDEX|
|non-cheap arr[b + 6]|false|true|false|FALLBACK|1 row|
The non-cheap query is (s = 'x' OR arr[b + 6] > 0) AND b < 0 AND arr[b + 6] <
10. There the CSE branch is taken with whole-stage codegen on and both modes
raise; that part is SPARK-60774 (first reference in a conditional branch,
evaluated before the conjunct in both generators). The disagreement reported
here is the one between the two modes: with a cheap common subexpression, or
with spark.sql.subexpressionElimination.filterExec.enabled=false, whole-stage
codegen does no elimination and answers, while GeneratePredicate still
eliminates and raises. As a control, SELECT * FROM t WHERE arr[5] < 10 raises
INVALID_ARRAY_INDEX in every configuration above.
Suggested direction: apply the same gate when GeneratePredicate builds a filter
condition, e.g. move FilterExec's test
(subexpressionElimination.filterExec.enabled and a non-cheap common
subexpression of the non-null-test conjuncts) into GeneratePredicateHelper and
use it both in FilterExec.doConsume and when FilterEvaluatorFactory creates the
predicate, so that the predicate is generated without elimination whenever
whole-stage codegen would take the non-CSE path. A fix of SPARK-60774 would
likely also remove this particular error, but the two modes would still
disagree on which subexpressions are eliminated in a filter condition.
Workaround: spark.sql.subexpressionElimination.enabled=false.
Related: SPARK-60332, SPARK-56032, SPARK-60774, SPARK-61197.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]