Max Gekk created SPARK-61459:
--------------------------------

             Summary: GeneratePredicate skips FilterExec's CSE gate: a filter 
raises only with whole-stage codegen off
                 Key: SPARK-61459
                 URL: https://issues.apache.org/jira/browse/SPARK-61459
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 5.0.0
            Reporter: Max Gekk


With whole-stage codegen off, FilterExec evaluates its condition through 
Predicate.create (FilterEvaluatorFactory), which under the default codegen 
factory mode builds a GeneratePredicate with subexpression elimination. Since 
SPARK-60332, GeneratePredicate evaluates each common subexpression of the 
top-level conjuncts just before the first conjunct whose tree contains it 
(CodegenContext.generateConjunctsWithSubexprCodes / firstReferencingExpression).

With whole-stage codegen on, FilterExec.doConsume takes its subexpression 
elimination branch (SPARK-56032) only when 
spark.sql.subexpressionElimination.enabled and 
spark.sql.subexpressionElimination.filterExec.enabled are on AND some common 
subexpression is not CollapseProject.isCheap (basicPhysicalOperators.scala, the 
condition before ctx.subexpressionEliminationForWholeStageCodegen). Otherwise 
it takes the non-CSE path, where every conjunct short-circuits inside itself.

GeneratePredicate applies neither the isCheap test nor 
spark.sql.subexpressionElimination.filterExec.enabled. So when a common 
subexpression's first reference is in a conditionally evaluated part of its 
first conjunct (here the right side of an OR), codegen off evaluates it on a 
row where the OR short-circuits, while codegen on does not. Under ANSI mode an 
error it raises fails the query only with whole-stage codegen off.

Reproducer (ANSI mode on; REPARTITION(1) keeps both rows in one Parquet file so 
that row-group pruning on b < 0 cannot skip the first row):

{code:sql}
SET spark.sql.ansi.enabled=true;
CREATE TABLE t (s STRING, b INT, arr ARRAY<INT>) USING parquet;
INSERT INTO t SELECT /*+ REPARTITION(1) */ * FROM VALUES
  ('x', 1, array(1, 2)), ('y', -1, array(1, 2, 3, 4, 5, 6)) AS d(s, b, arr);

SET spark.sql.codegen.wholeStage=true;
SELECT * FROM t WHERE (s = 'x' OR arr[5] > 0) AND b < 0 AND arr[5] < 10;
-- y  -1  [1,2,3,4,5,6]

SET spark.sql.codegen.wholeStage=false;
SELECT * FROM t WHERE (s = 'x' OR arr[5] > 0) AND b < 0 AND arr[5] < 10;
-- [INVALID_ARRAY_INDEX] The index 5 is out of bounds. The array has 2 
elements. ... SQLSTATE: 22003
{code}

On the first row the OR is TRUE at s = 'x' and b < 0 is FALSE, so And.eval 
never reaches arr[5]. arr[5] (GetArrayItem over a column and a literal) is 
common to the OR conjunct and the last conjunct; it is an ExtractValue over 
cheap children, so whole-stage codegen does no elimination, while 
GeneratePredicate evaluates it before the OR conjunct.

Observed on master at 183ccbe75d4 (ANSI on, same data; the arr[5] rows were run 
both on the table and on a temp view of the same rows, the arr[b + 6] rows on 
the temp view only):

||query||wholeStage||filterExec.enabled||subexpressionElimination.enabled||codegen.factoryMode||result||
|cheap arr[5]|true|true|true|FALLBACK|1 row|
|cheap arr[5]|true|false|true|FALLBACK|1 row|
|cheap arr[5]|false|true|true|FALLBACK|INVALID_ARRAY_INDEX|
|cheap arr[5]|false|false|true|FALLBACK|INVALID_ARRAY_INDEX|
|cheap arr[5]|false|true|false|FALLBACK|1 row|
|cheap arr[5]|false|true|true|NO_CODEGEN|1 row|
|non-cheap arr[b + 6]|true|true|true|FALLBACK|INVALID_ARRAY_INDEX|
|non-cheap arr[b + 6]|true|false|true|FALLBACK|1 row|
|non-cheap arr[b + 6]|false|true|true|FALLBACK|INVALID_ARRAY_INDEX|
|non-cheap arr[b + 6]|false|false|true|FALLBACK|INVALID_ARRAY_INDEX|
|non-cheap arr[b + 6]|false|true|false|FALLBACK|1 row|

The non-cheap query is (s = 'x' OR arr[b + 6] > 0) AND b < 0 AND arr[b + 6] < 
10. There the CSE branch is taken with whole-stage codegen on and both modes 
raise; that part is SPARK-60774 (first reference in a conditional branch, 
evaluated before the conjunct in both generators). The disagreement reported 
here is the one between the two modes: with a cheap common subexpression, or 
with spark.sql.subexpressionElimination.filterExec.enabled=false, whole-stage 
codegen does no elimination and answers, while GeneratePredicate still 
eliminates and raises. As a control, SELECT * FROM t WHERE arr[5] < 10 raises 
INVALID_ARRAY_INDEX in every configuration above.

Suggested direction: apply the same gate when GeneratePredicate builds a filter 
condition, e.g. move FilterExec's test 
(subexpressionElimination.filterExec.enabled and a non-cheap common 
subexpression of the non-null-test conjuncts) into GeneratePredicateHelper and 
use it both in FilterExec.doConsume and when FilterEvaluatorFactory creates the 
predicate, so that the predicate is generated without elimination whenever 
whole-stage codegen would take the non-CSE path. A fix of SPARK-60774 would 
likely also remove this particular error, but the two modes would still 
disagree on which subexpressions are eliminated in a filter condition.

Workaround: spark.sql.subexpressionElimination.enabled=false.

Related: SPARK-60332, SPARK-56032, SPARK-60774, SPARK-61197.




--
This message was sent by Atlassian Jira
(v10.3.26#10030026)

---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]

Reply via email to