[
https://issues.apache.org/jira/browse/SPARK-58428?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated SPARK-58428:
-----------------------------------
Labels: pull-request-available (was: )
> V2ExpressionBuilder loops forever on a FAILED_TO_EVALUATE-tagged foldable,
> hanging the optimizer during DSv2 pushdown
> ---------------------------------------------------------------------------------------------------------------------
>
> Key: SPARK-58428
> URL: https://issues.apache.org/jira/browse/SPARK-58428
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 4.1.0
> Reporter: Josh Rosen
> Priority: Major
> Labels: pull-request-available
>
> This is a report of an infinite loop optimizer hang present in Spark 4.1.0
> and later, reachable under default configuration during DSv2 pushdown.
> h3. Repro
> With stock configurations (ANSI mode on), run a pushdown-eligible query
> against a DSv2 source where a filter contains an expression that
> constant-folds to an error. This example uses a dummy JDBC source, but the
> same bug applies to other DSv2 sources, too:
> {code:java}
> SET
> spark.sql.catalog.d=org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog;
> SET spark.sql.catalog.d.url=jdbc:derby:memory:v2loopdb;
> SET spark.sql.catalog.d.driver=org.apache.derby.jdbc.EmbeddedDriver;
> SET spark.sql.catalog.d.create=true;
> CREATE NAMESPACE IF NOT EXISTS d.test;
> CREATE TABLE d.test.t (c INT);
> SELECT count(*) FROM d.test.t WHERE c = 1; -- control: 0
> SELECT * FROM d.test.t WHERE coalesce(c, 1 div 0) = 1; -- NEVER RETURNS{code}
> This gets stuck at an infinite loop at the following stack:
> {code:java}
> "main" #1 [5123] prio=5 os_prio=31 cpu=32747.43ms elapsed=409.55s
> tid=0x000000010324b5e0 nid=5123 runnable [0x000000016d585000]
> java.lang.Thread.State: RUNNABLE
> at
> org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.generateExpression(V2ExpressionBuilder.scala:98)
> at
> org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.$anonfun$generateExpressionWithNameByChildren$1(V2ExpressionBuilder.scala:465)
> at
> org.apache.spark.sql.catalyst.util.V2ExpressionBuilder$$Lambda/0x0000007802305cf8.apply(Unknown
> Source)
> at
> scala.collection.StrictOptimizedIterableOps.flatMap(StrictOptimizedIterableOps.scala:118)
> at
> scala.collection.StrictOptimizedIterableOps.flatMap$(StrictOptimizedIterableOps.scala:105)
> at scala.collection.immutable.Vector.flatMap(Vector.scala:116)
> at
> org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.generateExpressionWithNameByChildren(V2ExpressionBuilder.scala:465)
> at
> org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.generateExpressionWithName(V2ExpressionBuilder.scala:457)
> at
> org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.generateExpression(V2ExpressionBuilder.scala:152)
> at
> org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.generateExpression(V2ExpressionBuilder.scala:211)
> at
> org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.build(V2ExpressionBuilder.scala:39)
> at
> org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.buildPredicate(V2ExpressionBuilder.scala:43)
> at
> org.apache.spark.sql.execution.datasources.v2.PushablePredicate$.unapply(DataSourceV2Strategy.scala:1012)
> [...]
> {code}
> h3. Root cause
>
> {{V2ExpressionBuilder.generateExpression}} has [constant-folding
> logic|https://github.com/apache/spark/blob/8df89f20fc5c800ef4935c1b49a7525ec9df8921/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala#L96-L102]
> that recursively calls itself:
> {code:java}
> case _ if expr.contextIndependentFoldable
> &&
> SQLConf.get.getConfByKeyStrict[Boolean]("spark.sql.optimizer.datasourceV2ExprFolding")
> =>
> val constantExpr = ConstantFolding.constantFolding(expr)
> generateExpression(constantExpr, isPredicate) {code}
>
> If a node carries the {{FAILED_TO_EVALUATE}} tag then
> {{ConstantFolding.constantFolding}} passes it through unchanged, leading to
> an infinite loop because the nested {{generateExpression}} call is in a tail
> position and the Scala compiler's tail call elimination rewrites it to a jump.
>
> As a result, the loop continues indefinitely because it doesn't hit stack
> overflows or optimizer iteration limits.
> h3. Prior art on the suggested fix
> SPARK-50380 fixed the a similar "assumed a Literal came back" contract
> violation in {{{}ReorderAssociativeOperator{}}}: instead of assuming that
> {{.foldable}} implies that constant folding will return a {{{}Literal{}}}, it
> performs an explicit {{isInstanceOf}} check/match on the result. I think we
> should apply a similar check in V2ExpressionBuilder.generateExpression.
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
