Josh Rosen created SPARK-58428:
----------------------------------
Summary: V2ExpressionBuilder loops forever on a
FAILED_TO_EVALUATE-tagged foldable, hanging the optimizer during DSv2 pushdown
Key: SPARK-58428
URL: https://issues.apache.org/jira/browse/SPARK-58428
Project: Spark
Issue Type: Bug
Components: SQL
Affects Versions: 4.1.0
Reporter: Josh Rosen
This is a report of an infinite loop optimizer hang present in Spark 4.1.0 and
later, reachable under default configuration during DSv2 pushdown.
h3. Repro
With stock configurations (ANSI mode on), run a pushdown-eligible query against
a DSv2 source where a filter contains an expression that constant-folds to an
error. This example uses a dummy JDBC source, but the same bug applies to other
DSv2 sources, too:
{code:java}
SET
spark.sql.catalog.d=org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog;
SET spark.sql.catalog.d.url=jdbc:derby:memory:v2loopdb;
SET spark.sql.catalog.d.driver=org.apache.derby.jdbc.EmbeddedDriver;
SET spark.sql.catalog.d.create=true;
CREATE NAMESPACE IF NOT EXISTS d.test;
CREATE TABLE d.test.t (c INT);
SELECT count(*) FROM d.test.t WHERE c = 1; -- control: 0
SELECT * FROM d.test.t WHERE coalesce(c, 1 div 0) = 1; -- NEVER RETURNS{code}
This gets stuck at an infinite loop at the following stack:
{code:java}
"main" #1 [5123] prio=5 os_prio=31 cpu=32747.43ms elapsed=409.55s
tid=0x000000010324b5e0 nid=5123 runnable [0x000000016d585000]
java.lang.Thread.State: RUNNABLE
at
org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.generateExpression(V2ExpressionBuilder.scala:98)
at
org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.$anonfun$generateExpressionWithNameByChildren$1(V2ExpressionBuilder.scala:465)
at
org.apache.spark.sql.catalyst.util.V2ExpressionBuilder$$Lambda/0x0000007802305cf8.apply(Unknown
Source)
at
scala.collection.StrictOptimizedIterableOps.flatMap(StrictOptimizedIterableOps.scala:118)
at
scala.collection.StrictOptimizedIterableOps.flatMap$(StrictOptimizedIterableOps.scala:105)
at scala.collection.immutable.Vector.flatMap(Vector.scala:116)
at
org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.generateExpressionWithNameByChildren(V2ExpressionBuilder.scala:465)
at
org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.generateExpressionWithName(V2ExpressionBuilder.scala:457)
at
org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.generateExpression(V2ExpressionBuilder.scala:152)
at
org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.generateExpression(V2ExpressionBuilder.scala:211)
at
org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.build(V2ExpressionBuilder.scala:39)
at
org.apache.spark.sql.catalyst.util.V2ExpressionBuilder.buildPredicate(V2ExpressionBuilder.scala:43)
at
org.apache.spark.sql.execution.datasources.v2.PushablePredicate$.unapply(DataSourceV2Strategy.scala:1012)
[...]
{code}
h3. Root cause
{{V2ExpressionBuilder.generateExpression}} has [constant-folding
logic|https://github.com/apache/spark/blob/8df89f20fc5c800ef4935c1b49a7525ec9df8921/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/util/V2ExpressionBuilder.scala#L96-L102]
that recursively calls itself:
{code:java}
case _ if expr.contextIndependentFoldable
&&
SQLConf.get.getConfByKeyStrict[Boolean]("spark.sql.optimizer.datasourceV2ExprFolding")
=>
val constantExpr = ConstantFolding.constantFolding(expr)
generateExpression(constantExpr, isPredicate) {code}
If a node carries the {{FAILED_TO_EVALUATE}} tag then
{{ConstantFolding.constantFolding}} passes it through unchanged, leading to an
infinite loop because the nested {{generateExpression}} call is in a tail
position and the Scala compiler's tail call elimination rewrites it to a jump.
As a result, the loop continues indefinitely because it doesn't hit stack
overflows or optimizer iteration limits.
h3. Prior art on the suggested fix
SPARK-50380 fixed the a similar "assumed a Literal came back" contract
violation in {{{}ReorderAssociativeOperator{}}}: instead of assuming that
{{.foldable}} implies that constant folding will return a {{{}Literal{}}}, it
performs an explicit {{isInstanceOf}} check/match on the result. I think we
should apply a similar check in V2ExpressionBuilder.generateExpression.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
