AbhinavBattu opened a new pull request, #57974:
URL: https://github.com/apache/spark/pull/57974
### What changes were proposed in this pull request?
`V2ExpressionBuilder.generateExpression` handles a context independent
foldable
expression by constant folding it and recursing on the result, so that the
folded
value is translated by the `Literal` case:
```scala
case _ if expr.contextIndependentFoldable && <datasourceV2ExprFolding> =>
val constantExpr = ConstantFolding.constantFolding(expr)
generateExpression(constantExpr, isPredicate)
```
That assumes constant folding returns a literal. It does not.
`ConstantFolding`
returns an expression unchanged when it carries the `FAILED_TO_EVALUATE`
tag, which
is set when evaluation failed inside a conditional branch so that the error
is not
raised at planning time for a branch that may never be reached. The
recursive call
then re-enters the same case with the same expression and never makes
progress.
Because the call is in tail position it loops instead of raising
`StackOverflowError`, so the symptom is a query that never returns.
This PR recurses only when folding actually changed the expression.
This follows SPARK-50380, which made `ReorderAssociativeOperator` stop
assuming that
a foldable expression folds to a literal.
The guard is on the folding result rather than on the tag, so it also covers
any
future case where `constantFolding` returns its input unchanged.
### Why are the changes needed?
With default settings, a pushdown eligible query against a DSv2 source hangs
when a
filter contains an expression that fails to evaluate inside a conditional
branch:
```sql
SET
spark.sql.catalog.d=org.apache.spark.sql.execution.datasources.v2.jdbc.JDBCTableCatalog;
SET spark.sql.catalog.d.url=jdbc:derby:memory:v2loopdb;
SET spark.sql.catalog.d.driver=org.apache.derby.jdbc.EmbeddedDriver;
SET spark.sql.catalog.d.create=true;
CREATE NAMESPACE IF NOT EXISTS d.test;
CREATE TABLE d.test.t (c INT);
SELECT count(*) FROM d.test.t WHERE c = 1; -- control: 0
SELECT * FROM d.test.t WHERE coalesce(c, 1 div 0) = 1; -- never returns
```
Four conditions have to hold, which is why this was not hit earlier:
- ANSI mode, so `1 div 0` throws while the planner folds it
- the failing expression inside a conditional, so the error is deferred and
tagged
rather than raised
- a DSv2 source that pushes predicates, since `V2ExpressionBuilder` is only
used on
that path
- `spark.sql.optimizer.datasourceV2ExprFolding`, which defaults to true and
was
added in 4.1.0
Setting `spark.sql.optimizer.datasourceV2ExprFolding` to false avoids the
hang.
### Does this PR introduce _any_ user-facing change?
Yes. A query of the shape above previously hung and now completes. Such an
expression is no longer translated, so the predicate is not pushed to the
source and
is evaluated by Spark instead. Query results are unchanged.
### How was this patch tested?
Added a test to `DataSourceV2StrategySuite`, next to the existing
`datasourceV2ExprFolding` test. It builds `coalesce(c, 1 div 0) = 1`,
asserts that
constant folding tagged the failing branch, and then asserts that translation
produces no V2 expression. The tag assertion is there so that the test
cannot pass
for the wrong reason if the tagging behaviour changes.
Without this change the test does not terminate. The recursion is in tail
position
and does not block, so it cannot be interrupted by a time limit, which is
why the
test asserts the result instead of using `failAfter`. This matches existing
tests
for similar issues, for example "SPARK-48843: Prevent infinite loop with
BindParameters" in `ParametersSuite`.
```
build/sbt 'sql/testOnly *DataSourceV2StrategySuite *JDBCV2Suite'
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
