dongjoon-hyun opened a new pull request, #59818:
URL: https://github.com/apache/spark/pull/59818
### What changes were proposed in this pull request?
`FileSourceScanLike.metadata` builds `PushedFilters` for display. The scalar
subqueries have not run yet at that point, so it replaces each one with the
string literal `ScalarSubquery#<id>` and translates the data filters to V1
filters. The `In` arm of the V1 translator
(`DataSourceStrategy.translateLeafNodeFilter`) accepts an `In` once all its
list items are literals, and converts them with the Scala converter of the
column type. For a `DATE`, `TIMESTAMP`, `TIMESTAMP_NTZ` or `DECIMAL` column,
converting the string throws `ClassCastException`.
This PR leaves out of `PushedFilters` a data filter that has an `In` with a
scalar subquery in its list. The filter is still pushed down at run time, where
`pushedDownFilters` replaces each subquery with a literal of its result.
Comparisons with scalar subqueries, like `GreaterThan(c,ScalarSubquery#1)`, are
displayed as before, since they convert a value by the type of the literal.
### Why are the changes needed?
With Parquet tables `t1(d DATE)` and `t2(d DATE)`, this query fails before
any job runs:
```sql
SELECT * FROM t1 WHERE d IN (DATE'2020-01-05', (SELECT max(d) FROM t2))
```
```
java.lang.ClassCastException: class org.apache.spark.unsafe.types.UTF8String
cannot be cast to class java.lang.Integer
(org.apache.spark.unsafe.types.UTF8String is in unnamed module of loader 'app';
java.lang.Integer is in module java.base of loader 'bootstrap')
at
org.apache.spark.sql.catalyst.CatalystTypeConverters$DateConverter$.toScala(CatalystTypeConverters.scala:425)
...
at
org.apache.spark.sql.execution.datasources.DataSourceStrategy$.translateLeafNodeFilter(DataSourceStrategy.scala:680)
...
at
org.apache.spark.sql.execution.FileSourceScanLike.metadata(DataSourceScanExec.scala:553)
...
at
org.apache.spark.sql.execution.QueryExecution.explainString(QueryExecution.scala:531)
at
org.apache.spark.sql.execution.SQLExecution$.planDescription(SQLExecution.scala:197)
...
at org.apache.spark.sql.classic.Dataset.collect(Dataset.scala:1623)
```
`SQLExecution` builds the plan description of a query, which shows
`PushedFilters`, when the query starts. `Dataset.explain` throws the same
exception, and SQL `EXPLAIN` prints `Error occurred during query planning`
instead of the plan. `NOT IN`, an `IN` under `OR`, and `d IN ((SELECT min(d)
FROM t2), (SELECT max(d) FROM t2))` with
`spark.sql.execution.reuseSubquery=false` fail the same way. With the default
`true`, the two subqueries of the last query are merged into one, and the `IN`
is not translated.
This fails since
[v4.0.0](https://github.com/apache/spark/releases/tag/v4.0.0) (2025-05-19),
which added the pushdown of data filters with scalar subqueries (SPARK-43402)
and their display as `ScalarSubquery#<id>` (SPARK-44655). Before that, such
filters were not given to the scan.
### Does this PR introduce _any_ user-facing change?
Yes. These queries run, and EXPLAIN shows their plans. `PushedFilters` no
longer lists a filter with an `IN` that has scalar subqueries, whatever the
column type. For example, `In(c, [5,ScalarSubquery#12])` on an `INT` column was
displayed before. The filter is still pushed down to the data source, and query
results do not change.
### How was this patch tested?
Added a test to `SubquerySuite`, "SPARK-60577: FileSourceScanExec supports
push down IN data filter with scalar subquery". On `DATE`, `TIMESTAMP` and
`DECIMAL` columns of Parquet tables, it runs an `IN` with a literal and a
subquery, and an `IN` with two subqueries under
`spark.sql.execution.reuseSubquery=false`. It checks that EXPLAIN (simple and
formatted) and the query run, that `PushedFilters` has no `In(`, and through
the scan's `numOutputRows` that the `IN` still skips the row group. Without the
main change, the test fails with `ClassCastException`.
These also pass locally: `SubquerySuite`, `ExplainSuite`, `ExplainSuiteAE`,
the TPC-DS and TPC-H plan stability suites (no approved plan changes), and the
`explain*.sql` cases of `SQLQueryTestSuite`.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
