Vivek1106-04 opened a new pull request, #58912:
URL: https://github.com/apache/spark/pull/58912
### What changes were proposed in this pull request?
Type coercion of the compared sides may wrap the filtered column in a cast,
e.g.
`cast(part_col as bigint) = <scalar subquery>` when an INT partition column
is compared with a
BIGINT scalar subquery.
`DataSourceV2Strategy.translateScalarSubqueryFilterV2` literalizes the
scalar subquery once its result is known and translates the comparison with
`translateFilterV2`,
which pushes `CAST(part_col AS BIGINT) = <value>` through
`V2ExpressionBuilder`. The cast is
pushed but never unwrapped, so a source that only prunes on column
references ignores the filter.
The optimizer cannot unwrap it either, because the other side is not a
literal until runtime.
This PR adds `UnwrapCastInBinaryComparison.unwrapCastInExpression`, which
unwraps the cast of
every binary comparison, `In` and `InSet` of an expression. The rule's own
`apply` is rewritten on
top of it, so the optimizer and the runtime path share one function.
`translateScalarSubqueryFilterV2` calls it on the literalized filter:
- when the cast is unwrapped, the comparison is pushed in the column type,
e.g. `part_col = 3`;
- when the value is out of the column's range or is rounded by the
conversion, the unwrapping
produces a null-returning form that has no V2 predicate of its own, so it
is rewritten into what
it filters by before translation: `and(isnull(col), null)` keeps no row
and is pushed as `false`
(`AlwaysFalse`), `or(isnotnull(col), null)` keeps the rows where the
column is not null and is
pushed as `isnotnull(col)`. Only the whole filter is rewritten, as neither
is interchangeable
with its counterpart under a `Not`;
- when the cast cannot be unwrapped, e.g. a lossy one, the filter is pushed
as it was.
This is the sibling of SPARK-59301 (#58580), which did the same for the
dynamic partition pruning
IN filter. Unlike the IN case, the comparison rule rewrites out-of-range or
rounded values into
different comparisons or constant results, hence the handling above.
`InMemoryTableWithV2Filter`, the test table the DSv2 pruning tests use,
matched its `=` branch on
any predicate and cast the first child to `FieldReference`, so a pushed cast
failed it with a
`ClassCastException`. It now takes that branch only for a reference, as a
source that prunes on
column references does.
### Why are the changes needed?
A DSv2 scan whose partition column is compared with a wider-typed scalar
subquery received
`CAST(part_col AS BIGINT) = <value>`, which a source pruning on column
references cannot use, so
it read every partition. Reported in
https://github.com/apache/spark/pull/58580#issuecomment-5579953790.
Reproduction, with the in-memory V2 table taking runtime filters as V2
predicates: a 10-partition
table with an INT partition column and a BIGINT dimension column.
```sql
SELECT * FROM tbl WHERE part = (SELECT max(val) FROM dim)
```
Before this PR all 10 partitions are read, after it 1 is.
### Does this PR introduce _any_ user-facing change?
Yes. DSv2 sources now receive scalar subquery runtime filters whose column
is wrapped in a
lossless numeric cast, so the scan prunes partitions instead of reading all
of them. Query results
do not change.
### How was this patch tested?
- New unit tests for `translateScalarSubqueryFilterV2` in
`DataSourceV2StrategySuite`: bare and
nested columns, cast unwrapping for `=` and `>`, a value rounded by the
conversion, a value
above the column's range for `=` and for `<`, and a lossy cast that must
not be unwrapped.
- New end-to-end test in `DataSourceV2SQLSuiteV2Filter`, which asserts the
answer and that 1 of 10
partitions survives pruning.
- Control run with the two main-source changes reverted and the tests kept:
the unit test pushes
`CAST(cint AS long) = 1` instead of `cint = 1`, and the end-to-end test
reads all 10 partitions.
- `UnwrapCastInBinaryComparisonSuite`, `DataSourceV2StrategySuite`,
`DataSourceV2SQLSuiteV2Filter`,
`DataSourceV2CatalystRuntimeFilterSuite`, `DynamicPartitionPruningV1Suite`
and
`DynamicPartitionPruningV2Suite` pass.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 5)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
