snmvaughan opened a new issue, #7158:
URL: https://github.com/apache/datafusion-comet/issues/7158
### Describe the bug
With `spark.comet.parquet.rowFilterPushdown.enabled=true` on Spark 3.x, a
pushed-down filter that contains a planner-built cast fails at runtime even
though the file's schema matches the requested schema exactly:
```
Error evaluating filter predicate: External(ParquetSchemaConvert {
file_path: "", column: "[]", physical_type: "INT32", spark_type: "bigint" })
```
`SparkPhysicalExprAdapter::replace_with_spark_cast` treats every `CastExpr`
in the predicate as a Parquet file-versus-table schema conversion. Comet's
planner also builds `CastExpr`s, with `CastExpr::new`, to coerce
scalar-function arguments. `substring(s, 1, 2)` is one: Spark serializes Int32
literals, `SparkSubstring` takes Int64 positions, so the planner emits `CAST(1
AS Int64)`. On Spark 3.x `allow_type_promotion` is false, so the adapter turns
that into a deferred `INT32 -> bigint` rejection, which fires once the row
filter evaluates the predicate. The empty `column` is the tell: the cast is not
over a column.
### Steps to reproduce
- Native: the fix PR adds `pushed_substring_filter_with_row_filter_pushdown`
(in `native/core/src/parquet/schema_adapter.rs`), which fails without the fix.
It builds the predicate through `PhysicalPlanner` from the proto
`CometSubstring` serializes and scans a Parquet file whose schema equals the
requested schema.
- From Spark (not run end to end on stock Spark): Spark 3.4 or 3.5,
`spark.comet.parquet.rowFilterPushdown.enabled=true`, any Parquet table with a
string column, `SELECT * FROM t WHERE substring(s, 1, 2) = 'ab'`. It surfaced
as TPC-H Q22, the only TPC-H query that uses `substring`, failing when the
TPC-H suite ran with the flag enabled on Spark 3.4.
### Expected behavior
The query returns the matching rows.
### Additional context
The problem is hidden by default because row-level pushdown is off by
default, the TPC-H CI job does not enable it, and Spark 4.x allows `INT32 ->
INT64`. The Spark SQL test diffs do enable the flag. Any other planner-inserted
cast whose (source, target) pair `check_conversion` rejects would be affected
the same way.
### Willingness to contribute
I can contribute a fix for this bug independently. A fix is in the PR linked
from this issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
