ErikBPF commented on PR #5365: URL: https://github.com/apache/datafusion-comet/pull/5365#issuecomment-5518974053
Correction on the NYC pruning observation above: I traced the predicate shape before proposing a fix. The source Parquet column is annotated `Timestamp(isAdjustedToUTC=false, timeUnit=microseconds)`, so Spark 3.5 correctly infers `TIMESTAMP_NTZ`. My workload used ordinary `TIMESTAMP` bounds. Spark therefore analyzed each predicate as `cast(tpep_pickup_datetime AS timestamp) ...`; a direct Spark 3.5 plan probe confirms that using `TIMESTAMP_NTZ` bounds removes the cast. Comet did carry the filter into the native scan, but it becomes Comet’s Spark-compatible custom cast. DataFusion 55 pruning recognizes its own `CastExpr`/`TryCastExpr`, not that custom expression, so the zero page-index/row-group pruning is expected for this mixed LTZ/NTZ predicate. It is a broader cast-pruning limitation, not evidence of a Delta/DV-specific gap. A type-matched workload rerun would measure available Delta/file/Parquet pruning; I would not propose a timezone-sensitive cast rewrite from the current evidence. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
