sunchao commented on code in PR #5177:
URL: https://github.com/apache/datafusion-comet/pull/5177#discussion_r3918286005
##########
native/core/src/parquet/parquet_support.rs:
##########
@@ -195,6 +195,21 @@ fn parquet_convert_array(
list_arr.nulls().cloned(),
)))
}
+ (
+ Timestamp(TimeUnit::Millisecond, _),
+ Timestamp(TimeUnit::Microsecond, target_tz),
+ ) => {
+ // Spark's Parquet reader calls the checked `millisToMicros`
conversion for both
+ // direct and dictionary values, independent of CAST evaluation
mode:
+ //
https://github.com/apache/spark/blob/v4.2.0/sql/core/src/main/java/org/apache/spark/sql/execution/datasources/parquet/ParquetVectorUpdaterFactory.java#L817-L833
+ // `millisToMicros` uses `Math.multiplyExact`:
+ //
https://github.com/apache/spark/blob/v4.2.0/sql/api/src/main/scala/org/apache/spark/sql/catalyst/util/SparkDateTimeUtils.scala#L103-L108
+ let micros = array
+ .as_primitive::<TimestampMillisecondType>()
+ .try_unary::<_, TimestampMicrosecondType, _>(|value|
value.mul_checked(1_000))?
+ .with_timezone_opt(target_tz.clone());
Review Comment:
Rechecked `9172861`: all four predicate forms in your last reply (ordinary
comparison, short IN, null-safe equality, and nested timestamp filtering) now
pass the 64-query comparison matrix on both the head and base.
The broader pruning issue still has three remaining cases: IN lists above
DataFusion's 20-value cutoff, an unrelated INT32 filter column widened to
LongType, and dictionary-based exclusion that cannot be inferred from min/max
statistics. Each has a Spark/base-success versus head-overflow reproduction
with row-filter pushdown disabled. I posted the distinct reproducers and
validation in [the follow-up
review](https://github.com/apache/datafusion-comet/pull/5177#pullrequestreview-5094649307),
along with a separate empty-NOT-IN NULL-row finding. The nested surviving-row
overflow difference you documented remains pre-existing and is not one of these
new regressions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
