vinodkc commented on code in PR #56422:
URL: https://github.com/apache/spark/pull/56422#discussion_r3973474243
##########
sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFilters.scala:
##########
@@ -462,6 +467,16 @@ class ParquetFilters(
v.asInstanceOf[LocalTime].getLong(MICRO_OF_DAY)
}
+ // A LocalTime filter literal is only pushable against a TIME(MICROS) column
when it carries no
+ // sub-microsecond (nanosecond) component. TimeType is held internally as
nanos-of-day, so a
+ // literal can be finer-grained than the on-disk MICROS unit;
localTimeToMicros would then
+ // truncate it and push a bound that skips matching rows (e.g. `t <
12:00:00.000000001` truncates
+ // to `t < 12:00:00`, wrongly pruning a row at exactly 12:00:00; `!=` has
the symmetric
+ // false-negative). Sub-microsecond literals are therefore not pushed down;
the read falls back
+ // to a full scan, which is always correct.
+ private def isMicrosResolution(v: Any): Boolean =
+ v.asInstanceOf[LocalTime].getNano % 1000 == 0
Review Comment:
Fixed.
registering the UTC filterOps let me remove the ParquetFilters copy
isMicrosResolution now lives only in TimeTypeParquetOps.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
