vboo123 opened a new pull request, #56826: URL: https://github.com/apache/spark/pull/56826
### What changes were proposed in this pull request? This PR adds TIME type predicate pushdown support to the ORC reader in `OrcFilters.scala`. Two methods are extended: - `getPredicateLeafType`: adds a `TimeType` case that maps to `PredicateLeaf.Type.LONG`, grouped with the existing `TimestampNTZType`/`AnsiIntervalType` LONG cases. - `castLiteralValue`: adds a `TimeType` case that converts a `java.time.LocalTime` literal to its nanos-of-day representation via `DateTimeUtils.localTimeToNanos`. Two imports are added: `java.time.LocalTime` and `DateTimeUtils.localTimeToNanos`. The leaf-builder (`buildLeafSearchArgument`) is type-agnostic. It calls `getPredicateLeafType` and `castLiteralValue` generically for all comparison operators. Once both helpers handle `TimeType`, all supported comparisons (EqualTo, EqualNullSafe, LessThan, LessThanOrEqual, GreaterThan, GreaterThanOrEqual, In, IsNull, IsNotNull) push down automatically with no operator-specific code and no SARG-tree changes. A correctness note on the value unit: the literal is converted to **nanos-of-day**, not micros-of-day. The already-merged Parquet TIME pushdown (SPARK-51687) uses micros because Parquet's logical TIME type is defined in microseconds. ORC differs. The native ORC TIME read/write path (SPARK-54472) persists Spark's internal long representation, which stores nanos-of-day. The pushed-down literal must use nanos to match what is stored in the ORC file. Using micros here would produce a silent correctness bug where predicates filter against the wrong scale. ### Why are the changes needed? `OrcFilters` handled Date, Timestamp, and TimestampNTZ predicate pushdown but had no `TimeType` case. Without this change, any predicate on a TIME column over ORC is never pushed to the reader. The engine performs a full stripe/row-group scan and evaluates the filter only after reading all rows into memory. This is a sub-task of SPARK-57550 (extend TIME type support across Spark). The TIME data type was added by SPIP SPARK-51162 and shipped in Spark 4.1.0. Native ORC read/write for TIME landed in SPARK-54472. Predicate pushdown is the remaining gap for ORC to treat TIME on par with other temporal types. ### Does this PR introduce _any_ user-facing change? No change to query results. Queries that filter on a TIME column stored in ORC format now push the predicate to the ORC reader, reducing the number of rows and stripes read during a scan. ### How was this patch tested? A new test was added to `OrcFilterSuite` named `"SPARK-57571: filter pushdown - time"`, modeled on the existing `timestamp_ntz` sibling test. The test writes `LocalTime` values to an ORC file and asserts the correct SARG operator for IsNull, EqualTo, null-safe-equals, LessThan, LessThanOrEqual, GreaterThan, and GreaterThanOrEqual pushdown in both column-op-literal and literal-op-column forms. The test suite was not run locally (the dev desktop cannot complete a full Spark build without OOM). Verification relies on GitHub Actions CI on the fork. ### Was this patch authored or co-authored using generative AI tooling? Yes. Generated using Kiro (Claude Opus 4.8) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
