MaxGekk opened a new pull request, #56823:
URL: https://github.com/apache/spark/pull/56823
### What changes were proposed in this pull request?
This PR adds predicate pushdown for the `TIME` (`TimeType`) data type to the
ORC reader, matching the existing Parquet TIME filter pushdown (SPARK-51687).
Changes:
- `OrcFilters`:
- `getPredicateLeafType` now maps `TimeType` to `PredicateLeaf.Type.LONG`.
- `castLiteralValue` converts a `java.time.LocalTime` literal to
nanos-of-day via `DateTimeUtils.localTimeToNanos`, which is how ORC physically
stores `TIME` values (ORC `LONG`, see `OrcSerializer`/`OrcUtils`).
- The existing leaf handlers (`EqualTo`, `EqualNullSafe`, `LessThan`,
`LessThanOrEqual`, `GreaterThan`, `GreaterThanOrEqual`, `In`, `IsNull`) then
work for `TIME` unchanged.
- Add a `localTimeToLiteral` implicit to the Catalyst `dsl` package,
mirroring the existing `timestampNTZToLiteral`, so `TIME` literals can be used
in dsl predicates (used by the test).
- Add a test in `OrcFilterSuite` (also exercised by `OrcV1FilterSuite`).
### Why are the changes needed?
Native ORC read/write for `TIME` is already supported (SPARK-54472), but
`OrcFilters.getPredicateLeafType` had no `TimeType` case, so `TIME` predicates
were not pushed down to the ORC reader. This closes that gap and brings ORC to
parity with Parquet, avoiding unnecessary row scans for filters on `TIME`
columns.
### Does this PR introduce _any_ user-facing change?
No behavior change in query results. Filters on `TIME` columns over ORC are
now pushed down to the reader, which can improve scan performance.
### How was this patch tested?
Added `SPARK-57571: filter pushdown - time` in `OrcFilterSuite`, covering EQ
/ NULL_SAFE_EQUALS / LT / LTE / GT / GTE / IsNull in both `col op literal` and
`literal op col` forms. Ran:
```
build/sbt 'sql/testOnly *OrcFilterSuite *OrcV1FilterSuite'
```
All 40 tests passed. Also ran `./dev/scalastyle` (passed) and scalafmt on
the connect/api modules.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor (Claude Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
