stevomitric opened a new pull request, #57681:
URL: https://github.com/apache/spark/pull/57681
### What changes were proposed in this pull request?
Enable ORC predicate pushdown for nanosecond-precision timestamp columns
(`TIMESTAMP_LTZ(p)` / `TIMESTAMP_NTZ(p)`, `p` in [7, 9]).
`OrcFilters.getPredicateLeafType` / `castLiteralValue` route framework types
through `OrcTypeOps`, but both nanos ops inherited the default
`predicateLeafType = None`. A pushed filter on a nanos column therefore reached
the throwing arm of `getPredicateLeafType` and failed the query. This wires the
two nanos ops (`TimestampLTZNanosOrcOps` / `TimestampNTZNanosOrcOps`) into the
pushdown seam:
- `predicateLeafType` now returns `Some(PredicateLeaf.Type.TIMESTAMP)`,
matching the physical ORC category the types are stored in (`TIMESTAMP` for
NTZ, `TIMESTAMP_INSTANT` for LTZ).
- `castFilterLiteral` converts the external filter literal to a plain
`java.sql.Timestamp` whose local wall clock equals the value's nominal wall
clock, so the pushed value compares consistently against the ORC min/max
statistics:
- NTZ (`java.time.LocalDateTime`) -> `Timestamp.valueOf(ldt)`.
- LTZ (`java.time.Instant`) ->
`Timestamp.valueOf(LocalDateTime.ofInstant(i, UTC))`.
Two ORC-specific details drove the literal shape:
1. ORC's `SearchArgument` builder checks the literal's class by exact
equality (`value.getClass() == Type.getValueClass()`), so it rejects the
`OrcTimestamp` subclass used on the write path; the literal must be a plain
`java.sql.Timestamp`.
2. ORC evaluates a `TIMESTAMP` / `TIMESTAMP_INSTANT` predicate against the
stored value shifted into the JVM default zone (the default
`useUTCTimestamp=false` read mode Spark uses), so the literal must carry the
nominal wall clock in the local zone rather than the raw epoch instant. This
keeps pushdown correct regardless of the JVM time zone.
Stale comments in `TimestampNanosOrcOps` / `OrcTypeOps` / `OrcFilters` that
stated the nanos types had no pushdown are updated.
### Why are the changes needed?
Sub-task of SPARK-56822. Without this, a filter on a nanosecond-precision
timestamp ORC column throws `unsupportedOperationForDataTypeError` during
planning instead of pushing the predicate down (or safely skipping it).
### Does this PR introduce any user-facing change?
Yes. Predicates on nanosecond-precision timestamp ORC columns now push down
to ORC search arguments (enabling stripe/row-group skipping) instead of failing.
### How was this patch tested?
- New `OrcFilterSuite` test "SPARK-57823: filter pushdown - nanosecond
timestamp" asserts the expected `PredicateLeaf.Operator` is generated for all
comparison operators, both nanos types, and all precisions.
- New `OrcQuerySuite` test "SPARK-57823: ORC predicate pushdown returns
correct results for nanos timestamps" verifies end-to-end that results are
correct (`checkAnswer`) and that stripes are actually skipped
(`stripSparkFilter` count), across V1/V2, vectorized/non-vectorized readers,
both nanos types, and all precisions.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Code (Claude Opus 4.8)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
