dongjoon-hyun opened a new pull request, #59699:
URL: https://github.com/apache/spark/pull/59699
### What changes were proposed in this pull request?
This PR aims to fix ORC filter pushdown on `TIMESTAMP` columns whose ORC
files store sub-microsecond digits, like the files written by other engines.
Spark reads an ORC timestamp into `TimestampType` by truncating it to
microseconds, so a literal `v` matches the stored values from `v` to `v + 999`
nanoseconds. `OrcFilters` now builds the predicates on `TimestampType` with
that range.
| Predicate | Before | After |
| --- | --- | --- |
| `ts = v` | `ts = v` | `ts BETWEEN v AND v + 999ns` |
| `ts <=> v` | `ts <=> v` | `NOT ts IS NULL AND ts BETWEEN v AND v + 999ns` |
| `ts <= v` | `ts <= v` | `ts <= v + 999ns` |
| `ts > v` | `NOT ts <= v` | `NOT ts <= v + 999ns` |
| `ts IN (v1, v2)` | `ts IN (v1, v2)` | `ts BETWEEN v1 AND v1 + 999ns OR ts
BETWEEN v2 AND v2 + 999ns` |
`<`, `>=`, `IS NULL`, and `IS NOT NULL` are already exact, so they are
unchanged.
This works because SPARK-60436 (#59682) stopped the native ORC reader from
passing the `SearchArgument` through Kryo, which truncates the timestamp
literals to milliseconds. Otherwise, `v + 999ns` would become `v` again.
### Why are the changes needed?
ORC stores timestamps with nanosecond precision, and its timestamp
statistics have nanosecond precision since ORC 1.6.4 and 1.7.0 (ORC-611), and
ORC C++ 1.7.0 (ORC-663). So ORC compares a pushed down literal with the
nanosecond minimums and maximums, while Spark compares it with the values
truncated to microseconds.
- `=`, `<=>`, `IN`, and `<=` skip a stripe or row group whose minimum is
greater than `v` but at most `v + 999ns`, although such a minimum is read as
`v`.
- `>` is pushed down as `NOT ts <= v`, which has the same issue when it is
negated. So it is changed in the same way to stay exact.
For example, a file that stores `2020-05-25 10:11:13.000000789` is read as
`2020-05-25 10:11:13`, but the following query returns 0 instead of 1. The
result is correct with `spark.sql.orc.filterPushdown=false`.
```scala
spark.read.orc(path).where("ts = TIMESTAMP '2020-05-25 10:11:13'").count()
```
This affects files with sub-microsecond digits since Apache Spark 3.2.0
([v3.2.0](https://github.com/apache/spark/releases/tag/v3.2.0), 2021-10-06),
the first release with ORC 1.6 (SPARK-33295), which reads the nanosecond
statistics. The example above returns wrong results with Apache Spark 4.0.4,
4.1.3, and 4.2.0, in DSv1 and DSv2, with the vectorized ORC reader on and off.
Files written by Spark are not affected, because their `TIMESTAMP` values have
microsecond precision. ORC `timestamp with local time zone` columns are not
affected either, because ORC does not apply a `SearchArgument` to them when
Spark reads them as `timestamp` (`SchemaEvolution.isPPDSafeConversion` is
false).
The Hive ORC reader (`spark.sql.orc.impl=hive`) still gets the
`SearchArgument` through Kryo, which truncates `v + 999ns` back to `v`.
SPARK-60423 stops pushing down only the filters with sub-millisecond literals
there, so this PR does not change the results of the Hive ORC reader, and it is
out of the scope of this PR.
### Does this PR introduce _any_ user-facing change?
Yes. Predicates on `TIMESTAMP` columns of ORC files with sub-microsecond
digits return correct results with filter pushdown. Previously, they could skip
matching rows silently since Apache Spark 3.2.0.
Since ORC evaluates bloom filters only for `=`, `<=>`, and `IN`, these
predicates on `TIMESTAMP` columns no longer use ORC bloom filters. The minimum
and maximum statistics are used as before.
### How was this patch tested?
Pass the CIs with the newly added test case in `OrcFilterSuite`, which runs
in `OrcFilterSuite` (DSv2) and `OrcV1FilterSuite` (DSv1). It writes ORC files
with sub-microsecond digits through the ORC writer API and checks `=`, `<=>`,
`IN`, `<`, `<=`, `>`, `>=`, `!=`, and `NOT IN` with the vectorized ORC reader
and `spark.sql.datetime.java8API.enabled` on and off. It also checks that a
pushed down `=` keeps skipping the file without the value. Without the fix, it
fails.
```
[info] OrcFilterSuite:
[info] - SPARK-60449: filter pushdown - timestamp with sub-microsecond
digits *** FAILED *** (2 seconds)
[info] 0 did not equal 1 ts = TIMESTAMP'2020-05-25 10:11:13'
(OrcFilterSuite.scala:393)
...
[info] OrcV1FilterSuite:
[info] - SPARK-60449: filter pushdown - timestamp with sub-microsecond
digits *** FAILED *** (316 milliseconds)
[info] 0 did not equal 1 ts = TIMESTAMP'2020-05-25 10:11:13'
(OrcFilterSuite.scala:393)
```
`filter pushdown - timestamp` is updated for the new `BETWEEN` predicates.
The related test suites also pass.
```
$ build/sbt "sql/testOnly org.apache.spark.sql.execution.datasources.orc.*
*OrcV1AggregatePushDownSuite *OrcV2AggregatePushDownSuite *OrcCodecSuite
*OrcReadSchemaSuite *MergedOrcReadSchemaSuite *VectorizedOrcReadSchemaSuite
*OrcZipArchiveReadSuite *OrcTarArchiveReadSuite *OrcSevenZArchiveReadSuite
*FileBasedDataSourceSuite"
$ build/sbt -Phive "hive/testOnly org.apache.spark.sql.hive.orc.*"
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Claude Opus 5.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
