dongjoon-hyun opened a new pull request, #59682: URL: https://github.com/apache/spark/pull/59682
### What changes were proposed in this pull request? This PR aims to fix the native ORC reader (`spark.sql.orc.impl=native`) to push down the ORC `SearchArgument` without losing the sub-millisecond digits of timestamp literals. `OrcFileFormat` (DSv1) and `OrcPartitionReaderFactory` (DSv2) now pass the `SearchArgument` to the ORC readers, which set it directly on the ORC `Reader.Options` after `OrcInputFormat.buildOptions`. Previously, it was passed through the Hadoop `Configuration` via `OrcInputFormat.setSearchArgument`. - `OrcColumnarBatchReader.initialize` has a new overload with the `SearchArgument`. - `OrcUtils.createOrcMapreduceRecordReader` has a new `SearchArgument` parameter. - `OrcPartitionReaderFactory.createORCReader` no longer builds the `SearchArgument`, so it does not need to close the reader on a failure (SPARK-57529) any more. The `SearchArgument` is built while no ORC reader is open. The `SearchArgument` itself is built in the same way as before, for each file in the task. ### Why are the changes needed? `OrcInputFormat.setSearchArgument` writes the `SearchArgument` into `orc.kryo.sarg` with a plain `Kryo`, and `OrcInputFormat.buildOptions` reads it back. Kryo's default serializer of `java.util.Date` keeps only `getTime()`, so a `java.sql.Timestamp` literal loses its sub-millisecond digits, e.g. `2020-01-01 00:00:00.123456` becomes `2020-01-01 00:00:00.123`. Since ORC 1.6.4 (ORC-611), ORC timestamp statistics have nanosecond precision, so ORC compares the truncated literal with the nanosecond minimums and maximums. - `=`, `<=>`, `IN`, `<`, and `<=` skip a stripe or row group whose minimum is between the truncated literal and the literal, although it can hold matching rows. - `!=` and `NOT IN` skip a stripe or row group whose values are all equal to the truncated literal, because ORC evaluates `=` to `YES` for it. - `>` and `>=` stay correct, because the truncated literal only makes them less selective. This affects `TIMESTAMP` columns since Apache Spark 3.2.0 ([v3.2.0](https://github.com/apache/spark/releases/tag/v3.2.0), 2021-10-06), the first release with ORC 1.6 (SPARK-33295). It also affects the unreleased nanosecond timestamp types, `TIMESTAMP_NTZ(p)` and `TIMESTAMP_LTZ(p)` (SPARK-57823), even when they are read with the precision they are written with. Both DSv1 and DSv2 are affected, with the vectorized ORC reader on and off. For example, the following query returns 0 instead of 1. The result is correct with `spark.sql.orc.filterPushdown=false`. ```scala spark.sql("SELECT TIMESTAMP '2020-01-01 00:00:00.123456' AS ts").write.orc(path) spark.read.orc(path).where("ts = TIMESTAMP '2020-01-01 00:00:00.123456'").count() ``` The same issue is reproduced with Apache Spark 4.0.4, 4.1.3, 4.2.0, and 4.3.0-rc1. The Hive ORC reader (`spark.sql.orc.impl=hive`) has the same Kryo issue, but Hive's reader gets the `SearchArgument` only from the Hadoop configuration. So SPARK-60396 and SPARK-60423 stopped pushing down the affected predicates there instead. ### Does this PR introduce _any_ user-facing change? Yes. With the native ORC reader, predicates that compare a timestamp column with a literal with sub-millisecond digits return correct results. Previously, they could skip matching rows silently since Apache Spark 3.2.0. ### How was this patch tested? Pass the CIs with the newly added test case in `OrcQuerySuite`, which runs in `OrcV1QuerySuite` and `OrcV2QuerySuite`. It checks `=`, `<=>`, `IN`, `<`, `<=`, `>`, `>=`, `!=`, and `NOT IN` on `TIMESTAMP`, `TIMESTAMP_NTZ(p)`, and `TIMESTAMP_LTZ(p)` for all precisions, with the vectorized ORC reader on and off. It also checks that a pushed down `=` keeps skipping the files without the value. Without the fix, it fails. ``` [info] - SPARK-60436: ORC predicate pushdown on timestamps with sub-millisecond digits *** FAILED *** (2 seconds, 144 milliseconds) [info] Results do not match for query: ... [info] 'Filter ('ts = cast(2020-01-01 00:00:00.123456789 as timestamp)) ... [info] !== Correct Answer - 1 == == Spark Answer - 0 == [info] !struct<ts:timestamp> struct<> [info] ![2020-01-01 00:00:00.123456] (CheckAnswerHelper.scala:61) ``` The related test suites also pass. ``` $ build/sbt "sql/testOnly org.apache.spark.sql.execution.datasources.orc.* *OrcV1AggregatePushDownSuite *OrcV2AggregatePushDownSuite *OrcCodecSuite *OrcReadSchemaSuite *MergedOrcReadSchemaSuite *VectorizedOrcReadSchemaSuite *OrcZipArchiveReadSuite *OrcTarArchiveReadSuite *OrcSevenZArchiveReadSuite *FileBasedDataSourceSuite" $ build/sbt -Phive "hive/testOnly org.apache.spark.sql.hive.orc.*" ``` ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Opus 5.5 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
