SEPURI-SAI-KRISHNA opened a new pull request, #59584: URL: https://github.com/apache/spark/pull/59584
### What changes were proposed in this pull request? The native ORC readers (V1 `OrcFileFormat` and V2 `OrcPartitionReaderFactory`) now pass the pushed `SearchArgument` to the ORC reader with `Reader.Options.searchArgument`, instead of storing it in the Hadoop conf with `OrcInputFormat.setSearchArgument` for `OrcInputFormat.buildOptions` to decode. `OrcColumnarBatchReader.initialize` and `OrcUtils.createOrcMapreduceRecordReader` take the `SearchArgument` as a parameter for this. ### Why are the changes needed? ORC filter pushdown drops rows when a TIMESTAMP filter literal has sub-millisecond digits. With the default configuration, on 4.0.3, 4.2.0 (official binaries) and master: ```sql CREATE TABLE t USING orc AS SELECT 1 AS id, TIMESTAMP'2024-06-01 12:00:00.123456' AS ts; SELECT count(*) FROM t WHERE ts = TIMESTAMP'2024-06-01 12:00:00.123456'; -- 0, expected 1 SELECT count(*) FROM t WHERE ts <= TIMESTAMP'2024-06-01 12:00:00.123456'; -- 0, expected 1 CREATE TABLE m USING orc AS SELECT id, TIMESTAMP'2024-06-01 12:00:00.123' AS ts FROM range(10); SELECT count(*) FROM m WHERE ts != TIMESTAMP'2024-06-01 12:00:00.123456'; -- 0, expected 10 SELECT count(*) FROM m WHERE ts < TIMESTAMP'2024-06-01 12:00:00.123001'; -- 0, expected 10 ``` The same happens with `<=>` and `IN`, with the V1 and V2 readers, and with the vectorized reader on and off. All are correct with `spark.sql.orc.filterPushdown=false`. The `SearchArgument` built by `OrcFilters` has the exact literal, and the ORC statistics are exact too. But `OrcInputFormat.setSearchArgument` serializes the `SearchArgument` with a plain `Kryo`, whose serializer for `java.sql.Timestamp` only keeps milliseconds, so the reader evaluates `ts = 12:00:00.123`, `ts <= 12:00:00.123` or `ts != 12:00:00.123` against the statistics and skips stripes and row groups that hold matching rows. `>` and `>=` only get wider. It shows when the literal is within a millisecond of the minimum or maximum of a row group, stripe or file. The `SearchArgument` is built on the executor for each file, right before the reader is created, so there is no need to go through the conf. This also means the readers no longer write it into the broadcast Hadoop conf that the tasks of an executor share. The nanosecond timestamp types (SPARK-57823) are pushed down the same way and lost rows the same way, e.g. `ts = <TIMESTAMP_NTZ(9) value with sub-ms digits>`; they are fixed by this change too. `spark.sql.orc.impl=hive` has the same problem, but there Hive's `OrcInputFormat` reads the `SearchArgument` from the conf (`sarg.pushdown`) with a plain `Kryo` as well, so it cannot be fixed this way. It will be handled separately. ### Does this PR introduce _any_ user-facing change? Yes. Queries on ORC files with a TIMESTAMP filter whose literal has sub-millisecond digits now return the rows that were dropped before. Pushdown still skips the same stripes and row groups otherwise. ### How was this patch tested? New test in `OrcQuerySuite` (run by `OrcV1QuerySuite` and `OrcV2QuerySuite`): `=`, `<=>`, `<=`, `<`, `IN`, `!=` and `>=` with sub-millisecond literals on files of one value each, with the vectorized reader on and off, and checks that the files that do not match are still skipped. It fails without this change. A separate matrix compared pushdown on and off for these predicates and more (`BETWEEN`, `NOT`, `OR`, `NOT IN`), on a file of three row groups, a file of values on a millisecond, and values in 1900, 1969 and 9999, for V1 and V2, the vectorized reader on and off, and the UTC and America/Los_Angeles time zones: 20 of 176 cases returned fewer rows with pushdown before this change, none after, and the number of rows read from the files is the same before and after in all other cases. The `=` and `<=` cases of the nanosecond types, for V1 and V2, also return the right rows now. These suites pass: `OrcV1QuerySuite`, `OrcV2QuerySuite`, `OrcFilterSuite`, `OrcV1FilterSuite`, `OrcSourceV1Suite`, `OrcSourceV2Suite`, `OrcColumnarBatchReaderSuite`, `OrcPartitionReaderFactorySuite`, `OrcPartitionDiscoverySuite`, `OrcV1PartitionDiscoverySuite`, `OrcV1SchemaPruningSuite`, `OrcV2SchemaPruningSuite` and `OrcEncryptionSuite` in `sql/core`, and `HiveOrcQuerySuite`, `HiveOrcSourceSuite`, `HiveOrcPartitionDiscoverySuite`, `OrcHadoopFsRelationSuite` and `HiveOrcHadoopFsRelationSuite` in `sql/hive`. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
