SEPURI-SAI-KRISHNA opened a new pull request, #59574: URL: https://github.com/apache/spark/pull/59574
### What changes were proposed in this pull request? `ParquetFilters` and `OrcFilters` push a comparison of a `FLOAT` or `DOUBLE` column with 0.0 or -0.0 down in a form that keeps exactly the values that the Spark predicate keeps: | Spark predicate, `z` = 0.0 or -0.0 | Parquet | ORC | |---|---|---| | `f = z`, `f <=> z` | `or(eq(f, 0.0), eq(f, -0.0))` | `or(equals(f, 0.0), equals(f, -0.0))`, `nullSafeEquals` for `<=>` | | `f IN (z, ...)` | both zeros in the set | both zeros in the list | | `f != z` | `and(notEq(f, 0.0), notEq(f, -0.0))` | (pushed as `not(equals)`, covered above) | | `f < z`, `f >= z` | bound -0.0 | `lessThan` with bound -0.0 (`>=` is `not(lessThan)`) | | `f <= z`, `f > z` | bound 0.0 | `lessThanEquals` with bound 0.0 (`>` is `not(lessThanEquals)`) | Other values are pushed down as before. ### Why are the changes needed? Spark treats 0.0 and -0.0 as equal (`SELECT 0.0D = -0.0D` is true), but both formats tell them apart, and the pushed predicate compared with the literal unchanged. A row group, stripe or file that holds only the other zero was skipped, and the Filter above the scan never saw its rows. With the default configuration, on 3.5.3, 4.0.3 and 4.2.0 (official binaries) and on master: ```sql CREATE TABLE p USING parquet AS SELECT id, -0.0D * id AS f FROM range(0, 1000, 1, 1); SELECT count(*) FROM p WHERE f = 0; -- 0, expected 1000 CREATE TABLE o USING orc AS SELECT id, -0.0D * id AS f FROM range(1, 1000, 1, 1); SELECT count(*) FROM o WHERE f = 0; -- 0, expected 999 SELECT count(*) FROM o WHERE f >= 0; -- 0, expected 999 ``` With `spark.sql.parquet.filterPushdown=false` / `spark.sql.orc.filterPushdown=false` they return 1000, 999 and 999. Where the formats tell the zeros apart: - The Parquet dictionary filter and bloom filter, and the ORC bloom filter: a dictionary or bloom filter of -0.0 does not contain 0.0. This drops the Parquet rows above. - ORC statistics compare with `Double.compareTo`, so a min and max of -0.0 rule out `f = 0.0` and `f >= 0.0`. - Parquet statistics and the column index: Parquet 1.17 and earlier write -0.0 as the min and 0.0 as the max of a chunk of zeros, so these did not skip rows. Parquet 1.18 (SPARK-58869, master only) writes `FLOAT` and `DOUBLE` columns with the IEEE 754 total order and exact bounds, so on master they skip rows too. A range takes the zero on its side as the bound, rather than any zero that keeps enough values, because a pushed `Not` negates the pushed predicate: with a bound that keeps more values, `NOT (f < 0.0)` would become `not(lt(f, 0.0))`, which skips a file of -0.0 values. ORC builds `>` and `>=` that way, as `not(lessThanEquals)` and `not(lessThan)`. ### Does this PR introduce _any_ user-facing change? Yes. Queries that compare a `FLOAT` or `DOUBLE` column of a Parquet or ORC table with 0.0 or -0.0 now return the rows that hold the other zero, where they could return fewer rows before. ### How was this patch tested? New tests in `ParquetFilterSuite` and `OrcFilterSuite`, which run with DSv1 and DSv2. They write one file of 0.0, one of -0.0 and one of 1.0, for `FLOAT` and `DOUBLE`, and compare 30 predicates with each zero (`=`, `<=>`, `IN` below and above the IN threshold, `!=`, `<`, `<=`, `>`, `>=`, and `NOT` of them with `BooleanSimplification` excluded) with the same filter on the data in memory. The Parquet test writes the files once dictionary encoded and once with a bloom filter, checks that in the footer, and reads them with the record-level filter on. Both tests fail without this change. Each part of the change was also reverted on its own (equality, IN, not-equal and each of the four range bounds, for each format), and each of the 14 reverts fails the tests. `ParquetV1FilterSuite`, `ParquetV2FilterSuite`, `OrcFilterSuite`, `OrcV1FilterSuite` and the other suites in the `parquet` and `orc` packages pass. A separate matrix compared these predicates, and BETWEEN and OR built from them, with pushdown on and off over files of 0.0, -0.0, other values and NULL: Parquet and ORC, `FLOAT` and `DOUBLE`, DSv1 and DSv2, vectorized and not, dictionaries and bloom filters on and off, only the statistics, the column index or the dictionary filter on, the record-level filter and the storage filter on and off. 3,558 of 7,904 combinations were wrong before this change and none after. ### Was this patch authored or co-authored using generative AI tooling? Generated-by: Claude Code (Claude Opus 5.5) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
