Indian-boult opened a new pull request, #20742:
URL: https://github.com/apache/hudi/pull/20742
### Describe the issue this Pull Request addresses
Closes #20741
Data skipping re-applies a filter's term to the min and max stats of each
file (column stats) or partition (partition stats). That is only right when the
term is order-preserving in the column. `ConstantOperandRule` claimed it for
two kinds of terms where it does not hold, so files and partitions that hold
matching rows were pruned and the query returned too few rows:
- `BitwiseOr` (`a | c` and `c | a`) is not monotonic, even with a constant:
`1 | 2 = 3`, but the larger `2 | 2 = 2`. A file that holds 1 and 2 gets the
bounds `[3, 2]`, so `(a | 2) = 3` pruned it, though its row 1 matches. The rule
already noted this (#20449), and the oracle in #20439 has it as the
`bitwise_or` family.
- `DateAdd`, `DateSub`, `DateDiff` (end date) and `ShiftRight` accepted any
second operand. An operand that references the column is re-bound to the same
bound, so the term is evaluated at the min and at the max, but it is not
order-preserving: `date_add(d, dayofmonth(d))` maps 2024-01-31 to 2024-03-02
and the later 2024-02-01 to 2024-02-02, and `shiftright(a, a)` on a bigint maps
64 to 64 and 65 to 32 (a bigint shifts by the low 6 bits of the amount). `Add`,
`Multiply` and `ShiftLeft` already need a constant (#20457, #20532).
Repro on master, COW table with column stats on `a bigint` and `d date`, one
file per partition (`par1`: ids 1 and 2, `a` 1 and 2, `d` 2024-01-31 and
2024-02-01; `par2`: ids 3 and 4, `a` 64 and 65):
| filter | data skipping off | on, master | on, this PR |
| --- | --- | --- | --- |
| `(a \| 2) = 3`, `(2 \| a) = 3` | id 1 | no rows, 0 files | id 1, 2 files |
| `shiftright(a, a) = 64` | id 3 | no rows, 0 files | id 3, 2 files |
| `date_add(d, dayofmonth(d)) = date'2024-03-02'` | id 1 | no rows, 0 files
| id 1, 2 files |
| `date_sub(d, -dayofmonth(d)) = date'2024-03-02'` | id 1 | no rows, 0 files
| id 1, 2 files |
| `datediff(d, trunc(d, 'MM')) = 30` | id 1 | no rows, 0 files | id 1, 2
files |
| `shiftright(a, 1) = 32`, `date_add(d, 1) = ...`, `date_sub(d, 1) = ...`,
`datediff(d, date'2024-01-01') = 30` | right rows | right rows, 1 file | same
as master |
Partition stats on or off give the same result.
### Summary and Changelog
Data skipping no longer prunes through `a | c`, or through `date_add`,
`date_sub`, `datediff` or `shiftright` whose other operand is not a constant,
so these queries return every matching row. The same functions with a constant
operand prune as before.
- `ConstantOperandRule`: `BitwiseOr` is no longer matched. `DateAdd` and
`DateSub` need a constant number of days, `DateDiff` a constant start date, and
`ShiftRight` a constant shift amount. Constant means foldable, as `Add` already
checks. The class doc lists the guards.
- `TermRange`, `RangeRule`: three doc examples used `BitwiseOr` or
`date_add(d, dayofmonth(d))`, which no longer reach that code. They now use
`Add` and `left(s, length(s))`.
- Tests:
- `TestDataSkippingUtils#testNonOrderPreservingTransformationsSource`: the
six filters above over a file whose min and max map out of order (each keeps
the file), plus `shiftright(A, 1)` and `datediff` with a constant start date,
which still prune.
- `TestHoodieCatalystExpressionUtils`: `BitwiseOr` on either side and with
the column on both sides has no range. `DateAdd`, `DateSub`, `DateDiff` and
`ShiftRight` with an operand that references the column or calls `rand()` have
no range; with a foldable operand they still do. The two `BitwiseOr` assertions
of `testArithmeticTransformationsPreserveOrdering` move there, replaced by
`shiftright(l, 2)`.
- `TestDataSkippingQuery`: the repro table above, with partition stats off
and on. It checks the rows of the six filters with data skipping off and on. On
master it fails at `(a | 2) = 3`.
- `golden-index-filters.txt`: 51 lines change, each to `true:boolean`.
They are the inputs on `a | 1`, `1 | a`, `abs(a) | a`, `a | a`, `date_add(d,
dayofmonth(d))`, `date_sub(d, dayofmonth(d))` and `datediff(d, d)`, and 2
seeded trees on them. No other line changes.
### Impact
Queries with data skipping on that filter on `a | c`, or on `date_add`,
`date_sub`, `datediff` or `shiftright` with an operand that is not a constant,
now return every matching row. Those filters no longer prune files or
partitions. No public API or config change.
### Risk Level
low. The change only removes pruning, so it cannot drop a file that was kept
before. The cost is pruning for the shapes above that happen to be
order-preserving, for ex `a | 0` or `date_sub(d, dayofmonth(d))`. Constant
operands are unchanged: the golden file changes only on the lines listed above,
and the constant cases in `TestDataSkippingUtils` and the repro table prune as
before.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]