Indian-boult opened a new pull request, #20742:
URL: https://github.com/apache/hudi/pull/20742

   
   ### Describe the issue this Pull Request addresses
   
   Closes #20741
   
   Data skipping re-applies a filter's term to the min and max stats of each 
file (column stats) or partition (partition stats). That is only right when the 
term is order-preserving in the column. `ConstantOperandRule` claimed it for 
two kinds of terms where it does not hold, so files and partitions that hold 
matching rows were pruned and the query returned too few rows:
   
   - `BitwiseOr` (`a | c` and `c | a`) is not monotonic, even with a constant: 
`1 | 2 = 3`, but the larger `2 | 2 = 2`. A file that holds 1 and 2 gets the 
bounds `[3, 2]`, so `(a | 2) = 3` pruned it, though its row 1 matches. The rule 
already noted this (#20449), and the oracle in #20439 has it as the 
`bitwise_or` family.
   - `DateAdd`, `DateSub`, `DateDiff` (end date) and `ShiftRight` accepted any 
second operand. An operand that references the column is re-bound to the same 
bound, so the term is evaluated at the min and at the max, but it is not 
order-preserving: `date_add(d, dayofmonth(d))` maps 2024-01-31 to 2024-03-02 
and the later 2024-02-01 to 2024-02-02, and `shiftright(a, a)` on a bigint maps 
64 to 64 and 65 to 32 (a bigint shifts by the low 6 bits of the amount). `Add`, 
`Multiply` and `ShiftLeft` already need a constant (#20457, #20532).
   
   Repro on master, COW table with column stats on `a bigint` and `d date`, one 
file per partition (`par1`: ids 1 and 2, `a` 1 and 2, `d` 2024-01-31 and 
2024-02-01; `par2`: ids 3 and 4, `a` 64 and 65):
   
   | filter | data skipping off | on, master | on, this PR |
   | --- | --- | --- | --- |
   | `(a \| 2) = 3`, `(2 \| a) = 3` | id 1 | no rows, 0 files | id 1, 2 files |
   | `shiftright(a, a) = 64` | id 3 | no rows, 0 files | id 3, 2 files |
   | `date_add(d, dayofmonth(d)) = date'2024-03-02'` | id 1 | no rows, 0 files 
| id 1, 2 files |
   | `date_sub(d, -dayofmonth(d)) = date'2024-03-02'` | id 1 | no rows, 0 files 
| id 1, 2 files |
   | `datediff(d, trunc(d, 'MM')) = 30` | id 1 | no rows, 0 files | id 1, 2 
files |
   | `shiftright(a, 1) = 32`, `date_add(d, 1) = ...`, `date_sub(d, 1) = ...`, 
`datediff(d, date'2024-01-01') = 30` | right rows | right rows, 1 file | same 
as master |
   
   Partition stats on or off give the same result.
   
   ### Summary and Changelog
   
   Data skipping no longer prunes through `a | c`, or through `date_add`, 
`date_sub`, `datediff` or `shiftright` whose other operand is not a constant, 
so these queries return every matching row. The same functions with a constant 
operand prune as before.
   
   - `ConstantOperandRule`: `BitwiseOr` is no longer matched. `DateAdd` and 
`DateSub` need a constant number of days, `DateDiff` a constant start date, and 
`ShiftRight` a constant shift amount. Constant means foldable, as `Add` already 
checks. The class doc lists the guards.
   - `TermRange`, `RangeRule`: three doc examples used `BitwiseOr` or 
`date_add(d, dayofmonth(d))`, which no longer reach that code. They now use 
`Add` and `left(s, length(s))`.
   - Tests:
     - `TestDataSkippingUtils#testNonOrderPreservingTransformationsSource`: the 
six filters above over a file whose min and max map out of order (each keeps 
the file), plus `shiftright(A, 1)` and `datediff` with a constant start date, 
which still prune.
     - `TestHoodieCatalystExpressionUtils`: `BitwiseOr` on either side and with 
the column on both sides has no range. `DateAdd`, `DateSub`, `DateDiff` and 
`ShiftRight` with an operand that references the column or calls `rand()` have 
no range; with a foldable operand they still do. The two `BitwiseOr` assertions 
of `testArithmeticTransformationsPreserveOrdering` move there, replaced by 
`shiftright(l, 2)`.
     - `TestDataSkippingQuery`: the repro table above, with partition stats off 
and on. It checks the rows of the six filters with data skipping off and on. On 
master it fails at `(a | 2) = 3`.
     - `golden-index-filters.txt`: 51 lines change, each to `true:boolean`. 
They are the inputs on `a | 1`, `1 | a`, `abs(a) | a`, `a | a`, `date_add(d, 
dayofmonth(d))`, `date_sub(d, dayofmonth(d))` and `datediff(d, d)`, and 2 
seeded trees on them. No other line changes.
   
   ### Impact
   
   Queries with data skipping on that filter on `a | c`, or on `date_add`, 
`date_sub`, `datediff` or `shiftright` with an operand that is not a constant, 
now return every matching row. Those filters no longer prune files or 
partitions. No public API or config change.
   
   ### Risk Level
   
   low. The change only removes pruning, so it cannot drop a file that was kept 
before. The cost is pruning for the shapes above that happen to be 
order-preserving, for ex `a | 0` or `date_sub(d, dayofmonth(d))`. Constant 
operands are unchanged: the golden file changes only on the lines listed above, 
and the constant cases in `TestDataSkippingUtils` and the repro table prune as 
before.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to