wombatu-kun opened a new pull request, #20228:
URL: https://github.com/apache/hudi/pull/20228
### Describe the issue this Pull Request addresses
Closes #20227
### Summary and Changelog
Data skipping no longer treats `lower()`/`upper()` as order-preserving, so
filters such as `lower(col) = 'x'` stop pruning files that hold matching rows.
- `BaseHoodieCatalystExpressionUtils`: drops the `Lower` and `Upper` arms
from the order-preserving transformation whitelist. Case folding does not
preserve the binary ordering of strings (`'Zebra' < 'apple'`, but
`lower('Zebra') > 'apple'`), so the transformed min/max are no valid bounds.
- Tests: `TestHoodieCatalystExpressionUtils` pins that `Lower`/`Upper` no
longer match, and `TestDataSkippingUtils` covers files whose min/max straddle
the case boundary.
### Impact
Correctness fix: such queries could silently miss rows. Filters over
`lower(col)`/`upper(col)` no longer prune through the column stats of `col`; an
expression index on `lower(col)`/`upper(col)` still prunes them, since its
filters are rewritten to the indexed column before translation.
### Risk Level
low. It only removes pruning that was unsound. The new test cases fail
without the fix, the SQL repro from the issue returns the expected rows with
it, and `TestExpressionIndex` passes.
### Documentation Update
none
### Contributor's checklist
- [x] Read through [contributor's
guide](https://hudi.apache.org/contribute/how-to-contribute)
- [x] Enough context is provided in the sections above
- [x] Adequate tests were added if applicable
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
