fhan688 opened a new pull request, #20392: URL: https://github.com/apache/hudi/pull/20392
### Describe the issue this Pull Request addresses Prefix predicates can incorrectly exclude files containing matching records when data skipping uses column statistics. Currently, `StartsWith(column, prefix)` is translated into an equality-style range check: ```text minValue <= prefix AND maxValue >= prefix ``` For example, a file containing `abc1` and `abc9` is excluded by `name LIKE 'abc%'` because its minimum value is greater than `abc`, even though both records match. ### Summary and Changelog Correct prefix pruning to retain files whose statistics range may overlap the prefix range: ```text maxValue >= prefix AND (minValue <= prefix OR startsWith(minValue, prefix)) ``` - Preserve files when either minimum or maximum statistics are unavailable. - Apply this optimization only to direct binary string attributes; conservatively fall back for transformed expressions and non-binary collations. - Add regression coverage for matching ranges, singleton values, empty prefixes, missing statistics, Unicode, and compound predicates. - Add range-based checks ensuring matching values are never incorrectly pruned. - Add COW/MOR query regression tests comparing results with data skipping enabled and disabled, including `LIKE`, `startswith`, and `NOT LIKE`. **Validation** Executed with Spark 3.5.9 and Java 11: - `TestDataSkippingUtils`: 64 passed, 1 skipped. The skipped test requires Spark 4 non-binary collation support. - `TestDataSkippingQuery`: all 5 tests passed, including the new COW/MOR regression. - Scala style checks and `git diff --check` passed. ### Impact Prefix queries retain matching files and return correct results when data skipping is enabled. Predicates outside the supported optimization scope may scan additional files due to conservative fallback. Existing tables require no index rebuild. No configuration, storage-format, or public API changes are introduced. ### Risk Level low The change is limited to positive prefix-predicate translation. Boundary tests, matching-value checks, and COW/MOR query comparisons verify correctness. Conservative fallback can increase scanning but avoids unsafe pruning. ### Documentation Update none This restores the intended query behavior without adding configuration or features. ### Contributor's checklist - [ ] Contributor's guide reviewed by the PR author. - [x] Issue context and implementation details provided. - [x] Regression tests added and executed. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
