voonhous opened a new pull request, #20536: URL: https://github.com/apache/hudi/pull/20536
### Describe the issue this Pull Request addresses Closes #20534 With data skipping on, a filter on a large `IN` list fails the query with `StackOverflowError` (from about 4000 values) or `OutOfMemoryError` (about 2000 values). `Verdict` expanded `In`, `InSet` and `NOT IN` with `reduce(Or)`, a left-deep tree with one level per value, and Spark's analyzer and codegen recurse over it. Neither error is `NonFatal`, so `fallback` mode does not help. ### Summary and Changelog - `Verdict` builds the OR with Spark's `PredicateHelper.buildBalancedPredicate`, log2(n) levels deep. Pruning is unchanged. The helper exists in Spark 3.3 through 4.2. - `TestDataSkippingUtils#testLargeInListCompilesToBalancedOr`: `In`, `InSet` and `NOT IN` over 5000 values. Checks which files are kept and that the stats filter is at most 20 levels deep. On master it fails with "lookup filter is 5002 levels deep". ### Impact Queries with large `IN` lists and data skipping on no longer fail. Column stats table, 1 MB thread stacks, 4 GB heap: | IN values | before | after | | --- | --- | --- | | 1000 | 8.7 s | 1.2 s | | 2000 | `OutOfMemoryError` | 1.8 s | | 4000 / 8000 | `StackOverflowError` | 3.2 s / 6.1 s | | 16000 | - | 12.8 s | <details> <summary>Known limit: very large lists</summary> The stats filter is still two comparisons per value. At 32000 values, Janino runs out of a 4 GB heap compiling it. Capping that would mean checking only the list's overall min/max above some size, which costs pruning on sparse lists, so it is left out of this fix. </details> ### Risk Level low. The OR has the same leaves, so it evaluates to the same result. For one or two values the tree is identical, and the existing data skipping suites pass unchanged. ### Documentation Update none ### Contributor's checklist - [x] Read through [contributor's guide](https://hudi.apache.org/contribute/how-to-contribute) - [x] Enough context is provided in the sections above - [x] Adequate tests were added if applicable -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
