voonhous opened a new pull request, #20536:
URL: https://github.com/apache/hudi/pull/20536

   ### Describe the issue this Pull Request addresses
   
   Closes #20534
   
   With data skipping on, a filter on a large `IN` list fails the query with 
`StackOverflowError` (from about 4000 values) or `OutOfMemoryError` (about 2000 
values). `Verdict` expanded `In`, `InSet` and `NOT IN` with `reduce(Or)`, a 
left-deep tree with one level per value, and Spark's analyzer and codegen 
recurse over it. Neither error is `NonFatal`, so `fallback` mode does not help.
   
   ### Summary and Changelog
   
   - `Verdict` builds the OR with Spark's 
`PredicateHelper.buildBalancedPredicate`, log2(n) levels deep. Pruning is 
unchanged. The helper exists in Spark 3.3 through 4.2.
   - `TestDataSkippingUtils#testLargeInListCompilesToBalancedOr`: `In`, `InSet` 
and `NOT IN` over 5000 values. Checks which files are kept and that the stats 
filter is at most 20 levels deep. On master it fails with "lookup filter is 
5002 levels deep".
   
   ### Impact
   
   Queries with large `IN` lists and data skipping on no longer fail. Column 
stats table, 1 MB thread stacks, 4 GB heap:
   
   | IN values | before | after |
   | --- | --- | --- |
   | 1000 | 8.7 s | 1.2 s |
   | 2000 | `OutOfMemoryError` | 1.8 s |
   | 4000 / 8000 | `StackOverflowError` | 3.2 s / 6.1 s |
   | 16000 | - | 12.8 s |
   
   <details>
   <summary>Known limit: very large lists</summary>
   
   The stats filter is still two comparisons per value. At 32000 values, Janino 
runs out of a 4 GB heap compiling it. Capping that would mean checking only the 
list's overall min/max above some size, which costs pruning on sparse lists, so 
it is left out of this fix.
   
   </details>
   
   ### Risk Level
   
   low. The OR has the same leaves, so it evaluates to the same result. For one 
or two values the tree is identical, and the existing data skipping suites pass 
unchanged.
   
   ### Documentation Update
   
   none
   
   ### Contributor's checklist
   
   - [x] Read through [contributor's 
guide](https://hudi.apache.org/contribute/how-to-contribute)
   - [x] Enough context is provided in the sections above
   - [x] Adequate tests were added if applicable
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to