thomasrebele opened a new pull request, #6889: URL: https://github.com/apache/hive/pull/6889
### What changes were proposed in this pull request? Avoid stack overflow errors by fixing the was-already-done check of HivePreFilteringRule. This has been achieved by providing a HepPlanner copy hook in CalcitePlanner that notifies the HiveRulesRegistry about copied RelNodes. ### Why are the changes needed? The was-already-done check of HivePreFilteringRule did not work, because the [registered RelNode](https://github.com/apache/hive/blob/cccc6efe72932f6d80b54f9fa6087b50d1bfcc5c/ql/src/java/org/apache/hadoop/hive/ql/optimizer/calcite/rules/HivePreFilteringRule.java#L199-L202)s were replaced by [copies in the HepPlanner](https://github.com/apache/calcite/blob/c01f6b5519d4a906cf41bbe7843f5929662cf5fd/core/src/main/java/org/apache/calcite/plan/hep/HepPlanner.java#L963). Notifying the HiveRulesRegistry about the copies lets the HivePreFilteringRule track the nodes that have already been processed. Other PRs (e.g., see [this discussion](https://github.com/apache/hive/pull/5196#discussion_r1660939490)) have targeted to avoid related stack overflows by improving the logic of the rule to avoid adding predicates that already hold for the input. It is unclear to me whether we can make the logic good enough to cover all scenarios. Even my proposed PR won't be enough if there is a second rule that produces additional filters (so there could be a loop HivePreFilteringRule > other rule > HivePreFilteringRule > other rule > ...). At least my PR covers the simple case of HivePreFilteringRule creating the same filter repeatedly. ### Does this PR introduce _any_ user-facing change? No ### How was this patch tested? Added a unit test in TestCBORuleFiredOnlyOnce that uses the production Hive code for executing the HepProgram. This is an adaptation of @soumyakanti3578's test in a previous [PR for this ticket](https://github.com/apache/hive/pull/5379/changes#diff-c508b2fbcd1072d3d782188d4ceab85ecb2b1c3b5fbbf2be13782c723d706ecf). I used the following query which produces a stack overflow error with commit 208488089df71e8a0dafd95b2812e5e4864baed9. I ported this PR on top of that commit and with these changes the query can be executed (without a stack overflow error): ``` create table tab1 (id string, col string) stored as orc; select * from tab1 a join tab1 b on a.id = b.id where lower(a.col) not in ('a', 'b') or lower(a.col) is null; ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
