ahmed-mez opened a new issue, #16788: URL: https://github.com/apache/datafusion/issues/16788
### Describe the bug In PR [#16506](https://github.com/apache/datafusion/pull/16506), we added `#[recursive]` annotations (behind the `recursive_protection` feature flag) to a subset of recursive functions across the optimizer, physical planner, and expression evaluator. While this prevented stack-overflow panics on deeply nested plans, benchmarks showed a 1–2% slowdown in planning time when the feature is enabled. > “The benchmarks seem to imply this has some non trivial overhead (slows down planning by 1–2%). > I wonder if there is any way to restructure the affected codepaths to avoid the recursion (change them into an iterative version, for example, or Box more structures to reduce the stack size?)” > — @alamb > This epic will serve as a central place to explore multiple approaches (e.g., iterative rewrites, Box-based data structures to shrink stack usage, or other patterns) to reduce/eliminate deep recursion in key codepaths, remove the need for defensive macros, and regain any lost performance, all while still handling arbitrarily nested plans reliably. ### To Reproduce https://github.com/apache/datafusion/pull/16787 provides a reproducer. ### Expected behavior DataFusion doesn't overflow its stack. ### Additional context See the discussion in [#16506](https://github.com/apache/datafusion/pull/16506) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org --------------------------------------------------------------------- To unsubscribe, e-mail: github-unsubscr...@datafusion.apache.org For additional commands, e-mail: github-h...@datafusion.apache.org
