2010YOUY01 commented on issue #26481: URL: https://github.com/apache/datafusion/issues/26481#issuecomment-6462764771
> With the [current design](https://github.com/apache/datafusion/pull/26757), the analyzer phase is mostly a wrapper around the existing optimizer rules: every built-in rule is still a `PhysicalOptimizerRule` and runs in the analyzer through `RunInAnalyzer`. @jayzhan211 Is the idea that the analyzer should only contain rules required to make the plan valid, while all other optimizations should move to the optimizer? The ultimate goal should be to improve optimizer maintainability. Following this heuristic might actually work against that goal, while simply splitting the analyzer and optimizer already improves maintainability. ## Good to let analyzer run pure optimizations I think we should actually encourage more optimizations to stay in the analyzer, following a progressive refinement design. - Earlier rules: plan shapes are more canonical, making transformations easier (e.g., aggregates have one canonical shape). - Later rules: plan shapes become more diverse, making transformations harder (e.g., RepartitionExec inserted, aggregates have multiple equivalent physical shapes). To reduce overall complexity, I think the guideline should be: - If a rule doesn't introduce more plan-shape diversity, run it earlier in the transformation-friendly zone. - If a rule introduces more diversity, run it later, so fewer subsequent rules need to handle it. ## Why the analyzer/optimizer split is still useful Although the split may look somewhat cosmetic initially, it establishes a clear contract: - Optimizer rules: valid in, valid out. - Analyzer rules: no such restriction; enforcement restores validity afterward. This alone is already an improvement that removes uncertainty. ## Regarding hidden invariants For the existing hidden invariants for WindowTopN and JoinSelection, moving them from analyzer to optimizer seem to increase complexity, the prototype should be verifying it. They probably need other techniques to solve. 🤔 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
