zhuqi-lucas commented on PR #25688: URL: https://github.com/apache/datafusion/pull/25688#issuecomment-5930852416
Thanks @alamb and @2010YOUY01 for the careful reads. **Our use case is planning performance.** We maintain a downstream engine on DataFusion with a number of custom physical rewrites interleaved into the optimizer chain, among them materialized view matching and a distributed execution rule that wraps subtrees for remote execution. Because each of them can invalidate requirements the plan had already satisfied, several physical rules repeat in our chain, `EnsureRequirements` six times in a single planning pass. Most of those repeats return the plan unchanged and still pay the full cost. Measured on a snapshot endpoint over a 34-node plan: physical optimization is **118.3 ms across 36 rule calls**, of which `EnsureRequirements` alone is **93.9 ms, 79.4%**. **Four of those six passes are byte-identical no-ops**, and 28 of 35 rule passes in total leave the plan untouched. So what we want is a fingerprint of the physical plan, so that a rule handed an input it has already been observed to leave unchanged is skipped instead of re-run. That is #25356. On the same chain it takes `EnsureRequirements` from 6 executions to 3 and physical optimization from **87.0 ms to 48.5 ms**, with the rendered plans byte-identical. For context on how this PR came about: @alamb proposed the phase split as an alternate strategy while reviewing #25356, with the goal of being able to rerun the optimizer rules to convergence. I think the two are complementary rather than exclusive. The fingerprint does not require any reordering of rules, it does not depend on rules being idempotent, and it covers repetition that a downstream chain authors, which the upstream rule list cannot see. For our use case specifically that last part is what matters, since our repeats come from our own rewrites rather than from the upstream ordering. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
