askalt commented on issue #19351: URL: https://github.com/apache/datafusion/issues/19351#issuecomment-3823318860
> I think the main challenge is when ExecutionPlans need to coordinate sharing / communication across multiple nodes (e.g. dynamic filters) which is tricky Yes, it is a main we struggle with. `reset_plan_state` is enough fast for us (with all suggested optimizations, like https://github.com/apache/datafusion/pull/19792 and https://github.com/apache/datafusion/pull/19893). It seems, it would be good to have an ability to get and replace node physical expressions, this way we can rewrite dynamic filter/placeholder each time before re-use is required. Btw, there is PR implementing it for the DF plans, it would be great if you took a look: https://github.com/apache/datafusion/pull/20009 With these changes, we can extend `reset_plan_states` with the code like: ```rust let expressions = node.physical_expressions()?; let expressions_to_use = expressions.into_iter().map(|expr| { // Rewrite placeholder, dynamic fitler, or leave as is otherwise. }) let node = node.with_physical_expressions(expressions_to_use) ``` And then re-use a plan. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
