askalt commented on issue #21650: URL: https://github.com/apache/datafusion/issues/21650#issuecomment-4260789707
Our DF based DBMS project assumes that an execution plan can be reused, so for some time we struggled with the presence of a state in the plans (and currently we forcely rebuild the physical plan if dynamic filters are enabled or it is a recursive query). In our DF fork we stored a state in `TaskContext` for some time: this solution is described in details here https://github.com/apache/datafusion/issues/19351 and it is seems similar to @xudong963 suggestion: > Add a SharedStateRegistry to TaskContext — a HashMap<SharedStateId, Arc<dyn Any + Send + Sync>> where SharedStateId is a lightweight identifier (probably a u64 or a newtype around usize). Additionally it defines a plan state trait: ```rust /// Generic plan state. pub trait PlanState: std::fmt::Debug + std::any::Any + Send + Sync { /// Represent this state as any. fn as_any(&self) -> &dyn std::any::Any; /// Return plan metrics if some. fn metrics(&self) -> Option<crate::metrics::ExecutionPlanMetricsSet>; } ``` We used `ExecutionPlan` raw pointer addresses as `SharedStateId` as actually we need to establish the correspondence between plan nodes and its state after an execution is finished: e.g. to recover node metrics. > State initialization happens in a new prepare_execution phase — before the first execute() call We need to figure out a solution to keep it backward compatible, as currently any DF based code expects that a pure `.execute(...)` call is enough to launch a plan properly. e.g. keep a state in both plan and task context for some time and decide which one to use based on the `TaskContext` attached value. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
