edmondop opened a new issue, #25906: URL: https://github.com/apache/datafusion/issues/25906
## Is your feature request related to a problem or challenge? DataFusion executes physical plans by polling streams from the root. Operators such as sorts and joins may consume input before producing output, but those barriers are implemented inside individual operators. A caller has no common plan-level contract to start consuming a partition independently of downstream polling, observe when it has finished, and decide when its output may flow. This makes it difficult for an external execution controller to coordinate stages without coupling itself to operator-specific behavior. The same control would also support deterministic pause/resume tests, stage-level measurements, and coarse admission control before starting another buffered stage. ## Describe the solution you'd like Add a `StageBoundary` contract for plan nodes that can be primed per partition, report when each partition is ready, and release buffered output when the caller decides to continue. The caller derives execution order from the plan and owns the decision about when to release each boundary. The contract should describe how the boundary preserves the input's schema, partitioning, ordering, batches, and errors. It should not prescribe a production buffering strategy or make the boundary responsible for driver policy. ## Describe alternatives you've considered Keeping barriers inside operators leaves each operator with its own behavior and gives external callers no shared way to coordinate completion and release. A driver could build a custom buffering wrapper, but then the coordination interface would be specific to that wrapper rather than a capability exposed by the plan node. ## Additional context - Part of the broader [AQE issue #23194](https://github.com/apache/datafusion/issues/23194). - Implemented by [PR #25798](https://github.com/apache/datafusion/pull/25798). - Out of scope here: an AQE driver, runtime re-planning rules, statistics integration, and a production spillable boundary implementation. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
