Hey Everyone, I wanted to start this discussion to get feedback on PR #72407, which proposes adding a "draining" scheduling state to Airflow.
The Problem Currently, pausing a DAG stops it mid-flight. While running tasks are allowed to finish, queued and downstream tasks remain stranded until the DAG is unpaused. There is currently no native way to stop starting new runs while allowing those already in flight to complete. This is a significant pain point during upgrades and maintenance. The current workaround as described in the four-year-old Issue #22006 requires users to manually rewrite every DAG schedule to None, wait for runs to drain, and then restore the schedules afterward. This process is invasive and prone to error. Proposed Solution: The draining State The PR introduces a draining state that sits between active and paused. Key behaviors include: - No New Scheduled Runs: The scheduler creates no new runs for a draining DAG (including scheduled, asset-triggered, and partitioned/rollup paths). - Completion of In-Flight Runs: is_paused remains false during the drain, meaning task instances in existing runs are still scheduled and finish normally. - Automatic Convergence: Once no unfinished runs remain, the scheduler automatically moves the DAG to the paused state and writes a drain_completed audit log entry. The core property of this feature is that draining is transient, not a third resting state; it always converges to paused. Implementation Details Explicit run creation via manual triggers, TriggerDagRunOperator, asset materialization, or backfills remains allowed during draining, mirroring the behavior of a paused DAG. An earlier revision that blocked these actions was reverted to ensure draining is not stricter than the state it converges into. Links: - PR: https://github.com/apache/airflow/pull/72407 - Issue: https://github.com/apache/airflow/issues/22006 Thanks, Dheeraj
