Hi all, Jumping in with a related question. Skimming the original issue, the motivating case is explicitly instance-wide: "we don't want to fail/kill running DAGs... Once none of the DAGs are in a running state, we perform the upgrades... Once upgrades are done, we update the DAG schedules back to their original value". That reads as an operator preparing for a maintenance window, not a Dag owner reaching for a per-Dag control.
I have two UX concerns, from each side of that use case. The design reserves the drain-or-pause choice for the moment a Dag has runs in-flight, based on the logic that this is when the choice actually matters. However, for a Dag owner, having runs in flight isn't the exception, it's precisely the state a Dag is in when I need to pause it, because something running is going wrong. That's exactly the moment I want pause to still mean "stop it," not a fork to resolve under pressure. And as an operator preparing for an upgrade, having to toggle and confirm this per Dag, one at a time, across however many active Dags there are, is close to the same manual, error-prone process this is meant to replace. Would a bulk action cover the use-case better? Like, drain and pause everything currently active in one click, and resume everything that was active in one click afterward? That directly matches the upgrade scenario without adding a new decision to the single-Dag pause flow. A per-Dag version only justifies its complexity if there's a real need to gracefully drain one specific Dag independent of an upgrade. I haven't seen that use case motivated anywhere in this thread, every example so far points to draining everything before touching the cluster. Curious what others think, happy to be wrong here. Thanks, Constance On Thu, Sep 10, 2026 at 12:20 PM Shubham Raj <[email protected]> wrote: > Hey, > > I agree with the direction of the PR. Given that is_paused serves several > critical functions, overloading it to include draining behavior would > likely cause unintended side effects across the system. By keeping > "draining" as a distinct, transient state, we can avoid those complications > while effectively solving the need for graceful task completion. This will > also significant improvement for managing Dags during upgrades. > > > On Wed, 9 Sep 2026 at 20:15, Elad Kalif <[email protected]> wrote: > > > > The drain here is a transient which eventually converges to pause. > > > > I agree. drain is intermediate state. When draining is completed the > status > > should be changed to paused. > > > > On Wed, Sep 9, 2026 at 1:49 PM Jarek Potiuk <[email protected]> wrote: > > > > > > The drain here is a transient which eventually converges to pause. > > > > > > Yep. That was my thinking. You summarized it all in one sentence :) > > > > > > On Wed, Sep 9, 2026 at 7:03 AM Dheeraj Turaga <[email protected] > > > > > wrote: > > > > > > > Hi Amogh, > > > > > > > > On Jarek's thought, I agree with him that just changing the pause > > > behavior > > > > is probably not a good idea. > > > > I can see instances where a user may want to instantly pause a dag in > > one > > > > case > > > > but want to drain a dag in another case. I myself have such a need > for > > > > different dags. > > > > > > > > For example, during airflow upgrades, I would like to gracefully > drain > > my > > > > whole instance before upgrade. > > > > While during regular operation, I may want to prevent my dag from > > > launching > > > > further tasks. > > > > > > > > The drain here is a transient which eventually converges to pause. > > > > > > > > Draining keeps task scheduling enabled, and only automated dagrun > > > creation > > > > paths are gated. > > > > *is_paused* is set to true only when the dagrun are completed. Manual > > > > triggers are still allowed > > > > simialr to the behavior when *is_paused* set to false. > > > > > > > > It would be great to get your thoughts on the PR once you get a > chance > > to > > > > play with it. > > > > > > > > Dheeraj > > > > > > > > On Tue, Sep 8, 2026 at 11:19 PM Amogh Desai <[email protected]> > > > wrote: > > > > > > > > > Yep, *is_paused* does more than block new runs. The scheduler > checks > > it > > > > > directly when deciding which > > > > > tasks to run next. That is what freezes downstream tasks today, and > > it > > > is > > > > > also why I would push back on > > > > > "same is_paused but two buttons." > > > > > > > > > > If you are suggesting that "Pause and drain" sets *is_paused=True* > > > right > > > > > away, those two checks still need a way > > > > > to know "this one is a soft pause, let it keep going." That means > > > adding > > > > a > > > > > second flag anyway, just hidden behind > > > > > the same name. And *is_paused* shows up in other places like > backfill > > > > pause > > > > > logic and asset triggered runs. Each of > > > > > those would need checking to make sure a "soft paused" Dag still > acts > > > > like > > > > > an active one everywhere except the one > > > > > spot it should not. From the PR, it avoids all that by keeping > > > > > *is_paused=False* during drain, so every existing check > > > > > keeps working as it should. It only adds new checks in a few places > > > that > > > > > decide whether to start a new run. > > > > > > > > > > Thanks & Regards, > > > > > Amogh Desai > > > > > > > > > > > > > > > On Tue, Sep 8, 2026 at 4:29 PM Jarek Potiuk <[email protected]> > > wrote: > > > > > > > > > > > I like the idea Amogh. I think **just changing** pause behaviour > > > might > > > > > not > > > > > > be good idea for compatibility (someone might rely on it's > current > > > > > > behaviour). > > > > > > > > > > > > But maybe we could have different behaviour of two > > > > > > different buttons/behaviour: "Pause" (current behaviour - stop > > > current > > > > > > tasks and it's downstream), "Pause and drain" - set the Dag to > > > "paused" > > > > > but > > > > > > let the running dag_runs to continue until completion. > > > > > > > > > > > > It might however require some scheduler changes - I think > currently > > > > > > is_paused is used to select run-eligible tasks ? > > > > > > > > > > > > J. > > > > > > > > > > > > > > > > > > On Tue, Sep 8, 2026 at 7:41 AM Amogh Desai < > [email protected]> > > > > > wrote: > > > > > > > > > > > > > Thanks for picking a 4-year-old issue. > > > > > > > The ask itself makes sense. > > > > > > > > > > > > > > One design question that might eventually come up in this > thread > > / > > > PR > > > > > is: > > > > > > > why do we need a new persisted state > > > > > > > instead of just changing what is_paused does? i.e: keep gating > > new > > > > run > > > > > > > creation on is_paused, but stop freezing > > > > > > > downstream tasks in created runs. That would get you graceful > > > > draining > > > > > > > without a schema change or migration. > > > > > > > > > > > > > > From my reading, doing that would silently change the semantics > > of > > > > > > > *is_paused*. Today, freezes downstream tasks in-flight. > > > > > > > Some operators pause specifically to stop a run's progress, not > > > just > > > > to > > > > > > > block new ones. Folding drain behavior into *is_paused* > > > > > > > might change that for everyone already relying on the current > > > freeze > > > > + > > > > > > > pause behavior, with no option to opt out, yes? > > > > > > > Worth clarifying. > > > > > > > > > > > > > > Thanks & Regards, > > > > > > > Amogh Desai > > > > > > > > > > > > > > > > > > > > > On Sun, Sep 6, 2026 at 3:15 AM Dheeraj Turaga < > > > > [email protected] > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hey Everyone, > > > > > > > > > > > > > > > > I wanted to start this discussion to get feedback on PR > #72407, > > > > which > > > > > > > > proposes adding a "draining" scheduling state to Airflow. > > > > > > > > > > > > > > > > The Problem > > > > > > > > > > > > > > > > Currently, pausing a DAG stops it mid-flight. While running > > tasks > > > > are > > > > > > > > allowed to finish, queued and downstream tasks remain > stranded > > > > until > > > > > > the > > > > > > > > DAG is unpaused. There is currently no native way to stop > > > starting > > > > > new > > > > > > > runs > > > > > > > > while allowing those already in flight to complete. > > > > > > > > > > > > > > > > This is a significant pain point during upgrades and > > maintenance. > > > > The > > > > > > > > current workaround as described in the four-year-old Issue > > #22006 > > > > > > > requires > > > > > > > > users to manually rewrite every DAG schedule to None, wait > for > > > runs > > > > > to > > > > > > > > drain, and then restore the schedules afterward. This process > > is > > > > > > invasive > > > > > > > > and prone to error. > > > > > > > > > > > > > > > > Proposed Solution: The draining State > > > > > > > > > > > > > > > > The PR introduces a draining state that sits between active > and > > > > > paused. > > > > > > > Key > > > > > > > > behaviors include: > > > > > > > > > > > > > > > > - No New Scheduled Runs: The scheduler creates no new runs > > for > > > a > > > > > > > draining > > > > > > > > DAG (including scheduled, asset-triggered, and > > partitioned/rollup > > > > > > paths). > > > > > > > > - Completion of In-Flight Runs: is_paused remains false > > during > > > > the > > > > > > > drain, > > > > > > > > meaning task instances in existing runs are still scheduled > and > > > > > finish > > > > > > > > normally. > > > > > > > > - Automatic Convergence: Once no unfinished runs remain, > the > > > > > > scheduler > > > > > > > > automatically moves the DAG to the paused state and writes a > > > > > > > > drain_completed audit log entry. > > > > > > > > > > > > > > > > The core property of this feature is that draining is > > transient, > > > > not > > > > > a > > > > > > > > third resting state; it always converges to paused. > > > > > > > > > > > > > > > > Implementation Details > > > > > > > > > > > > > > > > Explicit run creation via manual triggers, > > TriggerDagRunOperator, > > > > > asset > > > > > > > > materialization, or backfills remains allowed during > draining, > > > > > > mirroring > > > > > > > > the behavior of a paused DAG. An earlier revision that > blocked > > > > these > > > > > > > > actions was reverted to ensure draining is not stricter than > > the > > > > > state > > > > > > it > > > > > > > > converges into. > > > > > > > > > > > > > > > > Links: > > > > > > > > > > > > > > > > - PR: https://github.com/apache/airflow/pull/72407 > > > > > > > > - Issue: https://github.com/apache/airflow/issues/22006 > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Dheeraj > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
