manipatnam opened a new pull request, #72054:
URL: https://github.com/apache/airflow/pull/72054
# Fix cleared backfill DAG runs getting stuck in `queued` state
## Problem
A DAG run that belongs to a **cancelled backfill** can get permanently stuck
in the `queued` state. The scheduler never promotes it to `running`, no matter
how much capacity is available, and clearing it repeatedly does not help.
### Root cause
Cancelling a backfill (`PUT /backfills/{id}/cancel`) does two things to the
`Backfill` row:
- sets `is_paused = True`
- sets `completed_at` to the cancel time
The scheduler's promotion query,
`DagRun.get_queued_dag_runs_to_set_running`,decides which queued runs to move
to `running`. It filters out any run whose backfill is paused:
```python
# don't set paused dag runs as running
not_(coalesce(cast("ColumnElement[bool]", Backfill.is_paused), False)),
```
This filter looks only at `is_paused` and ignores `completed_at`. That is
correct while a backfill is *live* (a user pauses it temporarily and expects
its runs to stop starting). But once a backfill is **completed/cancelled**,
`is_paused` is a stale flag that is never cleared — so any queued run still
associated with that backfill is skipped forever.
A run reaches this state through a normal, reasonable sequence:
1. A backfill reprocesses an existing run (the run gets `backfill_id` set
and `run_type = backfill`).
2. The user cancels the backfill → `is_paused = True`, `completed_at` set.
3. The user later clears that run to re-run it → the run returns to
`queued`,still associated with the now-cancelled backfill.
4. The scheduler skips it on every loop because `is_paused = True`. Stuck.
`last_scheduling_decision` stays `NULL` on the stuck run, confirming the
scheduler never even evaluates it.
## Fix
Treat `is_paused` as meaningful only while the backfill is still running. A
run whose backfill has already completed must remain schedulable:
```python
# don't set paused dag runs as running, but a completed backfill's
# is_paused flag is stale: a run cleared after the backfill was
# cancelled (cancel sets is_paused=True and completed_at) must still
# be schedulable, otherwise it is stuck in queued forever.
or_(
Backfill.completed_at.isnot(None),
not_(coalesce(cast("ColumnElement[bool]", Backfill.is_paused), False)),
),
```
This is the smallest possible change and targets the exact defective
assumption. Because the fix operates purely on the promotion query,
**previously stranded runs recover automatically** on the next scheduler loop —
no manual DB intervention or re-clearing is required.
## Reproduction
Minimal DAG:
```python
from airflow.sdk import DAG, chain
from airflow.providers.standard.operators.python import PythonOperator
from pendulum import datetime
with DAG(
dag_id="repro_backfill_clear_stuck",
schedule="@daily",
start_date=datetime(2026, 7, 1),
catchup=False,
max_active_runs=1,
) as dag:
a = PythonOperator(task_id="a", python_callable=lambda: None)
b = PythonOperator(task_id="b", python_callable=lambda: None)
chain(a, b)
```
Steps:
1. Unpause the DAG and let a scheduled run complete.
2. Create a backfill for a past interval:
```bash
airflow backfill create --dag-id repro_backfill_clear_stuck \
--from-date "2026-08-01T00:00:00+00:00" \
--to-date "2026-08-02T00:00:00+00:00"
```
3. **Cancel the backfill** (from the UI or API). This sets `is_paused =
True` and `completed_at` on the backfill.
4. Mark the backfill's DAG run as **failed**, then **clear** it.
5. Observe the run returns to `queued` and never progresses. Its
`last_scheduling_decision` stays `NULL`.
Verify the stuck state:
```sql
SELECT dr.run_id, dr.state, dr.backfill_id, b.is_paused, b.completed_at
FROM dag_run dr
JOIN backfill b ON b.id = dr.backfill_id
WHERE dr.dag_id = 'repro_backfill_clear_stuck'
AND dr.state = 'queued';
```
> Note: cancelling is required to reproduce. A backfill that completes
*normally*leaves `is_paused = False`, so its cleared > runs are promoted and
this bug does not surface.
With the fix applied, the run is promoted to `running` on the next scheduler
loop.
## Tests
Added
`test_backfill_runs_started_when_backfill_completed_despite_paused_flag`
in`airflow-core/tests/unit/jobs/test_scheduler_job.py`, next to the existing
`test_backfill_runs_not_started_when_backfill_paused`. It creates a backfill,
simulates a cancel (`is_paused = True` + `completed_at`), runs
`_start_queued_dagruns`, and asserts the runs are promoted (up to the
backfill's `max_active_runs`).
- Passes with the fix.
- **Fails without the fix** (0 runs promoted instead of 3), confirming it is
a genuine regression test.
The sibling `test_backfill_runs_not_started_when_backfill_paused` still
passes, confirming an *active* (non-completed) paused backfill continues to
hold its runs.
## Behavior notes and limitations
- **The run remains a backfill run.** The fix only changes promotion
eligibility; it does not alter `run_type`, `backfill_id`, or the
`BackfillDagRun` association. The run keeps its lower (`sort_ordinal`)
scheduling priority and remains governed by the backfill's `max_active_runs`.
- **The run's own data is recorded correctly.** State, start/end dates,
duration, task instances, and any data the DAG produces are written normally
when it runs.
- **Backfill aggregate metadata is not refreshed.** The parent
`Backfill.completed_at` stays frozen at the cancel time and
`_mark_backfills_complete` never revisits an already-completed backfill. So a
run that executes after the backfill was cancelled produces a reporting
inconsistency: the backfill shows completed earlier than the actual run. This
is a metadata/reporting artifact, not a data-integrity problem for the DAG's
outputs.
## Alternatives considered
- **Detach the run from the terminal backfill on clear** (set `backfill_id =
NULL`, drop the `BackfillDagRun` row, revert `run_type`). Cleaner lineage
semantics and avoids the stale-`completed_at` inconsistency, but touches the
hot `clear_task_instances` path and discards the "this interval was part of
backfill N" lineage.
- **Block clearing runs of completed/cancelled backfills** (return `409`).
Prevents new orphans but is a breaking UX change and does not recover
already-stuck runs.
Option A (this PR) was chosen for its minimal blast radius and automatic
recovery of existing stuck runs. Happy to switch direction if maintainers
prefer detaching the run or refreshing the backfill's `completed_at` when a run
becomes eligible again.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
