pharctor opened a new issue, #77300:
URL: https://github.com/apache/airflow/issues/77300
### Under which category would you file this issue?
Airflow Core
### Apache Airflow version
3.3.2. The code involved is unchanged on `main` (ea289c5) and on `v3-4-test`.
### What happened and how to reproduce it?
**What happened**
We cleared a few hundred task instances of a running Dag run whose Dag was
paused. The scheduler logged a `StaleDataError` from
`_update_dag_run_state_for_paused_dags`, then its loop died with
`PendingRollbackError`. The process then sat in `LocalExecutor.end()`
("Shutting down LocalExecutor; waiting for running tasks to finish"), so it
neither scheduled anything nor exited, until the container was restarted by
hand.
**Cause**
1. Clearing a task instance gives it a new primary key:
`prepare_db_for_next_try` sets `self.id = uuid7()`
([taskinstance.py#L1385](https://github.com/apache/airflow/blob/ea289c5bb41a6fa206f23dac210055547ea1bc7d/airflow-core/src/airflow/models/taskinstance.py#L1385)).
2. Every 60 s, `_update_dag_run_state_for_paused_dags` calls
`dag_run.update_state()` on each running run of a paused Dag
([scheduler_job_runner.py#L1992](https://github.com/apache/airflow/blob/ea289c5bb41a6fa206f23dac210055547ea1bc7d/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L1992)).
If a clear of that run commits between the read of its task instances and the
flush, the UPDATE on the old id matches 0 rows and SQLAlchemy raises
`StaleDataError`.
3. The function's `except Exception as e: # should not fail the scheduler`
([L2016](https://github.com/apache/airflow/blob/ea289c5bb41a6fa206f23dac210055547ea1bc7d/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L2016))
logs the error but does not roll the session back. On return,
`@provide_session`'s `create_session` commits
([session.py#L43](https://github.com/apache/airflow/blob/ea289c5bb41a6fa206f23dac210055547ea1bc7d/airflow-core/src/airflow/utils/session.py#L43)),
which raises `PendingRollbackError` because the flush failed; `create_session`
rolls back and re-raises it.
4. The timer is registered without `non_fatal=True`
([L2121](https://github.com/apache/airflow/blob/ea289c5bb41a6fa206f23dac210055547ea1bc7d/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L2121)),
so the exception escapes `timers.run()` and ends `_run_scheduler_loop`.
**Steps to reproduce**
Deterministic, on a fresh SQLite metadata DB with
`AIRFLOW__CORE__EXECUTOR=LocalExecutor`. Put this Dag in the Dags folder:
```python
import pendulum
from airflow.sdk import dag, task
@dag(schedule=None, start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
catchup=False)
def paused_sweep_repro():
@task
def first():
pass
@task
def second():
pass
first() >> second()
paused_sweep_repro()
```
Run `airflow db migrate`, `airflow dags reserialize` and `airflow dags
trigger paused_sweep_repro -r manual__repro`, then this script:
```python
"""Make the scheduler's paused-Dag sweep fail one flush and show what
escapes it.
Needs an initialized metadata DB holding the `paused_sweep_repro` Dag and a
run `manual__repro` of it.
"""
import datetime as dt
import sys
from unittest import mock
from sqlalchemy import select, update
from airflow.jobs.job import Job
from airflow.jobs.scheduler_job_runner import SchedulerJobRunner
from airflow.models.dag import DagModel
from airflow.models.dagrun import DagRun
from airflow.models.taskinstance import TaskInstance, uuid7
from airflow.utils import timezone
from airflow.utils.session import create_session
from airflow.utils.state import DagRunState, TaskInstanceState
DAG_ID, RUN_ID = "paused_sweep_repro", "manual__repro"
# A paused Dag whose run is still RUNNING and has a task instance updated
since the last scheduling decision:
# exactly the runs the sweep selects.
with create_session() as session:
session.execute(update(DagModel).where(DagModel.dag_id ==
DAG_ID).values(is_paused=True))
session.execute(
update(DagRun)
.where(DagRun.dag_id == DAG_ID, DagRun.run_id == RUN_ID)
.values(state=DagRunState.RUNNING,
last_scheduling_decision=timezone.utcnow() - dt.timedelta(hours=1))
)
session.execute(
update(TaskInstance)
.where(TaskInstance.dag_id == DAG_ID, TaskInstance.run_id == RUN_ID,
TaskInstance.task_id == "first")
.values(state=TaskInstanceState.RUNNING)
)
def update_state_after_concurrent_clear(self, *, session,
execute_callbacks=True):
"""Stand in for DagRun.update_state when a clear of this run commits
between its read and its flush."""
ti = session.scalars(
select(TaskInstance).where(TaskInstance.dag_id == self.dag_id,
TaskInstance.run_id == self.run_id)
).first()
# What the clear commits: TaskInstance.prepare_db_for_next_try gives the
row a new primary key.
table = TaskInstance.__table__
session.execute(update(table).where(table.c.id ==
ti.id).values(id=uuid7()))
ti.state = TaskInstanceState.UPSTREAM_FAILED
session.flush()
return [], None
runner = SchedulerJobRunner(job=Job())
with mock.patch.object(DagRun, "update_state",
update_state_after_concurrent_clear):
try:
runner._update_dag_run_state_for_paused_dags()
except Exception as e:
print(f"ESCAPED the sweep: {type(e).__name__}:
{str(e).splitlines()[0]}")
sys.exit(1)
print("Sweep returned normally: the scheduler loop would carry on.")
```
The stand-in for `DagRun.update_state` does what the concurrent clear's
commit does (re-keys the row through Core, behind the ORM) and then flushes, so
the race is hit on every run. Output on 3.3.2:
```
2026-11-19T10:00:03.329781Z [error ] Failed to update dag run state for
paused dags due to UPDATE statement on table 'task_instance' expected to update
1 row(s); 0 were matched.
...
sqlalchemy.orm.exc.StaleDataError: UPDATE statement on table 'task_instance'
expected to update 1 row(s); 0 were matched.
ESCAPED the sweep: PendingRollbackError: This Session's transaction has been
rolled back due to a previous exception during flush. To begin a new
transaction with this Session, first issue Session.rollback(). Original
exception was: UPDATE statement on table 'task_instance' expected to update 1
row(s); 0 were matched. (Background on this error at:
https://sqlalche.me/e/20/7s2a)
```
Outside the script it takes a clear of a paused Dag's running run, while
that run's task instances are still being updated, landing inside a sweep. A
larger clear holds its transaction open longer and widens the window.
### What you think should happen instead?
A failed sweep should cost one cycle, as the `# should not fail the
scheduler` comment intends: the scheduler loop keeps running, and the next
sweep, 60 s later, reads fresh rows.
### Operating System
Debian GNU/Linux 12 (bookworm), in the container
### Deployment
Docker-Compose
### Apache Airflow Provider(s)
_No response_
### Versions of Apache Airflow Providers
_No response_
### Official Helm Chart version
Not Applicable
### Kubernetes Version
Not Applicable
### Helm Chart configuration
Not Applicable
### Docker Image customizations
Not Applicable
### Anything else?
It happened once, on a clear of a few hundred task instances. #69487 added
`non_fatal` to `call_regular_interval` and cites this function's handler as the
existing pattern, but the handler does not cover the commit that
`@provide_session` runs after it. #72596 describes the same failed-flush shape
in two other periodic sweeps.
Investigated with Claude Code; the reproduction above was run on Airflow
3.3.2.
### Are you willing to submit PR?
- [X] Yes I am willing to submit a PR!
### Code of Conduct
- [X] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]