pharctor opened a new issue, #77300:
URL: https://github.com/apache/airflow/issues/77300

   ### Under which category would you file this issue?
   
   Airflow Core
   
   ### Apache Airflow version
   
   3.3.2. The code involved is unchanged on `main` (ea289c5) and on `v3-4-test`.
   
   ### What happened and how to reproduce it?
   
   **What happened**
   
   We cleared a few hundred task instances of a running Dag run whose Dag was 
paused. The scheduler logged a `StaleDataError` from 
`_update_dag_run_state_for_paused_dags`, then its loop died with 
`PendingRollbackError`. The process then sat in `LocalExecutor.end()` 
("Shutting down LocalExecutor; waiting for running tasks to finish"), so it 
neither scheduled anything nor exited, until the container was restarted by 
hand.
   
   **Cause**
   
   1. Clearing a task instance gives it a new primary key: 
`prepare_db_for_next_try` sets `self.id = uuid7()` 
([taskinstance.py#L1385](https://github.com/apache/airflow/blob/ea289c5bb41a6fa206f23dac210055547ea1bc7d/airflow-core/src/airflow/models/taskinstance.py#L1385)).
   2. Every 60 s, `_update_dag_run_state_for_paused_dags` calls 
`dag_run.update_state()` on each running run of a paused Dag 
([scheduler_job_runner.py#L1992](https://github.com/apache/airflow/blob/ea289c5bb41a6fa206f23dac210055547ea1bc7d/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L1992)).
 If a clear of that run commits between the read of its task instances and the 
flush, the UPDATE on the old id matches 0 rows and SQLAlchemy raises 
`StaleDataError`.
   3. The function's `except Exception as e:  # should not fail the scheduler` 
([L2016](https://github.com/apache/airflow/blob/ea289c5bb41a6fa206f23dac210055547ea1bc7d/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L2016))
 logs the error but does not roll the session back. On return, 
`@provide_session`'s `create_session` commits 
([session.py#L43](https://github.com/apache/airflow/blob/ea289c5bb41a6fa206f23dac210055547ea1bc7d/airflow-core/src/airflow/utils/session.py#L43)),
 which raises `PendingRollbackError` because the flush failed; `create_session` 
rolls back and re-raises it.
   4. The timer is registered without `non_fatal=True` 
([L2121](https://github.com/apache/airflow/blob/ea289c5bb41a6fa206f23dac210055547ea1bc7d/airflow-core/src/airflow/jobs/scheduler_job_runner.py#L2121)),
 so the exception escapes `timers.run()` and ends `_run_scheduler_loop`.
   
   **Steps to reproduce**
   
   Deterministic, on a fresh SQLite metadata DB with 
`AIRFLOW__CORE__EXECUTOR=LocalExecutor`. Put this Dag in the Dags folder:
   
   ```python
   import pendulum
   
   from airflow.sdk import dag, task
   
   
   @dag(schedule=None, start_date=pendulum.datetime(2026, 1, 1, tz="UTC"), 
catchup=False)
   def paused_sweep_repro():
       @task
       def first():
           pass
   
       @task
       def second():
           pass
   
       first() >> second()
   
   
   paused_sweep_repro()
   ```
   
   Run `airflow db migrate`, `airflow dags reserialize` and `airflow dags 
trigger paused_sweep_repro -r manual__repro`, then this script:
   
   ```python
   """Make the scheduler's paused-Dag sweep fail one flush and show what 
escapes it.
   
   Needs an initialized metadata DB holding the `paused_sweep_repro` Dag and a 
run `manual__repro` of it.
   """
   
   import datetime as dt
   import sys
   from unittest import mock
   
   from sqlalchemy import select, update
   
   from airflow.jobs.job import Job
   from airflow.jobs.scheduler_job_runner import SchedulerJobRunner
   from airflow.models.dag import DagModel
   from airflow.models.dagrun import DagRun
   from airflow.models.taskinstance import TaskInstance, uuid7
   from airflow.utils import timezone
   from airflow.utils.session import create_session
   from airflow.utils.state import DagRunState, TaskInstanceState
   
   DAG_ID, RUN_ID = "paused_sweep_repro", "manual__repro"
   
   # A paused Dag whose run is still RUNNING and has a task instance updated 
since the last scheduling decision:
   # exactly the runs the sweep selects.
   with create_session() as session:
       session.execute(update(DagModel).where(DagModel.dag_id == 
DAG_ID).values(is_paused=True))
       session.execute(
           update(DagRun)
           .where(DagRun.dag_id == DAG_ID, DagRun.run_id == RUN_ID)
           .values(state=DagRunState.RUNNING, 
last_scheduling_decision=timezone.utcnow() - dt.timedelta(hours=1))
       )
       session.execute(
           update(TaskInstance)
           .where(TaskInstance.dag_id == DAG_ID, TaskInstance.run_id == RUN_ID, 
TaskInstance.task_id == "first")
           .values(state=TaskInstanceState.RUNNING)
       )
   
   
   def update_state_after_concurrent_clear(self, *, session, 
execute_callbacks=True):
       """Stand in for DagRun.update_state when a clear of this run commits 
between its read and its flush."""
       ti = session.scalars(
           select(TaskInstance).where(TaskInstance.dag_id == self.dag_id, 
TaskInstance.run_id == self.run_id)
       ).first()
       # What the clear commits: TaskInstance.prepare_db_for_next_try gives the 
row a new primary key.
       table = TaskInstance.__table__
       session.execute(update(table).where(table.c.id == 
ti.id).values(id=uuid7()))
       ti.state = TaskInstanceState.UPSTREAM_FAILED
       session.flush()
       return [], None
   
   
   runner = SchedulerJobRunner(job=Job())
   with mock.patch.object(DagRun, "update_state", 
update_state_after_concurrent_clear):
       try:
           runner._update_dag_run_state_for_paused_dags()
       except Exception as e:
           print(f"ESCAPED the sweep: {type(e).__name__}: 
{str(e).splitlines()[0]}")
           sys.exit(1)
   print("Sweep returned normally: the scheduler loop would carry on.")
   ```
   
   The stand-in for `DagRun.update_state` does what the concurrent clear's 
commit does (re-keys the row through Core, behind the ORM) and then flushes, so 
the race is hit on every run. Output on 3.3.2:
   
   ```
   2026-11-19T10:00:03.329781Z [error    ] Failed to update dag run state for 
paused dags due to UPDATE statement on table 'task_instance' expected to update 
1 row(s); 0 were matched.
   ...
   sqlalchemy.orm.exc.StaleDataError: UPDATE statement on table 'task_instance' 
expected to update 1 row(s); 0 were matched.
   ESCAPED the sweep: PendingRollbackError: This Session's transaction has been 
rolled back due to a previous exception during flush. To begin a new 
transaction with this Session, first issue Session.rollback(). Original 
exception was: UPDATE statement on table 'task_instance' expected to update 1 
row(s); 0 were matched. (Background on this error at: 
https://sqlalche.me/e/20/7s2a)
   ```
   
   Outside the script it takes a clear of a paused Dag's running run, while 
that run's task instances are still being updated, landing inside a sweep. A 
larger clear holds its transaction open longer and widens the window.
   
   ### What you think should happen instead?
   
   A failed sweep should cost one cycle, as the `# should not fail the 
scheduler` comment intends: the scheduler loop keeps running, and the next 
sweep, 60 s later, reads fresh rows.
   
   ### Operating System
   
   Debian GNU/Linux 12 (bookworm), in the container
   
   ### Deployment
   
   Docker-Compose
   
   ### Apache Airflow Provider(s)
   
   _No response_
   
   ### Versions of Apache Airflow Providers
   
   _No response_
   
   ### Official Helm Chart version
   
   Not Applicable
   
   ### Kubernetes Version
   
   Not Applicable
   
   ### Helm Chart configuration
   
   Not Applicable
   
   ### Docker Image customizations
   
   Not Applicable
   
   ### Anything else?
   
   It happened once, on a clear of a few hundred task instances. #69487 added 
`non_fatal` to `call_regular_interval` and cites this function's handler as the 
existing pattern, but the handler does not cover the commit that 
`@provide_session` runs after it. #72596 describes the same failed-flush shape 
in two other periodic sweeps.
   
   Investigated with Claude Code; the reproduction above was run on Airflow 
3.3.2.
   
   ### Are you willing to submit PR?
   
   - [X] Yes I am willing to submit a PR!
   
   ### Code of Conduct
   
   - [X] I agree to follow this project's [Code of 
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

  • [I] Scheduler loop dies with P... pharctor (via GitHub) via commits

Reply via email to