1fanwang opened a new pull request, #77205:
URL: https://github.com/apache/airflow/pull/77205

    <!-- SPDX-License-Identifier: Apache-2.0
         https://www.apache.org/licenses/LICENSE-2.0 -->
   
   When an executor reports several failed tasks in one batch, for example when 
the workers running them are killed, and one of those task instances is deleted 
before the scheduler gets to it, the scheduler process exits. Deleting a task 
instance from the Task Instances page or through the REST API works on running 
tasks. It can land mid-batch because the scheduler commits after each failure 
it records, which releases the row locks on the rest of the batch. The 
scheduler log ends with:
   
   ```text
   sqlalchemy.orm.exc.StaleDataError: UPDATE statement on table 'task_instance' 
expected to update 1 row(s); 0 were matched.
   ```
   
   With retries left, it ends with ValueError: An archived task instance cannot 
be archived again. The heartbeat timeout check crashes the same way.
   
   Reloading a task instance whose row is gone used to mark it as changed, so 
failure handling went on to write a row that no longer exists. Now the reload 
leaves nothing to write and tells the caller the row is gone, and failure 
handling logs a warning and skips that task instance.
   
   https://github.com/apache/airflow/pull/76544 keeps the scheduler running 
when failure handling raises for any reason. Its fallback write also fails on a 
deleted row, so this root-cause fix complements it. A deleted task instance 
that was cleared while it was running still fails when its restart completes; 
this change leaves that path alone.
   
   related: https://github.com/apache/airflow/issues/76529
   
   ## Testing Done
   
   Setup: macOS 26.7.3 (arm64), Python 3.12.13, PostgreSQL 16.16 in Docker on 
127.0.0.1:25432, LocalExecutor, an apache/airflow checkout in $TREE prepared 
with uv sync --frozen, and the three files below in the current directory; on 
both commits the scheduler alone gets inject/sitecustomize.py on its 
PYTHONPATH, and that hook deletes the other task instance through the REST API 
right after the scheduler commits the first failure of the batch.
   
   Current main, 
https://github.com/apache/airflow/commit/3b35135afe9549e99088c4d557c62e5deafb125b:
   
   ```bash
   git -C "$TREE" checkout --quiet --detach 3b35135afe
   "$TREE/.venv/bin/python" run_e2e.py --label main --tree "$TREE" > 
run-main.txt
   grep -vE '^(TREE|POSTGRES) ' run-main.txt | sed -E 's# \(See 
https:[^)]*\)##; s# Learn more: https:[^ ]*##; s#https?://[^/ ]+##g'
   ```
   
   ```text
   STARTED api-server pid=72823
   STARTED dag-processor pid=72824
   STARTED scheduler pid=72825
   TRIGGERED staledata_e2e run_id=staledata-main
   RUNNING first: task pid 73468, LocalExecutor worker pid 73445 (child of 
scheduler pid 72825)
   RUNNING second: task pid 73469, LocalExecutor worker pid 73444 (child of 
scheduler pid 72825)
   SIGKILL worker pids [73445, 73444] and task pids [73468, 73469]
   INJECT second committed as failed; DELETE 
/api/v2/dags/staledata_e2e/dagRuns/staledata-main/taskInstances/first -> HTTP 
200
   SCHEDULER exited with code 1 (exited)
   AFTER staledata_e2e_after after-main: still queued after 60s
   TASK_INSTANCE rows (all attempts):
     staledata_e2e second 1 failed True
     staledata_e2e_after ping 0 None True
   {"label": "main", "scheduler_exit_code": 1}
   ```
   
   ```bash
   sed -n '/Received executor event with state failed/,/Exited execute loop/p' 
runs/main/scheduler.log | sed "s|$TREE/||g" | sed -E 's# \(See https:[^)]*\)##; 
s# Learn more: https:[^ ]*##; s#https?://[^/ ]+##g'
   ```
   
   ```text
   2026-11-17T15:00:20.033019Z [info     ] Received executor event with state 
failed for task instance 01a1ea05-d884-7a6f-babf-9f88d66ba6cc 
(coordinates=TaskInstanceKey(dag_id='staledata_e2e', task_id='second', 
run_id='staledata-main', try_number=1, map_index=-1)) 
[airflow.jobs.scheduler_job_runner.SchedulerJobRunner] 
loc=scheduler_job_runner.py:1584
   2026-11-17T15:00:20.033170Z [info     ] Received executor event with state 
failed for task instance 01a1ea05-d883-7c17-acd9-0af2caae8fca 
(coordinates=TaskInstanceKey(dag_id='staledata_e2e', task_id='first', 
run_id='staledata-main', try_number=1, map_index=-1)) 
[airflow.jobs.scheduler_job_runner.SchedulerJobRunner] 
loc=scheduler_job_runner.py:1584
   2026-11-17T15:00:20.046248Z [info     ] TaskInstance Finished: 
dag_id=staledata_e2e, task_id=second, run_id=staledata-main, map_index=-1, 
ti_id=01a1ea05-d884-7a6f-babf-9f88d66ba6cc, run_start_date=2026-11-17 
15:00:16.814869+00:00, run_end_date=None, run_duration=None, state=running, 
executor=LocalExecutor(parallelism=4), executor_state=failed, try_number=1, 
max_tries=0, pool=default_pool, queue=default, priority_weight=1, 
operator=_PythonDecoratedOperator, queued_dttm=2026-11-17 
15:00:15.016177+00:00, scheduled_dttm=2026-11-17 
15:00:15.001830+00:00,queued_by_job_id=1, pid=73469 
[airflow.jobs.scheduler_job_runner.SchedulerJobRunner] 
loc=scheduler_job_runner.py:1738
   2026-11-17T15:00:20.047802Z [error    ] Executor 
LocalExecutor(parallelism=4) reported that the task instance <TaskInstance: 
staledata_e2e.second staledata-main [running] 
ti_id=01a1ea05-d884-7a6f-babf-9f88d66ba6cc> finished with state failed, but the 
task instance's state attribute is running. [airflow.task] 
loc=taskinstance.py:2830
   2026-11-17T15:00:20.049873Z [info     ] Marking task as FAILED. 
dag_id=staledata_e2e, task_id=second, run_id=staledata-main, logical_date=, 
start_date=20261117T150016, end_date=20261117T150020 
[airflow.models.taskinstance] loc=taskinstance.py:2924
   INJECT second committed as failed; DELETE 
/api/v2/dags/staledata_e2e/dagRuns/staledata-main/taskInstances/first -> HTTP 
200
   2026-11-17T15:00:20.089455Z [info     ] TaskInstance Finished: 
dag_id=staledata_e2e, task_id=first, run_id=staledata-main, map_index=-1, 
ti_id=01a1ea05-d883-7c17-acd9-0af2caae8fca, run_start_date=2026-11-17 
15:00:16.814856+00:00, run_end_date=None, run_duration=None, state=running, 
executor=LocalExecutor(parallelism=4), executor_state=failed, try_number=1, 
max_tries=0, pool=default_pool, queue=default, priority_weight=1, 
operator=_PythonDecoratedOperator, queued_dttm=2026-11-17 
15:00:15.016177+00:00, scheduled_dttm=2026-11-17 
15:00:15.001830+00:00,queued_by_job_id=1, pid=73468 
[airflow.jobs.scheduler_job_runner.SchedulerJobRunner] 
loc=scheduler_job_runner.py:1738
   2026-11-17T15:00:20.092208Z [error    ] Executor 
LocalExecutor(parallelism=4) reported that the task instance <TaskInstance: 
staledata_e2e.first staledata-main [running] 
ti_id=01a1ea05-d883-7c17-acd9-0af2caae8fca> finished with state failed, but the 
task instance's state attribute is running. [airflow.task] 
loc=taskinstance.py:2830
   2026-11-17T15:00:20.093633Z [info     ] Marking task as FAILED. 
dag_id=staledata_e2e, task_id=first, run_id=staledata-main, logical_date=, 
start_date=20261117T150016, end_date=20261117T150020 
[airflow.models.taskinstance] loc=taskinstance.py:2924
   2026-11-17T15:00:20.098436Z [error    ] Exception when executing 
SchedulerJob._run_scheduler_loop 
[airflow.jobs.scheduler_job_runner.SchedulerJobRunner] 
loc=scheduler_job_runner.py:1976
   Traceback (most recent call last):
     File "airflow-core/src/airflow/jobs/scheduler_job_runner.py", line 1970, 
in _execute
       self._run_scheduler_loop()
     File "airflow-core/src/airflow/jobs/scheduler_job_runner.py", line 2193, 
in _run_scheduler_loop
       num_finished_events += self._process_executor_events(
                              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "airflow-core/src/airflow/jobs/scheduler_job_runner.py", line 1517, 
in _process_executor_events
       return SchedulerJobRunner.process_executor_events(
              ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
     File "airflow-core/src/airflow/jobs/scheduler_job_runner.py", line 1915, 
in process_executor_events
       ti.handle_failure(error=msg, session=session)
     File "airflow-core/src/airflow/utils/session.py", line 98, in wrapper
       return func(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^
     File "airflow-core/src/airflow/models/taskinstance.py", line 2927, in 
handle_failure
       TaskInstance.save_to_db(ti, session=session)
     File "airflow-core/src/airflow/utils/session.py", line 98, in wrapper
       return func(*args, **kwargs)
              ^^^^^^^^^^^^^^^^^^^^^
     File "airflow-core/src/airflow/models/taskinstance.py", line 2889, in 
save_to_db
       session.flush()
     File ".venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 
4360, in flush
       self._flush(objects)
     File ".venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 
4495, in _flush
       with util.safe_reraise():
            ^^^^^^^^^^^^^^^^^^^
     File ".venv/lib/python3.12/site-packages/sqlalchemy/util/langhelpers.py", 
line 122, in __exit__
       raise exc_value.with_traceback(exc_tb)
     File ".venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line 
4456, in _flush
       flush_context.execute()
     File ".venv/lib/python3.12/site-packages/sqlalchemy/orm/unitofwork.py", 
line 465, in execute
       rec.execute(self)
     File ".venv/lib/python3.12/site-packages/sqlalchemy/orm/unitofwork.py", 
line 641, in execute
       util.preloaded.orm_persistence.save_obj(
     File ".venv/lib/python3.12/site-packages/sqlalchemy/orm/persistence.py", 
line 86, in save_obj
       _emit_update_statements(
     File ".venv/lib/python3.12/site-packages/sqlalchemy/orm/persistence.py", 
line 949, in _emit_update_statements
       raise orm_exc.StaleDataError(
   sqlalchemy.orm.exc.StaleDataError: UPDATE statement on table 'task_instance' 
expected to update 1 row(s); 0 were matched.
   2026-11-17T15:00:20.105922Z [info     ] Shutting down LocalExecutor; waiting 
for running tasks to finish.  Signal again if you don't want to wait. 
[airflow.executors.local_executor.LocalExecutor] loc=local_executor.py:283
   2026-11-17T15:00:20.347192Z [info     ] Exited execute loop            
[airflow.jobs.scheduler_job_runner.SchedulerJobRunner] 
loc=scheduler_job_runner.py:1988
   ```
   
   With this change:
   
   ```bash
   git -C "$TREE" checkout --quiet --detach 6784d0c01f
   "$TREE/.venv/bin/python" run_e2e.py --label fix --tree "$TREE" > run-fix.txt
   grep -vE '^(TREE|POSTGRES) ' run-fix.txt | sed -E 's# \(See https:[^)]*\)##; 
s# Learn more: https:[^ ]*##; s#https?://[^/ ]+##g'
   ```
   
   ```text
   STARTED api-server pid=80140
   STARTED dag-processor pid=80141
   STARTED scheduler pid=80142
   TRIGGERED staledata_e2e run_id=staledata-fix
   RUNNING first: task pid 80299, LocalExecutor worker pid 80271 (child of 
scheduler pid 80142)
   RUNNING second: task pid 80300, LocalExecutor worker pid 80272 (child of 
scheduler pid 80142)
   SIGKILL worker pids [80271, 80272] and task pids [80299, 80300]
   INJECT second committed as failed; DELETE 
/api/v2/dags/staledata_e2e/dagRuns/staledata-fix/taskInstances/first -> HTTP 200
   SCHEDULER pid=80142 still running 6s after the kill
   AFTER staledata_e2e_after after-fix: success in 3s
   TASK_INSTANCE rows (all attempts):
     staledata_e2e second 1 failed True
     staledata_e2e_after ping 1 success True
   {"label": "fix", "scheduler_exit_code": null}
   ```
   
   ```bash
   sed -n '/Received executor event with state failed/,/no longer exists/p' 
runs/fix/scheduler.log | sed "s|$TREE/||g" | sed -E 's# \(See https:[^)]*\)##; 
s# Learn more: https:[^ ]*##; s#https?://[^/ ]+##g'
   grep -c Traceback runs/fix/scheduler.log
   ```
   
   ```text
   2026-11-17T15:02:11.550673Z [info     ] Received executor event with state 
failed for task instance 01a1ea07-8bda-73ad-bd86-dd102d711264 
(coordinates=TaskInstanceKey(dag_id='staledata_e2e', task_id='first', 
run_id='staledata-fix', try_number=1, map_index=-1)) 
[airflow.jobs.scheduler_job_runner.SchedulerJobRunner] 
loc=scheduler_job_runner.py:1584
   2026-11-17T15:02:11.550791Z [info     ] Received executor event with state 
failed for task instance 01a1ea07-8bdb-786d-bf7d-9950b63c88e0 
(coordinates=TaskInstanceKey(dag_id='staledata_e2e', task_id='second', 
run_id='staledata-fix', try_number=1, map_index=-1)) 
[airflow.jobs.scheduler_job_runner.SchedulerJobRunner] 
loc=scheduler_job_runner.py:1584
   2026-11-17T15:02:11.565201Z [info     ] TaskInstance Finished: 
dag_id=staledata_e2e, task_id=second, run_id=staledata-fix, map_index=-1, 
ti_id=01a1ea07-8bdb-786d-bf7d-9950b63c88e0, run_start_date=2026-11-17 
15:02:08.179954+00:00, run_end_date=None, run_duration=None, state=running, 
executor=LocalExecutor(parallelism=4), executor_state=failed, try_number=1, 
max_tries=0, pool=default_pool, queue=default, priority_weight=1, 
operator=_PythonDecoratedOperator, queued_dttm=2026-11-17 
15:02:06.519514+00:00, scheduled_dttm=2026-11-17 
15:02:06.504362+00:00,queued_by_job_id=1, pid=80300 
[airflow.jobs.scheduler_job_runner.SchedulerJobRunner] 
loc=scheduler_job_runner.py:1738
   2026-11-17T15:02:11.566836Z [error    ] Executor 
LocalExecutor(parallelism=4) reported that the task instance <TaskInstance: 
staledata_e2e.second staledata-fix [running] 
ti_id=01a1ea07-8bdb-786d-bf7d-9950b63c88e0> finished with state failed, but the 
task instance's state attribute is running. [airflow.task] 
loc=taskinstance.py:2834
   2026-11-17T15:02:11.569189Z [info     ] Marking task as FAILED. 
dag_id=staledata_e2e, task_id=second, run_id=staledata-fix, logical_date=, 
start_date=20261117T150208, end_date=20261117T150211 
[airflow.models.taskinstance] loc=taskinstance.py:2933
   INJECT second committed as failed; DELETE 
/api/v2/dags/staledata_e2e/dagRuns/staledata-fix/taskInstances/first -> HTTP 200
   2026-11-17T15:02:11.612812Z [info     ] TaskInstance Finished: 
dag_id=staledata_e2e, task_id=first, run_id=staledata-fix, map_index=-1, 
ti_id=01a1ea07-8bda-73ad-bd86-dd102d711264, run_start_date=2026-11-17 
15:02:08.179956+00:00, run_end_date=None, run_duration=None, state=running, 
executor=LocalExecutor(parallelism=4), executor_state=failed, try_number=1, 
max_tries=0, pool=default_pool, queue=default, priority_weight=1, 
operator=_PythonDecoratedOperator, queued_dttm=2026-11-17 
15:02:06.519514+00:00, scheduled_dttm=2026-11-17 
15:02:06.504362+00:00,queued_by_job_id=1, pid=80299 
[airflow.jobs.scheduler_job_runner.SchedulerJobRunner] 
loc=scheduler_job_runner.py:1738
   2026-11-17T15:02:11.615682Z [error    ] Executor 
LocalExecutor(parallelism=4) reported that the task instance <TaskInstance: 
staledata_e2e.first staledata-fix [running] 
ti_id=01a1ea07-8bda-73ad-bd86-dd102d711264> finished with state failed, but the 
task instance's state attribute is running. [airflow.task] 
loc=taskinstance.py:2834
   2026-11-17T15:02:11.616996Z [warning  ] Task instance no longer exists, 
skipping its failure handling [airflow.task] loc=taskinstance.py:2836 
task_instance_key=TaskInstanceKey(dag_id='staledata_e2e', task_id='first', 
run_id='staledata-fix', try_number=1, map_index=-1)
   0
   ```
   
   <details>
   <summary>Reproducer source: run_e2e.py</summary>
   
   ```python
   #!/usr/bin/env python3
   # Real-scheduler reproduction: two running tasks lose their LocalExecutor 
workers to SIGKILL, so the
   # executor reports both failures in one batch; the injected hook deletes the 
second task instance
   # through the REST API right after the scheduler commits the first failure.
   from __future__ import annotations
   
   import argparse
   import json
   import os
   import secrets
   import shutil
   import signal
   import subprocess
   import sys
   import time
   import urllib.request
   from pathlib import Path
   
   import psycopg2
   
   PG_DSN = "host=127.0.0.1 port=25432 user=airflow password=airflow"
   API = "http://127.0.0.1:18100";
   HERE = Path(__file__).resolve().parent
   
   
   def query(database: str, sql: str, params: tuple = ()) -> list[tuple]:
       with psycopg2.connect(f"{PG_DSN} dbname={database}") as connection, 
connection.cursor() as cursor:
           cursor.execute(sql, params)
           return cursor.fetchall()
   
   
   def wait_for(description: str, check, timeout: float):
       deadline = time.monotonic() + timeout
       while time.monotonic() < deadline:
           if result := check():
               return result
           time.sleep(0.2)
       raise TimeoutError(description)
   
   
   def main() -> None:
       parser = argparse.ArgumentParser()
       parser.add_argument("--label", required=True)
       parser.add_argument("--tree", required=True, type=Path)
       args = parser.parse_args()
       database = f"staledata_{args.label}"
       out = HERE / "runs" / args.label
       shutil.rmtree(out, ignore_errors=True)
       (out / "home" / "dags").mkdir(parents=True)
       shutil.copy(HERE / "staledata_e2e_dag.py", out / "home" / "dags")
       airflow = str(args.tree / ".venv" / "bin" / "airflow")
       head = subprocess.run(
           ["git", "-C", str(args.tree), "log", "-1", "--format=%h %s"], 
capture_output=True, text=True
       ).stdout.strip()
       dirty = subprocess.run(
           ["git", "-C", str(args.tree), "status", "--short", 
"--untracked-files=no"],
           capture_output=True,
           text=True,
       ).stdout.split("\n")
       print(f"TREE {head} (uncommitted tracked changes: {len([line for line in 
dirty if line])})")
   
       admin = psycopg2.connect(f"{PG_DSN} dbname=airflow")
       admin.autocommit = True
       with admin.cursor() as cursor:
           cursor.execute(f"DROP DATABASE IF EXISTS {database}")
           cursor.execute(f"CREATE DATABASE {database}")
           cursor.execute("SHOW server_version")
           print(f"POSTGRES {cursor.fetchone()[0]} database={database}")
       admin.close()
   
       env = {
           **os.environ,
           "AIRFLOW_HOME": str(out / "home"),
           "AIRFLOW__DATABASE__SQL_ALCHEMY_CONN": (
               
f"postgresql+psycopg2://airflow:[email protected]:25432/{database}"
           ),
           "AIRFLOW__CORE__EXECUTOR": "LocalExecutor",
           "AIRFLOW__CORE__PARALLELISM": "4",
           "AIRFLOW__CORE__DAGS_FOLDER": str(out / "home" / "dags"),
           "AIRFLOW__CORE__LOAD_EXAMPLES": "False",
           "AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION": "False",
           "AIRFLOW__CORE__EXECUTION_API_SERVER_URL": f"{API}/execution/",
           "AIRFLOW__CORE__SIMPLE_AUTH_MANAGER_ALL_ADMINS": "True",
           "AIRFLOW__API__BASE_URL": API,
           "AIRFLOW__API_AUTH__JWT_SECRET": secrets.token_urlsafe(32),
           "AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL": "2",
           "AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT": "600",
       }
       with open(out / "db-migrate.log", "wb") as log:
           subprocess.run([airflow, "db", "migrate"], env=env, stdout=log, 
stderr=subprocess.STDOUT, check=True)
   
       processes: dict[str, subprocess.Popen] = {}
   
       def launch(name: str, command: list[str], extra_env: dict[str, str] | 
None = None) -> None:
           log = open(out / f"{name}.log", "wb")
           processes[name] = subprocess.Popen(
               [airflow, *command],
               env={**env, **(extra_env or {})},
               stdout=log,
               stderr=subprocess.STDOUT,
               start_new_session=True,
           )
           print(f"STARTED {name} pid={processes[name].pid}")
   
       try:
           launch("api-server", ["api-server", "--host", "127.0.0.1", "--port", 
"18100", "--workers", "1"])
           launch("dag-processor", ["dag-processor"])
           launch(
               "scheduler",
               ["scheduler", "--skip-serve-logs"],
               {"PYTHONPATH": str(HERE / "inject"), "STALEDATA_API": API},
           )
   
           def api_up() -> bool:
               try:
                   with urllib.request.urlopen(f"{API}/api/v2/monitor/health", 
timeout=2) as response:
                       return response.status == 200
               except OSError:
                   return False
   
           wait_for("api-server health", api_up, 120)
           wait_for(
               "Dags parsed",
               lambda: len(query(database, "SELECT dag_id FROM 
serialized_dag")) >= 2,
               120,
           )
           run_id = f"staledata-{args.label}"
           subprocess.run(
               [airflow, "dags", "trigger", "staledata_e2e", "--run-id", 
run_id],
               env=env,
               capture_output=True,
               check=True,
           )
           print(f"TRIGGERED staledata_e2e run_id={run_id}")
           rows = wait_for(
               "both tasks running with a pid",
               lambda: (
                   found
                   if len(
                       found := query(
                           database,
                           "SELECT task_id, pid FROM task_instance WHERE dag_id 
= 'staledata_e2e' "
                           "AND state = 'running' AND pid IS NOT NULL ORDER BY 
task_id",
                       )
                   )
                   == 2
                   else None
               ),
               180,
           )
           workers = []
           for task_id, task_pid in rows:
               worker_pid = int(
                   subprocess.run(
                       ["ps", "-o", "ppid=", "-p", str(task_pid)], 
capture_output=True, text=True
                   ).stdout
               )
               parent_pid = int(
                   subprocess.run(
                       ["ps", "-o", "ppid=", "-p", str(worker_pid)], 
capture_output=True, text=True
                   ).stdout
               )
               print(
                   f"RUNNING {task_id}: task pid {task_pid}, LocalExecutor 
worker pid {worker_pid} "
                   f"(child of scheduler pid {parent_pid})"
               )
               workers.append(worker_pid)
           task_pids = [task_pid for _, task_pid in rows]
           for pid in [*workers, *task_pids]:
               os.kill(pid, signal.SIGKILL)
           killed_at = time.monotonic()
           print(f"SIGKILL worker pids {workers} and task pids {task_pids}")
   
           scheduler_log = out / "scheduler.log"
   
           def outcome() -> str | None:
               if processes["scheduler"].poll() is not None:
                   return "exited"
               text = scheduler_log.read_text(errors="replace")
               return "skipped" if "Task instance no longer exists" in text 
else None
   
           result = wait_for("scheduler outcome", outcome, 60)
           time.sleep(5)
           for line in scheduler_log.read_text(errors="replace").splitlines():
               if line.startswith("INJECT"):
                   print(line)
           code = processes["scheduler"].poll()
           elapsed = time.monotonic() - killed_at
           if code is None:
               print(f"SCHEDULER pid={processes['scheduler'].pid} still running 
{elapsed:.0f}s after the kill")
           else:
               print(f"SCHEDULER exited with code {code} ({result})")
   
           after_id = f"after-{args.label}"
           subprocess.run(
               [airflow, "dags", "trigger", "staledata_e2e_after", "--run-id", 
after_id],
               env=env,
               capture_output=True,
               check=True,
           )
           started = time.monotonic()
           try:
               state = wait_for(
                   "after run finished",
                   lambda: next(
                       (
                           s
                           for (s,) in query(
                               database, "SELECT state FROM dag_run WHERE 
run_id = %s", (after_id,)
                           )
                           if s in ("success", "failed")
                       ),
                       None,
                   ),
                   60,
               )
               print(f"AFTER staledata_e2e_after {after_id}: {state} in 
{time.monotonic() - started:.0f}s")
           except TimeoutError:
               (state,) = query(database, "SELECT state FROM dag_run WHERE 
run_id = %s", (after_id,))[0]
               print(f"AFTER staledata_e2e_after {after_id}: still {state} 
after 60s")
   
           print("TASK_INSTANCE rows (all attempts):")
           for row in query(
               database,
               "SELECT dag_id, task_id, try_number, state, working_set FROM 
task_instance "
               "ORDER BY dag_id, task_id, try_number",
           ):
               print("  " + " ".join(str(value) for value in row))
           print(json.dumps({"label": args.label, "scheduler_exit_code": code}))
       finally:
           for process in processes.values():
               if process.poll() is None:
                   os.killpg(process.pid, signal.SIGTERM)
           for process in processes.values():
               try:
                   process.wait(timeout=30)
               except subprocess.TimeoutExpired:
                   os.killpg(process.pid, signal.SIGKILL)
                   process.wait(timeout=10)
   
   
   if __name__ == "__main__":
       sys.exit(main())
   ```
   
   </details>
   
   <details>
   <summary>Reproducer source: inject/sitecustomize.py</summary>
   
   ```python
   # Loaded only by the `airflow scheduler` process (PYTHONPATH is set for that 
process alone).
   # Once the scheduler commits the failure of one staledata_e2e task while the 
other task of the
   # same executor-event batch is still loaded as running, delete that other 
task instance through
   # the public REST API, as a user clicking "Delete" in the UI would.
   from __future__ import annotations
   
   import json
   import os
   import sys
   import urllib.parse
   import urllib.request
   
   if os.path.basename(sys.argv[0]) == "airflow" and sys.argv[1:2] == 
["scheduler"]:
       from sqlalchemy import event
       from sqlalchemy.orm import Session
   
       _deleted: list[str] = []
   
       @event.listens_for(Session, "after_commit")
       def delete_other_task_instance(session: Session) -> None:
           if _deleted:
               return
           from airflow.models.taskinstance import TaskInstance
   
           loaded = [
               obj
               for obj in session.identity_map.values()
               if isinstance(obj, TaskInstance) and obj.dag_id == 
"staledata_e2e"
           ]
           handled = [ti for ti in loaded if ti.state in ("failed", 
"up_for_retry")]
           pending = [ti for ti in loaded if ti.state == "running"]
           if not handled or not pending:
               return
           target = pending[0]
           _deleted.append(target.task_id)
           api = os.environ["STALEDATA_API"]
           with urllib.request.urlopen(f"{api}/auth/token") as response:
               token = json.load(response)["access_token"]
           run_id = urllib.parse.quote(target.run_id, safe="")
           url = 
f"{api}/api/v2/dags/staledata_e2e/dagRuns/{run_id}/taskInstances/{target.task_id}"
           request = urllib.request.Request(url, method="DELETE", 
headers={"Authorization": f"Bearer {token}"})
           with urllib.request.urlopen(request) as response:
               status = response.status
           print(
               f"INJECT {handled[0].task_id} committed as {handled[0].state}; 
DELETE {url} -> HTTP {status}",
               file=sys.stderr,
               flush=True,
           )
   ```
   
   </details>
   
   <details>
   <summary>Reproducer source: staledata_e2e_dag.py</summary>
   
   ```python
   from __future__ import annotations
   
   import time
   
   from airflow.sdk import dag, task
   
   
   @dag(schedule=None)
   def staledata_e2e():
       @task(retries=0)
       def first():
           time.sleep(600)
   
       @task(retries=0)
       def second():
           time.sleep(600)
   
       first()
       second()
   
   
   @dag(schedule=None)
   def staledata_e2e_after():
       @task
       def ping():
           return "pong"
   
       ping()
   
   
   staledata_e2e()
   staledata_e2e_after()
   ```
   
   </details>
   
   <!--
   Thank you for contributing!
   
   Please provide above a brief description of the changes made in this pull 
request.
   Write a good git commit message following this guide: 
https://chris.beams.io/posts/git-commit/
   
   Please make sure that your code changes are covered with tests.
   And in case of new features or big changes remember to adjust the 
documentation.
   
   For user-facing UI changes, please attach before/after screenshots (or a 
short
   screen recording) so reviewers can assess the visual impact.
   
   Feel free to ping (in general) for the review if you do not see reaction for 
a few days
   (72 Hours is the minimum reaction time you can expect from volunteers) - we 
sometimes miss notifications.
   
   In case of an existing issue, reference it using one of the following:
   
   * closes: #ISSUE
   * related: #ISSUE
   -->
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes (please specify the tool below)
   
   Generated-by: GitHub Copilot CLI (gpt-6.1-sol, claude-opus-5.5) following 
[the 
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   
   ---
   
   * Read the Pull Request Guidelines for more information: 
https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines.
 Note: commit author/co-author name and email in commits become permanently 
public when merged.
   * For fundamental code changes, an Airflow Improvement Proposal (AIP) is 
needed: 
https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals.
   * When adding dependency, check compliance with the ASF 3rd Party License 
Policy: https://www.apache.org/legal/resolved.html#category-x.
   * For significant user-facing changes create newsfragment: 
{pr_number}.significant.rst, in airflow-core/newsfragments: 
https://github.com/apache/airflow/tree/main/airflow-core/newsfragments. You can 
add this file in a follow-up commit after the PR is created so you know the PR 
number.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to