1fanwang opened a new pull request, #77205:
URL: https://github.com/apache/airflow/pull/77205
<!-- SPDX-License-Identifier: Apache-2.0
https://www.apache.org/licenses/LICENSE-2.0 -->
When an executor reports several failed tasks in one batch, for example when
the workers running them are killed, and one of those task instances is deleted
before the scheduler gets to it, the scheduler process exits. Deleting a task
instance from the Task Instances page or through the REST API works on running
tasks. It can land mid-batch because the scheduler commits after each failure
it records, which releases the row locks on the rest of the batch. The
scheduler log ends with:
```text
sqlalchemy.orm.exc.StaleDataError: UPDATE statement on table 'task_instance'
expected to update 1 row(s); 0 were matched.
```
With retries left, it ends with ValueError: An archived task instance cannot
be archived again. The heartbeat timeout check crashes the same way.
Reloading a task instance whose row is gone used to mark it as changed, so
failure handling went on to write a row that no longer exists. Now the reload
leaves nothing to write and tells the caller the row is gone, and failure
handling logs a warning and skips that task instance.
https://github.com/apache/airflow/pull/76544 keeps the scheduler running
when failure handling raises for any reason. Its fallback write also fails on a
deleted row, so this root-cause fix complements it. A deleted task instance
that was cleared while it was running still fails when its restart completes;
this change leaves that path alone.
related: https://github.com/apache/airflow/issues/76529
## Testing Done
Setup: macOS 26.7.3 (arm64), Python 3.12.13, PostgreSQL 16.16 in Docker on
127.0.0.1:25432, LocalExecutor, an apache/airflow checkout in $TREE prepared
with uv sync --frozen, and the three files below in the current directory; on
both commits the scheduler alone gets inject/sitecustomize.py on its
PYTHONPATH, and that hook deletes the other task instance through the REST API
right after the scheduler commits the first failure of the batch.
Current main,
https://github.com/apache/airflow/commit/3b35135afe9549e99088c4d557c62e5deafb125b:
```bash
git -C "$TREE" checkout --quiet --detach 3b35135afe
"$TREE/.venv/bin/python" run_e2e.py --label main --tree "$TREE" >
run-main.txt
grep -vE '^(TREE|POSTGRES) ' run-main.txt | sed -E 's# \(See
https:[^)]*\)##; s# Learn more: https:[^ ]*##; s#https?://[^/ ]+##g'
```
```text
STARTED api-server pid=72823
STARTED dag-processor pid=72824
STARTED scheduler pid=72825
TRIGGERED staledata_e2e run_id=staledata-main
RUNNING first: task pid 73468, LocalExecutor worker pid 73445 (child of
scheduler pid 72825)
RUNNING second: task pid 73469, LocalExecutor worker pid 73444 (child of
scheduler pid 72825)
SIGKILL worker pids [73445, 73444] and task pids [73468, 73469]
INJECT second committed as failed; DELETE
/api/v2/dags/staledata_e2e/dagRuns/staledata-main/taskInstances/first -> HTTP
200
SCHEDULER exited with code 1 (exited)
AFTER staledata_e2e_after after-main: still queued after 60s
TASK_INSTANCE rows (all attempts):
staledata_e2e second 1 failed True
staledata_e2e_after ping 0 None True
{"label": "main", "scheduler_exit_code": 1}
```
```bash
sed -n '/Received executor event with state failed/,/Exited execute loop/p'
runs/main/scheduler.log | sed "s|$TREE/||g" | sed -E 's# \(See https:[^)]*\)##;
s# Learn more: https:[^ ]*##; s#https?://[^/ ]+##g'
```
```text
2026-11-17T15:00:20.033019Z [info ] Received executor event with state
failed for task instance 01a1ea05-d884-7a6f-babf-9f88d66ba6cc
(coordinates=TaskInstanceKey(dag_id='staledata_e2e', task_id='second',
run_id='staledata-main', try_number=1, map_index=-1))
[airflow.jobs.scheduler_job_runner.SchedulerJobRunner]
loc=scheduler_job_runner.py:1584
2026-11-17T15:00:20.033170Z [info ] Received executor event with state
failed for task instance 01a1ea05-d883-7c17-acd9-0af2caae8fca
(coordinates=TaskInstanceKey(dag_id='staledata_e2e', task_id='first',
run_id='staledata-main', try_number=1, map_index=-1))
[airflow.jobs.scheduler_job_runner.SchedulerJobRunner]
loc=scheduler_job_runner.py:1584
2026-11-17T15:00:20.046248Z [info ] TaskInstance Finished:
dag_id=staledata_e2e, task_id=second, run_id=staledata-main, map_index=-1,
ti_id=01a1ea05-d884-7a6f-babf-9f88d66ba6cc, run_start_date=2026-11-17
15:00:16.814869+00:00, run_end_date=None, run_duration=None, state=running,
executor=LocalExecutor(parallelism=4), executor_state=failed, try_number=1,
max_tries=0, pool=default_pool, queue=default, priority_weight=1,
operator=_PythonDecoratedOperator, queued_dttm=2026-11-17
15:00:15.016177+00:00, scheduled_dttm=2026-11-17
15:00:15.001830+00:00,queued_by_job_id=1, pid=73469
[airflow.jobs.scheduler_job_runner.SchedulerJobRunner]
loc=scheduler_job_runner.py:1738
2026-11-17T15:00:20.047802Z [error ] Executor
LocalExecutor(parallelism=4) reported that the task instance <TaskInstance:
staledata_e2e.second staledata-main [running]
ti_id=01a1ea05-d884-7a6f-babf-9f88d66ba6cc> finished with state failed, but the
task instance's state attribute is running. [airflow.task]
loc=taskinstance.py:2830
2026-11-17T15:00:20.049873Z [info ] Marking task as FAILED.
dag_id=staledata_e2e, task_id=second, run_id=staledata-main, logical_date=,
start_date=20261117T150016, end_date=20261117T150020
[airflow.models.taskinstance] loc=taskinstance.py:2924
INJECT second committed as failed; DELETE
/api/v2/dags/staledata_e2e/dagRuns/staledata-main/taskInstances/first -> HTTP
200
2026-11-17T15:00:20.089455Z [info ] TaskInstance Finished:
dag_id=staledata_e2e, task_id=first, run_id=staledata-main, map_index=-1,
ti_id=01a1ea05-d883-7c17-acd9-0af2caae8fca, run_start_date=2026-11-17
15:00:16.814856+00:00, run_end_date=None, run_duration=None, state=running,
executor=LocalExecutor(parallelism=4), executor_state=failed, try_number=1,
max_tries=0, pool=default_pool, queue=default, priority_weight=1,
operator=_PythonDecoratedOperator, queued_dttm=2026-11-17
15:00:15.016177+00:00, scheduled_dttm=2026-11-17
15:00:15.001830+00:00,queued_by_job_id=1, pid=73468
[airflow.jobs.scheduler_job_runner.SchedulerJobRunner]
loc=scheduler_job_runner.py:1738
2026-11-17T15:00:20.092208Z [error ] Executor
LocalExecutor(parallelism=4) reported that the task instance <TaskInstance:
staledata_e2e.first staledata-main [running]
ti_id=01a1ea05-d883-7c17-acd9-0af2caae8fca> finished with state failed, but the
task instance's state attribute is running. [airflow.task]
loc=taskinstance.py:2830
2026-11-17T15:00:20.093633Z [info ] Marking task as FAILED.
dag_id=staledata_e2e, task_id=first, run_id=staledata-main, logical_date=,
start_date=20261117T150016, end_date=20261117T150020
[airflow.models.taskinstance] loc=taskinstance.py:2924
2026-11-17T15:00:20.098436Z [error ] Exception when executing
SchedulerJob._run_scheduler_loop
[airflow.jobs.scheduler_job_runner.SchedulerJobRunner]
loc=scheduler_job_runner.py:1976
Traceback (most recent call last):
File "airflow-core/src/airflow/jobs/scheduler_job_runner.py", line 1970,
in _execute
self._run_scheduler_loop()
File "airflow-core/src/airflow/jobs/scheduler_job_runner.py", line 2193,
in _run_scheduler_loop
num_finished_events += self._process_executor_events(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "airflow-core/src/airflow/jobs/scheduler_job_runner.py", line 1517,
in _process_executor_events
return SchedulerJobRunner.process_executor_events(
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
File "airflow-core/src/airflow/jobs/scheduler_job_runner.py", line 1915,
in process_executor_events
ti.handle_failure(error=msg, session=session)
File "airflow-core/src/airflow/utils/session.py", line 98, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "airflow-core/src/airflow/models/taskinstance.py", line 2927, in
handle_failure
TaskInstance.save_to_db(ti, session=session)
File "airflow-core/src/airflow/utils/session.py", line 98, in wrapper
return func(*args, **kwargs)
^^^^^^^^^^^^^^^^^^^^^
File "airflow-core/src/airflow/models/taskinstance.py", line 2889, in
save_to_db
session.flush()
File ".venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line
4360, in flush
self._flush(objects)
File ".venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line
4495, in _flush
with util.safe_reraise():
^^^^^^^^^^^^^^^^^^^
File ".venv/lib/python3.12/site-packages/sqlalchemy/util/langhelpers.py",
line 122, in __exit__
raise exc_value.with_traceback(exc_tb)
File ".venv/lib/python3.12/site-packages/sqlalchemy/orm/session.py", line
4456, in _flush
flush_context.execute()
File ".venv/lib/python3.12/site-packages/sqlalchemy/orm/unitofwork.py",
line 465, in execute
rec.execute(self)
File ".venv/lib/python3.12/site-packages/sqlalchemy/orm/unitofwork.py",
line 641, in execute
util.preloaded.orm_persistence.save_obj(
File ".venv/lib/python3.12/site-packages/sqlalchemy/orm/persistence.py",
line 86, in save_obj
_emit_update_statements(
File ".venv/lib/python3.12/site-packages/sqlalchemy/orm/persistence.py",
line 949, in _emit_update_statements
raise orm_exc.StaleDataError(
sqlalchemy.orm.exc.StaleDataError: UPDATE statement on table 'task_instance'
expected to update 1 row(s); 0 were matched.
2026-11-17T15:00:20.105922Z [info ] Shutting down LocalExecutor; waiting
for running tasks to finish. Signal again if you don't want to wait.
[airflow.executors.local_executor.LocalExecutor] loc=local_executor.py:283
2026-11-17T15:00:20.347192Z [info ] Exited execute loop
[airflow.jobs.scheduler_job_runner.SchedulerJobRunner]
loc=scheduler_job_runner.py:1988
```
With this change:
```bash
git -C "$TREE" checkout --quiet --detach 6784d0c01f
"$TREE/.venv/bin/python" run_e2e.py --label fix --tree "$TREE" > run-fix.txt
grep -vE '^(TREE|POSTGRES) ' run-fix.txt | sed -E 's# \(See https:[^)]*\)##;
s# Learn more: https:[^ ]*##; s#https?://[^/ ]+##g'
```
```text
STARTED api-server pid=80140
STARTED dag-processor pid=80141
STARTED scheduler pid=80142
TRIGGERED staledata_e2e run_id=staledata-fix
RUNNING first: task pid 80299, LocalExecutor worker pid 80271 (child of
scheduler pid 80142)
RUNNING second: task pid 80300, LocalExecutor worker pid 80272 (child of
scheduler pid 80142)
SIGKILL worker pids [80271, 80272] and task pids [80299, 80300]
INJECT second committed as failed; DELETE
/api/v2/dags/staledata_e2e/dagRuns/staledata-fix/taskInstances/first -> HTTP 200
SCHEDULER pid=80142 still running 6s after the kill
AFTER staledata_e2e_after after-fix: success in 3s
TASK_INSTANCE rows (all attempts):
staledata_e2e second 1 failed True
staledata_e2e_after ping 1 success True
{"label": "fix", "scheduler_exit_code": null}
```
```bash
sed -n '/Received executor event with state failed/,/no longer exists/p'
runs/fix/scheduler.log | sed "s|$TREE/||g" | sed -E 's# \(See https:[^)]*\)##;
s# Learn more: https:[^ ]*##; s#https?://[^/ ]+##g'
grep -c Traceback runs/fix/scheduler.log
```
```text
2026-11-17T15:02:11.550673Z [info ] Received executor event with state
failed for task instance 01a1ea07-8bda-73ad-bd86-dd102d711264
(coordinates=TaskInstanceKey(dag_id='staledata_e2e', task_id='first',
run_id='staledata-fix', try_number=1, map_index=-1))
[airflow.jobs.scheduler_job_runner.SchedulerJobRunner]
loc=scheduler_job_runner.py:1584
2026-11-17T15:02:11.550791Z [info ] Received executor event with state
failed for task instance 01a1ea07-8bdb-786d-bf7d-9950b63c88e0
(coordinates=TaskInstanceKey(dag_id='staledata_e2e', task_id='second',
run_id='staledata-fix', try_number=1, map_index=-1))
[airflow.jobs.scheduler_job_runner.SchedulerJobRunner]
loc=scheduler_job_runner.py:1584
2026-11-17T15:02:11.565201Z [info ] TaskInstance Finished:
dag_id=staledata_e2e, task_id=second, run_id=staledata-fix, map_index=-1,
ti_id=01a1ea07-8bdb-786d-bf7d-9950b63c88e0, run_start_date=2026-11-17
15:02:08.179954+00:00, run_end_date=None, run_duration=None, state=running,
executor=LocalExecutor(parallelism=4), executor_state=failed, try_number=1,
max_tries=0, pool=default_pool, queue=default, priority_weight=1,
operator=_PythonDecoratedOperator, queued_dttm=2026-11-17
15:02:06.519514+00:00, scheduled_dttm=2026-11-17
15:02:06.504362+00:00,queued_by_job_id=1, pid=80300
[airflow.jobs.scheduler_job_runner.SchedulerJobRunner]
loc=scheduler_job_runner.py:1738
2026-11-17T15:02:11.566836Z [error ] Executor
LocalExecutor(parallelism=4) reported that the task instance <TaskInstance:
staledata_e2e.second staledata-fix [running]
ti_id=01a1ea07-8bdb-786d-bf7d-9950b63c88e0> finished with state failed, but the
task instance's state attribute is running. [airflow.task]
loc=taskinstance.py:2834
2026-11-17T15:02:11.569189Z [info ] Marking task as FAILED.
dag_id=staledata_e2e, task_id=second, run_id=staledata-fix, logical_date=,
start_date=20261117T150208, end_date=20261117T150211
[airflow.models.taskinstance] loc=taskinstance.py:2933
INJECT second committed as failed; DELETE
/api/v2/dags/staledata_e2e/dagRuns/staledata-fix/taskInstances/first -> HTTP 200
2026-11-17T15:02:11.612812Z [info ] TaskInstance Finished:
dag_id=staledata_e2e, task_id=first, run_id=staledata-fix, map_index=-1,
ti_id=01a1ea07-8bda-73ad-bd86-dd102d711264, run_start_date=2026-11-17
15:02:08.179956+00:00, run_end_date=None, run_duration=None, state=running,
executor=LocalExecutor(parallelism=4), executor_state=failed, try_number=1,
max_tries=0, pool=default_pool, queue=default, priority_weight=1,
operator=_PythonDecoratedOperator, queued_dttm=2026-11-17
15:02:06.519514+00:00, scheduled_dttm=2026-11-17
15:02:06.504362+00:00,queued_by_job_id=1, pid=80299
[airflow.jobs.scheduler_job_runner.SchedulerJobRunner]
loc=scheduler_job_runner.py:1738
2026-11-17T15:02:11.615682Z [error ] Executor
LocalExecutor(parallelism=4) reported that the task instance <TaskInstance:
staledata_e2e.first staledata-fix [running]
ti_id=01a1ea07-8bda-73ad-bd86-dd102d711264> finished with state failed, but the
task instance's state attribute is running. [airflow.task]
loc=taskinstance.py:2834
2026-11-17T15:02:11.616996Z [warning ] Task instance no longer exists,
skipping its failure handling [airflow.task] loc=taskinstance.py:2836
task_instance_key=TaskInstanceKey(dag_id='staledata_e2e', task_id='first',
run_id='staledata-fix', try_number=1, map_index=-1)
0
```
<details>
<summary>Reproducer source: run_e2e.py</summary>
```python
#!/usr/bin/env python3
# Real-scheduler reproduction: two running tasks lose their LocalExecutor
workers to SIGKILL, so the
# executor reports both failures in one batch; the injected hook deletes the
second task instance
# through the REST API right after the scheduler commits the first failure.
from __future__ import annotations
import argparse
import json
import os
import secrets
import shutil
import signal
import subprocess
import sys
import time
import urllib.request
from pathlib import Path
import psycopg2
PG_DSN = "host=127.0.0.1 port=25432 user=airflow password=airflow"
API = "http://127.0.0.1:18100"
HERE = Path(__file__).resolve().parent
def query(database: str, sql: str, params: tuple = ()) -> list[tuple]:
with psycopg2.connect(f"{PG_DSN} dbname={database}") as connection,
connection.cursor() as cursor:
cursor.execute(sql, params)
return cursor.fetchall()
def wait_for(description: str, check, timeout: float):
deadline = time.monotonic() + timeout
while time.monotonic() < deadline:
if result := check():
return result
time.sleep(0.2)
raise TimeoutError(description)
def main() -> None:
parser = argparse.ArgumentParser()
parser.add_argument("--label", required=True)
parser.add_argument("--tree", required=True, type=Path)
args = parser.parse_args()
database = f"staledata_{args.label}"
out = HERE / "runs" / args.label
shutil.rmtree(out, ignore_errors=True)
(out / "home" / "dags").mkdir(parents=True)
shutil.copy(HERE / "staledata_e2e_dag.py", out / "home" / "dags")
airflow = str(args.tree / ".venv" / "bin" / "airflow")
head = subprocess.run(
["git", "-C", str(args.tree), "log", "-1", "--format=%h %s"],
capture_output=True, text=True
).stdout.strip()
dirty = subprocess.run(
["git", "-C", str(args.tree), "status", "--short",
"--untracked-files=no"],
capture_output=True,
text=True,
).stdout.split("\n")
print(f"TREE {head} (uncommitted tracked changes: {len([line for line in
dirty if line])})")
admin = psycopg2.connect(f"{PG_DSN} dbname=airflow")
admin.autocommit = True
with admin.cursor() as cursor:
cursor.execute(f"DROP DATABASE IF EXISTS {database}")
cursor.execute(f"CREATE DATABASE {database}")
cursor.execute("SHOW server_version")
print(f"POSTGRES {cursor.fetchone()[0]} database={database}")
admin.close()
env = {
**os.environ,
"AIRFLOW_HOME": str(out / "home"),
"AIRFLOW__DATABASE__SQL_ALCHEMY_CONN": (
f"postgresql+psycopg2://airflow:[email protected]:25432/{database}"
),
"AIRFLOW__CORE__EXECUTOR": "LocalExecutor",
"AIRFLOW__CORE__PARALLELISM": "4",
"AIRFLOW__CORE__DAGS_FOLDER": str(out / "home" / "dags"),
"AIRFLOW__CORE__LOAD_EXAMPLES": "False",
"AIRFLOW__CORE__DAGS_ARE_PAUSED_AT_CREATION": "False",
"AIRFLOW__CORE__EXECUTION_API_SERVER_URL": f"{API}/execution/",
"AIRFLOW__CORE__SIMPLE_AUTH_MANAGER_ALL_ADMINS": "True",
"AIRFLOW__API__BASE_URL": API,
"AIRFLOW__API_AUTH__JWT_SECRET": secrets.token_urlsafe(32),
"AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL": "2",
"AIRFLOW__SCHEDULER__TASK_INSTANCE_HEARTBEAT_TIMEOUT": "600",
}
with open(out / "db-migrate.log", "wb") as log:
subprocess.run([airflow, "db", "migrate"], env=env, stdout=log,
stderr=subprocess.STDOUT, check=True)
processes: dict[str, subprocess.Popen] = {}
def launch(name: str, command: list[str], extra_env: dict[str, str] |
None = None) -> None:
log = open(out / f"{name}.log", "wb")
processes[name] = subprocess.Popen(
[airflow, *command],
env={**env, **(extra_env or {})},
stdout=log,
stderr=subprocess.STDOUT,
start_new_session=True,
)
print(f"STARTED {name} pid={processes[name].pid}")
try:
launch("api-server", ["api-server", "--host", "127.0.0.1", "--port",
"18100", "--workers", "1"])
launch("dag-processor", ["dag-processor"])
launch(
"scheduler",
["scheduler", "--skip-serve-logs"],
{"PYTHONPATH": str(HERE / "inject"), "STALEDATA_API": API},
)
def api_up() -> bool:
try:
with urllib.request.urlopen(f"{API}/api/v2/monitor/health",
timeout=2) as response:
return response.status == 200
except OSError:
return False
wait_for("api-server health", api_up, 120)
wait_for(
"Dags parsed",
lambda: len(query(database, "SELECT dag_id FROM
serialized_dag")) >= 2,
120,
)
run_id = f"staledata-{args.label}"
subprocess.run(
[airflow, "dags", "trigger", "staledata_e2e", "--run-id",
run_id],
env=env,
capture_output=True,
check=True,
)
print(f"TRIGGERED staledata_e2e run_id={run_id}")
rows = wait_for(
"both tasks running with a pid",
lambda: (
found
if len(
found := query(
database,
"SELECT task_id, pid FROM task_instance WHERE dag_id
= 'staledata_e2e' "
"AND state = 'running' AND pid IS NOT NULL ORDER BY
task_id",
)
)
== 2
else None
),
180,
)
workers = []
for task_id, task_pid in rows:
worker_pid = int(
subprocess.run(
["ps", "-o", "ppid=", "-p", str(task_pid)],
capture_output=True, text=True
).stdout
)
parent_pid = int(
subprocess.run(
["ps", "-o", "ppid=", "-p", str(worker_pid)],
capture_output=True, text=True
).stdout
)
print(
f"RUNNING {task_id}: task pid {task_pid}, LocalExecutor
worker pid {worker_pid} "
f"(child of scheduler pid {parent_pid})"
)
workers.append(worker_pid)
task_pids = [task_pid for _, task_pid in rows]
for pid in [*workers, *task_pids]:
os.kill(pid, signal.SIGKILL)
killed_at = time.monotonic()
print(f"SIGKILL worker pids {workers} and task pids {task_pids}")
scheduler_log = out / "scheduler.log"
def outcome() -> str | None:
if processes["scheduler"].poll() is not None:
return "exited"
text = scheduler_log.read_text(errors="replace")
return "skipped" if "Task instance no longer exists" in text
else None
result = wait_for("scheduler outcome", outcome, 60)
time.sleep(5)
for line in scheduler_log.read_text(errors="replace").splitlines():
if line.startswith("INJECT"):
print(line)
code = processes["scheduler"].poll()
elapsed = time.monotonic() - killed_at
if code is None:
print(f"SCHEDULER pid={processes['scheduler'].pid} still running
{elapsed:.0f}s after the kill")
else:
print(f"SCHEDULER exited with code {code} ({result})")
after_id = f"after-{args.label}"
subprocess.run(
[airflow, "dags", "trigger", "staledata_e2e_after", "--run-id",
after_id],
env=env,
capture_output=True,
check=True,
)
started = time.monotonic()
try:
state = wait_for(
"after run finished",
lambda: next(
(
s
for (s,) in query(
database, "SELECT state FROM dag_run WHERE
run_id = %s", (after_id,)
)
if s in ("success", "failed")
),
None,
),
60,
)
print(f"AFTER staledata_e2e_after {after_id}: {state} in
{time.monotonic() - started:.0f}s")
except TimeoutError:
(state,) = query(database, "SELECT state FROM dag_run WHERE
run_id = %s", (after_id,))[0]
print(f"AFTER staledata_e2e_after {after_id}: still {state}
after 60s")
print("TASK_INSTANCE rows (all attempts):")
for row in query(
database,
"SELECT dag_id, task_id, try_number, state, working_set FROM
task_instance "
"ORDER BY dag_id, task_id, try_number",
):
print(" " + " ".join(str(value) for value in row))
print(json.dumps({"label": args.label, "scheduler_exit_code": code}))
finally:
for process in processes.values():
if process.poll() is None:
os.killpg(process.pid, signal.SIGTERM)
for process in processes.values():
try:
process.wait(timeout=30)
except subprocess.TimeoutExpired:
os.killpg(process.pid, signal.SIGKILL)
process.wait(timeout=10)
if __name__ == "__main__":
sys.exit(main())
```
</details>
<details>
<summary>Reproducer source: inject/sitecustomize.py</summary>
```python
# Loaded only by the `airflow scheduler` process (PYTHONPATH is set for that
process alone).
# Once the scheduler commits the failure of one staledata_e2e task while the
other task of the
# same executor-event batch is still loaded as running, delete that other
task instance through
# the public REST API, as a user clicking "Delete" in the UI would.
from __future__ import annotations
import json
import os
import sys
import urllib.parse
import urllib.request
if os.path.basename(sys.argv[0]) == "airflow" and sys.argv[1:2] ==
["scheduler"]:
from sqlalchemy import event
from sqlalchemy.orm import Session
_deleted: list[str] = []
@event.listens_for(Session, "after_commit")
def delete_other_task_instance(session: Session) -> None:
if _deleted:
return
from airflow.models.taskinstance import TaskInstance
loaded = [
obj
for obj in session.identity_map.values()
if isinstance(obj, TaskInstance) and obj.dag_id ==
"staledata_e2e"
]
handled = [ti for ti in loaded if ti.state in ("failed",
"up_for_retry")]
pending = [ti for ti in loaded if ti.state == "running"]
if not handled or not pending:
return
target = pending[0]
_deleted.append(target.task_id)
api = os.environ["STALEDATA_API"]
with urllib.request.urlopen(f"{api}/auth/token") as response:
token = json.load(response)["access_token"]
run_id = urllib.parse.quote(target.run_id, safe="")
url =
f"{api}/api/v2/dags/staledata_e2e/dagRuns/{run_id}/taskInstances/{target.task_id}"
request = urllib.request.Request(url, method="DELETE",
headers={"Authorization": f"Bearer {token}"})
with urllib.request.urlopen(request) as response:
status = response.status
print(
f"INJECT {handled[0].task_id} committed as {handled[0].state};
DELETE {url} -> HTTP {status}",
file=sys.stderr,
flush=True,
)
```
</details>
<details>
<summary>Reproducer source: staledata_e2e_dag.py</summary>
```python
from __future__ import annotations
import time
from airflow.sdk import dag, task
@dag(schedule=None)
def staledata_e2e():
@task(retries=0)
def first():
time.sleep(600)
@task(retries=0)
def second():
time.sleep(600)
first()
second()
@dag(schedule=None)
def staledata_e2e_after():
@task
def ping():
return "pong"
ping()
staledata_e2e()
staledata_e2e_after()
```
</details>
<!--
Thank you for contributing!
Please provide above a brief description of the changes made in this pull
request.
Write a good git commit message following this guide:
https://chris.beams.io/posts/git-commit/
Please make sure that your code changes are covered with tests.
And in case of new features or big changes remember to adjust the
documentation.
For user-facing UI changes, please attach before/after screenshots (or a
short
screen recording) so reviewers can assess the visual impact.
Feel free to ping (in general) for the review if you do not see reaction for
a few days
(72 Hours is the minimum reaction time you can expect from volunteers) - we
sometimes miss notifications.
In case of an existing issue, reference it using one of the following:
* closes: #ISSUE
* related: #ISSUE
-->
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (please specify the tool below)
Generated-by: GitHub Copilot CLI (gpt-6.1-sol, claude-opus-5.5) following
[the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
---
* Read the Pull Request Guidelines for more information:
https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines.
Note: commit author/co-author name and email in commits become permanently
public when merged.
* For fundamental code changes, an Airflow Improvement Proposal (AIP) is
needed:
https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals.
* When adding dependency, check compliance with the ASF 3rd Party License
Policy: https://www.apache.org/legal/resolved.html#category-x.
* For significant user-facing changes create newsfragment:
{pr_number}.significant.rst, in airflow-core/newsfragments:
https://github.com/apache/airflow/tree/main/airflow-core/newsfragments. You can
add this file in a follow-up commit after the PR is created so you know the PR
number.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]