1fanwang opened a new pull request, #77204:
URL: https://github.com/apache/airflow/pull/77204

   <!-- SPDX-License-Identifier: Apache-2.0
         https://www.apache.org/licenses/LICENSE-2.0 -->
   
   When a KubernetesExecutor worker pod is deleted while its task runs, for 
example by a forced delete, the executor's pod watch normally reports it and 
the task fails within seconds. After 30 seconds without pod changes, the watch 
read times out (the client timeout from 
https://github.com/apache/airflow/pull/39551). The watcher logs "Kubernetes 
watch timed out waiting for events. Restarting watch." and a second later 
starts a new watch from the resource version the old one started at, usually 0. 
A watch from 0 lists only existing pods, so a pod deleted during that second is 
never reported. Its task stays running and holds an executor slot until 
"Detected a task instance without a heartbeat", five minutes later by default.
   
   The watcher now keeps the resource version of the last change event it 
handled, so the next watch after a read timeout starts there and replays the 
deletion. Below, the task fails two seconds after the deletion. The watcher 
skips the ADDED events that open a watch from 0, since each carries its pod's 
own, possibly expired, version. If the server reports that version as too old 
(410), the watcher restarts from the current state in place instead of 
crashing, which is how the provider's async pod event watch already handles it.
   
   ## Testing Done
   
   Both runs use the reproducer below, which runs a native scheduler, Dag 
processor and execution API on macOS with SQLite, starts KubernetesExecutor 
pods in a kind cluster running Kubernetes 1.37.0 from one worker image built at 
3b35135afe9549e99088c4d557c62e5deafb125b, and force-deletes the running task 
pod as soon as the watcher logs its read timeout.
   
   Current main:
   
   ```bash
   "$MAIN_TREE/.venv/bin/python" watch_gap_e2e.py --label main --tree 
"$MAIN_TREE" --image aip97-watch-worker:3b35135a \
       --kubeconfig "$KUBECONFIG" --context kind-aip97-watch --output 
raw/main-2 > raw/main-2.driver.log 2>&1; echo "exit=$?" >> raw/main-2.driver.log
   sed -n '/TI_RUNNING/,$p' raw/main-2.driver.log | sed -E 's/ \(See 
https:[^)]*\)//; s/ Learn more: https:[^ ]*//'
   ```
   
   ```text
   {"utc": "2026-11-17T14:57:30.954547+00:00", "event": "TI_RUNNING", "ti": 
{"state": "running", "start_date": "2026-11-17 14:57:28.435273", "end_date": 
null, "last_heartbeat_at": "2026-11-17 14:57:30.669467"}}
   {"utc": "2026-11-17T14:57:55.839606+00:00", "event": 
"POD_FORCE_DELETED_AFTER_WATCH_TIMEOUT", "pod": 
"watch-gap-long-running-p1kk98ot", "timeout_line_seen_at": 
"2026-11-17T14:57:55.830244+00:00", "phase": "Running", "pod_resource_version": 
"2057"}
   {"utc": "2026-11-17T15:03:00.736384+00:00", "event": "TI_FINISHED", "ti": 
{"state": "failed", "start_date": "2026-11-17 14:57:28.435273", "end_date": 
"2026-11-17 15:03:00.117391", "last_heartbeat_at": "2026-11-17 
14:57:55.964167"}}
   
   SCHEDULER_LOG (watcher and task state lines)
   2026-11-17T14:57:21.394058Z [info     ] Event: and now my watch begins 
starting at resource_version: 0
   2026-11-17T14:57:25.665453Z [info     ] Received executor event with state 
queued for task instance 01a1ea03-41e1-7a12-8801-18f716bbd659 
(coordinates=TaskInstanceKey(dag_id='watch_gap', task_id='long_running', 
run_id='watch-gap-main', try_number=1, map_index=-1))
   2026-11-17T14:57:25.832225Z [info     ] Event: 
watch-gap-long-running-p1kk98ot is Running, annotations: <omitted>
   2026-11-17T14:57:55.829606Z [info     ] Kubernetes watch timed out waiting 
for events. Restarting watch.
   2026-11-17T14:57:56.836865Z [info     ] Event: and now my watch begins 
starting at resource_version: 0
   2026-11-17T14:58:26.841850Z [info     ] Kubernetes watch timed out waiting 
for events. Restarting watch.
   2026-11-17T14:58:27.844106Z [info     ] Event: and now my watch begins 
starting at resource_version: 0
   2026-11-17T14:58:57.849079Z [info     ] Kubernetes watch timed out waiting 
for events. Restarting watch.
   2026-11-17T14:58:58.850544Z [info     ] Event: and now my watch begins 
starting at resource_version: 0
   2026-11-17T14:59:28.854866Z [info     ] Kubernetes watch timed out waiting 
for events. Restarting watch.
   2026-11-17T14:59:29.855997Z [info     ] Event: and now my watch begins 
starting at resource_version: 0
   2026-11-17T14:59:59.859803Z [info     ] Kubernetes watch timed out waiting 
for events. Restarting watch.
   2026-11-17T15:00:00.869966Z [info     ] Event: and now my watch begins 
starting at resource_version: 0
   2026-11-17T15:00:30.875219Z [info     ] Kubernetes watch timed out waiting 
for events. Restarting watch.
   2026-11-17T15:00:31.878657Z [info     ] Event: and now my watch begins 
starting at resource_version: 0
   2026-11-17T15:01:01.881422Z [info     ] Kubernetes watch timed out waiting 
for events. Restarting watch.
   2026-11-17T15:01:02.884357Z [info     ] Event: and now my watch begins 
starting at resource_version: 0
   2026-11-17T15:01:32.889459Z [info     ] Kubernetes watch timed out waiting 
for events. Restarting watch.
   2026-11-17T15:01:33.890294Z [info     ] Event: and now my watch begins 
starting at resource_version: 0
   2026-11-17T15:02:03.894720Z [info     ] Kubernetes watch timed out waiting 
for events. Restarting watch.
   2026-11-17T15:02:04.897976Z [info     ] Event: and now my watch begins 
starting at resource_version: 0
   2026-11-17T15:02:34.902935Z [info     ] Kubernetes watch timed out waiting 
for events. Restarting watch.
   2026-11-17T15:02:35.911784Z [info     ] Event: and now my watch begins 
starting at resource_version: 0
   2026-11-17T15:03:00.116303Z [error    ] Detected a task instance without a 
heartbeat: <TaskInstance: watch_gap.long_running watch-gap-main [running] 
ti_id=01a1ea03-41e1-7a12-8801-18f716bbd659>
   2026-11-17T15:03:00.117901Z [info     ] Marking task as FAILED. 
dag_id=watch_gap, task_id=long_running, run_id=watch-gap-main, logical_date=, 
start_date=20261117T145728, end_date=20261117T150300
   2026-11-17T15:03:00.656800Z [info     ] Received executor event with state 
failed for task instance 01a1ea03-41e1-7a12-8801-18f716bbd659 
(coordinates=TaskInstanceKey(dag_id='watch_gap', task_id='long_running', 
run_id='watch-gap-main', try_number=1, map_index=-1))
   2026-11-17T15:03:05.917141Z [info     ] Kubernetes watch timed out waiting 
for events. Restarting watch.
   2026-11-17T15:03:06.922587Z [info     ] Event: and now my watch begins 
starting at resource_version: 0
   
   EXECUTOR_GAUGES (value changes)
   2026-11-17T14:57:19.457262+00:00 airflow.executor.open_slots=4
   2026-11-17T14:57:19.457329+00:00 airflow.executor.running_tasks=0
   2026-11-17T14:57:25.687757+00:00 airflow.executor.open_slots=3
   2026-11-17T14:57:25.687804+00:00 airflow.executor.running_tasks=1
   2026-11-17T15:03:00.656705+00:00 airflow.executor.open_slots=4
   2026-11-17T15:03:00.656964+00:00 airflow.executor.running_tasks=0
   exit=0
   ```
   
   This branch, same scenario:
   
   ```bash
   "$FIX_TREE/.venv/bin/python" watch_gap_e2e.py --label fix --tree "$FIX_TREE" 
--image aip97-watch-worker:3b35135a \
       --kubeconfig "$KUBECONFIG" --context kind-aip97-watch --output raw/fix-1 
> raw/fix-1.driver.log 2>&1; echo "exit=$?" >> raw/fix-1.driver.log
   sed -n '/TI_RUNNING/,$p' raw/fix-1.driver.log | sed -E 's/ \(See 
https:[^)]*\)//; s/ Learn more: https:[^ ]*//'
   ```
   
   ```text
   {"utc": "2026-11-17T14:55:48.578966+00:00", "event": "TI_RUNNING", "ti": 
{"state": "running", "start_date": "2026-11-17 14:55:46.290364", "end_date": 
null, "last_heartbeat_at": "2026-11-17 14:55:48.504518"}}
   {"utc": "2026-11-17T14:56:13.592856+00:00", "event": 
"POD_FORCE_DELETED_AFTER_WATCH_TIMEOUT", "pod": 
"watch-gap-long-running-ax1lqh71", "timeout_line_seen_at": 
"2026-11-17T14:56:13.583828+00:00", "phase": "Running", "pod_resource_version": 
"1889"}
   {"utc": "2026-11-17T14:56:16.603907+00:00", "event": "TI_FINISHED", "ti": 
{"state": "failed", "start_date": "2026-11-17 14:55:46.290364", "end_date": 
"2026-11-17 14:56:15.617775", "last_heartbeat_at": "2026-11-17 
14:56:13.751881"}}
   
   SCHEDULER_LOG (watcher and task state lines)
   2026-11-17T14:55:39.975056Z [info     ] Event: and now my watch begins 
starting at resource_version: 0
   2026-11-17T14:55:43.456271Z [info     ] Received executor event with state 
queued for task instance 01a1ea01-b50f-7f08-a3b6-74199847397b 
(coordinates=TaskInstanceKey(dag_id='watch_gap', task_id='long_running', 
run_id='watch-gap-fix', try_number=1, map_index=-1))
   2026-11-17T14:55:43.585363Z [info     ] Event: 
watch-gap-long-running-ax1lqh71 is Running, annotations: <omitted>
   2026-11-17T14:56:13.583721Z [info     ] Kubernetes watch timed out waiting 
for events. Restarting watch.
   2026-11-17T14:56:14.593859Z [info     ] Event: and now my watch begins 
starting at resource_version: 1845
   2026-11-17T14:56:14.601954Z [info     ] Event: 
watch-gap-long-running-ax1lqh71 is Running, annotations: <omitted>
   2026-11-17T14:56:14.602683Z [info     ] Event: Pod 
watch-gap-long-running-ax1lqh71 deleted before it could complete, annotations: 
<omitted>
   2026-11-17T14:56:15.614028Z [info     ] Changing state of 
KubernetesResults(key=TaskInstanceUuid(id=UUID('01a1ea01-b50f-7f08-a3b6-74199847397b')),
 state=<TaskInstanceState.FAILED: 'failed'>, 
pod_name='watch-gap-long-running-ax1lqh71', namespace='watch-gap-fix', 
resource_version='1889', failure_details=None) to failed
   2026-11-17T14:56:15.614698Z [info     ] Received executor event with state 
failed for task instance 01a1ea01-b50f-7f08-a3b6-74199847397b 
(coordinates=TaskInstanceKey(dag_id='watch_gap', task_id='long_running', 
run_id='watch-gap-fix', try_number=1, map_index=-1))
   2026-11-17T14:56:15.616890Z [error    ] Executor 
KubernetesExecutor(parallelism=4) reported that the task instance 
<TaskInstance: watch_gap.long_running watch-gap-fix [running] 
ti_id=01a1ea01-b50f-7f08-a3b6-74199847397b> finished with state failed, but the 
task instance's state attribute is running.
   2026-11-17T14:56:15.618150Z [info     ] Marking task as FAILED. 
dag_id=watch_gap, task_id=long_running, run_id=watch-gap-fix, logical_date=, 
start_date=20261117T145546, end_date=20261117T145615
   
   EXECUTOR_GAUGES (value changes)
   2026-11-17T14:55:38.285914+00:00 airflow.executor.open_slots=4
   2026-11-17T14:55:38.285955+00:00 airflow.executor.running_tasks=0
   2026-11-17T14:55:43.483620+00:00 airflow.executor.open_slots=3
   2026-11-17T14:55:43.483664+00:00 airflow.executor.running_tasks=1
   2026-11-17T14:56:16.637923+00:00 airflow.executor.open_slots=4
   2026-11-17T14:56:16.637974+00:00 airflow.executor.running_tasks=0
   exit=0
   ```
   
   This branch with the saved version expired. Here the watcher runs on its own 
against the cluster while the driver patches an unrelated pod five times a 
second, so the server drops the old version from its watch history.
   
   ```bash
   "$FIX_TREE/.venv/bin/python" watch_expire_e2e.py --label fix --kubeconfig 
"$KUBECONFIG" --context kind-aip97-watch \
       --log raw/expire-fix-2.log > raw/expire-fix-2.log 2>&1; echo "exit=$?" 
>> raw/expire-fix-2.log
   cat raw/expire-fix-2.log
   ```
   
   ```text
   2026-11-17T15:11:39.567094Z [info     ] Event: and now my watch begins 
starting at resource_version: 0 
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
 loc=kubernetes_executor_utils.py:170
   2026-11-17T15:11:39.771307Z [info     ] Event: task-pod Pending, 
annotations: <omitted> 
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
 loc=kubernetes_executor_utils.py:340
   2026-11-17T15:11:39.772770Z [info     ] Event: task-pod Pending, 
annotations: <omitted> 
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
 loc=kubernetes_executor_utils.py:340
   2026-11-17T15:11:39.778963Z [info     ] Event: task-pod Pending, 
annotations: <omitted> 
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
 loc=kubernetes_executor_utils.py:340
   2026-11-17T15:11:40.173488Z [info     ] Event: task-pod Pending, 
annotations: <omitted> 
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
 loc=kubernetes_executor_utils.py:340
   2026-11-17T15:11:40.227411Z [info     ] Event: task-pod is Running, 
annotations: <omitted> 
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
 loc=kubernetes_executor_utils.py:392
   DRIVER: task-pod is Running; patching pod churn in watch-expire-churn 5 
times a second
   2026-11-17T15:12:10.224521Z [info     ] Kubernetes watch timed out waiting 
for events. Restarting watch. 
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
 loc=kubernetes_executor_utils.py:126
   2026-11-17T15:12:11.230316Z [info     ] Event: and now my watch begins 
starting at resource_version: 4706 
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
 loc=kubernetes_executor_utils.py:170
   2026-11-17T15:12:41.236580Z [info     ] Kubernetes watch timed out waiting 
for events. Restarting watch. 
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
 loc=kubernetes_executor_utils.py:126
   2026-11-17T15:12:42.239313Z [info     ] Event: and now my watch begins 
starting at resource_version: 4706 
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
 loc=kubernetes_executor_utils.py:170
   2026-11-17T15:13:12.244990Z [info     ] Kubernetes watch timed out waiting 
for events. Restarting watch. 
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
 loc=kubernetes_executor_utils.py:126
   2026-11-17T15:13:13.249424Z [info     ] Event: and now my watch begins 
starting at resource_version: 4706 
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
 loc=kubernetes_executor_utils.py:170
   2026-11-17T15:13:13.267228Z [info     ] Kubernetes resource version 4706 is 
too old. Restarting watch from the current state. 
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
 loc=kubernetes_executor_utils.py:130
   2026-11-17T15:13:13.267476Z [info     ] Event: and now my watch begins 
starting at resource_version: 0 
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
 loc=kubernetes_executor_utils.py:170
   2026-11-17T15:13:13.275348Z [info     ] Event: task-pod is Running, 
annotations: <omitted> 
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
 loc=kubernetes_executor_utils.py:392
   DRIVER: 450 churn patches in 95s, outcome: is too old
   2026-11-17T15:13:43.274413Z [info     ] Kubernetes watch timed out waiting 
for events. Restarting watch. 
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
 loc=kubernetes_executor_utils.py:126
   2026-11-17T15:13:44.284986Z [info     ] Event: and now my watch begins 
starting at resource_version: 0 
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
 loc=kubernetes_executor_utils.py:170
   2026-11-17T15:13:44.292173Z [info     ] Event: task-pod is Running, 
annotations: <omitted> 
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
 loc=kubernetes_executor_utils.py:392
   DRIVER: watcher process alive: True exitcode: None
   2026-11-17T15:13:44.478040Z [info     ] Event: task-pod is Running, 
annotations: <omitted> 
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
 loc=kubernetes_executor_utils.py:392
   2026-11-17T15:13:44.481512Z [info     ] Event: Pod task-pod deleted before 
it could complete, annotations: <omitted> 
[airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils.KubernetesJobWatcher]
 loc=kubernetes_executor_utils.py:376
   DRIVER: deleted task-pod
   DRIVER: watcher reported task-pod failed
   exit=0
   ```
   
   <details>
   <summary>Reproducer source: setup.sh</summary>
   
   ```bash
   #!/usr/bin/env bash
   # MAIN_TREE: checkout of apache/main at 
3b35135afe9549e99088c4d557c62e5deafb125b; FIX_TREE: checkout of this branch.
   set -euo pipefail
   export DOCKER_CONTEXT=colima
   export KUBECONFIG="$PWD/kind/kubeconfig"
   mkdir -p kind
   cat > kind/kind-config.yaml <<'YAML'
   kind: Cluster
   apiVersion: kind.x-k8s.io/v1alpha4
   name: aip97-watch
   networking:
     apiServerAddress: "127.0.0.1"
     apiServerPort: 18445
   nodes:
     - role: control-plane
       image: 
kindest/node:v1.37.0@sha256:a1ed56cfb0e7b93589bdf97c8cd566405a265939e3620fc4f5de89adff580ae5
   YAML
   kind create cluster --config kind/kind-config.yaml --kubeconfig 
"$KUBECONFIG" --wait 180s
   for tree in "$MAIN_TREE" "$FIX_TREE"; do
       (cd "$tree" && uv sync --frozen && uv pip install --python 
.venv/bin/python datadog==0.56.0)
   done
   ./build_worker_image.sh "$MAIN_TREE" 
3b35135afe9549e99088c4d557c62e5deafb125b image/context 
aip97-watch-worker:3b35135a
   kind load docker-image aip97-watch-worker:3b35135a --name aip97-watch
   ```
   
   </details>
   
   <details>
   <summary>Reproducer source: watch_gap_e2e.py</summary>
   
   ```python
   #!/usr/bin/env python3
   from __future__ import annotations
   
   import argparse
   import contextlib
   import datetime
   import json
   import os
   import re
   import secrets
   import shutil
   import signal
   import socket
   import sqlite3
   import subprocess
   import threading
   import time
   from collections.abc import Callable
   from pathlib import Path
   from typing import Any
   
   import yaml
   from kubernetes import client, config
   
   API_PORT: int = 18090
   STATSD_PORT: int = 18095
   HOST_ALIAS: str = "host.lima.internal"
   DAG_ID: str = "watch_gap"
   TIMEOUT_LINE: str = "Kubernetes watch timed out waiting for events. 
Restarting watch."
   
   
   def utc() -> str:
       return datetime.datetime.now(datetime.timezone.utc).isoformat()
   
   
   def emit(event: str, **values: Any) -> None:
       print(json.dumps({"utc": utc(), "event": event, **values}, default=str), 
flush=True)
   
   
   def read_ti(database: Path) -> dict[str, Any] | None:
       with contextlib.suppress(sqlite3.OperationalError):
           with sqlite3.connect(f"file:{database}?mode=ro", uri=True, 
timeout=5) as connection:
               connection.row_factory = sqlite3.Row
               row = connection.execute(
                   "SELECT state, start_date, end_date, last_heartbeat_at FROM 
task_instance WHERE dag_id = ?",
                   (DAG_ID,),
               ).fetchone()
               return dict(row) if row else None
       return None
   
   
   def wait_for(predicate: Callable[[], Any], description: str, timeout: float, 
interval: float = 0.5) -> Any:
       deadline: float = time.monotonic() + timeout
       while time.monotonic() < deadline:
           if value := predicate():
               return value
           time.sleep(interval)
       raise TimeoutError(description)
   
   
   def main() -> None:
       parser = argparse.ArgumentParser()
       parser.add_argument("--label", required=True)
       parser.add_argument("--tree", required=True, type=Path)
       parser.add_argument("--image", required=True)
       parser.add_argument("--kubeconfig", required=True, type=Path)
       parser.add_argument("--context", required=True)
       parser.add_argument("--output", required=True, type=Path)
       arguments = parser.parse_args()
       output: Path = arguments.output
       home: Path = output / "home"
       (home / "dags").mkdir(parents=True)
       shutil.copyfile(src=Path(__file__).with_name("watch_gap_dag.py"), 
dst=home / "dags" / "watch_gap_dag.py")
       namespace: str = f"watch-gap-{arguments.label}"
       repository, tag = arguments.image.rsplit(":", 1)
       worker_env: dict[str, str] = {
           "AIRFLOW__CORE__EXECUTOR": "LocalExecutor",
           "AIRFLOW__CORE__LOAD_EXAMPLES": "False",
           "AIRFLOW__CORE__DAGS_FOLDER": "/opt/watch-gap/dags",
           "AIRFLOW__CORE__EXECUTION_API_SERVER_URL": 
f"http://{HOST_ALIAS}:{API_PORT}/execution/";,
       }
       template: dict[str, Any] = {
           "apiVersion": "v1",
           "kind": "Pod",
           "metadata": {"name": "watch-gap-template"},
           "spec": {
               "restartPolicy": "Never",
               "automountServiceAccountToken": False,
               "containers": [
                   {
                       "name": "base",
                       "image": arguments.image,
                       "imagePullPolicy": "Never",
                       "env": [{"name": name, "value": value} for name, value 
in worker_env.items()],
                       "resources": {"requests": {"cpu": "100m", "memory": 
"256Mi"}},
                   }
               ],
           },
       }
       (home / "pod-template.yaml").write_text(yaml.safe_dump(template))
       database: Path = home / "airflow.db"
       environment: dict[str, str] = {
           **os.environ,
           "PYTHONUNBUFFERED": "1",
           "AIRFLOW_HOME": str(home),
           "AIRFLOW__DATABASE__SQL_ALCHEMY_CONN": f"sqlite:///{database}",
           "AIRFLOW__CORE__DAGS_FOLDER": str(home / "dags"),
           "AIRFLOW__CORE__LOAD_EXAMPLES": "False",
           "AIRFLOW__CORE__EXECUTOR": "KubernetesExecutor",
           "AIRFLOW__CORE__PARALLELISM": "4",
           "AIRFLOW__CORE__EXECUTION_API_SERVER_URL": 
f"http://{HOST_ALIAS}:{API_PORT}/execution/";,
           "AIRFLOW__DAG_PROCESSOR__REFRESH_INTERVAL": "1",
           "AIRFLOW__SCHEDULER__SCHEDULER_HEARTBEAT_SEC": "1",
           "AIRFLOW__API_AUTH__JWT_SECRET": secrets.token_urlsafe(64),
           "AIRFLOW__METRICS__STATSD_ON": "True",
           "AIRFLOW__METRICS__STATSD_DATADOG_ENABLED": "True",
           "AIRFLOW__METRICS__STATSD_HOST": "127.0.0.1",
           "AIRFLOW__METRICS__STATSD_PORT": str(STATSD_PORT),
           "KUBECONFIG": str(arguments.kubeconfig),
           "AIRFLOW__KUBERNETES_EXECUTOR__CONFIG_FILE": 
str(arguments.kubeconfig),
           "AIRFLOW__KUBERNETES_EXECUTOR__CLUSTER_CONTEXT": arguments.context,
           "AIRFLOW__KUBERNETES_EXECUTOR__IN_CLUSTER": "False",
           "AIRFLOW__KUBERNETES_EXECUTOR__NAMESPACE": namespace,
           "AIRFLOW__KUBERNETES_EXECUTOR__POD_TEMPLATE_FILE": str(home / 
"pod-template.yaml"),
           "AIRFLOW__KUBERNETES_EXECUTOR__WORKER_CONTAINER_REPOSITORY": 
repository,
           "AIRFLOW__KUBERNETES_EXECUTOR__WORKER_CONTAINER_TAG": tag,
       }
       airflow: str = str(arguments.tree / ".venv" / "bin" / "airflow")
       config.load_kube_config(config_file=str(arguments.kubeconfig), 
context=arguments.context)
       kube = client.CoreV1Api()
       
kube.create_namespace(body=client.V1Namespace(metadata=client.V1ObjectMeta(name=namespace)))
       emit("SETUP", tree=str(arguments.tree), image=arguments.image, 
namespace=namespace)
   
       stop = threading.Event()
       datagrams_path: Path = output / "statsd-datagrams.log"
       receiver = socket.socket(family=socket.AF_INET, type=socket.SOCK_DGRAM)
       receiver.bind(("127.0.0.1", STATSD_PORT))
       receiver.settimeout(0.2)
   
       def capture() -> None:
           with datagrams_path.open(mode="a") as datagrams:
               while not stop.is_set():
                   with contextlib.suppress(TimeoutError):
                       packet, _ = receiver.recvfrom(65535)
                       for line in packet.decode().splitlines():
                           datagrams.write(f"{utc()} {line}\n")
                       datagrams.flush()
   
       capture_thread = threading.Thread(target=capture, daemon=True)
       capture_thread.start()
       processes: list[subprocess.Popen[bytes]] = []
       scheduler_log: Path = output / "scheduler.log"
       try:
           subprocess.run(args=[airflow, "db", "migrate"], env=environment, 
check=True, timeout=300,
                          stdout=subprocess.DEVNULL)
           for name, command in (
               ("api", ["api-server", "--apps", "execution", "--host", 
"127.0.0.1", "--port", str(API_PORT)]),
               ("dag-processor", ["dag-processor"]),
               ("scheduler", ["scheduler"]),
           ):
               with (output / f"{name}.log").open(mode="wb") as log:
                   processes.append(
                       subprocess.Popen(
                           args=[airflow, *command], env=environment, 
stdout=log, stderr=subprocess.STDOUT,
                           start_new_session=True,
                       )
                   )
           wait_for(lambda: _dag_serialized(database), "Dag not serialized", 
timeout=180)
           wait_for(lambda: "now my watch begins" in 
scheduler_log.read_text(errors="replace"),
                    "pod watcher never started", timeout=180)
           subprocess.run(
               args=[airflow, "dags", "trigger", DAG_ID, "--run-id", 
f"watch-gap-{arguments.label}"],
               env=environment, check=True, timeout=120, 
stdout=subprocess.DEVNULL,
           )
           running: dict[str, Any] = wait_for(
               lambda: (ti if (ti := read_ti(database)) and ti["state"] == 
"running" and ti["last_heartbeat_at"] else None),
               "task instance never ran", timeout=300,
           )
           emit("TI_RUNNING", ti=running)
           pod = wait_for(lambda: 
next(iter(kube.list_namespaced_pod(namespace=namespace).items), None),
                          "no worker pod", timeout=30)
           pod_name: str = pod.metadata.name
           wait_for(lambda: f"Event: {pod_name} is Running" in 
scheduler_log.read_text(errors="replace"),
                    "watcher never saw the pod running", timeout=60)
           offset: int = scheduler_log.stat().st_size
   
           def timeout_line() -> bool:
               with scheduler_log.open(mode="rb") as log:
                   log.seek(offset)
                   return TIMEOUT_LINE.encode() in log.read()
   
           wait_for(timeout_line, "watcher never timed out", timeout=120, 
interval=0.02)
           detected_at: str = utc()
           deleted = kube.delete_namespaced_pod(name=pod_name, 
namespace=namespace, grace_period_seconds=0)
           emit("POD_FORCE_DELETED_AFTER_WATCH_TIMEOUT", pod=pod_name, 
timeout_line_seen_at=detected_at,
                phase=deleted.status.phase, 
pod_resource_version=deleted.metadata.resource_version)
           final: dict[str, Any] = wait_for(
               lambda: (ti if (ti := read_ti(database)) and ti["state"] not in 
("running", "queued", "scheduled") else None),
               "task instance never finished", timeout=600, interval=1,
           )
           emit("TI_FINISHED", ti=final)
           time.sleep(10)
       finally:
           for process in processes:
               with contextlib.suppress(ProcessLookupError):
                   os.killpg(process.pid, signal.SIGTERM)
           for process in processes:
               try:
                   process.wait(timeout=20)
               except subprocess.TimeoutExpired:
                   with contextlib.suppress(ProcessLookupError):
                       os.killpg(process.pid, signal.SIGKILL)
           stop.set()
           capture_thread.join(timeout=2)
           receiver.close()
           kube.delete_namespace(name=namespace)
       summarize(output=output)
   
   
   def _dag_serialized(database: Path) -> bool:
       with contextlib.suppress(sqlite3.OperationalError):
           with sqlite3.connect(f"file:{database}?mode=ro", uri=True, 
timeout=5) as connection:
               return connection.execute("SELECT 1 FROM serialized_dag WHERE 
dag_id = ?", (DAG_ID,)).fetchone() is not None
       return False
   
   
   def summarize(output: Path) -> None:
       keep = re.compile(
           r"watch timed out|my watch begins|is Running|too old|deleted before 
it could complete|Changing state of|"
           r"Received executor event|without a heartbeat|Marking task as 
FAILED|reported that the task instance"
       )
       print("\nSCHEDULER_LOG (watcher and task state lines)")
       for line in (output / 
"scheduler.log").read_text(errors="replace").splitlines():
           if keep.search(line):
               print(re.sub(r" \[airflow\.\S+\]| loc=\S+", "", line)[:400])
       print("\nEXECUTOR_GAUGES (value changes)")
       last: dict[str, str] = {}
       for line in (output / "statsd-datagrams.log").read_text().splitlines():
           if match := 
re.search(r"(airflow\.executor\.(?:running_tasks|open_slots)):([\d.]+)\|g", 
line):
               if last.get(match[1]) != match[2]:
                   last[match[1]] = match[2]
                   print(line.split(" ", 1)[0], f"{match[1]}={match[2]}")
   
   
   if __name__ == "__main__":
       main()
   ```
   
   </details>
   
   <details>
   <summary>Reproducer source: watch_gap_dag.py</summary>
   
   ```python
   from __future__ import annotations
   
   import time
   
   import pendulum
   
   from airflow.sdk import DAG, task
   
   
   @task
   def long_running() -> None:
       print("WATCH_GAP_TASK_STARTED", flush=True)
       time.sleep(900)
   
   
   with DAG(
       dag_id="watch_gap",
       schedule=None,
       start_date=pendulum.datetime(2026, 1, 1, tz="UTC"),
       catchup=False,
       is_paused_upon_creation=False,
       default_args={"retries": 0},
   ):
       long_running()
   ```
   
   </details>
   
   <details>
   <summary>Reproducer source: watch_expire_e2e.py</summary>
   
   ```python
   #!/usr/bin/env python3
   from __future__ import annotations
   
   import argparse
   import multiprocessing
   import os
   import time
   from pathlib import Path
   
   CHURN_NAMESPACE: str = "watch-expire-churn"
   SCHEDULER_JOB_ID: str = "777"
   
   
   def wait_for_line(log: Path, needles: tuple[str, ...], offset: int, timeout: 
float) -> str | None:
       deadline: float = time.monotonic() + timeout
       while True:
           with log.open(mode="rb") as handle:
               handle.seek(offset)
               text: str = handle.read().decode(errors="replace")
           for needle in needles:
               if needle in text:
                   return needle
           if time.monotonic() >= deadline:
               return None
           time.sleep(0.2)
   
   
   def main() -> None:
       parser = argparse.ArgumentParser()
       parser.add_argument("--label", required=True)
       parser.add_argument("--kubeconfig", required=True)
       parser.add_argument("--context", required=True)
       parser.add_argument("--log", required=True, type=Path)
       arguments = parser.parse_args()
       os.environ.update(
           {
               "AIRFLOW_HOME": str(arguments.log.with_suffix(".home")),
               "KUBECONFIG": arguments.kubeconfig,
               "AIRFLOW__KUBERNETES_EXECUTOR__CONFIG_FILE": 
arguments.kubeconfig,
               "AIRFLOW__KUBERNETES_EXECUTOR__CLUSTER_CONTEXT": 
arguments.context,
               "AIRFLOW__KUBERNETES_EXECUTOR__IN_CLUSTER": "False",
           }
       )
       from kubernetes import client, config
   
       from 
airflow.providers.cncf.kubernetes.executors.kubernetes_executor_utils import 
KubernetesJobWatcher
       from airflow.providers.cncf.kubernetes.kube_config import KubeConfig
   
       config.load_kube_config(config_file=arguments.kubeconfig, 
context=arguments.context)
       kube = client.CoreV1Api()
       namespace: str = f"watch-expire-{arguments.label}"
       for name in (namespace, CHURN_NAMESPACE):
           
kube.create_namespace(body=client.V1Namespace(metadata=client.V1ObjectMeta(name=name)))
       kube.create_namespaced_pod(
           namespace=CHURN_NAMESPACE,
           body={
               "apiVersion": "v1",
               "kind": "Pod",
               "metadata": {"name": "churn"},
               "spec": {
                   "nodeSelector": {"watch-expire": "never"},
                   "containers": [{"name": "c", "image": 
"aip97-watch-worker:3b35135a"}],
               },
           },
       )
       watcher = KubernetesJobWatcher(
           namespace=namespace,
           watcher_queue=multiprocessing.Queue(),
           resource_version="0",
           scheduler_job_id=SCHEDULER_JOB_ID,
           kube_config=KubeConfig(),
       )
       watcher.start()
       try:
           if not wait_for_line(arguments.log, ("now my watch begins",), 
offset=0, timeout=120):
               raise TimeoutError("watcher did not start")
           kube.create_namespaced_pod(
               namespace=namespace,
               body={
                   "apiVersion": "v1",
                   "kind": "Pod",
                   "metadata": {
                       "name": "task-pod",
                       "labels": {"airflow-worker": SCHEDULER_JOB_ID},
                       "annotations": {"dag_id": "d", "task_id": "t", "run_id": 
"r", "try_number": "1"},
                   },
                   "spec": {
                       "terminationGracePeriodSeconds": 0,
                       "containers": [
                           {"name": "base", "image": 
"aip97-watch-worker:3b35135a", "imagePullPolicy": "Never",
                            "command": ["sleep", "900"]}
                       ],
                   },
               },
           )
           if not wait_for_line(arguments.log, ("Event: task-pod is Running",), 
offset=0, timeout=120):
               raise TimeoutError("watcher never saw task-pod running")
           print("DRIVER: task-pod is Running; patching pod churn in", 
CHURN_NAMESPACE, "5 times a second", flush=True)
           offset: int = arguments.log.stat().st_size
           started: float = time.monotonic()
           outcome: str | None = None
           count: int = 0
           while outcome is None and time.monotonic() - started < 300:
               kube.patch_namespaced_pod(
                   name="churn", namespace=CHURN_NAMESPACE, body={"metadata": 
{"annotations": {"n": str(count)}}}
               )
               count += 1
               time.sleep(0.2)
               if count % 25 == 0:
                   outcome = wait_for_line(
                       arguments.log, ("is too old", "Unknown error in 
KubernetesJobWatcher"), offset=offset, timeout=0
                   )
           print(f"DRIVER: {count} churn patches in {time.monotonic() - 
started:.0f}s, outcome: {outcome}", flush=True)
           offset = arguments.log.stat().st_size
           if not wait_for_line(arguments.log, ("now my watch begins",), 
offset=offset, timeout=120):
               raise TimeoutError("watcher stopped watching")
           print("DRIVER: watcher process alive:", watcher.is_alive(), 
"exitcode:", watcher.exitcode, flush=True)
           kube.delete_namespaced_pod(name="task-pod", namespace=namespace, 
grace_period_seconds=0)
           print("DRIVER: deleted task-pod", flush=True)
           reported = watcher.watcher_queue.get(timeout=60)
           print("DRIVER: watcher reported", reported.pod_name, reported.state, 
flush=True)
       finally:
           watcher.terminate()
           watcher.join(timeout=10)
           for name in (namespace, CHURN_NAMESPACE):
               kube.delete_namespace(name=name)
   
   
   if __name__ == "__main__":
       main()
   ```
   
   </details>
   
   <details>
   <summary>Reproducer source: Dockerfile</summary>
   
   ```dockerfile
   FROM 
python:3.12-slim-bookworm@sha256:782412e85d0f0984994c290652577d4018aff08145c85b262bb63dc0c7522254
   ENV PYTHONDONTWRITEBYTECODE=1 \
       PYTHONUNBUFFERED=1 \
       PIP_DISABLE_PIP_VERSION_CHECK=1 \
       PIP_NO_CACHE_DIR=1 \
       AIRFLOW_HOME=/opt/watch-gap/home
   WORKDIR /opt/watch-gap
   RUN python -m pip install uv==0.13.8
   COPY requirements.txt ./
   RUN uv pip install --system --no-deps -r requirements.txt
   COPY source ./source
   RUN uv pip install --system --no-deps --no-sources \
       -e ./source/airflow-core -e ./source/task-sdk \
       ./source/providers/standard ./source/providers/common/compat 
./source/providers/cncf/kubernetes \
       && python -c "import airflow, airflow.sdk; print('airflow', 
airflow.__version__)"
   COPY source-commit.txt watch_gap_dag.py ./
   RUN mkdir -p dags home && cp watch_gap_dag.py dags/
   ```
   
   </details>
   
   <details>
   <summary>Reproducer source: build_worker_image.sh</summary>
   
   ```bash
   #!/usr/bin/env bash
   # Licensed to the Apache Software Foundation (ASF) under one
   # or more contributor license agreements.  See the NOTICE file
   # distributed with this work for additional information
   # regarding copyright ownership.  The ASF licenses this file
   # to you under the Apache License, Version 2.0 (the
   # "License"); you may not use this file except in compliance
   # with the License.  You may obtain a copy of the License at
   #
   #   http://www.apache.org/licenses/LICENSE-2.0
   #
   # Unless required by applicable law or agreed to in writing,
   # software distributed under the License is distributed on an
   # "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
   # KIND, either express or implied.  See the License for the
   # specific language governing permissions and limitations
   # under the License.
   #
   # Usage: build_worker_image.sh <airflow-worktree> <commit> <context-dir> 
<image-tag>
   set -euo pipefail
   worktree="$1"
   commit="$2"
   context="$3"
   tag="$4"
   demo="$(cd "$(dirname "$0")" && pwd)"
   
   rm -rf "$context"
   mkdir -p "$context/source"
   git -C "$worktree" archive --format=tar "$commit" \
       airflow-core task-sdk shared providers/standard providers/common/compat 
providers/cncf/kubernetes \
       | tar -x -C "$context/source"
   git -C "$worktree" rev-parse "$commit" > "$context/source-commit.txt"
   (cd "$worktree" && uv export --frozen --no-hashes --no-dev 
--no-emit-workspace --no-header \
       --package apache-airflow-core --package apache-airflow-task-sdk \
       --package apache-airflow-providers-cncf-kubernetes --package 
apache-airflow-providers-standard \
       --package apache-airflow-providers-common-compat -o 
"$context/requirements.txt")
   cp "$demo/Dockerfile" "$demo/watch_gap_dag.py" "$context/"
   docker build --platform linux/arm64 --tag "$tag" "$context"
   docker image inspect "$tag" --format '{{.Id}} {{.Architecture}} {{.Os}}'
   ```
   
   </details>
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   - [X] Yes (please specify the tool below)
   
   Generated-by: GitHub Copilot CLI (gpt-6.1-sol, claude-opus-5.5) following 
[the 
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   
   ---
   
   * Read the Pull Request Guidelines for more information: 
https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines.
 Note: commit author/co-author name and email in commits become permanently 
public when merged.
   * For fundamental code changes, an Airflow Improvement Proposal (AIP) is 
needed: 
https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals.
   * When adding dependency, check compliance with the ASF 3rd Party License 
Policy: https://www.apache.org/legal/resolved.html#category-x.
   * For significant user-facing changes create newsfragment: 
{pr_number}.significant.rst, in airflow-core/newsfragments: 
https://github.com/apache/airflow/tree/main/airflow-core/newsfragments. You can 
add this file in a follow-up commit after the PR is created so you know the PR 
number.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to