potiuk opened a new issue, #66396: URL: https://github.com/apache/airflow/issues/66396
### Apache Airflow Provider versions `apache-airflow-providers-cncf-kubernetes >= 10.15.0` (verified on `main`; introduced by #61839, released in 10.15.0) ### Apache Airflow version 3.x with `KubernetesExecutor` and **2 or more schedulers**. ### What happened PR #61839 added a periodic call to `_adopt_completed_pods` from inside `KubernetesExecutor.sync()`, gated by `[scheduler] orphaned_tasks_check_interval` (default 300 s): https://github.com/apache/airflow/blob/main/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py#L271-L275 `_adopt_completed_pods` selects every Succeeded pod whose `airflow-worker` label is **not** the current scheduler's label and PATCHes it with the current scheduler's label so its `KubernetesJobWatcher` will see the change and DELETE the pod: https://github.com/apache/airflow/blob/main/providers/cncf/kubernetes/src/airflow/providers/cncf/kubernetes/executors/kubernetes_executor.py#L684-L714 ```python query_kwargs = { "field_selector": "status.phase=Succeeded", "label_selector": ( "kubernetes_executor=True," f"airflow-worker!={new_worker_id_label},{POD_EXECUTOR_DONE_KEY}!=True" ), } ``` With **N schedulers (N ≥ 2)** running concurrently, every `orphaned_tasks_check_interval` each scheduler iterates over every Succeeded pod that doesn't carry **its own** label and PATCHes it. Schedulers fight each other: - Scheduler A relabels every Succeeded pod owned by B and C → A's watcher receives the update event and starts DELETEing - Scheduler B does the same a few seconds later → relabels A's freshly patched pods to B → B's watcher takes over - Scheduler C the same At steady state with high pod churn (we have a user reporting 1000+ pods created in a couple of minutes — see context below), this manifests as: - Heavy `PATCH /api/v1/namespaces/.../pods/...` traffic against the kube API server, multiplied by N - Heavy `_list_pods` traffic — see #35599 for the known per-call cost at scale (15–30 s with 500 pods, returning the full V1PodList) - Tasks stalling in `scheduled` / `queued` because every scheduler loop is burning seconds inside `_list_pods` + `patch_namespaced_pod` and not doing useful scheduling - `delete_worker_pods=False` does NOT help — the periodic adoption code path doesn't gate on `delete_worker_pods`, it goes through the watcher's delete ### What you think should happen instead The periodic `_adopt_completed_pods` should only adopt pods owned by schedulers that are **no longer alive** — i.e., scope the `airflow-worker!=<my_label>` query to also exclude `airflow-worker IN (<set of currently-active scheduler job IDs>)`. The set of active scheduler job IDs is already known to the scheduler via the `SchedulerJob` rows. With that scoping: - Single-scheduler deployment: no behavior change. - Multi-scheduler deployment: each scheduler only adopts orphaned pods whose owning scheduler is gone — no thrash, no cross-scheduler relabeling. The original goal of #61839 (closing #57553 — completed pods leaking after scheduler restart) is still achieved. ### How to reproduce 1. Deploy Airflow 3.x with `KubernetesExecutor` and at least 2 schedulers using `cncf.kubernetes >= 10.15.0`. 2. Run a workload that produces enough Succeeded pods to keep them visible across the 5-minute `orphaned_tasks_check_interval` window (anything sustained for > 5 min works). 3. Observe `PATCH /api/v1/namespaces/.../pods/...` traffic against the kube API server — each Succeeded pod gets PATCHed by every scheduler that doesn't currently own its label, on every interval tick. 4. The reproducer in the original PR (#61839, #57553) still works for the single-scheduler "completed pods leak after restart" case, so any fix needs to keep that scenario working. ### Operating system N/A (the bug is in the Python code path, not OS-specific). ### Versions of Apache Airflow Providers - `apache-airflow-providers-cncf-kubernetes` 10.15.0 — present on `main`. ### Deployment Official Apache Airflow Helm Chart and any other Kubernetes deployment running multiple schedulers + KubernetesExecutor. ### Anything else - Caught while triaging a user report on the Airflow user mailing list: 3 schedulers, Airflow 3.2.1, "thrashing between the schedulers trying to PATCH/DELETE pods after task completion" with `delete_worker_pods=False` not helping. Their workaround was to move workloads to Celery, which bypasses this code path entirely. - Workaround for affected users until a fix lands: set `[scheduler] orphaned_tasks_check_interval` very high (e.g., 86400) and rely on a separate `delete_worker_pods` cron for cleanup. - Original closing-issue / introducing-PR / related-perf-issue: - #57553 (the leak this regression was meant to fix) - #61839 (the introducing PR) - #35599 (the underlying `_list_pods` cost at scale that compounds the impact) ### Are you willing to submit PR? - [X] Yes I am willing to submit a PR! (per the discussion in the user thread, I offered to draft a fix that scopes the periodic adoption to dead schedulers only) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
