arkadiuszbach opened a new pull request, #69034: URL: https://github.com/apache/airflow/pull/69034
## Summary PR #61627 installed a SIGTERM/SIGINT handler in `ActivitySubprocess.wait()` that forwarded the signal to the task subprocess via `os.kill(self.pid, ...)`, causing the operator's `on_kill()` to run whenever the supervisor was signalled. On KubernetesExecutor this kills the task on every routine pod shutdown. `dumb-init` runs as PID 1 with `DUMB_INIT_SETSID=1` (the image default), so it broadcasts SIGTERM to the whole process group — the task subprocess already receives it directly. After #61627 the supervisor forwarded it a second time, so the task could receive SIGTERM twice; either delivery makes the child handler call `on_kill()` and abort the task the moment the pod starts terminating. That is wrong for a graceful pod shutdown (spot interruption, scale-down, rolling update). For operators that release external resources in `on_kill()` (Databricks tearing down a job cluster, ADF cancelling a pipeline, Dataproc, Bash terminating its child, etc.) this destroys in-flight work we want to let finish. This change makes the task warm-shut-down instead, in two parts: 1. **Supervisor** — replace the forward-and-kill handler with a *warm-shutdown* handler that does **not** signal the task. It logs the received signal and lets the supervisor stay in its monitoring loop so the running task continues to completion. The handler is installed in `_PythonCoordinator.execute_task` around **both** `start()` (the RUNNING transition) and `wait()` (run + terminal-state report + log upload), so no window is left where the default SIGTERM disposition could tear the supervisor down. Previous dispositions are restored on exit so a reused Celery prefork worker doesn't leak the handler. 2. **KubernetesExecutor pod template** — set `DUMB_INIT_SETSID=0`, so dumb-init delivers pod-termination signals only to its direct child (the supervisor), not to the whole process group. Without this the task subprocess would still receive the group-broadcast SIGTERM directly and call `on_kill()`, defeating the warm shutdown. This mirrors what Celery worker pods already do. `on_kill()` is still reached through the legitimate kill paths (heartbeat-failure / server-terminated kills, overtime termination, or an explicit success/failure transition) — all of which already drive `self.kill(...)` and are unchanged. ## Why In CeleryExecutor, SIGTERM never reaches the supervisor and `on_kill()` is not called (and shouldn't be — for some operators `on_kill()` stops the remote work: job clusters, pipelines, etc.). Instead we want the running job to finish. The official docker-compose and Helm chart set `DUMB_INIT_SETSID=0` on the Celery worker ([worker-deployment.yaml](https://github.com/apache/airflow/blob/0619574dd215787b18ef0e760f54d891eb9b42ae/chart/templates/workers/worker-deployment.yaml#L324)), so only the Celery MainProcess (which coordinates work) receives SIGTERM. It does not forward it to the pool workers; it enters a warm shutdown and waits for running tasks to finish, so `on_kill()` is never called. KubernetesExecutor has no MainProcess to shield the supervisor: dumb-init (PID 1) sends SIGTERM directly to the supervisor. So we need to replicate what Celery does — a SIGTERM handler that keeps the supervisor alive — otherwise Python's default disposition terminates it immediately. ```text Celery: dumb-init (PID 1) → starts the Celery MainProcess Celery MainProcess (PID 7) → receives SIGTERM from dumb-init, starts warm shutdown ForkPoolWorker-1 (PID 11) → runs supervisor code; no SIGTERM (DUMB_INIT_SETSID=0) ForkPoolWorker-2 (PID 12) → same as worker-1 Kubernetes: dumb-init (PID 1) → starts the supervisor directly supervisor (PID 7) → receives SIGTERM from dumb-init; needs a handler to wait for the task to finish, else it stops it ``` #61627 partially improved KubernetesExecutor. **Before** it, on Airflow 3.2.2, when a task pod received SIGTERM (`DUMB_INIT_SETSID=1`): - The supervisor got SIGTERM from dumb-init and stopped without waiting for the child; the pod stopped; KubernetesExecutor got the `DELETED` event and marked the task failed. - The child the supervisor spawned might or might not have run `on_kill()` — depending on whether SIGTERM arrived before or after it registered its own handler ([task_runner.py#L1254](https://github.com/apache/airflow/blob/3.2.2/task-sdk/src/airflow/sdk/execution_time/task_runner.py#L1254)) (the supervisor didn't forward it pre-fix; dumb-init delivered it via the group because `DUMB_INIT_SETSID=1`). - Net result: regardless of timing, the task failed. With `safe-to-evict=false` this is rarer, since the pod is mostly signalled only on node upgrades or manual deletion. **After** #61627: - SIGTERM lands **before** the supervisor registers its handler → Python's default disposition fires, the supervisor stops without waiting for the child (which may or may not have been spawned), the pod stops, the executor gets `DELETED`, the task is failed. - SIGTERM lands **after** the supervisor registers its handler but **before** the child registers its own → the child's default disposition fires, it stops, the pod stops, the task is failed. - SIGTERM lands **after** both registered → `on_kill()` runs (up to twice — see below). For a no-op `on_kill()` (e.g. PythonOperator) the task completes and the pod sits in `Terminating`; but for operators with a real `on_kill()` (Databricks, ADF, Dataproc, Bash) it aborts the running task — exactly what we want to avoid. > **Note on "twice":** the task main process can receive SIGTERM both from the group broadcast and from the supervisor's forward. Standard (non-realtime) signals coalesce if a second arrives while the first is still pending, so the two deliveries may collapse into one. Either way `on_kill()` fires at least once and the task is aborted. This is fundamentally a race condition. In Celery, tasks/supervisors don't start until Celery has bootstrapped and registered its warm-shutdown handler, so a SIGTERM arriving early can't fail a task. In KubernetesExecutor, the pod can be deleted before the supervisor registers its handler, the executor sees `DELETED`, and the task is failed — even though user code may not have started yet (queued, not running), in which case we'd rather retry. This PR: - Sets `DUMB_INIT_SETSID=0` so only the supervisor receives SIGTERM (as in CeleryExecutor); `on_kill()` is skipped. - Registers the supervisor's SIGTERM handler **before** the task transitions to RUNNING, removing the race: if user code hasn't started, the pod can be safely recreated; once it has, the supervisor holds pod termination until the task completes. A follow-up PR will add the KubernetesExecutor change to recreate the pod on a `DELETED` event when the task hadn't started yet. ## Note With this fix, once a task has reached the RUNNING state it will be gracefully terminated — the supervisor holds pod termination until the running task completes instead of aborting it. The remaining case is when the supervisor receives SIGTERM **before** it has registered its handler: the task fails because the signal lands during that small startup window. On busy Airflow instances with a lot going on, this can happen often enough to matter. The follow-up PR will fix that case by recreating the task pod on the `DELETED` event when user code hadn't started yet, so it retries instead of failing. might be related: https://github.com/apache/airflow/discussions/62978 --- ##### Was generative AI tooling used to co-author this PR? - [X] Yes — Claude Code (Opus 4.8) Generated-by: Claude Code (Opus 4.8) following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
