The GitHub Actions job "Tests (AMD)" on airflow.git/fix/supervisor-sigterm-handling has failed. Run started by GitHub user arkadiuszbach (triggered by arkadiuszbach).
Head commit for run: 009d07dba257925d414c77f75f7d01b3762665ab / bach.ab <[email protected]> Warm-shutdown supervisor on SIGTERM instead of killing the running task PR #61627 installed a SIGTERM/SIGINT handler in `ActivitySubprocess.wait()` that forwards the signal to the task subprocess via `os.kill(self.pid, ...)`, so the operator's `on_kill()` runs when the supervisor is signalled. On KubernetesExecutor this kills the task on every routine pod shutdown. `dumb-init` runs as PID 1 with DUMB_INIT_SETSID=1 (the image default), so it broadcasts SIGTERM to the whole process group — the task subprocess already receives it directly. After #61627 the supervisor forwards it a second time, so the task gets SIGTERM twice; either delivery makes the child handler call `on_kill()` and abort the task the moment the pod starts terminating. That is wrong for a graceful pod shutdown (spot interruption, scale-down, rolling update). For operators that release external resources in `on_kill()` (Databricks tearing down a job cluster, ADF cancelling a pipeline, etc.) this destroys in-flight work we want to let finish. This change makes the task warm-shut-down instead, in two parts: 1. supervisor: replace the forward-and-kill handler with a *warm-shutdown* handler that does NOT signal the task. It logs the received signal and keeps the supervisor in its monitoring loop so the running task continues to completion. The handler is installed in `_PythonCoordinator.execute_task` around both `start()` (the RUNNING transition) and `wait()` (run + terminal-state report + log upload), so no window is left where the default SIGTERM disposition could tear the supervisor down. Previous dispositions are restored on exit so a reused Celery prefork worker doesn't leak the handler. 2. KubernetesExecutor pod template: set DUMB_INIT_SETSID=0, so dumb-init delivers pod-termination signals only to its direct child (the supervisor), not to the whole process group. Without this the task subprocess would still receive the group-broadcast SIGTERM directly and call `on_kill()`, defeating the warm shutdown. This mirrors what Celery worker pods already do. `on_kill()` is still reached through the legitimate kill paths (heartbeat-failure / server-terminated kills, overtime termination, or an explicit success/failure transition) — all of which already drive `self.kill(...)` and are unchanged. Report URL: https://github.com/apache/airflow/actions/runs/28238788272 With regards, GitHub Actions via GitBox --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
