ChickenBenny opened a new issue, #75394: URL: https://github.com/apache/airflow/issues/75394
### Under which category would you file this issue? Task SDK ### Apache Airflow version main ### What happened and how to reproduce it? Since #74869, the supervisor enforces `execution_timeout` for every task, using the value from `TIRunContext`. But it only acts `killed_task_cleanup_time` after the timeout, and the Python runner still keeps its own timers. With `execution_timeout=10s` and the default `killed_task_cleanup_time=60`: | Time | Python task | Go / TypeScript / Java task | |---|---|---| | 10s | Runner's SIGALRM raises `AirflowTaskTimeout`, `on_kill()` runs | Nothing, the task keeps running | | 70s | SIGTERM, only if still running | SIGTERM, the first signal the task gets | | 130s | SIGKILL, only if still alive | SIGKILL, only if still alive | Problems: 1. **Language SDK tasks overrun by `killed_task_cleanup_time`** (#74717). 2. **Python has several timers:** the runner's SIGALRM, the supervisor backstop, and `asyncio.wait_for` in `BaseAsyncOperator.execute`, which reads `task.execution_timeout` and raises `TimeoutError` without calling `on_kill()`. 3. **The timeout is not recorded as the reason.** A language SDK task stopped by SIGTERM reports a generic failure. If it exits without reporting and has retries left, the supervisor does not report `UP_FOR_RETRY` (it is in `STATES_SENT_DIRECTLY`), so the task instance stays `running` until the scheduler marks it "killed externally". 4. **A task that swallows the cancellation succeeds**, e.g. a Go task that returns `nil` after `ctx.Done()`. A Python task cannot do this by accident, since `AirflowTaskTimeout` is a `BaseException`. ### What you think should happen instead? The supervisor keeps the only clock, and the task process only reacts to a signal: - **Supervisor:** at `execution_timeout`, signal the task process, then SIGKILL after `killed_task_cleanup_time`. Record that the task timed out, and report the outcome with the timeout as the reason, including `RetryTask` or a failed state when the runtime exits without reporting. - **Language SDK runtimes:** no change. Go, TypeScript and Java (#74690) already cancel the task's context on SIGTERM; they just get it at the deadline. - **Python task runner:** drop the SIGALRM timer and the `asyncio.wait_for` timer, and raise `AirflowTaskTimeout` when the supervisor signals a timeout, so `on_kill()`, callbacks and retries behave as today. A signal rather than a comms message, because the supervisor only sends frames in response to a request (`comms.py`), and all four runtimes would need a protocol change to handle a pushed message. 1. **Which signal tells a Python task it timed out?** SIGTERM already means a server-requested stop (`on_kill()` without `AirflowTaskTimeout`). SIGUSR1 is unused in the task process (it would need resetting in `_reset_signals`, like SIGUSR2). SIGALRM is taken by `airflow.sdk.execution_time.timeout`, which Dag parsing and providers use. 2. Should a task that reports success after its timeout be failed by the supervisor? 3. Should the TypeScript runtime's 30s `ABORT_GRACE_PERIOD_MS` follow `killed_task_cleanup_time`? Out of scope: deferral (the runner's `TODO: handle timeout in case of deferral`), and letting a language SDK task opt out of `[core] default_task_execution_timeout`. ### Operating System _No response_ ### Deployment None ### Apache Airflow Provider(s) _No response_ ### Versions of Apache Airflow Providers _No response_ ### Official Helm Chart version Not Applicable ### Kubernetes Version _No response_ ### Helm Chart configuration _No response_ ### Docker Image customizations _No response_ ### Anything else? _No response_ ### Are you willing to submit PR? - [x] Yes I am willing to submit a PR! ### Code of Conduct - [x] I agree to follow this project's [Code of Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
