cmettler opened a new pull request, #65738:
URL: https://github.com/apache/airflow/pull/65738
When a running TaskInstance is forcibly transitioned out of `running`
(scheduler reset, REST PATCH, etc.), the next heartbeat from the still-running
task-runner returns HTTP 409 and the supervisor kills the task. On Linux this
produced two bugs:
1. **Orphaned subprocesses** — any Popen child the task-runner had spawned
(`@task.virtualenv`, `DockerOperator`, `BashOperator`, Cosmos dbt, etc.) was
reparented to PID 1 and kept running until it finished on its own, wasting
CPU/RAM/API quota.
2. **Supervisor crash** — ~60s later `_cleanup_open_sockets()` closed the
selector while `_service_subprocess()` was still polling it, raising
`ValueError: I/O operation on closed epoll object` (regression from #51180).
### Fix
Place the task-runner in its own session via `os.setsid()` immediately after
fork, then have `kill()` signal the whole process group via
`os.killpg(os.getpgid(pid), sig)`. This reaches every subprocess the
task-runner spawned. Grandchildren without a SIGTERM handler exit promptly and
close their inherited pipes, so the supervisor drains `_open_sockets` normally
and never enters the cleanup-the-selector-mid-loop path.
`killpg`/`getpgid` fall back to `self._process.send_signal(sig)` on
`ProcessLookupError` or `PermissionError`, preserving behaviour when the group
has vanished or permissions are lacking.
### Tests
- `test_kill_signals_process_group` — primary path uses `killpg`.
- `test_kill_falls_back_to_send_signal_when_group_signal_fails` (4 params:
`{ProcessLookupError, PermissionError} × {getpgid, killpg}`).
- `test_child_is_session_leader` — real-fork regression: asserts child's
PGID == child's PID after `ActivitySubprocess.start()`.
- Two existing kill tests updated to mock `os.getpgid` / `os.killpg`
explicitly.
- All 136 supervisor tests pass (`uv run --project task-sdk pytest
task-sdk/tests/task_sdk/execution_time/test_supervisor.py`).
closes: #65505
---
##### Was generative AI tooling used to co-author this PR?
- [X] Yes (Claude Opus 4.7 (1M context))
Generated-by: Claude Opus 4.7 (1M context) following [the
guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
