arham766 opened a new pull request, #69508: URL: https://github.com/apache/airflow/pull/69508
The task-supervisor is a single-threaded `selectors` event loop, and the `SetXCom` branch of `_handle_request` ran `client.xcoms.set(...)` — a synchronous httpx POST that serializes and uploads the entire payload — inline on that loop. For a large XCom value the POST outlasts `task_instance_heartbeat_timeout` (default 300s), `_send_heartbeat_if_needed` never gets a turn, and the scheduler purges the still-running task as a zombie. That's #64628 (18-minute upload for a ~300MB payload in the report). **Fix** - `SetXCom`/`DeleteXCom` handlers now run on a lazily-created single-worker `ThreadPoolExecutor`; the event loop keeps servicing sockets and heartbeats while the upload is in flight. - Responses are drained back on the event-loop thread (`_drain_pending_requests`, called from `_service_subprocess`), so all socket writes stay single-threaded. Errors map to the same `ErrorResponse` shapes as the existing inline handling, so the child is never left waiting. - A self-pipe wakes the selector when a worker finishes, so responses aren't delayed by the `select()` timeout. - Ordering is preserved: single worker + head-of-queue draining, and the task-runner protocol blocks on each request per child socket, so at most one offloaded request per child is in flight anyway. - The current OTel context is attached on the worker thread so the outbound HTTP call stays linked to the task span. - `InProcessTestSupervisor` (`dag.test()`) overrides the submission to stay fully synchronous — no behavior change there. This follows the approach from #64743 (closed as a stale draft, not on technical grounds), rebased onto the `request_handlers.py` refactor, with the review findings from that PR baked in (guarded `exc.response`, guarded `.json()`, non-blocking executor shutdown in `_cleanup_open_sockets`). **Testing** New regression test `test_slow_set_xcom_does_not_block_heartbeats`: the mocked `xcoms.set` blocks until a heartbeat is observed, so on `main` it deadlocks into failure (verified: fails on `main`, passes here). Full `TestHandleRequest` passes; existing `set_xcom`/`delete_xcom` cases were adapted to wait for the drained response. Known limitation, stated for the record: JSON serialization inside httpx holds the GIL, so a multi-second stall during serialization is still possible — but it's bounded and far below the heartbeat timeout, and the dominant cost (the network upload) now fully overlaps heartbeats. closes: #64628 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
