amoghrajesh opened a new issue, #71707:
URL: https://github.com/apache/airflow/issues/71707
**Under which category would you file this issue?** Task SDK
**Apache Airflow version:** main (current
`task-sdk/src/airflow/sdk/execution_time/supervisor.py`, where
`_FORK_EXEC_PLATFORMS = {"darwin"}` is unchanged) — likely affects any 3.x
release with the same gate.
**What happened and how to reproduce it?**
Issue Description:
Tasks that make outbound HTTPS calls (cloud-provider hooks/operators built
on `google-auth`/`urllib3`/`requests` — e.g. `GKEStartPodOperator` or a
Google-provider sensor) can hang indefinitely with no further progress.
`execution_timeout` does not recover the task (that's a separate, related
problem tracked in #53337 — this issue is about the hang itself).
`py-spy` dumps of a hung worker, taken minutes apart, are byte-identical —
confirming a true deadlock, not a slow call. Every active thread in the process
— the task's own thread, plus unrelated background threads (an OpenTelemetry
metrics-exporter thread, a `google-auth` credential-refresh thread, an
OpenLineage listener thread) — is frozen at the identical point, each building
a fresh `SSLContext` for a *different* remote host:
```
__new__ (ssl.py:440)
create_urllib3_context (urllib3/util/ssl_.py:252)
_ssl_wrap_socket_and_match_hostname (urllib3/connection.py:928)
connect (urllib3/connection.py:812)
...
```
That rules out a network-specific hang — `SSLContext` construction itself is
blocked, process-wide.
The task subprocess is created via a bare `os.fork()` in
`ActivitySubprocess.start()`
(`task-sdk/src/airflow/sdk/execution_time/supervisor.py`), since
`_should_use_exec()` only returns `True` on `darwin`:
```python
_FORK_EXEC_PLATFORMS = {"darwin"}
def _should_use_exec() -> bool:
return sys.platform in _FORK_EXEC_PLATFORMS
```
The supervisor process is multithreaded by the time any task starts (OTel
exporter threads, per-connection `google-auth` background refresh threads, an
OpenLineage listener, etc. are normal background threads in a long-running
worker). `os.fork()` copies the address space, but only the calling thread
survives in the child. If any sibling thread was mid-way through OpenSSL's
global lock (taken inside `SSL_CTX_new()`, which every `ssl.SSLContext()` call
goes through) at the exact instant of the fork, the child inherits that lock
already held, permanently — the thread that held it no longer exists to release
it. Every later `SSLContext` construction in that child, from any thread, then
blocks forever.
Steps to reproduce (a timing race, won't reproduce every attempt):
1. Run a task via the Task SDK on Linux, using a hook/operator that opens
outbound HTTPS connections.
2. Ensure the supervisor has several concurrent background threads also
doing HTTPS/TLS work at task-launch time (OTel exporter, one or more
`google-auth` `Credentials` objects with regional-access-boundary refresh
threads, OpenLineage listener).
3. Launch tasks in volume. Intermittently, a task hangs forever with no log
output past its last HTTPS-triggering line.
4. `py-spy dump --pid <hung-pid> --subprocesses` while hung — every active
thread shows the same `ssl.py:440` frame.
**What you think should happen instead?**
The task should complete or eventually be killed and retried — not hang
indefinitely due to C-library lock state inherited from an unrelated background
thread.
Same hazard class already fixed for macOS (#64874, #65691 — Objective-C
runtime corruption after bare `fork()`) and hit again independently on Linux
(#65942/#65943 — Edge Worker forking a 22-thread process, corrupting Python's
own import-lock state). Here the poisoned lock is OpenSSL's, and the affected
fork is the Task SDK's own `ActivitySubprocess.start()`, not an executor-level
fork — so the existing `_should_use_exec()` gate doesn't cover it, since the
original macOS fix's reasoning ("Linux's resolver has no ObjC dependency") is
specific to that hazard and doesn't generalize to OpenSSL's own fork-unsafe
locking.
Following the Edge Worker precedent (#65943), a plausible fix shape is an
opt-in "fresh interpreter instead of bare fork" setting for
`ActivitySubprocess.start()` on Linux, defaulting to current behavior to avoid
a blanket performance regression — open question for maintainers whether that
reuses `execute_tasks_new_python_interpreter` or needs a dedicated setting.
**Operating System:** Linux (the hazard is generic CPython/OpenSSL
fork-safety, not distro-specific)
**Deployment:** Other Docker-based deployment
**Apache Airflow Provider(s):** google, cncf-kubernetes (trigger examples
observed; not provider-specific — any HTTPS-heavy hook can be the victim)
**Anything else?**
- Related but distinct: #53337 (moving `execution_timeout` enforcement to
the supervisor) would let a hang like this get killed and retried automatically
— valuable, but doesn't fix the hang itself.
- Prior art for this hazard class: #64874, #65691 (macOS/ObjC),
#65942/#65943 (Linux/Edge Worker, Python import locks). CPython itself emits
`DeprecationWarning: This process is multi-threaded, use of fork() may lead to
deadlocks in the child` in this exact scenario.
- Can share redacted `py-spy` dumps if useful for triage.
**Are you willing to submit PR?**
- [ ] Yes I am willing to submit a PR!
**Code of Conduct**
- [X] I agree to follow this project's Code of Conduct
---
Drafted-by: Claude Code (Sonnet 5); reviewed by @amoghrajesh before posting
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
