diogosilva30 opened a new issue, #65942:
URL: https://github.com/apache/airflow/issues/65942
### Under which category would you file this issue?
Providers
### Apache Airflow version
3.0+ (tested with providers built from `main` branch)
### What happened and how to reproduce it?
**Issue Description**
Edge worker tasks intermittently deadlock at startup and never execute. The
process hangs permanently with no error output until the scheduler zombie task
threshold (default 180s) marks the task as failed.
**Root cause**
The edge worker main process runs **22 threads** at runtime (asyncio event
loop, `ThreadPoolExecutor`, `httpx`/`httpcore` connection pools, API server
HTTP clients).
`EdgeWorker._launch_job()` in
`providers/edge3/src/airflow/providers/edge3/cli/worker.py` uses
`multiprocessing.Process` with the default `fork` start method:
```python
def _launch_job(self, workload: ExecuteTask) -> tuple[Process,
Queue[Exception]]:
results_queue: Queue[Exception] = Queue()
process = Process(
target=self._run_job_via_supervisor,
kwargs={"workload": workload, "results_queue": results_queue},
)
process.start() # ← os.fork() here, copies 22 threads' lock state
return process, results_queue
```
`os.fork()` copies the entire address space — including **locked import
locks** held by the other 21 threads. Since only the forking thread survives in
the child, those locks are never released. Any subsequent `import` in the child
that needs one of them deadlocks permanently.
Python 3.12 emits a deprecation warning on every task launch:
```
DeprecationWarning: This process (pid=XXXXXX) is multi-threaded, use of
fork() may lead to deadlocks in the child.
```
**Steps to reproduce**
1. Deploy edge workers with concurrency >= 2 on Linux (Python 3.12)
2. Trigger multiple tasks to the edge worker queue
3. Observe intermittent task hangs — tasks never start executing, eventually
marked as zombie/failed
**Reproduction script** (run on an edge worker pod):
```python
import os, sys, threading, time, signal
# Simulate edge worker's thread profile
stop = threading.Event()
def import_loop():
while not stop.is_set():
import importlib
importlib.import_module('json')
importlib.import_module('email.mime.text')
importlib.import_module('http.client')
time.sleep(0.001)
threads = [threading.Thread(target=import_loop, daemon=True) for _ in
range(20)]
for t in threads:
t.start()
time.sleep(0.5)
success = 0
for i in range(10):
r, w = os.pipe()
pid = os.fork()
if pid == 0:
os.close(r)
signal.alarm(3) # 3s timeout to detect deadlock
try:
from airflow.sdk.execution_time.supervisor import supervise
os.write(w, b'1')
except:
os.write(w, b'0')
os._exit(0)
else:
os.close(w)
result = os.read(r, 1)
os.close(r)
os.waitpid(pid, 0)
if result == b'1':
success += 1
stop.set()
print(f'fork: {success}/10 succeeded')
# Typical result: 0-1/10 (9-10 deadlocked, killed by SIGALRM)
```
**Observed on live edge worker pod:**
```
$ kubectl exec deploy/airflow-worker -- python3 -c "import threading;
print(threading.active_count())"
22
$ kubectl exec deploy/airflow-worker -- python3 -c "import multiprocessing;
print(multiprocessing.get_start_method())"
fork
```
**Notes on why existing mitigations don't apply:**
- `AIRFLOW__CORE__EXECUTE_TASKS_NEW_PYTHON_INTERPRETER` is only read by the
**Celery executor** path (`celery_executor_utils.py`). The edge3
`_run_job_via_supervisor` → `supervise()` → `ActivitySubprocess.start()` path
never consults it.
- `ActivitySubprocess.start()` has its own fork+exec for macOS
(`_FORK_EXEC_PLATFORMS`) but that's the **inner** fork (supervisor → task), not
the **outer** fork (`_launch_job` → supervisor) where the 22-thread problem
occurs.
- `forkserver`/`spawn` start methods can't be a drop-in fix because
`_launch_job` passes `self._run_job_via_supervisor` (a bound method on
`EdgeWorker` which carries unpicklable state: HTTP clients, running-job dicts,
multiprocessing.Queue).
### What you think should happen instead?
`_launch_job` should not `fork()` from a multi-threaded process without
clearing inherited lock state. Possible approaches:
1. Register `os.register_at_fork(after_in_child=...)` to clear
`importlib._bootstrap._module_locks` and `_blocking_on` in the child (quick
workaround)
2. Refactor `_launch_job` to pass a picklable task descriptor and use
`mp.get_context('forkserver').Process(...)` (proper fix)
3. Use `subprocess.Popen` to launch a fresh Python interpreter that calls
`_run_job_via_supervisor` (similar to `execute_tasks_new_python_interpreter`
approach in Celery)
### Operating System
Linux (Debian-based container, Python 3.12.13)
### Versions of Apache Airflow Providers
```
apache-airflow-providers-edge3 (latest, 2026-04-27)
```
### Official Helm Chart version
Not Applicable
### Kubernetes Version
v1.30 (RKE2)
### Helm Chart configuration
Not Applicable — issue is in the edge3 provider Python code, not Helm chart.
### Docker Image customizations
Custom image based on official Airflow image with additional pip packages.
The issue is independent of image customizations — it's inherent to the edge3
provider's use of `multiprocessing.Process` with `fork` start method in a
multi-threaded process.
### Anything else?
- Reproduction rate: ~95% under thread contention
- Python 3.14 will change the default `multiprocessing` start method away
from `fork`, which will **also break** edge workers since `EdgeWorker` is not
picklable for `spawn`/`forkserver`
- Related prior art in the codebase: `ActivitySubprocess.start()` already
handles fork+exec for macOS via `_FORK_EXEC_PLATFORMS`; Celery executor has
`execute_tasks_new_python_interpreter`
### Are you willing to submit PR?
- [x] Yes I am willing to submit a PR!
### Code of Conduct
- [x] I agree to follow this project's [Code of
Conduct](https://github.com/apache/airflow/blob/main/CODE_OF_CONDUCT.md)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
