diogosilva30 commented on PR #65943:
URL: https://github.com/apache/airflow/pull/65943#issuecomment-4370641743
@jscheffl it just happened again in our prod environment. After Sunday some
tasks started randomly failing on edge worker.
The logs (anonymized):
```
[2026-05-04 11:08:08] INFO - Stats instance was created in PID 1 but
accessed in PID 2408809. Re-initializing. source=airflow.stats loc=stats.py:57
[2026-05-04 11:08:08] ERROR - Failed to import plugin
/opt/airflow/dags/repo/plugins/common/operators/monitoring/internal_active_sessions.py
source=airflow.plugins_manager loc=plugins_manager.py:298
ModuleNotFoundError: No module named 'common'
...
[2026-05-04 11:08:08] ERROR - Startup reschedule limit exceeded
reschedule_count=3 max_reschedules=3 source=task loc=task_runner.py:631
[2026-05-04 11:08:09] WARNING - Process exited abnormally exit_code=1
source=task
```
Both `common` and `monitoring` are modules in
`AIRFLOW__CORE__PLUGINS_FOLDER` (`/opt/airflow/dags/repo/plugins`). They're
present on disk — git-sync keeps them up to date. The `ModuleNotFoundError`
isn't a missing file problem, it's the forked child process seeing a corrupted
import state inherited from the parent.
<details>
<summary>Full anonymized logs</summary>
```
[2026-05-04 11:08:08] INFO - Stats instance was created in PID 1 but
accessed in PID 2408809. Re-initializing. source=airflow.stats loc=stats.py:57
[2026-05-04 11:08:08] ERROR - Failed to import plugin
/opt/airflow/dags/repo/plugins/common/operators/monitoring/internal_active_sessions.py
source=airflow.plugins_manager loc=plugins_manager.py:298
ModuleNotFoundError: No module named 'common'
File
"/home/airflow/.local/lib/python3.12/site-packages/airflow/plugins_manager.py",
line 291 in load_plugins_from_plugin_directory
File "<frozen importlib._bootstrap_external>", line 999 in exec_module
File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_removed
File
"/opt/airflow/dags/repo/plugins/common/operators/monitoring/internal_active_sessions.py",
line 13 in <module>
[2026-05-04 11:08:08] ERROR - Failed to import plugin
/opt/airflow/dags/repo/plugins/common/operators/monitoring/internal_agent_states.py
source=airflow.plugins_manager loc=plugins_manager.py:298
ModuleNotFoundError: No module named 'common'
File
"/opt/airflow/dags/repo/plugins/common/operators/monitoring/internal_agent_states.py",
line 11 in <module>
[2026-05-04 11:08:08] ERROR - Failed to import plugin
/opt/airflow/dags/repo/plugins/common/operators/monitoring/internal_call_data.py
source=airflow.plugins_manager loc=plugins_manager.py:298
ModuleNotFoundError: No module named 'common'
File
"/opt/airflow/dags/repo/plugins/common/operators/monitoring/internal_call_data.py",
line 9 in <module>
[2026-05-04 11:08:08] ERROR - Failed to import plugin
/opt/airflow/dags/repo/plugins/common/operators/monitoring/app_logged_in_users.py
source=airflow.plugins_manager loc=plugins_manager.py:298
ModuleNotFoundError: No module named 'monitoring'
File
"/opt/airflow/dags/repo/plugins/common/operators/monitoring/app_logged_in_users.py",
line 13 in <module>
[2026-05-04 11:08:08] ERROR - Failed to import plugin
/opt/airflow/dags/repo/plugins/common/operators/monitoring/app_user_stats.py
source=airflow.plugins_manager loc=plugins_manager.py:298
ModuleNotFoundError: No module named 'common'
File
"/opt/airflow/dags/repo/plugins/common/operators/monitoring/app_user_stats.py",
line 14 in <module>
[2026-05-04 11:08:08] ERROR - Failed to import plugin
/opt/airflow/dags/repo/plugins/common/operators/monitoring/tenant_info.py
source=airflow.plugins_manager loc=plugins_manager.py:298
ModuleNotFoundError: No module named 'monitoring'
File
"/opt/airflow/dags/repo/plugins/common/operators/monitoring/tenant_info.py",
line 8 in <module>
[2026-05-04 11:08:08] ERROR - Failed to import plugin
/opt/airflow/dags/repo/plugins/common/operators/monitoring/internal_api_stats.py
source=airflow.plugins_manager loc=plugins_manager.py:298
ModuleNotFoundError: No module named 'common'
File
"/opt/airflow/dags/repo/plugins/common/operators/monitoring/internal_api_stats.py",
line 8 in <module>
[2026-05-04 11:08:08] ERROR - Failed to import plugin
/opt/airflow/dags/repo/plugins/common/operators/monitoring/vendor_data.py
source=airflow.plugins_manager loc=plugins_manager.py:298
ModuleNotFoundError: No module named 'common'
File
"/opt/airflow/dags/repo/plugins/common/operators/monitoring/vendor_data.py",
line 8 in <module>
[2026-05-04 11:08:08] ERROR - Failed to import plugin
/opt/airflow/dags/repo/plugins/monitoring/cache.py
source=airflow.plugins_manager loc=plugins_manager.py:298
ModuleNotFoundError: No module named 'monitoring'
File "/opt/airflow/dags/repo/plugins/monitoring/cache.py", line 18 in
<module>
[2026-05-04 11:08:08] ERROR - Failed to import:
/opt/airflow/dags/repo/dags/stg/monitoring/region/test_dag.py
source=airflow.models.dagbag.BundleDagBag loc=dagbag.py:415
ModuleNotFoundError: No module named 'common'
[2026-05-04 11:08:08] ERROR - Dag not found during start up
dag_id=stg__monitoring__region__test_staging_dag
bundle=BundleInfo(name='dags-folder', version=None)
[2026-05-04 11:08:08] ERROR - Startup reschedule limit exceeded
reschedule_count=3 max_reschedules=3
[2026-05-04 11:08:09] WARNING - Process exited abnormally exit_code=1
```
</details>
<details>
<summary>Pod spec (anonymized)</summary>
```yaml
spec:
containers:
- name: edge-worker
image: internal-registry.example.net/monitoring/airflow:3.1.8
args: [edge, worker, '-q', stg__general__region]
env:
- name: AIRFLOW__CORE__PLUGINS_FOLDER
value: /opt/airflow/dags/repo/plugins
- name: AIRFLOW__EDGE__API_URL
value:
https://airflow.stg.monitoring.example.com/edge_worker/v1/rpcapi
volumeMounts:
- mountPath: /opt/airflow/dags
name: dags
- name: git-sync
image: internal-registry.example.net/monitoring/git-sync:4.3.0
args: [--repo=..., --root=/dags, --link=repo, --period=60s]
volumeMounts:
- mountPath: /dags
name: dags
volumes:
- name: dags
emptyDir: {}
```
</details>
---
This connects directly to your question about the thread count. The edge
worker runs `asyncio.run(edge_worker.start())` and by the time `_launch_job()`
fires it's already running 22+ OS threads:
1. The asyncio event loop main thread
2. asyncio's default `ThreadPoolExecutor` — created lazily the first time
`anyio`/`aiofiles` calls `loop.run_in_executor(None, ...)` in
`_push_logs_in_chunks`. Python's default pool size is `min(32, os.cpu_count() +
4)` — on our 16-core nodes that's 20 threads, all kept alive as idle workers.
You can verify from inside a running pod:
```python
import os
from collections import Counter
task_dir = '/proc/1/task'
tids = os.listdir(task_dir)
wchans = [open(f'{task_dir}/{tid}/wchan').read().strip() for tid in tids]
print(f'Total threads: {len(tids)}')
print(Counter(wchans).most_common())
# → [('futex_wait_queue_me', 20), ('ep_poll', 1), ...]
```
When `Process(...).start()` calls `os.fork()` from a 22-thread process, the
child inherits all thread states but only the forking thread survives. If any
thread was mid-import, the child sees the import lock permanently held →
`ModuleNotFoundError` for modules that are physically on disk.
Are you seeing the `DeprecationWarning: This process is multi-threaded, use
of fork() may lead to deadlocks` warning in your own deployments? On ours it
fires on every task launch.
---
**Proposal: hook onto `core.execute_tasks_new_python_interpreter`**
Rather than a hard switch, Airflow already has
[`core.execute_tasks_new_python_interpreter`](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/plugins.html#when-are-plugins-re-loaded)
for exactly this tradeoff. What if the edge worker just honours that setting?
Default stays `False` (fork) to keep existing behaviour — no change for users
who don't opt in. Users who want the safer path flip it to `True` and get
`subprocess.Popen`, same as the other executors.
Happy to update the PR to implement it that way if you're on board.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
