diogosilva30 commented on issue #65942:
URL: https://github.com/apache/airflow/issues/65942#issuecomment-4326206243
### Production logs showing the failure mode
In addition to full deadlocks, we also observe cases where the forked child
process **doesn't** deadlock but has **corrupted import state**, causing all
plugin and DAG imports to fail. These are from a task attempt on an edge worker
(PID 1 → child PID 1435804):
```
[2026-04-27 10:39:11] INFO - Stats instance was created in PID 1 but
accessed in PID 1435804. Re-initializing.
source=airflow.stats loc=stats.py:57
[2026-04-27 10:39:12] ERROR - Failed to import plugin
/opt/airflow/dags/repo/plugins/common/operators/observability/vcc_active_sessions.py
source=airflow.plugins_manager loc=plugins_manager.py:298
ModuleNotFoundError: No module named 'common'
File "airflow/plugins_manager.py", line 291 in
load_plugins_from_plugin_directory
File "<frozen importlib._bootstrap_external>", line 999 in exec_module
File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_cleaned
[2026-04-27 10:39:12] ERROR - Failed to import plugin
/opt/airflow/dags/repo/plugins/observability/cache.py
source=airflow.plugins_manager loc=plugins_manager.py:298
ModuleNotFoundError: No module named 'observability'
File "airflow/plugins_manager.py", line 291 in
load_plugins_from_plugin_directory
File "<frozen importlib._bootstrap_external>", line 999 in exec_module
File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_cleaned
```
(Same `ModuleNotFoundError` repeated for all plugin files that use
inter-plugin imports.)
This cascades into a DAG import failure:
```
[2026-04-27 10:39:12] ERROR - Failed to import:
/opt/airflow/dags/repo/dags/stg/.../monit_producer.py
source=airflow.models.dagbag.BundleDagBag loc=dagbag.py:415
ModuleNotFoundError: No module named 'common'
[2026-04-27 10:39:12] ERROR - Dag not found during start up
dag_id=stg__observability__us__monit_producer_scllab
source=task loc=task_runner.py:664
[2026-04-27 10:39:12] ERROR - Startup reschedule limit exceeded
reschedule_count=3 max_reschedules=3
source=task loc=task_runner.py:631
[2026-04-27 10:39:12] WARNING - Process exited abnormally exit_code=1
```
**Key observations:**
- The "Stats instance was created in PID 1 but accessed in PID 1435804" line
confirms the child was forked from the main edge worker process
- Plugin imports fail because `sys.path` / import state inherited from the
multi-threaded parent is corrupted
- The task ultimately fails with exit code 1 after exhausting reschedule
attempts (3/3)
- This is the **non-deadlock variant** of the same root cause — the fork
sometimes doesn't hang but still produces broken Python import state
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
