diogosilva30 commented on issue #65942:
URL: https://github.com/apache/airflow/issues/65942#issuecomment-4326220714
### Production logs showing the non-deadlock failure variant
In addition to full deadlocks, the forked child sometimes **doesn't** hang
but has **corrupted import state**, causing all plugin and DAG imports to fail.
**Repo structure** (standard Airflow plugins layout with inter-package
imports):
```
plugins/
├── __init__.py
├── common/
│ ├── __init__.py
│ └── operators/
│ ├── __init__.py
│ └── mypackage/
│ ├── __init__.py
│ ├── operator_a.py # imports: from
common.operators.mypackage.utils import ...
│ ├── operator_b.py # imports: from mypackage.config import ...
│ └── utils.py
└── mypackage/
├── __init__.py
├── config.py
└── cache.py # imports: from mypackage.config import ...
```
**Sample DAG** (nothing unusual — standard operator usage):
```python
from airflow.sdk import dag, task
from common.operators.mypackage.operator_a import MyOperator
@dag(schedule="*/3 * * * *")
def my_dag():
extract = MyOperator(task_id="extract_data", ...)
my_dag()
```
**Logs from a forked task child process** (PID 1 → child PID 1435804):
```
INFO - Stats instance was created in PID 1 but accessed in PID 1435804.
Re-initializing.
source=airflow.stats
ERROR - Failed to import plugin
plugins/common/operators/mypackage/operator_a.py
source=airflow.plugins_manager loc=plugins_manager.py:298
ModuleNotFoundError: No module named 'common'
File "airflow/plugins_manager.py", line 291 in
load_plugins_from_plugin_directory
File "<frozen importlib._bootstrap_external>", line 999 in exec_module
File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_cleaned
ERROR - Failed to import plugin plugins/mypackage/cache.py
source=airflow.plugins_manager loc=plugins_manager.py:298
ModuleNotFoundError: No module named 'mypackage'
File "airflow/plugins_manager.py", line 291 in
load_plugins_from_plugin_directory
File "<frozen importlib._bootstrap_external>", line 999 in exec_module
File "<frozen importlib._bootstrap>", line 488 in _call_with_frames_cleaned
```
(Same `ModuleNotFoundError` repeated for every plugin file that uses
inter-package imports.)
This cascades into DAG import failure and task death:
```
ERROR - Failed to import: dags/my_dag.py
source=airflow.models.dagbag.BundleDagBag loc=dagbag.py:415
ModuleNotFoundError: No module named 'common'
ERROR - Dag not found during start up
source=task loc=task_runner.py:664
ERROR - Startup reschedule limit exceeded reschedule_count=3
max_reschedules=3
source=task loc=task_runner.py:631
WARNING - Process exited abnormally exit_code=1
```
**Key observations:**
- "Stats instance was created in PID 1 but accessed in PID 1435804" confirms
the child was forked from the edge worker main process
- Plugin imports fail because import lock / `sys.modules` state inherited
from the 22-thread parent is corrupted
- The task fails with exit code 1 after exhausting reschedule attempts (3/3)
- This is the **non-deadlock variant** of the same root cause — fork
sometimes doesn't hang but still produces broken Python import machinery
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
