jscheffl commented on PR #65943: URL: https://github.com/apache/airflow/pull/65943#issuecomment-4382449253
> @jscheffl it just happened again in our prod environment. After Sunday some tasks started randomly failing on edge worker. > > The logs (anonymized): > > ``` > [2026-05-04 11:08:08] INFO - Stats instance was created in PID 1 but accessed in PID 2408809. Re-initializing. source=airflow.stats loc=stats.py:57 > [2026-05-04 11:08:08] ERROR - Failed to import plugin /opt/airflow/dags/repo/plugins/common/operators/monitoring/internal_active_sessions.py source=airflow.plugins_manager loc=plugins_manager.py:298 > ModuleNotFoundError: No module named 'common' > ... > [2026-05-04 11:08:08] ERROR - Startup reschedule limit exceeded reschedule_count=3 max_reschedules=3 source=task loc=task_runner.py:631 > [2026-05-04 11:08:09] WARNING - Process exited abnormally exit_code=1 source=task > ``` Interesting, is it intended on your end to load plugins on the worker actually? Did not realize after adding anyio that this runs the IO calls just in a background thread. Thought this lib is just making async IO calls on OS level. > **Proposal: hook onto `core.execute_tasks_new_python_interpreter`** > > Rather than a hard switch, Airflow already has [`core.execute_tasks_new_python_interpreter`](https://airflow.apache.org/docs/apache-airflow/stable/administration-and-deployment/plugins.html#when-are-plugins-re-loaded) for exactly this tradeoff. What if the edge worker just honours that setting? Default stays `False` (fork) to keep existing behaviour — no change for users who don't opt in. Users who want the safer path flip it to `True` and get `subprocess.Popen`, same as the other executors. I still wonder about the case and that it is not happening on our side and heard no reports previously and code is productive since 12+months. Anyway I'd accept a switlcih like with this flag as it is also in CeleryExecutor as an optional switch/flag. But the Exception serialization which was originally removed should stay as this is uploaded back to task logs such that a user can see more details of the root of the failure in task logs. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
