potiuk commented on PR #60087: URL: https://github.com/apache/airflow/pull/60087#issuecomment-3708410863
Just to be constructive: I think the right solution (@ashb @amoghrajesh @kaxil ?) is - after we complete isolation work - is to review all the intialisation code that happens in all our processes and make sure that we: a) do not start any threads ourselves before forking in any of our components that use forking b) do not run any custom code before forking (because we are not able to prevent it from running threads) That will likely require to make something that I've been talking for a long while (and as far as I remember @ashb ackknowledged that we have a goal to do that during/after task isolation) - that instead of doing a lot of things during the `import airflow` (including local settings and conf initialization) - we switch to explicit initialization of all components when needed when particular commands of airflow start. This is also nicely accompanied by our `shared` components separation - we already have a few components that require implicit initialization as discused with @amoghrajesh. We will have to explicitly initialize some of our shared libraries (differently in task.sdk and differently in airflow.core. We will also likely have to do some tests (prek hooks?) to prevent accidental adding of such imports. I've done similar (but simpler check in Breeze - where we only make sure to run specific imports for auto-completion - because auto-completeion runs outside of venv of Breeze and excessive imports while doing it are a) slow b) imported extra libraries might not be on PYTHONPATH - and it saved me numerous times from adding a top-level import that would have caused auto-complete failures. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
