potiuk commented on issue #47608:
URL: https://github.com/apache/airflow/issues/47608#issuecomment-2727541575

   > Yes, I believe there was an email thread regarding this started by 
[@potiuk](https://github.com/potiuk)
   
   Now.. I recall one thing. - the only problem with getting rid of sequential 
executor, is that it will potentially make it impossible in some cases to run 
"production like" airflow configuration on Mac OS (i.e. harming development 
environment for some people). 
   
   The problem is that some people suffered with Local Executor doing forks, 
where sequential executor did not do forks. Which is potentially disruptive 
especially for people who are running on MacOS
   
   Example discussion here https://github.com/apache/airflow/discussions/44243 
explaining the problem
   
   Discussion that I started was here : 
https://lists.apache.org/thread/cc7wfz9w8d1npopj72nlxnh52dpzbls9
   
   I am not sure if we can do much for MacOS until we get rid of the potential 
initializations that are executed through `airflow.__init__`  which I 
**believe** is the root of the problem. The main problem is that if any library 
starts any threads before first fork happens, this often leads to SEGV problems 
- and this is inherent problem of running Python on MacOS - that is not really 
solvable on a system level. See for example discussion here: 
https://discuss.python.org/t/concerns-regarding-deprecation-of-fork-with-alive-threads/33555
   
   What I think triggers this problem is combination of factors:
   
   * macOS 
   * configuration in local settings that imports some libraries that start 
threads (mostly for logging as logging is initlalized on airflow `__init__.py` 
effectively
   * the fact that we need to import `airflow.__init__.py` to do anything - 
including importing task-sdk package on execution and dag file processing side
   * and then we fork in case of sequential executor just before we start task 
- if some threads are started by any of the 3rd-party libraries that randomly 
leads to SIGSEGV
   
   The problem with it is that it's next to impossible to tell user what was 
the root cause - some import somewhere of some library ...
   
   I think we can deal with it in two ways:
   
   * rearchitect initial import to not do all the initialization on just 
importing airflow
   * maybe (not sure if possible) we could check if there are any started 
threads before forking and "FAIL" - asking user to change configuration of 
logging or look at their libraries....
   
   
   
   
   
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to