Yay! Moar deletions! Just one comment to that - we **know** that forking + threading in MacOS causes SEGV (Segmentation Violation) issues when you are using some additional libraries that can get initialized by **just** importing them and they start threads, when it happens before we attempt to make the first fork with Local Executor.
This is a known issue in MacOS + Python, it's racy (i.e. it might or might not happen at any given execution) and there is not much we can do - with exception of **not using** those libraries or using some workarounds. An example of that - that might be happening when logging configuration uses some libraries and initialises loggers (like AWS with boto). In this case setting `NO_PROXY=true" env variable helps for example. But this is case-by-case and it's hard to both - know what the root cause is for SIGSEGV and finding remediations is more of a "don't use this or that". Another option for MacOS users is to use containerized / dockerized airflow rather than in their local environment. We should be aware of that - because that's what was the feature of the Sequential Executor - it did not do forking when executing tasks. We might add better protection or detection over time, but for now we should be wary of running airflow locally with more "production-like" settings and we should likely gather more information about those cases to find a better remedy. That's calculated risk, really - we know it's an issue, and we know some users will have problems with running airflow locally with Local Executor, but we think it's better to remove those old executors anyway. Let's monitor and try to improve that part over time. J. On Tue, Apr 1, 2025 at 11:52 AM Amogh Desai <amoghde...@apache.org> wrote: > Hi All, > > I’m excited to announce that the *Sequential Executor* and *Debug Executor* > have been removed from the in-tree executors in our repository, thanks to > the efforts of the community! (Discussed in > https://lists.apache.org/thread/65jw3o3mn7qlc56f9bt2146fmoy0z5t6). > > These executors were primarily used for local testing and debugging but we > have the LocalExecutor( > > https://airflow.apache.org/docs/apache-airflow/stable/core-concepts/executor/local.html > ) > which can achieve better results since it supports parallelism too. The > LocalExecutor has also been ported over to use the new Airflow 3 way of > running tasks using task SDK and task execution interface + API server. > > One of the reasons the Sequential Executor was kept around for so long was > due to some "single" process restrictions while using Sqlite. With the > newer versions of Sqlite supporting WAL mode, LocalExecutor now works well > with Sqlite, making the SequentialExecutor redundant. [3] helped in > accomplishing that. > > This was a long shot for Airflow 3 but we have managed to do this even > before the release candidate, thanks to everyone who helped in reviewing > the code and helping me get this one through. > > *What does this mean for you?* > If you were using these executors, you should switch to alternatives. > *LocalExecutor* is the easiest and the most natural replacement to any of > those, but *CeleryExecutor* is a good option too if you do not mind an > extra worker being spun up. > > I do not anticipate any major issues from this change, but do shout out on > Slack, on #airflow-3-dev channel for help if you encounter something. > > Thanks & Best Regards, > Amogh > > [1] - https://github.com/apache/airflow/issues/48522 > [2] - https://github.com/apache/airflow/issues/48523 > [3] - https://github.com/apache/airflow/pull/44839 >
