Hello Everyone, I would like to start discussion about potential suspension and eventually removal of Daskexecutor provider.
After some recent changes and moving DaskExecutor from core to provider we have now an open option to suspend and eventually remove Dask Executor from our codebase. After we have the process tested and tried with Qubole, for me this is the next candidate to remove and get rid of some burden it involves. I believe this executor is hardly used by anyone - IMHO it does not really add a value to our Executor's offering and it causes a number of CI/ maintenance issues for us, while we have little to no experience and incentive and knowledge to manage it. It slows us down, makes our CI flaky, sometimes delays our release process etc etc. Some of the past discussions about it (this is the third time I am (re) starting this discussion - the first time it was in 2020 when we discussed removing it for Airflow 2.0. * https://lists.apache.org/thread/6stgcpjt5jb3xfw92oo1j486j33c8v7m * https://lists.apache.org/thread/875fpgb7vfpmtxrmt19jmo8d3p6mgqnh Some of the recent problems with Dask executor impacting stability of our CI / tests/ impacting release process are manifested in those issues and PRs over the last few years (and this is just a subset of issues and PRs dealing with the stability of it. * https://github.com/apache/airflow/pull/35659 * https://github.com/apache/airflow/issues/32778 * https://github.com/apache/airflow/pull/35473 * https://github.com/apache/airflow/pull/32991 * https://github.com/apache/airflow/pull/30258 * https://github.com/apache/airflow/pull/22046 * https://github.com/apache/airflow/pull/22017 Why now? What changed? One of the fantastic properties of the "providers" approach and moving the executor from core to provider - we can now ACTUALLY remove DaskExecutor without breaking any of our SemVer promises. This is something I was working on for quite a while and with some of the recent changes - making our Executor API stable and part of our public interface, introducing full life-cycle process for providers this is now entirely possible to remove all the maintenance burden while keeping our SemVer promises - so that we can do it without bumping Airflow to version 3. How does it work ? Simply - provider will remain released, and it will remain working in the version it has been released for the last time with all the future versions of Airflow, But at the same time we can remove the code from main of Airflow, stop running tests, stop caring about the provider impacting the constraints and generally speaking. If we decide to do somethin about, we have three options: 1) Suspend the daskexecutor provider 2) Remove it 3) We can also follow Suspend -> wait few months -> remove) pattern All of them are easy, all of them bring us much welcome relief in our release and CI and maintenance burden. All of them are well documented and tried and all that we need is just to agree which scenario we want to follow. In case 1) we will stop releasing new versions, but in case any new "development" is done on the dask side that will break currently released providers, it will be on someone who will take on the task to bring it back from suspension - fix all dependency issues, make sure tests are green and then it's just a matter of PR to bring it back. This is really a way to say "Hey Dask Executor community, if you want to keep any future development, it's on you to bring it back from suspension and take the task to solve all the difficulties we currently have to leave with in Airflow community". In case 2) we deliberately decide we do not want to maintain it any more and if someone would like to bring it back, they will have to convince the community and get the VOTE or LAZY CONSENSUS to bring it back (and then it could be even different provider package, not compatible etc.). WDYT? Are there any DaskExecutor users around or someone you know of who would rely on it heavily? Are there any reasons why we should keep it as is ? Which of the above scenarios would you like to see? I would love to hear your opinions on that. If we see that we would like to follow 1) or 2) I am happy to pass the message to the Dask community and let them know that they will need to be involved in case they will want to bring back Dask from suspension or removal - I did that before for some of the past issues and I am happy to be the messenger (hopefull no-one shoots me). J.
