Ok. I am ok with leaving Dask preinstalled, I think we will already benefit from moving it to provider and moving tests to the provider tests.
On Fri, Jul 14, 2023 at 3:08 PM Hussein Awala <[email protected]> wrote: > We all agree on moving the 5 remote executors to their respective > providers. The main question now is whether we should pre-install all the > providers or not. > > IMHO we should pre-install Celery, Kubernetes, and Dask providers until > Airflow 3.0. Not doing so would be a breaking change for some users, and it > would go against the Airflow deprecation policy. > > Once Airflow 3.0 is released, we should remove the pre-installed providers > and instead prompt the user to install the provider specific to its > executor. > > On Fri 14 Jul 2023 at 14:54, utkarsh sharma <[email protected]> > wrote: > > > I'm too in favor of moving Dask Executor into its own provider, which > will > > make the airflow's codebase more pluggable and orthogonal. Big +1 :) > > > > Thanks, > > Utkarsh Sharma > > > > On Fri, Jul 14, 2023 at 2:03 AM Ferruzzi, Dennis > > <[email protected]> wrote: > > > > > I'm not sure how much of a train wreck it would turn into, but moving > > Dask > > > Executor to a provider seems logical to me. > > > > > > Maybe in the far-flung future of Airflow 3.0 we could move celery and > k8s > > > into their own as well and make it truly pluggable and > executor-agnostic, > > > but I agree that at this time they are too deeply integrated. > > > > > > > > > - ferruzzi > > > > > > > > > ________________________________ > > > From: Oliveira, Niko <[email protected]> > > > Sent: Wednesday, July 12, 2023 10:40 AM > > > To: [email protected] > > > Subject: RE: [EXTERNAL][DISCUSS] Moving Dask Executor to a separate > > > (optional?) dask provider > > > > > > CAUTION: This email originated from outside of the organization. Do not > > > click links or open attachments unless you can confirm the sender and > > know > > > the content is safe. > > > > > > > > > > > > I think in a perfect world we'd only have the completely vendor neutral > > > executors pre-installed (Local, Sequential, Debug) and anything else > > would > > > need to be specifically installed by admins/users. I think if we were > > > starting from scratch this would make the most sense, but clearly > > > Kubernetes and Celery executors are so ubiquitous that it'd cause too > > much > > > wreckage to not install them, but I'd like to push for Dask to _not_ be > > > installed by default. If this causes too much wreckage then perhaps we > > > should deprecate that (though I'm not sure exactly what that would look > > > like in this context), but it's difficult to measure how many folks are > > > using the Dask executor. Perhaps we have data from the yearly > > > questionnaire/survey we send? > > > > > > ________________________________ > > > From: Jarek Potiuk <[email protected]> > > > Sent: Wednesday, July 12, 2023 8:05:54 AM > > > To: [email protected] > > > Subject: [EXTERNAL] [DISCUSS] Moving Dask Executor to a separate > > > (optional?) dask provider > > > > > > CAUTION: This email originated from outside of the organization. Do not > > > click links or open attachments unless you can confirm the sender and > > know > > > the content is safe. > > > > > > > > > > > > Hello Everyone, > > > > > > A small follow up after K8S/Celery executors being moved: > > > https://lists.apache.org/thread/7gyw7ty9vm0pokjxq7y3b1zw6mrlxfm8 > > > > > > We are in the process of moving Celery / Kubernetes executor (Celery > > almost > > > complete and I am working on K8S next + some common discovery and > config > > > moving) > > > > > > But there is one more "questionable" executor - i.e. Dask executor, > still > > > living in Airflow Core. > > > > > > When it comes to Celery/Kubernetes, we decided to make the two > providers > > > preinstalled, because it makes most sense - we are also going to get > the > > > basic documentation in the "core" airflow documentation so that it is > > > easier discoverable and prominently visible - also because of the > > > vendor-neutrality. > > > > > > However when it comes to Dask I am not sure about its status and > whether > > we > > > should make it preinstalled ? > > > > > > I guess there is no doubt to move it to a provider - this has only the > > > benefits same as Celery/K8S move. But whether it should be preinstalled > > > with Airflow - I am not sure. I do not know how frequently Dask > executor > > > (and Dask) is used by people using Airflow, but I personally do not > think > > > it should be as "closely" connected with Airflow as Celery/Kubernetes > > ones. > > > > > > If we do not make it preinstalled, it is somewhat (but not too much, > > > really) breaking change. We still might choose to install dask provider > > in > > > the PROD reference image, so it will continue to work if you use the > > image, > > > and when you are installing airflow in venv you will only have to > specify > > > `pip install apache-airflow[dask]` or manually install > > > `apache-airflow-providers-daskexecutor` (for now at least this is the > > name > > > I could reserve in PyPI). So this is not really breaking, it just > > requires > > > another dependency to be installed. But some pipelines of installing > > > Airflow might get broken because it won't be pre-installed - so this > is a > > > borderline breaking. > > > > > > WDYT? Should we make the dask executor pre-installed or not? > > > > > > J. > > > > > >
