Should we also create apache-airflow-providers-dask for DaskExecutor? ---- Best Wishes *Andrey Anshin*
On Sun, 29 Jan 2023 at 13:21, Jarek Potiuk <[email protected]> wrote: > Hello Everyone, > > As a follow-up to AIP-51 - when it is completed (with few more quirks > like the one described by Andrey in the "Rendered Task Instance > Fields" discussion) - it should now be entirely possible to move > Kubernetes Executor, Celery Executor and related "joined" executors to > respective providers. > > There is of course question where to move CeleryKubernetesExecutor, > but we have prior art with Transfer Operators and we might choose to > add it to "cncf.kubernetes" and make the celery provider an optional > dependency of K8S provider. > > This has multiple advantages, and one of the biggest one I can think > of is that we could evolve K8S executor faster than Airflow itself and > we would avoid quite a lot of complexities involved in parallel > modifications of K8S code in provider and executor (it happened in the > past that we had do to add very high min-airflow version for > "cncf.kubernetes" provider in order to make this happens, and we could > likely avoid similar problems by making sure K8S executor is used as > "yet another executor" - compliant with the AIP-51 interface and we > could evolve it much faster. > > That would also give celery provider some "real" meaning - so far > celery provider was merely exposing celery queue sensor, but when we > move CeleryExecutor to the provider, the provider would finally be a > useful one. > > Doing this requires a few small things: > * we likely need to add dynamic import in the old "executors" package > following PEP 562 (and the way we've done operators) and deprecation > notice in case someone uses them from there. > * we likely need to add "cncf.kubernetes" and "celery" packages as > pre-installed providers - alongside http, fttp, common.sql, sqlite, > imap. > > I **think**, after AIP-51 gets implemented, this would be a fully > backwards-compatible change - it's just a matter of proper management > of the dependencies. We could add min-cncf.kubernetes/celery versions > in Airflow 2.6, so that we are sure 2.6+ airflow uses only newer > providers and kubernetes/celery code and this would be fully > transparent for the users, I believe. > > J. >
