Hello Everyone, As a follow-up to AIP-51 - when it is completed (with few more quirks like the one described by Andrey in the "Rendered Task Instance Fields" discussion) - it should now be entirely possible to move Kubernetes Executor, Celery Executor and related "joined" executors to respective providers.
There is of course question where to move CeleryKubernetesExecutor, but we have prior art with Transfer Operators and we might choose to add it to "cncf.kubernetes" and make the celery provider an optional dependency of K8S provider. This has multiple advantages, and one of the biggest one I can think of is that we could evolve K8S executor faster than Airflow itself and we would avoid quite a lot of complexities involved in parallel modifications of K8S code in provider and executor (it happened in the past that we had do to add very high min-airflow version for "cncf.kubernetes" provider in order to make this happens, and we could likely avoid similar problems by making sure K8S executor is used as "yet another executor" - compliant with the AIP-51 interface and we could evolve it much faster. That would also give celery provider some "real" meaning - so far celery provider was merely exposing celery queue sensor, but when we move CeleryExecutor to the provider, the provider would finally be a useful one. Doing this requires a few small things: * we likely need to add dynamic import in the old "executors" package following PEP 562 (and the way we've done operators) and deprecation notice in case someone uses them from there. * we likely need to add "cncf.kubernetes" and "celery" packages as pre-installed providers - alongside http, fttp, common.sql, sqlite, imap. I **think**, after AIP-51 gets implemented, this would be a fully backwards-compatible change - it's just a matter of proper management of the dependencies. We could add min-cncf.kubernetes/celery versions in Airflow 2.6, so that we are sure 2.6+ airflow uses only newer providers and kubernetes/celery code and this would be fully transparent for the users, I believe. J.
