Big +1 to this proposal. Thanks for initiating it, Jarek. This will accelerate innovation for K8s and celery executor and encourage other providers, like Google and Amazon, to include cloud native executors in their packages.
Couple of questions/concerns: 1. How does this impact work of release managers? I see them already working pretty hard to release providers and Airflow core. I wonder if this makes their task even harder, especially as these require more rigorous testing as compared to providers. 2. As part of AIP-44, should we create an internal API for executors? I recall a discussion in Slack (related to ECS) where we aspire to remove all database access from providers. Best, Shubham On 2023-01-29, 1:21 AM, "Jarek Potiuk" <[email protected]> wrote: CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Hello Everyone, As a follow-up to AIP-51 - when it is completed (with few more quirks like the one described by Andrey in the "Rendered Task Instance Fields" discussion) - it should now be entirely possible to move Kubernetes Executor, Celery Executor and related "joined" executors to respective providers. There is of course question where to move CeleryKubernetesExecutor, but we have prior art with Transfer Operators and we might choose to add it to "cncf.kubernetes" and make the celery provider an optional dependency of K8S provider. This has multiple advantages, and one of the biggest one I can think of is that we could evolve K8S executor faster than Airflow itself and we would avoid quite a lot of complexities involved in parallel modifications of K8S code in provider and executor (it happened in the past that we had do to add very high min-airflow version for "cncf.kubernetes" provider in order to make this happens, and we could likely avoid similar problems by making sure K8S executor is used as "yet another executor" - compliant with the AIP-51 interface and we could evolve it much faster. That would also give celery provider some "real" meaning - so far celery provider was merely exposing celery queue sensor, but when we move CeleryExecutor to the provider, the provider would finally be a useful one. Doing this requires a few small things: * we likely need to add dynamic import in the old "executors" package following PEP 562 (and the way we've done operators) and deprecation notice in case someone uses them from there. * we likely need to add "cncf.kubernetes" and "celery" packages as pre-installed providers - alongside http, fttp, common.sql, sqlite, imap. I **think**, after AIP-51 gets implemented, this would be a fully backwards-compatible change - it's just a matter of proper management of the dependencies. We could add min-cncf.kubernetes/celery versions in Airflow 2.6, so that we are sure 2.6+ airflow uses only newer providers and kubernetes/celery code and this would be fully transparent for the users, I believe. J.
