I would love to see this. I think it would legitimize the interface a bit more and also help to encourage folks to not abuse/leak it in the future.
AIP-51 is close to completion, I'd say 80%. We've boiled off most of the easier items and what's left is a few tricky decouplings (I have a PR for one of the trickier ones here [1] if anyone has some free time :)). There are also lots of small items that aren't covered by the AIP, such as the one Andrey called out, which are not necessarily leaked abstractions from the base interface but are in a roundabout way still coupled to a particular executor or execution environment. There will be a long tail of these and we should start tracking them (perhaps a GH label?). Cheers, Niko [1]: https://github.com/apache/airflow/pull/29055 ________________________________ From: Jarek Potiuk <ja...@potiuk.com> Sent: Sunday, January 29, 2023 1:20:38 AM To: dev@airflow.apache.org Subject: [EXTERNAL] [DISCUSSION] Move K8S and Celery Executors (and related) to respective providers? CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. Hello Everyone, As a follow-up to AIP-51 - when it is completed (with few more quirks like the one described by Andrey in the "Rendered Task Instance Fields" discussion) - it should now be entirely possible to move Kubernetes Executor, Celery Executor and related "joined" executors to respective providers. There is of course question where to move CeleryKubernetesExecutor, but we have prior art with Transfer Operators and we might choose to add it to "cncf.kubernetes" and make the celery provider an optional dependency of K8S provider. This has multiple advantages, and one of the biggest one I can think of is that we could evolve K8S executor faster than Airflow itself and we would avoid quite a lot of complexities involved in parallel modifications of K8S code in provider and executor (it happened in the past that we had do to add very high min-airflow version for "cncf.kubernetes" provider in order to make this happens, and we could likely avoid similar problems by making sure K8S executor is used as "yet another executor" - compliant with the AIP-51 interface and we could evolve it much faster. That would also give celery provider some "real" meaning - so far celery provider was merely exposing celery queue sensor, but when we move CeleryExecutor to the provider, the provider would finally be a useful one. Doing this requires a few small things: * we likely need to add dynamic import in the old "executors" package following PEP 562 (and the way we've done operators) and deprecation notice in case someone uses them from there. * we likely need to add "cncf.kubernetes" and "celery" packages as pre-installed providers - alongside http, fttp, common.sql, sqlite, imap. I **think**, after AIP-51 gets implemented, this would be a fully backwards-compatible change - it's just a matter of proper management of the dependencies. We could add min-cncf.kubernetes/celery versions in Airflow 2.6, so that we are sure 2.6+ airflow uses only newer providers and kubernetes/celery code and this would be fully transparent for the users, I believe. J.
