Yeah, alright. In that case I agree. I just wanted to make sure we weren't making it harder to onboard a new user.
________________________________ From: Jarek Potiuk <[email protected]> Sent: Monday, January 30, 2023 10:18 AM To: [email protected] Subject: RE: [EXTERNAL][DISCUSSION] Move K8S and Celery Executors (and related) to respective providers? CAUTION: This email originated from outside of the organization. Do not click links or open attachments unless you can confirm the sender and know the content is safe. > Would this make it harder for a new user to find which executors are > available and get started? On one hand, for better or worse it makes the > executor choice more of a conscious decision. On the other hand unless we're > still setting one as the default and packaging it together - and you did say > they'd be added as requirements so i assume this is your intent - then it is > additional research and configuration to get Airflow running. I think not. We do not have to remove doc references in Airflow about those. It's merely where the packages are, but IMHO we still can (and should) list all the available "community executors" and explain how they work in the Airflow docs - pointing helpfully to implementation sitting in providers. On Mon, Jan 30, 2023 at 6:32 PM Ferruzzi, Dennis <[email protected]> wrote: > > Would this make it harder for a new user to find which executors are > available and get started? On one hand, for better or worse it makes the > executor choice more of a conscious decision. On the other hand unless we're > still setting one as the default and packaging it together - and you did say > they'd be added as requirements so i assume this is your intent - then it is > additional research and configuration to get Airflow running. > > I do like the idea of decoupling executor improvements from core > improvements. I'm al for pretty much anything that helps make the experience > more modular and pluggable, in general. > > > > ________________________________ > From: Jarek Potiuk <[email protected]> > Sent: Sunday, January 29, 2023 1:20 AM > To: [email protected] > Subject: [EXTERNAL] [DISCUSSION] Move K8S and Celery Executors (and related) > to respective providers? > > CAUTION: This email originated from outside of the organization. Do not click > links or open attachments unless you can confirm the sender and know the > content is safe. > > > > Hello Everyone, > > As a follow-up to AIP-51 - when it is completed (with few more quirks > like the one described by Andrey in the "Rendered Task Instance > Fields" discussion) - it should now be entirely possible to move > Kubernetes Executor, Celery Executor and related "joined" executors to > respective providers. > > There is of course question where to move CeleryKubernetesExecutor, > but we have prior art with Transfer Operators and we might choose to > add it to "cncf.kubernetes" and make the celery provider an optional > dependency of K8S provider. > > This has multiple advantages, and one of the biggest one I can think > of is that we could evolve K8S executor faster than Airflow itself and > we would avoid quite a lot of complexities involved in parallel > modifications of K8S code in provider and executor (it happened in the > past that we had do to add very high min-airflow version for > "cncf.kubernetes" provider in order to make this happens, and we could > likely avoid similar problems by making sure K8S executor is used as > "yet another executor" - compliant with the AIP-51 interface and we > could evolve it much faster. > > That would also give celery provider some "real" meaning - so far > celery provider was merely exposing celery queue sensor, but when we > move CeleryExecutor to the provider, the provider would finally be a > useful one. > > Doing this requires a few small things: > * we likely need to add dynamic import in the old "executors" package > following PEP 562 (and the way we've done operators) and deprecation > notice in case someone uses them from there. > * we likely need to add "cncf.kubernetes" and "celery" packages as > pre-installed providers - alongside http, fttp, common.sql, sqlite, > imap. > > I **think**, after AIP-51 gets implemented, this would be a fully > backwards-compatible change - it's just a matter of proper management > of the dependencies. We could add min-cncf.kubernetes/celery versions > in Airflow 2.6, so that we are sure 2.6+ airflow uses only newer > providers and kubernetes/celery code and this would be fully > transparent for the users, I believe. > > J.
