I like what Daniel suggested. But I guess we’ll probably need to split a few executors from existing providers for this? If that’s the case, are we going to have those executors exist in both side for some time? or is there a better for it?
Best, Wei > On Sep 17, 2024, at 1:16 AM, Jens Scheffler <j_scheff...@gmx.de.INVALID> > wrote: > > Hi all, > > yes, diverged a bit. I'll add an agenda item for the next Airflow 3 dev > call. > > Main question as outcome would be: Shall we split the Providers if we > are (ongoing) also planning to split the deployment dependencies of > Scheduler, Worker, Webserver etc? > > --> > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+3+Dev+call%3A+Meeting+Notes#Airflow3Devcall:MeetingNotes-19September2024 > > Jens > > On 16.09.24 16:54, Vincent Beck wrote: >> On one side it makes sense to me, and I actually like the thinking >> "providers should only be for DAG authors". That makes it simple to figure >> whether something should belong to providers. If we go that way then FAB >> would no longer be a provider but a plugin which would be one step closer to >> not install providers on the webserver. >> >> On the other side, I can see some concerns/questions: >> >> - Where would these plugins be in the codebase? And more importantly, how >> would they be released? As part of Airflow? As a separate release management >> like apache-airflow-plugins? >> >> - Some executors (e.g. AwsEcsExecutor), depend on hooks (AWS hook). If we >> move executors outside of providers, then we need the plugin to depend on >> some providers? >> >> PS: we diverged from the original topic, we might want to create a separate >> thread for that. >> >> On 2024/09/14 12:52:25 Ash Berlin-Taylor wrote: >>>> Provider package name: "edge", so gets to package "airflow.providers.edge" >>> I'm not so sure about this name. I have no problems with the"edge" part, I >>> am mostly questioning if this should be a provider, or should it be another >>> kind of module. For instance it won't have an Operator, Sensor or Hook, nor >>> need anything else provided by the Provider Manager. >>> >>> Similarly now I think about it for Celery executor, it's not got anything >>> for use in dags does it? >>> >>> My thinking here is: if a dag author uses it directly -> "provider" >>> If it's only used by the deployment manager -> something else, possibly >>> "plugin"? >>> >>> Things that I think should fall in this second category would be the FAB, >>> Celery and Edge executors. cncf.kubernetes is mixed as it has the executor >>> and an operator. >>> >>> What do others think? >>> >>> On 8 September 2024 14:05:51 BST, Jens Scheffler >>> <j_scheff...@gmx.de.INVALID> wrote: >>>> Hi Devs, >>>> >>>> as we had a naming discussion for AIP-69 in >>>> https://lists.apache.org/thread/br1jfoc8p1wjzk74c09srjgr29spytfy and PRs >>>> are ready, Elad proposed to have at least a lazy consensus to close the >>>> naming before merge. >>>> >>>> Not having real counts leave "Distributed" and "Edge" close-by with most >>>> positive feedback. As nobody objected and the term "Edge" is shorter... >>>> That means: >>>> >>>> - Provider package name: "edge", so gets to package >>>> "airflow.providers.edge" >>>> >>>> - Executor class: "EdgeExecutor" >>>> >>>> - Remotely running worker name: "Edge Worker" - called via `airflow edge >>>> worker [options]` (similar like celery) >>>> >>>> >>>> If somebody wants to have a look to the implementation, several PRs are >>>> split for better review: >>>> >>>> - "Mothership" PR with the complete implementation if you want to have a >>>> test drive: https://github.com/apache/airflow/pull/40900 (=3800 LoC) >>>> >>>> - Split 1: (Full) Edge provider package: >>>> https://github.com/apache/airflow/pull/41729 (3700 LoC) >>>> >>>> - Split 1.1: (Empty) Provider package (Empty boilerplate) >>>> https://github.com/apache/airflow/pull/42046 (450 LoC) >>>> >>>> - Split 1.2: DB Models for Edge Provider >>>> https://github.com/apache/airflow/pull/42047 (1.1 + 950 LoC) >>>> >>>> - Split 1.3: EdgeExecutor >>>> https://github.com/apache/airflow/pull/42048 (1.1+1.2 + 650 LoC) >>>> >>>> - Split 1.4: Rest API Endpoint >>>> https://github.com/apache/airflow/pull/42049 (1.1+1.2 + 1050 LoC) >>>> >>>> - Split 1.5: Worker CLI >>>> https://github.com/apache/airflow/pull/42050 (1.1+1.2 + 650 LoC) >>>> >>>> - Split 1.6: Leftover glue to bring all together >>>> https://github.com/apache/airflow/pull/42051 (1.1-1.5 + 25 LoC) >>>> >>>> - Split 2: (Small) Adjustments in Core: >>>> https://github.com/apache/airflow/pull/41730 (<10 LoC) >>>> >>>> - Split 3: Breeze adjustments to develop/start via CLI >>>> https://github.com/apache/airflow/pull/41731 (<200 LoC) >>>> >>>> >>>> If anyone objects to the consensus here, let me/devlist know - otherwise >>>> this will be effective by Thursday 12th of September 2024 18.00 am PST >>>> (which coincidentally is when >>>> Airflow Summit is completed in San Francisco) - Also looking forward for >>>> approvals on the codebase / PRs. PRs are ready to have something working. >>>> >>>> Jens >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >>>> For additional commands, e-mail: dev-h...@airflow.apache.org >>>> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org >> For additional commands, e-mail: dev-h...@airflow.apache.org >> > > --------------------------------------------------------------------- > To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org > For additional commands, e-mail: dev-h...@airflow.apache.org > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@airflow.apache.org For additional commands, e-mail: dev-h...@airflow.apache.org
