Yep. Thanks, Ash - indeed that is closely related and if do this, I think 9506 should be fixed together. I added it to the doc.
I also resolved some of the questions and comments in the doc and added a short "Alternative" - i.e. what is the alternative - monolith Airflow release as we did so far. I think at the call today we should start from this "alternative" - because maybe the gains that we have by releasing separate packages are not as big as the complexity we are adding. I am a bit on the fence on that one myself - I am not 100 % sure if the gains are really as big, I would love to hear what others think about it :) J. On Sat, Sep 5, 2020 at 11:28 AM Ash Berlin-Taylor <a...@apache.org> wrote: > Very closely related issue https://github.com/apache/airflow/issues/9506 > > On 5 September 2020 08:01:11 BST, Jarek Potiuk <jarek.pot...@polidea.com> > wrote: > >And we have a new addition from Kamil about the need to extend slightly > >plugin mechanism to be able to cover dynamically "Connections", > >"Connection > >Form" and "Extra Links" - those are indeed the "core -> Providers" > >dependencies that we still have. > > > >They seem to be easy to handle by making providers "plugins" and > >extending > >the plugin mechanism a bit. Thanks Kamil for the thoughtful input! > > > >On Fri, Sep 4, 2020 at 8:08 PM Jarek Potiuk <jarek.pot...@polidea.com> > >wrote: > > > >> Just a short reminder - for some more comments/review on the "PIP > >package > >> model of Airflow 2.0" doc > >> > >> > > > https://docs.google.com/document/d/1vV67Qomk_rxVuy1Tj_vrjaNq3Eh-V6n6aLDnOy7gVWk/edit# > >> > >> I've added one small addition - in this model we want to make sure > >that > >> there are no dependencies of core packages on any of the providers - > >we do > >> not run such checks yet but it's easy to add. > >> > >> J > >> > >> > >> On Wed, Sep 2, 2020 at 5:46 PM Jarek Potiuk > ><jarek.pot...@polidea.com> > >> wrote: > >> > >>> Cool! > >>> > >>> If you have comments on particular sections/paragraphs - it's easier > >to > >>> keep track of it and respond in the doc. If you have some general > >>> statements, and some summary of your thinking after the review - > >it's best > >>> to respond to the email :) > >>> > >>> I am ok with both and will aggregate it eventually. > >>> > >>> J. > >>> > >>> > >>> On Wed, Sep 2, 2020 at 4:38 PM Vikram Koka <vik...@astronomer.io> > >wrote: > >>> > >>>> Jarek, > >>>> > >>>> Thank you, this is very helpful. > >>>> I assume that you would like comments in the document itself? > >>>> Or, would you like them in email? > >>>> > >>>> Best regards, > >>>> Vikram > >>>> > >>>> > >>>> On Wed, Sep 2, 2020 at 12:43 AM Jarek Potiuk > ><jarek.pot...@polidea.com> > >>>> wrote: > >>>> > >>>> > As promised during the last call I prepared the proposal on how > >we can > >>>> > approach the package model for Airflow 2.0 - including the > >"Provider > >>>> > Packages" approach. > >>>> > > >>>> > https://s.apache.org/airflow-2-0-package-model > >>>> > > >>>> > I would like to discuss it at our next meeting on Monday. I'd > >love to > >>>> > hear your comments. > >>>> > > >>>> > J. > >>>> > > >>>> > > >>>> > On Thu, Aug 27, 2020 at 10:23 AM Jarek Potiuk < > >>>> jarek.pot...@polidea.com> > >>>> > wrote: > >>>> > > > >>>> > > +1 Kevin on the call :). > >>>> > > > >>>> > > On Wed, Aug 26, 2020 at 12:59 PM Kaxil Naik > ><kaxiln...@gmail.com> > >>>> wrote: > >>>> > >> > >>>> > >> Thanks Kevin, Looking forward to see you on the next call. > >>>> > >> > >>>> > >> On Wed, Aug 26, 2020, 08:54 Kevin Yang <yrql...@gmail.com> > >wrote: > >>>> > >> > >>>> > >> > Thank you Kaxil, this is extremely helpful. We'll try to > >join at > >>>> > least the > >>>> > >> > next meeting trying to see if we can provide more > >perspectives on > >>>> > >> > SmartSensor and anything else we can help. > >>>> > >> > > >>>> > >> > > >>>> > >> > Cheers, > >>>> > >> > Kevin Y > >>>> > >> > > >>>> > >> > On Tue, Aug 25, 2020 at 4:28 PM Kaxil Naik > ><kaxiln...@gmail.com> > >>>> > wrote: > >>>> > >> > > >>>> > >> > > Hi all, > >>>> > >> > > > >>>> > >> > > I have created a document to summarize the discussion from > >our > >>>> > second dev > >>>> > >> > > call for Airflow 2.0. > >>>> > >> > > > >>>> > >> > > Thank you all who joined the call. > >>>> > >> > > > >>>> > >> > > *Doc Link*: > >>>> > >> > > > >>>> > >> > > > >>>> > >> > > >>>> > > >>>> > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#2:24Aug2020 > <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020> > >>>> > >< > https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020 > > > >>>> > < > >>>> > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020 > >>>> > > >>>> > >> > < > >>>> > > >>>> > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020 > >>>> > > > >>>> > >> > > < > >>>> > >> > > >>>> > > >>>> > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020 > >>>> > >> > > > >>>> > >> > > > >>>> > >> > > To all those who attended, can you please double-check and > >add > >>>> if I > >>>> > have > >>>> > >> > > missed anything? > >>>> > >> > > > >>>> > >> > > To all those who didn't join, if you disagree to anything > >in the > >>>> > Summary > >>>> > >> > > please voice your opinion. > >>>> > >> > > > >>>> > >> > > Including the Summary here too (might potentially break > >>>> formatting): > >>>> > >> > > > >>>> > >> > > *Key Decisions* > >>>> > >> > > > >>>> > >> > > - *Smart Sensors – *in 2.0 or 2.1 > >>>> > >> > > - AIP-17 > >>>> > >> > > < > >>>> > >> > > > >>>> > >> > > >>>> > > >>>> > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17%3A+Consolidate+and+de-duplicate+sensor+tasks+in+airflow+Smart+Sensor > >>>> > >> > > > > >>>> > >> > > | > >>>> > >> > > PR: https://github.com/apache/airflow/pull/5499 > >>>> > >> > > - We have not come to a conclusion yet on whether > >this > >>>> should > >>>> > be > >>>> > >> > > included in 2.0 or not. The majority is towards > >adding it > >>>> in > >>>> > 2.0 > >>>> > >> > (as > >>>> > >> > > it > >>>> > >> > > supports Airflow 2.0's Scalability story) and > >marking it > >>>> as > >>>> > >> > > *experimental*. > >>>> > >> > > - There were some questions raised around supporting > >this > >>>> new > >>>> > >> > > feature. So we decided that *everyone would take a > >look at > >>>> > the PR > >>>> > >> > > itself and we will spend a few minutes in the next > >>>> meeting to > >>>> > >> > decide > >>>> > >> > > whether it is 2.0 or not*. > >>>> > >> > > - *Simplification of KubernetesExecutor / > >>>> KubernetesPodOperator* > >>>> > >> > > - PR: https://github.com/apache/airflow/pull/10393 > >>>> > >> > > - This will be part of *Airflow 2.0* > >>>> > >> > > - *Airflow Upgrade Check* (airflow upgrade-check)* > >command * > >>>> > >> > > - WIP PR: PR: > >https://github.com/apache/airflow/pull/9467 > >>>> | > >>>> > Design > >>>> > >> > > Doc: > >>>> > >> > > > >>>> > >> > > > >>>> > >> > > >>>> > > >>>> > > > https://docs.google.com/document/d/17tB9KZrH871q3AEafqR_i2I7Nrn-OT7le_P49G65VzM/edit#heading=h.vv80w6y621gv > >>>> > >> > > - *Scope*: > >>>> > >> > > - Users bash script won’t be included but > >anything in > >>>> the > >>>> > core > >>>> > >> > > Airflow would be covered > >>>> > >> > > - > >>>> > >> > > > >>>> > >> > > *DAG Definitions*: > >>>> > >> > > - Changes in Path for contrib to Providers > >packages > >>>> > >> > > - DAG Interfaces: changes in arguments of a > >DAG / > >>>> > >> > BaseOperator > >>>> > >> > > - *Configurations*: > >>>> > >> > > - Option to auto-replace deprecated configs > >with new > >>>> > options > >>>> > >> > > - *Run-time Core items*: > >>>> > >> > > - Changes like "Connection type can't be > >null". The > >>>> > >> > > upgrade-check should at least shown warning if > >it > >>>> can't > >>>> > >> > > provide option to > >>>> > >> > > detect the type. > >>>> > >> > > - *CLI refactor is out-of-scope* > >>>> > >> > > - Automatic refactor is *out-of-scope* as it > >is too > >>>> > difficult > >>>> > >> > > to cover all the cases in the Users bash > >scripts. > >>>> > >> > > - This will be covered by docs or by showing > >>>> warnings > >>>> > via the > >>>> > >> > > upgrade-check command > >>>> > >> > > - *Experimental API to New API refactor is > >>>> out-of-scope* > >>>> > (will > >>>> > >> > be > >>>> > >> > > covered by Migration docs) > >>>> > >> > > - We agreed that the airflow upgrade-check command > >*needs > >>>> to > >>>> > be > >>>> > >> > > available in the last release before Airflow 2.0* > >(1.10.x > >>>> or > >>>> > >> > 1.11.x) > >>>> > >> > > - Potential problems with time-consuming DB Migration > >were > >>>> also > >>>> > >> > > discussed. If we identify such a DB Migration (example > >the > >>>> one > >>>> > >> > involving > >>>> > >> > > TaskInstance table) should be noted separately in > >>>> Updating.md to > >>>> > >> > > provide a > >>>> > >> > > warning to the users. > >>>> > >> > > - *DEV Calls Feedback* > >>>> > >> > > - We agreed on having *Weekly calls from 7 September > >>>> onwards* > >>>> > >> > > - Calls will start with a 5-min reviewing the > >progress > >>>> from > >>>> > the > >>>> > >> > last > >>>> > >> > > call towards 2.0 > >>>> > >> > > - *Process* > >>>> > >> > > - A *2.0.0-test* branch will be created on 10 Sep > >2020 > >>>> > >> > > - Changelog: > >>>> > >> > > - The current way of Changelog is OK. We don't > >need > >>>> further > >>>> > >> > > categorization like Webserver, Scheduler etc. > >>>> > >> > > - Separate Changelog would be created for > >Providers > >>>> > Packages > >>>> > >> > > - We need to figure a way to tag/label PRs & > >Issues > >>>> with > >>>> > correct > >>>> > >> > > categories. Some options that were discussed > >were: > >>>> > >> > > - Adding labels on the PRs & Issues via Bot > >>>> > >> > > - A field in PR template for PR authors to > >add, the > >>>> bot > >>>> > would > >>>> > >> > > then read the field which would be used to > >label > >>>> the PR > >>>> > >> > > - Add rules, for example Committers needs to > >add > >>>> > appropriate > >>>> > >> > > labels to the PR before merging it. We could > >have a > >>>> > >> > > scheduled Github > >>>> > >> > > Actions workflow that would fail if it finds > >PRs > >>>> without > >>>> > >> > > labels. > >>>> > >> > > > >>>> > >> > > *Things to Discuss Next* > >>>> > >> > > > >>>> > >> > > - *7 September* > >>>> > >> > > - Progress, Current Work & Discussions > >>>> > >> > > - API > >>>> > >> > > - Providers Packages > >>>> > >> > > - Discuss open questions > >>>> > >> > > - Improvements to SubDags / Concept of TaskGroup > >>>> > >> > > - AIP-34 < > >>>> https://github.com/apache/airflow/pull/10153> > >>>> > >> > > - *14 September* > >>>> > >> > > - Process: > >>>> > >> > > - When should we defer the in-scope items to > >post-2.0 > >>>> > >> > > - Completion by a date? > >>>> > >> > > - Progress by a date? > >>>> > >> > > - Progress, Current Work & Discussions > >>>> > >> > > - Scheduler HA > >>>> > >> > > - Docs Improvements > >>>> > >> > > - Helm Chart > >>>> > >> > > - Discuss the issue with sources > >>>> > >> > > > >>>> > >> > > > >>>> > >> > > Regards, > >>>> > >> > > Kaxil > >>>> > >> > > > >>>> > >> > > >>>> > > > >>>> > > > >>>> > > > >>>> > > -- > >>>> > > > >>>> > > Jarek Potiuk > >>>> > > Polidea | Principal Software Engineer > >>>> > > > >>>> > > M: +48 660 796 129 > >>>> > > > >>>> > > > >>>> > > >>>> > > >>>> > -- > >>>> > > >>>> > Jarek Potiuk > >>>> > Polidea | Principal Software Engineer > >>>> > > >>>> > M: +48 660 796 129 > >>>> > > >>>> > >>> > >>> > >>> -- > >>> > >>> Jarek Potiuk > >>> Polidea <https://www.polidea.com/> | Principal Software Engineer > >>> > >>> M: +48 660 796 129 <+48660796129> > >>> [image: Polidea] <https://www.polidea.com/> > >>> > >>> > >> > >> -- > >> > >> Jarek Potiuk > >> Polidea <https://www.polidea.com/> | Principal Software Engineer > >> > >> M: +48 660 796 129 <+48660796129> > >> [image: Polidea] <https://www.polidea.com/> > >> > >> > > > >-- > > > >Jarek Potiuk > >Polidea <https://www.polidea.com/> | Principal Software Engineer > > > >M: +48 660 796 129 <+48660796129> > >[image: Polidea] <https://www.polidea.com/> > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
