Very closely related issue https://github.com/apache/airflow/issues/9506
On 5 September 2020 08:01:11 BST, Jarek Potiuk <jarek.pot...@polidea.com> wrote: >And we have a new addition from Kamil about the need to extend slightly >plugin mechanism to be able to cover dynamically "Connections", >"Connection >Form" and "Extra Links" - those are indeed the "core -> Providers" >dependencies that we still have. > >They seem to be easy to handle by making providers "plugins" and >extending >the plugin mechanism a bit. Thanks Kamil for the thoughtful input! > >On Fri, Sep 4, 2020 at 8:08 PM Jarek Potiuk <jarek.pot...@polidea.com> >wrote: > >> Just a short reminder - for some more comments/review on the "PIP >package >> model of Airflow 2.0" doc >> >> >https://docs.google.com/document/d/1vV67Qomk_rxVuy1Tj_vrjaNq3Eh-V6n6aLDnOy7gVWk/edit# >> >> I've added one small addition - in this model we want to make sure >that >> there are no dependencies of core packages on any of the providers - >we do >> not run such checks yet but it's easy to add. >> >> J >> >> >> On Wed, Sep 2, 2020 at 5:46 PM Jarek Potiuk ><jarek.pot...@polidea.com> >> wrote: >> >>> Cool! >>> >>> If you have comments on particular sections/paragraphs - it's easier >to >>> keep track of it and respond in the doc. If you have some general >>> statements, and some summary of your thinking after the review - >it's best >>> to respond to the email :) >>> >>> I am ok with both and will aggregate it eventually. >>> >>> J. >>> >>> >>> On Wed, Sep 2, 2020 at 4:38 PM Vikram Koka <vik...@astronomer.io> >wrote: >>> >>>> Jarek, >>>> >>>> Thank you, this is very helpful. >>>> I assume that you would like comments in the document itself? >>>> Or, would you like them in email? >>>> >>>> Best regards, >>>> Vikram >>>> >>>> >>>> On Wed, Sep 2, 2020 at 12:43 AM Jarek Potiuk ><jarek.pot...@polidea.com> >>>> wrote: >>>> >>>> > As promised during the last call I prepared the proposal on how >we can >>>> > approach the package model for Airflow 2.0 - including the >"Provider >>>> > Packages" approach. >>>> > >>>> > https://s.apache.org/airflow-2-0-package-model >>>> > >>>> > I would like to discuss it at our next meeting on Monday. I'd >love to >>>> > hear your comments. >>>> > >>>> > J. >>>> > >>>> > >>>> > On Thu, Aug 27, 2020 at 10:23 AM Jarek Potiuk < >>>> jarek.pot...@polidea.com> >>>> > wrote: >>>> > > >>>> > > +1 Kevin on the call :). >>>> > > >>>> > > On Wed, Aug 26, 2020 at 12:59 PM Kaxil Naik ><kaxiln...@gmail.com> >>>> wrote: >>>> > >> >>>> > >> Thanks Kevin, Looking forward to see you on the next call. >>>> > >> >>>> > >> On Wed, Aug 26, 2020, 08:54 Kevin Yang <yrql...@gmail.com> >wrote: >>>> > >> >>>> > >> > Thank you Kaxil, this is extremely helpful. We'll try to >join at >>>> > least the >>>> > >> > next meeting trying to see if we can provide more >perspectives on >>>> > >> > SmartSensor and anything else we can help. >>>> > >> > >>>> > >> > >>>> > >> > Cheers, >>>> > >> > Kevin Y >>>> > >> > >>>> > >> > On Tue, Aug 25, 2020 at 4:28 PM Kaxil Naik ><kaxiln...@gmail.com> >>>> > wrote: >>>> > >> > >>>> > >> > > Hi all, >>>> > >> > > >>>> > >> > > I have created a document to summarize the discussion from >our >>>> > second dev >>>> > >> > > call for Airflow 2.0. >>>> > >> > > >>>> > >> > > Thank you all who joined the call. >>>> > >> > > >>>> > >> > > *Doc Link*: >>>> > >> > > >>>> > >> > > >>>> > >> > >>>> > >>>> >https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#2:24Aug2020 >>>> ><https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020> >>>> > < >>>> >https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020 >>>> > >>>> > >> > < >>>> > >>>> >https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020 >>>> > > >>>> > >> > > < >>>> > >> > >>>> > >>>> >https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%232:24Aug2020 >>>> > >> > > >>>> > >> > > >>>> > >> > > To all those who attended, can you please double-check and >add >>>> if I >>>> > have >>>> > >> > > missed anything? >>>> > >> > > >>>> > >> > > To all those who didn't join, if you disagree to anything >in the >>>> > Summary >>>> > >> > > please voice your opinion. >>>> > >> > > >>>> > >> > > Including the Summary here too (might potentially break >>>> formatting): >>>> > >> > > >>>> > >> > > *Key Decisions* >>>> > >> > > >>>> > >> > > - *Smart Sensors – *in 2.0 or 2.1 >>>> > >> > > - AIP-17 >>>> > >> > > < >>>> > >> > > >>>> > >> > >>>> > >>>> >https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-17%3A+Consolidate+and+de-duplicate+sensor+tasks+in+airflow+Smart+Sensor >>>> > >> > > > >>>> > >> > > | >>>> > >> > > PR: https://github.com/apache/airflow/pull/5499 >>>> > >> > > - We have not come to a conclusion yet on whether >this >>>> should >>>> > be >>>> > >> > > included in 2.0 or not. The majority is towards >adding it >>>> in >>>> > 2.0 >>>> > >> > (as >>>> > >> > > it >>>> > >> > > supports Airflow 2.0's Scalability story) and >marking it >>>> as >>>> > >> > > *experimental*. >>>> > >> > > - There were some questions raised around supporting >this >>>> new >>>> > >> > > feature. So we decided that *everyone would take a >look at >>>> > the PR >>>> > >> > > itself and we will spend a few minutes in the next >>>> meeting to >>>> > >> > decide >>>> > >> > > whether it is 2.0 or not*. >>>> > >> > > - *Simplification of KubernetesExecutor / >>>> KubernetesPodOperator* >>>> > >> > > - PR: https://github.com/apache/airflow/pull/10393 >>>> > >> > > - This will be part of *Airflow 2.0* >>>> > >> > > - *Airflow Upgrade Check* (airflow upgrade-check)* >command * >>>> > >> > > - WIP PR: PR: >https://github.com/apache/airflow/pull/9467 >>>> | >>>> > Design >>>> > >> > > Doc: >>>> > >> > > >>>> > >> > > >>>> > >> > >>>> > >>>> >https://docs.google.com/document/d/17tB9KZrH871q3AEafqR_i2I7Nrn-OT7le_P49G65VzM/edit#heading=h.vv80w6y621gv >>>> > >> > > - *Scope*: >>>> > >> > > - Users bash script won’t be included but >anything in >>>> the >>>> > core >>>> > >> > > Airflow would be covered >>>> > >> > > - >>>> > >> > > >>>> > >> > > *DAG Definitions*: >>>> > >> > > - Changes in Path for contrib to Providers >packages >>>> > >> > > - DAG Interfaces: changes in arguments of a >DAG / >>>> > >> > BaseOperator >>>> > >> > > - *Configurations*: >>>> > >> > > - Option to auto-replace deprecated configs >with new >>>> > options >>>> > >> > > - *Run-time Core items*: >>>> > >> > > - Changes like "Connection type can't be >null". The >>>> > >> > > upgrade-check should at least shown warning if >it >>>> can't >>>> > >> > > provide option to >>>> > >> > > detect the type. >>>> > >> > > - *CLI refactor is out-of-scope* >>>> > >> > > - Automatic refactor is *out-of-scope* as it >is too >>>> > difficult >>>> > >> > > to cover all the cases in the Users bash >scripts. >>>> > >> > > - This will be covered by docs or by showing >>>> warnings >>>> > via the >>>> > >> > > upgrade-check command >>>> > >> > > - *Experimental API to New API refactor is >>>> out-of-scope* >>>> > (will >>>> > >> > be >>>> > >> > > covered by Migration docs) >>>> > >> > > - We agreed that the airflow upgrade-check command >*needs >>>> to >>>> > be >>>> > >> > > available in the last release before Airflow 2.0* >(1.10.x >>>> or >>>> > >> > 1.11.x) >>>> > >> > > - Potential problems with time-consuming DB Migration >were >>>> also >>>> > >> > > discussed. If we identify such a DB Migration (example >the >>>> one >>>> > >> > involving >>>> > >> > > TaskInstance table) should be noted separately in >>>> Updating.md to >>>> > >> > > provide a >>>> > >> > > warning to the users. >>>> > >> > > - *DEV Calls Feedback* >>>> > >> > > - We agreed on having *Weekly calls from 7 September >>>> onwards* >>>> > >> > > - Calls will start with a 5-min reviewing the >progress >>>> from >>>> > the >>>> > >> > last >>>> > >> > > call towards 2.0 >>>> > >> > > - *Process* >>>> > >> > > - A *2.0.0-test* branch will be created on 10 Sep >2020 >>>> > >> > > - Changelog: >>>> > >> > > - The current way of Changelog is OK. We don't >need >>>> further >>>> > >> > > categorization like Webserver, Scheduler etc. >>>> > >> > > - Separate Changelog would be created for >Providers >>>> > Packages >>>> > >> > > - We need to figure a way to tag/label PRs & >Issues >>>> with >>>> > correct >>>> > >> > > categories. Some options that were discussed >were: >>>> > >> > > - Adding labels on the PRs & Issues via Bot >>>> > >> > > - A field in PR template for PR authors to >add, the >>>> bot >>>> > would >>>> > >> > > then read the field which would be used to >label >>>> the PR >>>> > >> > > - Add rules, for example Committers needs to >add >>>> > appropriate >>>> > >> > > labels to the PR before merging it. We could >have a >>>> > >> > > scheduled Github >>>> > >> > > Actions workflow that would fail if it finds >PRs >>>> without >>>> > >> > > labels. >>>> > >> > > >>>> > >> > > *Things to Discuss Next* >>>> > >> > > >>>> > >> > > - *7 September* >>>> > >> > > - Progress, Current Work & Discussions >>>> > >> > > - API >>>> > >> > > - Providers Packages >>>> > >> > > - Discuss open questions >>>> > >> > > - Improvements to SubDags / Concept of TaskGroup >>>> > >> > > - AIP-34 < >>>> https://github.com/apache/airflow/pull/10153> >>>> > >> > > - *14 September* >>>> > >> > > - Process: >>>> > >> > > - When should we defer the in-scope items to >post-2.0 >>>> > >> > > - Completion by a date? >>>> > >> > > - Progress by a date? >>>> > >> > > - Progress, Current Work & Discussions >>>> > >> > > - Scheduler HA >>>> > >> > > - Docs Improvements >>>> > >> > > - Helm Chart >>>> > >> > > - Discuss the issue with sources >>>> > >> > > >>>> > >> > > >>>> > >> > > Regards, >>>> > >> > > Kaxil >>>> > >> > > >>>> > >> > >>>> > > >>>> > > >>>> > > >>>> > > -- >>>> > > >>>> > > Jarek Potiuk >>>> > > Polidea | Principal Software Engineer >>>> > > >>>> > > M: +48 660 796 129 >>>> > > >>>> > > >>>> > >>>> > >>>> > -- >>>> > >>>> > Jarek Potiuk >>>> > Polidea | Principal Software Engineer >>>> > >>>> > M: +48 660 796 129 >>>> > >>>> >>> >>> >>> -- >>> >>> Jarek Potiuk >>> Polidea <https://www.polidea.com/> | Principal Software Engineer >>> >>> M: +48 660 796 129 <+48660796129> >>> [image: Polidea] <https://www.polidea.com/> >>> >>> >> >> -- >> >> Jarek Potiuk >> Polidea <https://www.polidea.com/> | Principal Software Engineer >> >> M: +48 660 796 129 <+48660796129> >> [image: Polidea] <https://www.polidea.com/> >> >> > >-- > >Jarek Potiuk >Polidea <https://www.polidea.com/> | Principal Software Engineer > >M: +48 660 796 129 <+48660796129> >[image: Polidea] <https://www.polidea.com/>
