Thanks, Vikram for updating the planning page. On Wed, Sep 9, 2020 at 5:43 PM Vikram Koka <vik...@astronomer.io> wrote:
> Thanks Kaxil, this looks right to me as well. > I updated the main Airflow 2.0 planning page as well to reflect the current > scope and milestones based on this meeting. > > *Doc Link*: > https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0+-+Planning > > I also wanted to thank Kevin from the AirBnb team for joining the call on > Monday and offering to help with SmartSensors support as needed. I am > excited that this feature will be part of the 2.0 release. > > Vikram > > > On Wed, Sep 9, 2020 at 3:41 AM Jarek Potiuk <jarek.pot...@polidea.com> > wrote: > > > LGTM! Thanks, Kaxil for putting this together. It is really helpful! > > > > On Wed, Sep 9, 2020 at 12:29 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > > > > > Hi all, > > > > > > I have created a document to summarize the discussion from our third > dev > > > call for Airflow 2.0. > > > > > > Thank you all who joined the call. > > > > > > *Doc Link*: > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#3:7Sep2020 > <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%233:7Sep2020> > > < > https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%233:7Sep2020 > > > > > < > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%233:7Sep2020 > > > > > > > > > To all those who attended, can you please double-check and add if I > have > > > missed anything? > > > > > > To all those who didn't join, if you disagree to anything in the > Summary > > > please > > > voice your opinion. > > > > > > Including the Summary here too (might potentially break formatting): > > > > > > *Key Decisions* > > > > > > - *Smart Sensors* > > > - Will be included in 2.0 as an *early-access* feature with a > clear > > > note that *this feature might potentially change in future > Airflow > > > version with breaking changes*. > > > - Airbnb team would be happy to help on the support side > answering > > > questions related to Smart Sensor. > > > - Add docs around different execution modes for Sensor: *Poke > > > mode*, *Reschedule > > > mode* vs *Smart Sensor* > > > - *Providers Packages* > > > - We had a consensus on *releasing providers packages separately* > > > mainly > > > because of the following reasons: > > > - Separate cadence for providers compared to Airflow, so bugs > in > > > operator/hooks can be fixed lot faster. > > > - Enterprises generally would not like to upgrade the “core” > > > (Scheduler) as a small bug can break the deployment and > > > affect all the DAGs > > > - Breaking library changes (new version of a library) can be > > fixed > > > with a new version of Backport/Providers > > > - Upgrades of backport providers are not “that” destructive > i.e. > > > even if you upgrade to a newer version and find a bug, you > > > could go back to > > > the previous version without causing any issues at all. > > > - Open questions / Action Items: > > > - How would users figure out “breaking changes” with CALVER > > > Versioning (which is very clear with SEMVER)? > > > - Use plugin Mechanism to: > > > - Register Connections from an external provider to allow > > > custom field or hide existing form fields. > > > - Register Operator Extra links > > > < > > > https://airflow.apache.org/docs/stable/howto/define_extra_link.html> > > > for > > > operators in providers so that a change is not required in > > > Airflow > > > - Backport providers will only be supported/released for > *three > > > months after 2.0.0 released* > > > - *Timeline to Airflow 2.0* > > > - Only *critical fixes* (fixes to bugs that takedown Production > > > system) will be backported to 1.10.x core for *six months* after > > > Airflow 2.0 is released. > > > > > > Date > > > > > > Milestone > > > > > > Week of 7 Sep 2020 Create the 2.0.0-test branch > > > > > > While the scope is fluid, we would be rebasing this test branch from > > > master. After we completely freeze the scope, we would only cherrypick > > > commits from Airflow Master to v2-0-test branch if they are “in-scope”. > > > Normal development would continue on Master branch i.e. PRs would be > > > created against Airflow Master. > > > Week of 28 Sep 2020 Cut Functionally complete 2.0 alpha release > > > Week of 12 Oct 2020 Cut first 2.0 beta release > > > > > > Beta snapshots would be published to the Airflow Community to test and > > > create issues to make sure Airflow is functioning and backwards > > compatible. > > > Week of 19 Oct 2020 Cut bridge release based on 1.10.x - jump-off point > > to > > > 2.0. Probably 1.10.13 or 1.10.14 containing upgrade check scripts for > 2.0 > > > Week of 26 Oct 2020 Cut second 2.0 beta release > > > Week of 9 Nov 2020 Cut third 2.0 beta release > > > Week of 23 Nov 2020 Cut first 2.0 release candidate (2.0.0rc1) > > > > > > *Things to Discuss Next* > > > > > > - *14 September (Subject to change)* > > > - API > > > - Progress, Current Work & Discussions > > > - Any open questions? > > > - Improvements to SubDags / Concept of TaskGroup > > > - *AIP-34*: > > > > > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-34+TaskGroup%3A+A+UI+task+grouping+concept+as+an+alternative+to+SubDagOperator > > > > > > - *PR*: https://github.com/apache/airflow/pull/10153 > > > - Any concerns on including this PR to 2.0 ?? > > > - Do we want to support both TaskGroup & SubDags? > > > - Process: > > > - When should we defer the in-scope items to post-2.0 > > > - Completion by a date? > > > - Progress by a date? > > > - Progress, Current Work & Discussions > > > - Scheduler HA > > > - Docs Improvements > > > - Helm Chart > > > - Discuss the issue with sources > > > > > > > > > > > > Regards, > > > Kaxil > > > > > > > > > -- > > > > Jarek Potiuk > > Polidea <https://www.polidea.com/> | Principal Software Engineer > > > > M: +48 660 796 129 <+48660796129> > > [image: Polidea] <https://www.polidea.com/> > > >
