Thanks Kaxil, this looks right to me as well. I updated the main Airflow 2.0 planning page as well to reflect the current scope and milestones based on this meeting.
*Doc Link*: https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+2.0+-+Planning I also wanted to thank Kevin from the AirBnb team for joining the call on Monday and offering to help with SmartSensors support as needed. I am excited that this feature will be part of the 2.0 release. Vikram On Wed, Sep 9, 2020 at 3:41 AM Jarek Potiuk <jarek.pot...@polidea.com> wrote: > LGTM! Thanks, Kaxil for putting this together. It is really helpful! > > On Wed, Sep 9, 2020 at 12:29 PM Kaxil Naik <kaxiln...@gmail.com> wrote: > > > Hi all, > > > > I have created a document to summarize the discussion from our third dev > > call for Airflow 2.0. > > > > Thank you all who joined the call. > > > > *Doc Link*: > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#3:7Sep2020 > <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%233:7Sep2020> > > < > https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%233:7Sep2020 > > > > > > To all those who attended, can you please double-check and add if I have > > missed anything? > > > > To all those who didn't join, if you disagree to anything in the Summary > > please > > voice your opinion. > > > > Including the Summary here too (might potentially break formatting): > > > > *Key Decisions* > > > > - *Smart Sensors* > > - Will be included in 2.0 as an *early-access* feature with a clear > > note that *this feature might potentially change in future Airflow > > version with breaking changes*. > > - Airbnb team would be happy to help on the support side answering > > questions related to Smart Sensor. > > - Add docs around different execution modes for Sensor: *Poke > > mode*, *Reschedule > > mode* vs *Smart Sensor* > > - *Providers Packages* > > - We had a consensus on *releasing providers packages separately* > > mainly > > because of the following reasons: > > - Separate cadence for providers compared to Airflow, so bugs in > > operator/hooks can be fixed lot faster. > > - Enterprises generally would not like to upgrade the “core” > > (Scheduler) as a small bug can break the deployment and > > affect all the DAGs > > - Breaking library changes (new version of a library) can be > fixed > > with a new version of Backport/Providers > > - Upgrades of backport providers are not “that” destructive i.e. > > even if you upgrade to a newer version and find a bug, you > > could go back to > > the previous version without causing any issues at all. > > - Open questions / Action Items: > > - How would users figure out “breaking changes” with CALVER > > Versioning (which is very clear with SEMVER)? > > - Use plugin Mechanism to: > > - Register Connections from an external provider to allow > > custom field or hide existing form fields. > > - Register Operator Extra links > > < > > https://airflow.apache.org/docs/stable/howto/define_extra_link.html> > > for > > operators in providers so that a change is not required in > > Airflow > > - Backport providers will only be supported/released for *three > > months after 2.0.0 released* > > - *Timeline to Airflow 2.0* > > - Only *critical fixes* (fixes to bugs that takedown Production > > system) will be backported to 1.10.x core for *six months* after > > Airflow 2.0 is released. > > > > Date > > > > Milestone > > > > Week of 7 Sep 2020 Create the 2.0.0-test branch > > > > While the scope is fluid, we would be rebasing this test branch from > > master. After we completely freeze the scope, we would only cherrypick > > commits from Airflow Master to v2-0-test branch if they are “in-scope”. > > Normal development would continue on Master branch i.e. PRs would be > > created against Airflow Master. > > Week of 28 Sep 2020 Cut Functionally complete 2.0 alpha release > > Week of 12 Oct 2020 Cut first 2.0 beta release > > > > Beta snapshots would be published to the Airflow Community to test and > > create issues to make sure Airflow is functioning and backwards > compatible. > > Week of 19 Oct 2020 Cut bridge release based on 1.10.x - jump-off point > to > > 2.0. Probably 1.10.13 or 1.10.14 containing upgrade check scripts for 2.0 > > Week of 26 Oct 2020 Cut second 2.0 beta release > > Week of 9 Nov 2020 Cut third 2.0 beta release > > Week of 23 Nov 2020 Cut first 2.0 release candidate (2.0.0rc1) > > > > *Things to Discuss Next* > > > > - *14 September (Subject to change)* > > - API > > - Progress, Current Work & Discussions > > - Any open questions? > > - Improvements to SubDags / Concept of TaskGroup > > - *AIP-34*: > > > > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-34+TaskGroup%3A+A+UI+task+grouping+concept+as+an+alternative+to+SubDagOperator > > > > - *PR*: https://github.com/apache/airflow/pull/10153 > > - Any concerns on including this PR to 2.0 ?? > > - Do we want to support both TaskGroup & SubDags? > > - Process: > > - When should we defer the in-scope items to post-2.0 > > - Completion by a date? > > - Progress by a date? > > - Progress, Current Work & Discussions > > - Scheduler HA > > - Docs Improvements > > - Helm Chart > > - Discuss the issue with sources > > > > > > > > Regards, > > Kaxil > > > > > -- > > Jarek Potiuk > Polidea <https://www.polidea.com/> | Principal Software Engineer > > M: +48 660 796 129 <+48660796129> > [image: Polidea] <https://www.polidea.com/> >
