LGTM! Thanks, Kaxil for putting this together. It is really helpful! On Wed, Sep 9, 2020 at 12:29 PM Kaxil Naik <kaxiln...@gmail.com> wrote:
> Hi all, > > I have created a document to summarize the discussion from our third dev > call for Airflow 2.0. > > Thank you all who joined the call. > > *Doc Link*: > > https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#3:7Sep2020 > <https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-%233:7Sep2020> > > To all those who attended, can you please double-check and add if I have > missed anything? > > To all those who didn't join, if you disagree to anything in the Summary > please > voice your opinion. > > Including the Summary here too (might potentially break formatting): > > *Key Decisions* > > - *Smart Sensors* > - Will be included in 2.0 as an *early-access* feature with a clear > note that *this feature might potentially change in future Airflow > version with breaking changes*. > - Airbnb team would be happy to help on the support side answering > questions related to Smart Sensor. > - Add docs around different execution modes for Sensor: *Poke > mode*, *Reschedule > mode* vs *Smart Sensor* > - *Providers Packages* > - We had a consensus on *releasing providers packages separately* > mainly > because of the following reasons: > - Separate cadence for providers compared to Airflow, so bugs in > operator/hooks can be fixed lot faster. > - Enterprises generally would not like to upgrade the “core” > (Scheduler) as a small bug can break the deployment and > affect all the DAGs > - Breaking library changes (new version of a library) can be fixed > with a new version of Backport/Providers > - Upgrades of backport providers are not “that” destructive i.e. > even if you upgrade to a newer version and find a bug, you > could go back to > the previous version without causing any issues at all. > - Open questions / Action Items: > - How would users figure out “breaking changes” with CALVER > Versioning (which is very clear with SEMVER)? > - Use plugin Mechanism to: > - Register Connections from an external provider to allow > custom field or hide existing form fields. > - Register Operator Extra links > < > https://airflow.apache.org/docs/stable/howto/define_extra_link.html> > for > operators in providers so that a change is not required in > Airflow > - Backport providers will only be supported/released for *three > months after 2.0.0 released* > - *Timeline to Airflow 2.0* > - Only *critical fixes* (fixes to bugs that takedown Production > system) will be backported to 1.10.x core for *six months* after > Airflow 2.0 is released. > > Date > > Milestone > > Week of 7 Sep 2020 Create the 2.0.0-test branch > > While the scope is fluid, we would be rebasing this test branch from > master. After we completely freeze the scope, we would only cherrypick > commits from Airflow Master to v2-0-test branch if they are “in-scope”. > Normal development would continue on Master branch i.e. PRs would be > created against Airflow Master. > Week of 28 Sep 2020 Cut Functionally complete 2.0 alpha release > Week of 12 Oct 2020 Cut first 2.0 beta release > > Beta snapshots would be published to the Airflow Community to test and > create issues to make sure Airflow is functioning and backwards compatible. > Week of 19 Oct 2020 Cut bridge release based on 1.10.x - jump-off point to > 2.0. Probably 1.10.13 or 1.10.14 containing upgrade check scripts for 2.0 > Week of 26 Oct 2020 Cut second 2.0 beta release > Week of 9 Nov 2020 Cut third 2.0 beta release > Week of 23 Nov 2020 Cut first 2.0 release candidate (2.0.0rc1) > > *Things to Discuss Next* > > - *14 September (Subject to change)* > - API > - Progress, Current Work & Discussions > - Any open questions? > - Improvements to SubDags / Concept of TaskGroup > - *AIP-34*: > > https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-34+TaskGroup%3A+A+UI+task+grouping+concept+as+an+alternative+to+SubDagOperator > > - *PR*: https://github.com/apache/airflow/pull/10153 > - Any concerns on including this PR to 2.0 ?? > - Do we want to support both TaskGroup & SubDags? > - Process: > - When should we defer the in-scope items to post-2.0 > - Completion by a date? > - Progress by a date? > - Progress, Current Work & Discussions > - Scheduler HA > - Docs Improvements > - Helm Chart > - Discuss the issue with sources > > > > Regards, > Kaxil > -- Jarek Potiuk Polidea <https://www.polidea.com/> | Principal Software Engineer M: +48 660 796 129 <+48660796129> [image: Polidea] <https://www.polidea.com/>
