Hi all, I have created a document to summarize the discussion from our third dev call for Airflow 2.0.
Thank you all who joined the call. *Doc Link*: https://cwiki.apache.org/confluence/display/AIRFLOW/Meeting+Notes#MeetingNotes-#3:7Sep2020 To all those who attended, can you please double-check and add if I have missed anything? To all those who didn't join, if you disagree to anything in the Summary please voice your opinion. Including the Summary here too (might potentially break formatting): *Key Decisions* - *Smart Sensors* - Will be included in 2.0 as an *early-access* feature with a clear note that *this feature might potentially change in future Airflow version with breaking changes*. - Airbnb team would be happy to help on the support side answering questions related to Smart Sensor. - Add docs around different execution modes for Sensor: *Poke mode*, *Reschedule mode* vs *Smart Sensor* - *Providers Packages* - We had a consensus on *releasing providers packages separately* mainly because of the following reasons: - Separate cadence for providers compared to Airflow, so bugs in operator/hooks can be fixed lot faster. - Enterprises generally would not like to upgrade the “core” (Scheduler) as a small bug can break the deployment and affect all the DAGs - Breaking library changes (new version of a library) can be fixed with a new version of Backport/Providers - Upgrades of backport providers are not “that” destructive i.e. even if you upgrade to a newer version and find a bug, you could go back to the previous version without causing any issues at all. - Open questions / Action Items: - How would users figure out “breaking changes” with CALVER Versioning (which is very clear with SEMVER)? - Use plugin Mechanism to: - Register Connections from an external provider to allow custom field or hide existing form fields. - Register Operator Extra links <https://airflow.apache.org/docs/stable/howto/define_extra_link.html> for operators in providers so that a change is not required in Airflow - Backport providers will only be supported/released for *three months after 2.0.0 released* - *Timeline to Airflow 2.0* - Only *critical fixes* (fixes to bugs that takedown Production system) will be backported to 1.10.x core for *six months* after Airflow 2.0 is released. Date Milestone Week of 7 Sep 2020 Create the 2.0.0-test branch While the scope is fluid, we would be rebasing this test branch from master. After we completely freeze the scope, we would only cherrypick commits from Airflow Master to v2-0-test branch if they are “in-scope”. Normal development would continue on Master branch i.e. PRs would be created against Airflow Master. Week of 28 Sep 2020 Cut Functionally complete 2.0 alpha release Week of 12 Oct 2020 Cut first 2.0 beta release Beta snapshots would be published to the Airflow Community to test and create issues to make sure Airflow is functioning and backwards compatible. Week of 19 Oct 2020 Cut bridge release based on 1.10.x - jump-off point to 2.0. Probably 1.10.13 or 1.10.14 containing upgrade check scripts for 2.0 Week of 26 Oct 2020 Cut second 2.0 beta release Week of 9 Nov 2020 Cut third 2.0 beta release Week of 23 Nov 2020 Cut first 2.0 release candidate (2.0.0rc1) *Things to Discuss Next* - *14 September (Subject to change)* - API - Progress, Current Work & Discussions - Any open questions? - Improvements to SubDags / Concept of TaskGroup - *AIP-34*: https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-34+TaskGroup%3A+A+UI+task+grouping+concept+as+an+alternative+to+SubDagOperator - *PR*: https://github.com/apache/airflow/pull/10153 - Any concerns on including this PR to 2.0 ?? - Do we want to support both TaskGroup & SubDags? - Process: - When should we defer the in-scope items to post-2.0 - Completion by a date? - Progress by a date? - Progress, Current Work & Discussions - Scheduler HA - Docs Improvements - Helm Chart - Discuss the issue with sources Regards, Kaxil
