Hello, fellow Airflowers! I wanted to start this discussion in hopes of "kicking the tires" and agreeing on a timeline for the Airflow 2.0 release.
It seems to me that our 1.10 branch is straying farther and farther from master, and with that, there are two significant consequences. Releases become harder as we are rewriting PRs for the release branch, and we are delaying releasing of already-merged features and upgrades. But I want to add _____ feature which would be so awesome in a 2.0 press release! Our community has a lot of REALLY cool projects and efforts, and yes, it would be awesome to have a shiny 2.0 release fully packed with amazing features. Still, we are sacrificing two things with this mindset: timeliness and stability. The longer we wait on releasing many of the already added features of our master branch (e.g., a more stable typing system, the ability to submit K8sExecutor pods as YAML files, etc.) the more we risk falling behind to other projects in this field. We should also consider that the 2.0 release is ALREADY going to take a monumental testing effort, and adding more features will only make that worse. With Apache Spark, some of the most significant recent features released were the native Kubernetes integrations <https://spark.apache.org/docs/2.3.0/running-on-kubernetes.html>. This major paradigm shift was a crucial feature for Spark to remain competitive in the data processing space, and the community released it in the innocuous-sounding 2.3 release. Had they waited for spark 3.0, this feature would still be unreleased. The release process for 2.0 will take a lot of work/I'm not sure we have time for it. This reason is why it is ALL the more crucial that we start this process now. This will only get harder, and we risk more bugs as master diverges farther from 1.10. So what should we do now? Spring Cleaning of Necessary 2.0 tasks A while back, we created a spring cleaning list of 2.0 necessary fixes. Let's review, triage the tickets no longer relevant, and give the highest priority to the remaining tickets. Determine Breaking Changes Of all of the significant efforts in play right now, let's determine which of those efforts are breaking. A major version bump is the only time we can reasonably expect users to accept modifying DAGs, so let's categorize those issues and prioritize them. Release API and Scheduler HA Of the remaining efforts that could potentially cause breaking changes, I believe that the new API, which replaces the “Experimental API”, and the Scheduler HA have the most potential upside for users. Let's make sure to include these in our planning for releasing 2.0. Set a Timeline for 1.10 releases One question users will ask when this release comes out is, "how long will the 1.10.x release stream continue to be enhanced?" In creating a timeline, we want to ensure that customers are not left high and dry while also preventing any form of Python 2/3 issue where users feel no urgency to update. I propose a 2 month window where the 1.10.x release stream will continue getting critical bug fixes, but not any additional features. Enhance Tests to Avoid Regressions Let’s make sure that any new commits have tests suitable for both streams while we get ready for 2.0. Let's continue our work on writing system tests. Let’s all contribute scenarios so that we can ensure that these are covered before we launch 2.0. I'm ecstatic for the next year in Apache Airflow. Everything is speeding up, and by cutting a 2.0 release, I think we can ensure that we keep up with the pace of our growing community and demand.
