Thanks Xingtong and Jark for kicking off and driving the discussion! It is really good to see we finally start talking about Flink 2.0. There are so many great ideas that require breaking API changes and so many tech debts need to be cleaned up. With the Flink 2.0 ahead, we will be more fast-paced to bring Flink to the next level. +1 for your proposal.
Best regards, Jing On Tue, Apr 25, 2023 at 3:55 PM Chesnay Schepler <ches...@apache.org> wrote: > This is definitely a good discussion so have. > > Some thoughts: > > One aspect that wasn't mentioned is what this release means going > forward. I already waited a decade for 2.0; don't really want to wait > another one to see Flink 3.0. > We should discuss how regularly we will ship major releases from now on. > Let's avoid again making breaking changes because we "gotta do it now > because 3.0 isn't happening anytime soon". > (e.g., every 2 years or something) > > Related to that we need to figure out how long 1.x will be supported and > in what way (features+patches vs only patches). > > The timeline/branch/release-manager bits sound good to me. > > > /There are also opinions that we should stay focused as much as > //possible on the breaking changes only. Incremental / > non-breaking//improvements and features, or anything that can be added > in 2.x minor ////releases, should not block the 2.0 release./ > > I would definitely agree with this. I'd much rather focus on resolving > technical debt and setting us up for improvements later than trying to > tackle both at the same time. > The "marketing perspective" of having big key features to me just > doesn't make sense considering what features we shipped with 1.x > releases in the past years. > If that means 2.0 comes along faster, then that's a bonus in my book. > We may of course ship features (e.g., Java 17 which basically comes for > free if we drop the Scala APIs), but they shouldn't be a focus. > > > /With breaking API changes, we may need multiple 2.0-alpha/beta > versions to collect feedback./ > > Personally I wouldn't even aim for a big 2.0 release. I think that will > become quiet a mess and very difficult to actually get feedback on. > My thinking goes rather in the area of defining Milestone releases, each > Milestone targeting specific changes. > For example, one milestone could cleanup the REST API (+ X,Y,Z), while > another removes deprecated APIs, etc etc. > Depending on the scope we could iterate quite fast on these. > (Note that I haven't thought this through yet from the dev workflow > perspective, but it'd likely require longer-living feature branches) > > There are some clear benefits to this approach; if we'd drop deprecated > APIs in M1 then we could already offers users a version of Flink that > works with Java 17. > > On 25/04/2023 13:09, Xintong Song wrote: > > Hi everyone, > > > > I'd like to start a discussion on planning for a Flink 2.0 release. > > > > AFAIK, in the past years this topic has been mentioned from time to time, > > in mailing lists, jira tickets and offline discussions. However, few > > concrete steps have been taken, due to the significant determination and > > efforts it requires and distractions from other prioritized focuses. > After > > a series of offline discussions in the recent weeks, with folks mostly > from > > our team internally as well as a few from outside Alibaba / Ververica > > (thanks for insights from Becket and Robert), we believe it's time to > kick > > this off in the community. > > > > Below are some of our thoughts about the 2.0 release. Looking forward to > > your opinions and feedback. > > > > > > ## Why plan for release 2.0? > > > > > > Flink 1.0.0 was released in March 2016. In the past 7 years, many new > > features have been added and the project has become different from what > it > > used to be. So what is Flink now? What will it become in the next 3-5 > > years? What do we think of Flink's position in the industry? We believe > > it's time to rethink these questions, and draw a roadmap towards another > > milestone, a milestone that worths a new major release. > > > > > > In addition, we are still providing backwards compatibility (maybe not > > perfectly but largely) with APIs that we designed and claimed stable 7 > > years ago. While such backwards compatibility helps users to stick with > the > > latest Flink releases more easily, it sometimes, and more and more over > > time, also becomes a burden for maintenance and a limitation for new > > features and improvements. It's probably time to have a comprehensive > > review and clean-up over all the public APIs. > > > > > > Furthermore, next year is the 10th year for Flink as an Apache project. > > Flink joined the Apache incubator in April 2014, and became a top-level > > project in December 2014. That makes 2024 a perfect time for bringing out > > the release 2.0 milestone. And for such a major release, we'd expect it > > takes one year or even longer to prepare for, which means we probably > > should start now. > > > > > > ## What should we focus on in release 2.0? > > > > > > - Roadmap discussion - How do we define and position Flink for now > and > > in future? This is probably something we lacked. I believe some > people have > > thought about it, but at least it's not explicitly discussed and > aligned in > > the community. Ideally, the 2.0 release should be a result of the > roadmap. > > - Breaking changes - Important improvements, bugfixes, technical > debts > > that involve breaking of API backwards compatibility, which can only > be > > carried out in major releases. > > - With breaking API changes, we may need multiple 2.0-alpha/beta > > versions to collect feedback. > > - Key features - Significant features and improvements (e.g., new > user > > stories, architectural upgrades) that may change how users use Flink > and > > its position in the industry. Some items from this category may also > > involve API breaking changes or significant behavior changes. > > - There are also opinions that we should stay focused as much as > > possible on the breaking changes only. Incremental / non-breaking > > improvements and features, or anything that can be added in 2.x > minor > > releases, should not block the 2.0 release. > > > > It might be better to discuss the detailed technical items later in > another > > thread, to keep the current discussion focused on the overall proposal, > and > > to leave time for all parties to think about their technical plans. For > > your reference, I've attached a preliminary list of work items proposed > by > > Alibaba / Ververica [1]. Note that the listed items are still being > > carefully evaluated and prioritized, and may change in future. > > > > > > ## How do we manage the release? > > > > > > #### Release Process > > > > > > We'd expect the release process for Flink 2.0 to be different from the > 1.x > > releases. > > > > > > A major difference is that, we think the timeline-based release > management > > may not be suitable. The idea behind the timeline-based approach is that > we > > can have more frequent releases and deliver completed features to users > > earlier, while incompleted features can be postponed to the next release > > which won't be too late with the short release cycle. However, for > breaking > > changes that can only take place in major releases, the price for > missing a > > release is too high. > > > > > > Alternatively, we probably should discuss and agree on a list of > must-have > > work items. That doesn't mean keep postponing the release upon a few > > delayed features. In fact, we would need to closely monitor the progress > of > > the must-have items during the entire release cycle, making sure they are > > taken care of by contributors with enough expertise and capacities. > > > > > > #### Timeline > > > > > > The release cycle should be decided according to the feature list, > > especially the must-have items that we plan to do in the release. > However, > > a target feature freeze date would still be helpful when making the plan, > > so that we don't pack too many things into the release. We propose to aim > > for a feature freeze around mid 2024, so that in case must-have items are > > delayed, we still have a good chance to make the release happen by the > end > > of 2024. > > > > > > #### Branch > > > > > > A longer release cycle also means we probably should keep shiping the 1.x > > releases while preparing for the 2.0 release. We may cut a release-1 from > > master, on which we can keep developing and release 1.x releases. The > > version on the master branch will then become '2.0-SNAPSHOT'. > > > > > > #### Release Manager > > > > > > Given the new and to-be-explored release process, longer cycle and higher > > synchronization requirements, we'd expect the 2.0 release to be more > > challenging than previous 1.x releases. Therefore, we'd like to propose > to > > assemble a release management team with 4-5 experienced PMC members. Jark > > and I would like to volunteer as 2 of the release managers. > > > > > > Looking forward to your thoughts. > > > > > > Best, > > > > Jark & Xintong > > > > > > [1] > > > https://docs.google.com/document/d/1_PMGl5RuDQGlV99_gL3y7OiRsF0DgCk91Coua6hFXhE/edit?usp=sharing > > >
