Thanks all for your opinion. @Chesnay: That is a risk, but I hope the people responsible for individual FLIPs plan accordingly. Extending the time till the feature freeze should not mean that we are extending the scope of the release. Ideally, features are done before FF, and they use the time till the freeze for additional testing and documentation polishing. This FF will be virtual, there should be less disruption than a physical conference with all the travelling. Do you have a different proposal for the timing?
I'm currently considering splitting the feature freeze and the release branch creation. Similar to the Linux kernel development, we could have a "merge window" and a stabilization phase. At the end of the stabilization phase, we cut the release branch and open the next merge window (I'll start a separate thread regarding this towards the end of this release cycle, if I still like the idea then) On Wed, Aug 5, 2020 at 12:04 PM Chesnay Schepler <ches...@apache.org> wrote: > I'm a bit concerned about end of October, because it means we have Flink > forward, which usually means at least 1 week of little-to-no activity, > and then 1 week until feature-freeze. > > On 05/08/2020 11:56, jincheng sun wrote: > > +1 for end of October from me as well. > > > > Best, > > Jincheng > > > > > > Kostas Kloudas <kklou...@gmail.com> 于2020年8月5日周三 下午4:59写道： > > > >> +1 for end of October from me as well. > >> > >> Cheers, > >> Kostas > >> > >> On Wed, Aug 5, 2020 at 9:59 AM Till Rohrmann <trohrm...@apache.org> > wrote: > >> > >>> +1 for end of October from my side as well. > >>> > >>> Cheers, > >>> Till > >>> > >>> On Tue, Aug 4, 2020 at 9:46 PM Stephan Ewen <se...@apache.org> wrote: > >>> > >>>> The end of October sounds good from my side, unless it collides with > >> some > >>>> holidays that affect many committers. > >>>> > >>>> Feature-wise, I believe we can definitely make good use of the time to > >>> wrap > >>>> up some critical threads (like finishing the FLIP-27 source efforts). > >>>> > >>>> So +1 to the end of October from my side. > >>>> > >>>> Best, > >>>> Stephan > >>>> > >>>> > >>>> On Tue, Aug 4, 2020 at 8:59 AM Robert Metzger <rmetz...@apache.org> > >>> wrote: > >>>>> Thanks a lot for commenting on the feature freeze date. > >>>>> > >>>>> You are raising a few good points on the timing. > >>>>> If we have already (2 months before) concerns regarding the deadline, > >>>> then > >>>>> I agree that we should move it till the end of October. > >>>>> > >>>>> We then just need to be careful not to run into the Christmas season > >> at > >>>> the > >>>>> end of December. > >>>>> > >>>>> If nobody objects within a few days, I'll update the feature freeze > >>> date > >>>> in > >>>>> the Wiki. > >>>>> > >>>>> > >>>>> On Tue, Aug 4, 2020 at 7:52 AM Kurt Young <ykt...@gmail.com> wrote: > >>>>> > >>>>>> Regarding setting the feature freeze date to late September, I have > >>>> some > >>>>>> concern that it might make > >>>>>> the development time of 1.12 too short. > >>>>>> > >>>>>> One reason for this is we took too much time (about 1.5 month, from > >>> mid > >>>>> of > >>>>>> May to beginning of July) > >>>>>> for testing 1.11. It's not ideal but further squeeze the > >> development > >>>> time > >>>>>> of 1.12 won't make this better. > >>>>>> Besides, AFAIK July & August is also a popular vacation season for > >>>>>> European. Given the fact most > >>>>>> committers of Flink come from Europe, I think we should also take > >>> this > >>>>>> into consideration. > >>>>>> > >>>>>> It's also true that the first week of October is the national > >> holiday > >>>> of > >>>>>> China, so I'm wondering whether the > >>>>>> end of October could be a candidate feature freeze date. > >>>>>> > >>>>>> Best, > >>>>>> Kurt > >>>>>> > >>>>>> > >>>>>> On Tue, Jul 28, 2020 at 2:41 AM Robert Metzger < > >> rmetz...@apache.org> > >>>>>> wrote: > >>>>>> > >>>>>>> Hi all, > >>>>>>> > >>>>>>> Thanks a lot for the responses so far. I've put them into this > >> Wiki > >>>>> page: > >>>>>>> https://cwiki.apache.org/confluence/display/FLINK/1.12+Release > >> to > >>>> keep > >>>>>>> track of them. Ideally, post JIRA tickets for your feature, then > >>> the > >>>>>> status > >>>>>>> will update automatically in the wiki :) > >>>>>>> > >>>>>>> Please keep posting features here, or add them to the Wiki > >> yourself > >>>> 🙏 > >>>>>>> @Prasanna kumar <prasannakumarram...@gmail.com>: Dynamic Auto > >>>> Scaling > >>>>>> is a > >>>>>>> feature request the community is well-aware of. Till has posted > >>>>>>> "Reactive-scaling mode" as a feature he's working on for the 1.12 > >>>>>> release. > >>>>>>> This work will introduce the basic building blocks and partial > >>>> support > >>>>>> for > >>>>>>> the feature you are requesting. > >>>>>>> Proper support for dynamic scaling, while maintaining Flink's > >> high > >>>>>>> performance (throughout, low latency) and correctness is a > >>> difficult > >>>>> task > >>>>>>> that needs a lot of work. It will probably take a little bit of > >>> time > >>>>> till > >>>>>>> this is fully available. > >>>>>>> > >>>>>>> Cheers, > >>>>>>> Robert > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> On Thu, Jul 23, 2020 at 2:27 PM Till Rohrmann < > >>> trohrm...@apache.org> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Thanks for being our release managers for the 1.12 release > >> Dian & > >>>>>> Robert! > >>>>>>>> Here are some features I would like to work on for this > >> release: > >>>>>>>> # Features > >>>>>>>> > >>>>>>>> ## Finishing pipelined region scheduling ( > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-16430) > >>>>>>>> With the pipelined region scheduler we want to implement a > >>>> scheduler > >>>>>>> which > >>>>>>>> can serve streaming as well as batch workloads alike while > >> being > >>>> able > >>>>>> to > >>>>>>>> run jobs under constrained resources. The latter is > >> particularly > >>>>>>> important > >>>>>>>> for bounded streaming jobs which, currently, are not well > >>>> supported. > >>>>>>>> ## Reactive-scaling mode > >>>>>>>> Being able to react to newly available resources and rescaling > >> a > >>>>>> running > >>>>>>>> job accordingly will make Flink's operation much easier because > >>>>>> resources > >>>>>>>> can then be controlled by an external tool (e.g. GCP > >> autoscaling, > >>>> K8s > >>>>>>>> horizontal pod scaler, etc.). In this release we want to make a > >>> big > >>>>>> step > >>>>>>>> towards this direction. As a first step we want to support the > >>>>>> execution > >>>>>>> of > >>>>>>>> jobs with a parallelism which is lower than the specified > >>>> parallelism > >>>>>> in > >>>>>>>> case that Flink lost a TaskManager or could not acquire enough > >>>>>> resources. > >>>>>>>> # Maintenance/Stability > >>>>>>>> > >>>>>>>> ## JM / TM finished task reconciliation ( > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-17075) > >>>>>>>> This prevents the system from going out of sync if a task state > >>>>> change > >>>>>>> from > >>>>>>>> the TM to the JM is lost. > >>>>>>>> > >>>>>>>> ## Make metrics services work with Kubernetes deployments ( > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-11127) > >>>>>>>> Invert the direction in which the MetricFetcher connects to the > >>>>>>>> MetricQueryFetchers. That way it will no longer be necessary to > >>>>> expose > >>>>>> on > >>>>>>>> K8s for every TaskManager a port on which the > >> MetricQueryFetcher > >>>>> runs. > >>>>>>> This > >>>>>>>> will then make the deployment of Flink clusters on K8s easier. > >>>>>>>> > >>>>>>>> ## Handle long-blocking operations during job submission > >>> (savepoint > >>>>>>>> restore) (https://issues.apache.org/jira/browse/FLINK-16866) > >>>>>>>> Submitting a Flink job can involve the interaction with > >> external > >>>>>> systems > >>>>>>>> (blocking operations). Depending on the job the interactions > >> can > >>>> take > >>>>>> so > >>>>>>>> long that it exceeds the submission timeout which reports a > >>> failure > >>>>> on > >>>>>>> the > >>>>>>>> client side even though the actual submission succeeded. By > >>>>> decoupling > >>>>>>> the > >>>>>>>> creation of the ExecutionGraph from the job submission, we can > >>> make > >>>>> the > >>>>>>> job > >>>>>>>> submission non-blocking which will solve this problem. > >>>>>>>> > >>>>>>>> ## Make IDs more intuitive to ease debugging (FLIP-118) ( > >>>>>>>> https://issues.apache.org/jira/browse/FLINK-15679) > >>>>>>>> By making the internal Flink IDs compositional or logging how > >>> they > >>>>>> belong > >>>>>>>> together, we can make the debugging of Flink's operations much > >>>>> easier. > >>>>>>>> Cheers, > >>>>>>>> Till > >>>>>>>> > >>>>>>>> > >>>>>>>> On Thu, Jul 23, 2020 at 7:48 AM Canbin Zheng < > >>>> felixzhen...@gmail.com > >>>>>>>> wrote: > >>>>>>>> > >>>>>>>>> Hi All, > >>>>>>>>> > >>>>>>>>> Thanks for bring-up this discussion, Robert! > >>>>>>>>> Congratulations on becoming the release manager of 1.12， Dian > >>> and > >>>>>>> Robert > >>>>>>>> ! > >>>>>>>>> ---------- > >>>>>>>>> Here are some of my thoughts of the features for native > >>>> integration > >>>>>>> with > >>>>>>>>> Kubernetes in Flink 1.12: > >>>>>>>>> > >>>>>>>>> 1. Support user-specified pod templates > >>>>>>>>> Description: > >>>>>>>>> The current approach of introducing new configuration > >>> options > >>>>> for > >>>>>>>> each > >>>>>>>>> aspect of pod specification a user might wish is becoming > >>>> unwieldy, > >>>>>> we > >>>>>>>> have > >>>>>>>>> to maintain more and more Flink side Kubernetes configuration > >>>>> options > >>>>>>> and > >>>>>>>>> users have to learn the gap between the declarative model > >> used > >>> by > >>>>>>>>> Kubernetes and the configuration model used by Flink. It's a > >>>> great > >>>>>>>>> improvement to allow users to specify pod templates as > >> central > >>>>> places > >>>>>>> for > >>>>>>>>> all customization needs for the jobmanager and taskmanager > >>> pods. > >>>>>>>>> Benefits: > >>>>>>>>> Users can leverage many of the advanced K8s features that > >>> the > >>>>>> Flink > >>>>>>>>> community does not support explicitly, such as volume > >> mounting, > >>>> DNS > >>>>>>>>> configuration, pod affinity/anti-affinity setting, etc. > >>>>>>>>> > >>>>>>>>> 2. Support running PyFlink on Kubernetes > >>>>>>>>> Description: > >>>>>>>>> Support running PyFlink on Kubernetes, including session > >>>>> cluster > >>>>>>> and > >>>>>>>>> application cluster. > >>>>>>>>> Benefits: > >>>>>>>>> Running python application in a containerized > >> environment. > >>>>>>>>> 3. Support built-in init-Container > >>>>>>>>> Description: > >>>>>>>>> We need a built-in init-Container to help solve > >> dependency > >>>>>>> management > >>>>>>>>> in a containerized environment, especially in the application > >>>> mode. > >>>>>>>>> Benefits: > >>>>>>>>> Separate the base Flink image from dynamic dependencies. > >>>>>>>>> > >>>>>>>>> 4. Support accessing secured services via K8s secrets > >>>>>>>>> Description: > >>>>>>>>> Kubernetes Secrets > >>>>>>>>> <https://kubernetes.io/docs/concepts/configuration/secret/> > >>> can > >>>> be > >>>>>>> used > >>>>>>>> to > >>>>>>>>> provide credentials for a Flink application to access secured > >>>>>> services. > >>>>>>>> It > >>>>>>>>> helps people who want to use a user-specified K8s Secret > >>> through > >>>> an > >>>>>>>>> environment variable. > >>>>>>>>> Benefits: > >>>>>>>>> Improve user experience. > >>>>>>>>> > >>>>>>>>> 5. Support configuring replica of JobManager Deployment in > >>>>> ZooKeeper > >>>>>> HA > >>>>>>>>> setups > >>>>>>>>> Description: > >>>>>>>>> Make the *replica* of Deployment configurable in the > >>>> ZooKeeper > >>>>> HA > >>>>>>>>> setups. > >>>>>>>>> Benefits: > >>>>>>>>> Achieve faster failover. > >>>>>>>>> > >>>>>>>>> 6. Support to configure limit for CPU requirement > >>>>>>>>> Description: > >>>>>>>>> To leverage the Kubernetes feature of container > >>> request/limit > >>>>>> CPU. > >>>>>>>>> Benefits: > >>>>>>>>> Reduce cost. > >>>>>>>>> > >>>>>>>>> Regards, > >>>>>>>>> Canbin Zheng > >>>>>>>>> > >>>>>>>>> Harold.Miao <miaohong...@gmail.com> 于2020年7月23日周四 下午12:44写道： > >>>>>>>>> > >>>>>>>>>> I'm excited to hear about this feature, very, very, very > >>>> highly > >>>>>>>>> encouraged > >>>>>>>>>> > >>>>>>>>>> Prasanna kumar <prasannakumarram...@gmail.com> > >> 于2020年7月23日周四 > >>>>>>>> 上午12:10写道： > >>>>>>>>>>> Hi Flink Dev Team, > >>>>>>>>>>> > >>>>>>>>>>> Dynamic AutoScaling Based on the incoming data load would > >>> be > >>>> a > >>>>>>> great > >>>>>>>>>>> feature. > >>>>>>>>>>> > >>>>>>>>>>> We should be able have some rule say If the load > >> increased > >>> by > >>>>>> 20% , > >>>>>>>> add > >>>>>>>>>>> extra resource should be added. > >>>>>>>>>>> Or time based say during these peak hours the pipeline > >>> should > >>>>>> scale > >>>>>>>>>>> automatically by 50%. > >>>>>>>>>>> > >>>>>>>>>>> This will help a lot in cost reduction. > >>>>>>>>>>> > >>>>>>>>>>> EMR cluster provides a similar feature for SPARK based > >>>>>> application. > >>>>>>>>>>> Thanks, > >>>>>>>>>>> Prasanna. > >>>>>>>>>>> > >>>>>>>>>>> On Wed, Jul 22, 2020 at 5:40 PM Robert Metzger < > >>>>>>> rmetz...@apache.org> > >>>>>>>>>>> wrote: > >>>>>>>>>>> > >>>>>>>>>>>> Hi all, > >>>>>>>>>>>> > >>>>>>>>>>>> Now that the 1.11 release is out, it is time to plan > >> for > >>>> the > >>>>>> next > >>>>>>>>> major > >>>>>>>>>>>> Flink release. > >>>>>>>>>>>> > >>>>>>>>>>>> Some items: > >>>>>>>>>>>> > >>>>>>>>>>>> 1. > >>>>>>>>>>>> > >>>>>>>>>>>> Dian Fu and me volunteer to be the release managers > >>> for > >>>>>> Flink > >>>>>>>>> 1.12. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> 1. > >>>>>>>>>>>> > >>>>>>>>>>>> Timeline: We propose to stick to our approximate 4 > >>> month > >>>>>>> release > >>>>>>>>>>> cycle, > >>>>>>>>>>>> thus the release should be done by late October. > >> Given > >>>>> that > >>>>>>>>> there’s > >>>>>>>>>> a > >>>>>>>>>>>> holiday week in China at the beginning of October, I > >>>>> propose > >>>>>>> to > >>>>>>>> do > >>>>>>>>>> the > >>>>>>>>>>>> feature freeze on master by late September. > >>>>>>>>>>>> > >>>>>>>>>>>> 2. > >>>>>>>>>>>> > >>>>>>>>>>>> Collecting features: It would be good to have a > >> rough > >>>>>> overview > >>>>>>>> of > >>>>>>>>>> the > >>>>>>>>>>>> features that will likely be ready to be merged by > >>> late > >>>>>>>> September, > >>>>>>>>>> and > >>>>>>>>>>>> that > >>>>>>>>>>>> we want in the release. > >>>>>>>>>>>> Based on the discussion, we will update the Roadmap > >> on > >>>> the > >>>>>>> Flink > >>>>>>>>>>> website > >>>>>>>>>>>> again! > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> 1. > >>>>>>>>>>>> > >>>>>>>>>>>> Test instabilities and blockers: I would like to > >>> avoid a > >>>>>>>> situation > >>>>>>>>>>> where > >>>>>>>>>>>> we have many blocking issues or build instabilities > >> at > >>>> the > >>>>>>> time > >>>>>>>> of > >>>>>>>>>> the > >>>>>>>>>>>> feature freeze. To achieve that, we will try to > >> check > >>>>> every > >>>>>>>> build > >>>>>>>>>>>> instability within a week, to decide if it is a > >>> blocker > >>>>>> (make > >>>>>>>> sure > >>>>>>>>>> to > >>>>>>>>>>>> use > >>>>>>>>>>>> the “test-stability” label for those tickets!) > >>>>>>>>>>>> Blocker issues will need to have somebody assigned > >>>>>>> (responsible) > >>>>>>>>>>> within > >>>>>>>>>>>> a week, and we want to see progress on all blocker > >>>> issues > >>>>>>>>>> (downgrade, > >>>>>>>>>>>> resolution, a good plan how to proceed if it is more > >>>>>>>> complicated) > >>>>>>>>>>>> 2. > >>>>>>>>>>>> > >>>>>>>>>>>> Quality and stability of new features: In order to > >>> have > >>>> a > >>>>>>> short > >>>>>>>>>>> feature > >>>>>>>>>>>> freeze phase, we encourage developers to only merge > >>>>>>> well-tested > >>>>>>>>> and > >>>>>>>>>>>> documented features. In our experience, the feature > >>>> freeze > >>>>>>> works > >>>>>>>>>> best > >>>>>>>>>>> if > >>>>>>>>>>>> new features are complete, and the community can > >> focus > >>>>> fully > >>>>>>> on > >>>>>>>>>>>> addressing > >>>>>>>>>>>> newly found bugs and voting the release. > >>>>>>>>>>>> By having a smooth release process, the next > >>>> merge-window > >>>>>> for > >>>>>>>> the > >>>>>>>>>> next > >>>>>>>>>>>> release will come sooner. > >>>>>>>>>>>> > >>>>>>>>>>>> > >>>>>>>>>>>> Let me know what you think about our items, and share > >>> which > >>>>>>>> features > >>>>>>>>>> you > >>>>>>>>>>>> want in Flink 1.12. > >>>>>>>>>>>> > >>>>>>>>>>>> Best, > >>>>>>>>>>>> > >>>>>>>>>>>> Robert & Dian > >>>>>>>>>>>> > >>>>>>>>>> > >>>>>>>>>> -- > >>>>>>>>>> > >>>>>>>>>> Best Regards, > >>>>>>>>>> Harold Miao > >>>>>>>>>> > >
