+1 for end of October from me as well. Cheers, Kostas
On Wed, Aug 5, 2020 at 9:59 AM Till Rohrmann <trohrm...@apache.org> wrote: > +1 for end of October from my side as well. > > Cheers, > Till > > On Tue, Aug 4, 2020 at 9:46 PM Stephan Ewen <se...@apache.org> wrote: > > > The end of October sounds good from my side, unless it collides with some > > holidays that affect many committers. > > > > Feature-wise, I believe we can definitely make good use of the time to > wrap > > up some critical threads (like finishing the FLIP-27 source efforts). > > > > So +1 to the end of October from my side. > > > > Best, > > Stephan > > > > > > On Tue, Aug 4, 2020 at 8:59 AM Robert Metzger <rmetz...@apache.org> > wrote: > > > > > Thanks a lot for commenting on the feature freeze date. > > > > > > You are raising a few good points on the timing. > > > If we have already (2 months before) concerns regarding the deadline, > > then > > > I agree that we should move it till the end of October. > > > > > > We then just need to be careful not to run into the Christmas season at > > the > > > end of December. > > > > > > If nobody objects within a few days, I'll update the feature freeze > date > > in > > > the Wiki. > > > > > > > > > On Tue, Aug 4, 2020 at 7:52 AM Kurt Young <ykt...@gmail.com> wrote: > > > > > > > Regarding setting the feature freeze date to late September, I have > > some > > > > concern that it might make > > > > the development time of 1.12 too short. > > > > > > > > One reason for this is we took too much time (about 1.5 month, from > mid > > > of > > > > May to beginning of July) > > > > for testing 1.11. It's not ideal but further squeeze the development > > time > > > > of 1.12 won't make this better. > > > > Besides, AFAIK July & August is also a popular vacation season for > > > > European. Given the fact most > > > > committers of Flink come from Europe, I think we should also take > this > > > > into consideration. > > > > > > > > It's also true that the first week of October is the national holiday > > of > > > > China, so I'm wondering whether the > > > > end of October could be a candidate feature freeze date. > > > > > > > > Best, > > > > Kurt > > > > > > > > > > > > On Tue, Jul 28, 2020 at 2:41 AM Robert Metzger <rmetz...@apache.org> > > > > wrote: > > > > > > > > > Hi all, > > > > > > > > > > Thanks a lot for the responses so far. I've put them into this Wiki > > > page: > > > > > https://cwiki.apache.org/confluence/display/FLINK/1.12+Release to > > keep > > > > > track of them. Ideally, post JIRA tickets for your feature, then > the > > > > status > > > > > will update automatically in the wiki :) > > > > > > > > > > Please keep posting features here, or add them to the Wiki yourself > > 🙏 > > > > > > > > > > @Prasanna kumar <prasannakumarram...@gmail.com>: Dynamic Auto > > Scaling > > > > is a > > > > > feature request the community is well-aware of. Till has posted > > > > > "Reactive-scaling mode" as a feature he's working on for the 1.12 > > > > release. > > > > > This work will introduce the basic building blocks and partial > > support > > > > for > > > > > the feature you are requesting. > > > > > Proper support for dynamic scaling, while maintaining Flink's high > > > > > performance (throughout, low latency) and correctness is a > difficult > > > task > > > > > that needs a lot of work. It will probably take a little bit of > time > > > till > > > > > this is fully available. > > > > > > > > > > Cheers, > > > > > Robert > > > > > > > > > > > > > > > > > > > > On Thu, Jul 23, 2020 at 2:27 PM Till Rohrmann < > trohrm...@apache.org> > > > > > wrote: > > > > > > > > > > > Thanks for being our release managers for the 1.12 release Dian & > > > > Robert! > > > > > > > > > > > > Here are some features I would like to work on for this release: > > > > > > > > > > > > # Features > > > > > > > > > > > > ## Finishing pipelined region scheduling ( > > > > > > https://issues.apache.org/jira/browse/FLINK-16430) > > > > > > With the pipelined region scheduler we want to implement a > > scheduler > > > > > which > > > > > > can serve streaming as well as batch workloads alike while being > > able > > > > to > > > > > > run jobs under constrained resources. The latter is particularly > > > > > important > > > > > > for bounded streaming jobs which, currently, are not well > > supported. > > > > > > > > > > > > ## Reactive-scaling mode > > > > > > Being able to react to newly available resources and rescaling a > > > > running > > > > > > job accordingly will make Flink's operation much easier because > > > > resources > > > > > > can then be controlled by an external tool (e.g. GCP autoscaling, > > K8s > > > > > > horizontal pod scaler, etc.). In this release we want to make a > big > > > > step > > > > > > towards this direction. As a first step we want to support the > > > > execution > > > > > of > > > > > > jobs with a parallelism which is lower than the specified > > parallelism > > > > in > > > > > > case that Flink lost a TaskManager or could not acquire enough > > > > resources. > > > > > > > > > > > > # Maintenance/Stability > > > > > > > > > > > > ## JM / TM finished task reconciliation ( > > > > > > https://issues.apache.org/jira/browse/FLINK-17075) > > > > > > This prevents the system from going out of sync if a task state > > > change > > > > > from > > > > > > the TM to the JM is lost. > > > > > > > > > > > > ## Make metrics services work with Kubernetes deployments ( > > > > > > https://issues.apache.org/jira/browse/FLINK-11127) > > > > > > Invert the direction in which the MetricFetcher connects to the > > > > > > MetricQueryFetchers. That way it will no longer be necessary to > > > expose > > > > on > > > > > > K8s for every TaskManager a port on which the MetricQueryFetcher > > > runs. > > > > > This > > > > > > will then make the deployment of Flink clusters on K8s easier. > > > > > > > > > > > > ## Handle long-blocking operations during job submission > (savepoint > > > > > > restore) (https://issues.apache.org/jira/browse/FLINK-16866) > > > > > > Submitting a Flink job can involve the interaction with external > > > > systems > > > > > > (blocking operations). Depending on the job the interactions can > > take > > > > so > > > > > > long that it exceeds the submission timeout which reports a > failure > > > on > > > > > the > > > > > > client side even though the actual submission succeeded. By > > > decoupling > > > > > the > > > > > > creation of the ExecutionGraph from the job submission, we can > make > > > the > > > > > job > > > > > > submission non-blocking which will solve this problem. > > > > > > > > > > > > ## Make IDs more intuitive to ease debugging (FLIP-118) ( > > > > > > https://issues.apache.org/jira/browse/FLINK-15679) > > > > > > By making the internal Flink IDs compositional or logging how > they > > > > belong > > > > > > together, we can make the debugging of Flink's operations much > > > easier. > > > > > > > > > > > > Cheers, > > > > > > Till > > > > > > > > > > > > > > > > > > On Thu, Jul 23, 2020 at 7:48 AM Canbin Zheng < > > felixzhen...@gmail.com > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > Thanks for bring-up this discussion, Robert! > > > > > > > Congratulations on becoming the release manager of 1.12， Dian > and > > > > > Robert > > > > > > ! > > > > > > > > > > > > > > ---------- > > > > > > > Here are some of my thoughts of the features for native > > integration > > > > > with > > > > > > > Kubernetes in Flink 1.12: > > > > > > > > > > > > > > 1. Support user-specified pod templates > > > > > > > Description: > > > > > > > The current approach of introducing new configuration > options > > > for > > > > > > each > > > > > > > aspect of pod specification a user might wish is becoming > > unwieldy, > > > > we > > > > > > have > > > > > > > to maintain more and more Flink side Kubernetes configuration > > > options > > > > > and > > > > > > > users have to learn the gap between the declarative model used > by > > > > > > > Kubernetes and the configuration model used by Flink. It's a > > great > > > > > > > improvement to allow users to specify pod templates as central > > > places > > > > > for > > > > > > > all customization needs for the jobmanager and taskmanager > pods. > > > > > > > Benefits: > > > > > > > Users can leverage many of the advanced K8s features that > the > > > > Flink > > > > > > > community does not support explicitly, such as volume mounting, > > DNS > > > > > > > configuration, pod affinity/anti-affinity setting, etc. > > > > > > > > > > > > > > 2. Support running PyFlink on Kubernetes > > > > > > > Description: > > > > > > > Support running PyFlink on Kubernetes, including session > > > cluster > > > > > and > > > > > > > application cluster. > > > > > > > Benefits: > > > > > > > Running python application in a containerized environment. > > > > > > > > > > > > > > 3. Support built-in init-Container > > > > > > > Description: > > > > > > > We need a built-in init-Container to help solve dependency > > > > > management > > > > > > > in a containerized environment, especially in the application > > mode. > > > > > > > Benefits: > > > > > > > Separate the base Flink image from dynamic dependencies. > > > > > > > > > > > > > > 4. Support accessing secured services via K8s secrets > > > > > > > Description: > > > > > > > Kubernetes Secrets > > > > > > > <https://kubernetes.io/docs/concepts/configuration/secret/> > can > > be > > > > > used > > > > > > to > > > > > > > provide credentials for a Flink application to access secured > > > > services. > > > > > > It > > > > > > > helps people who want to use a user-specified K8s Secret > through > > an > > > > > > > environment variable. > > > > > > > Benefits: > > > > > > > Improve user experience. > > > > > > > > > > > > > > 5. Support configuring replica of JobManager Deployment in > > > ZooKeeper > > > > HA > > > > > > > setups > > > > > > > Description: > > > > > > > Make the *replica* of Deployment configurable in the > > ZooKeeper > > > HA > > > > > > > setups. > > > > > > > Benefits: > > > > > > > Achieve faster failover. > > > > > > > > > > > > > > 6. Support to configure limit for CPU requirement > > > > > > > Description: > > > > > > > To leverage the Kubernetes feature of container > request/limit > > > > CPU. > > > > > > > Benefits: > > > > > > > Reduce cost. > > > > > > > > > > > > > > Regards, > > > > > > > Canbin Zheng > > > > > > > > > > > > > > Harold.Miao <miaohong...@gmail.com> 于2020年7月23日周四 下午12:44写道： > > > > > > > > > > > > > > > I'm excited to hear about this feature, very, very, very > > highly > > > > > > > encouraged > > > > > > > > > > > > > > > > > > > > > > > > Prasanna kumar <prasannakumarram...@gmail.com> 于2020年7月23日周四 > > > > > > 上午12:10写道： > > > > > > > > > > > > > > > > > Hi Flink Dev Team, > > > > > > > > > > > > > > > > > > Dynamic AutoScaling Based on the incoming data load would > be > > a > > > > > great > > > > > > > > > feature. > > > > > > > > > > > > > > > > > > We should be able have some rule say If the load increased > by > > > > 20% , > > > > > > add > > > > > > > > > extra resource should be added. > > > > > > > > > Or time based say during these peak hours the pipeline > should > > > > scale > > > > > > > > > automatically by 50%. > > > > > > > > > > > > > > > > > > This will help a lot in cost reduction. > > > > > > > > > > > > > > > > > > EMR cluster provides a similar feature for SPARK based > > > > application. > > > > > > > > > > > > > > > > > > Thanks, > > > > > > > > > Prasanna. > > > > > > > > > > > > > > > > > > On Wed, Jul 22, 2020 at 5:40 PM Robert Metzger < > > > > > rmetz...@apache.org> > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > > > Now that the 1.11 release is out, it is time to plan for > > the > > > > next > > > > > > > major > > > > > > > > > > Flink release. > > > > > > > > > > > > > > > > > > > > Some items: > > > > > > > > > > > > > > > > > > > > 1. > > > > > > > > > > > > > > > > > > > > Dian Fu and me volunteer to be the release managers > for > > > > Flink > > > > > > > 1.12. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. > > > > > > > > > > > > > > > > > > > > Timeline: We propose to stick to our approximate 4 > month > > > > > release > > > > > > > > > cycle, > > > > > > > > > > thus the release should be done by late October. Given > > > that > > > > > > > there’s > > > > > > > > a > > > > > > > > > > holiday week in China at the beginning of October, I > > > propose > > > > > to > > > > > > do > > > > > > > > the > > > > > > > > > > feature freeze on master by late September. > > > > > > > > > > > > > > > > > > > > 2. > > > > > > > > > > > > > > > > > > > > Collecting features: It would be good to have a rough > > > > overview > > > > > > of > > > > > > > > the > > > > > > > > > > features that will likely be ready to be merged by > late > > > > > > September, > > > > > > > > and > > > > > > > > > > that > > > > > > > > > > we want in the release. > > > > > > > > > > Based on the discussion, we will update the Roadmap on > > the > > > > > Flink > > > > > > > > > website > > > > > > > > > > again! > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > 1. > > > > > > > > > > > > > > > > > > > > Test instabilities and blockers: I would like to > avoid a > > > > > > situation > > > > > > > > > where > > > > > > > > > > we have many blocking issues or build instabilities at > > the > > > > > time > > > > > > of > > > > > > > > the > > > > > > > > > > feature freeze. To achieve that, we will try to check > > > every > > > > > > build > > > > > > > > > > instability within a week, to decide if it is a > blocker > > > > (make > > > > > > sure > > > > > > > > to > > > > > > > > > > use > > > > > > > > > > the “test-stability” label for those tickets!) > > > > > > > > > > Blocker issues will need to have somebody assigned > > > > > (responsible) > > > > > > > > > within > > > > > > > > > > a week, and we want to see progress on all blocker > > issues > > > > > > > > (downgrade, > > > > > > > > > > resolution, a good plan how to proceed if it is more > > > > > > complicated) > > > > > > > > > > > > > > > > > > > > 2. > > > > > > > > > > > > > > > > > > > > Quality and stability of new features: In order to > have > > a > > > > > short > > > > > > > > > feature > > > > > > > > > > freeze phase, we encourage developers to only merge > > > > > well-tested > > > > > > > and > > > > > > > > > > documented features. In our experience, the feature > > freeze > > > > > works > > > > > > > > best > > > > > > > > > if > > > > > > > > > > new features are complete, and the community can focus > > > fully > > > > > on > > > > > > > > > > addressing > > > > > > > > > > newly found bugs and voting the release. > > > > > > > > > > By having a smooth release process, the next > > merge-window > > > > for > > > > > > the > > > > > > > > next > > > > > > > > > > release will come sooner. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > Let me know what you think about our items, and share > which > > > > > > features > > > > > > > > you > > > > > > > > > > want in Flink 1.12. > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > > > > > > > > > > > > > > Robert & Dian > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > > > > > > > > > > Best Regards, > > > > > > > > Harold Miao > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
