Hi Rong Rong! I would add the security / kerberos threads to the roadmap. They seem to be advanced enough in the discussions so that there is clarity what will come.
For the window operator with slicing, I would personally like to see the discussion advance and have some more clarity and consensus on the feature before adding it to the roadmap. Not having that in the first version of the roadmap does not mean there will be no activity. And when the discussion advances well in the next weeks, we can update the roadmap soon. What do you think? Best, Stephan On Thu, Feb 14, 2019 at 5:46 PM Rong Rong <walter...@gmail.com> wrote: > Hi Stephan, > > Thanks for the clarification, yes I think these issues has already been > discussed in previous mailing list threads [1,2,3]. > > I also agree that updating the "official" roadmap every release is a very > good idea to avoid frequent update. > One question I might've been a bit confusion is: are we suggesting to keep > one roadmap on the documentation site (e.g. [4]) per release, or simply > just one most up-to-date roadmap in the main website [5] ? > Just like the release notes in every release, the former will probably > provide a good tracker for users to look back at previous roadmaps as well > I am assuming. > > Thanks, > Rong > > [1] > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Improvement-to-Flink-Window-Operator-with-Slicing-td25750.html > [2] > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-security-improvements-td21068.html > [3] > http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-Kerberos-Improvement-td25983.html > > [4] https://ci.apache.org/projects/flink/flink-docs-release-1.7/ > [5] https://flink.apache.org/ > > On Thu, Feb 14, 2019 at 2:26 AM Stephan Ewen <se...@apache.org> wrote: > >> I think the website is better as well. >> >> I agree with Fabian that the wiki is not so visible, and visibility is >> the main motivation. >> This type of roadmap overview would not be updated by everyone - letting >> committers update the roadmap means the listed threads are actually >> happening at the moment. >> >> >> On Thu, Feb 14, 2019 at 11:14 AM Fabian Hueske <fhue...@gmail.com> wrote: >> >>> Hi, >>> >>> I like the idea of putting the roadmap on the website because it is much >>> more visible (and IMO more credible, obligatory) there. >>> However, I share the concerns about frequent updates. >>> >>> It think it would be great to update the "official" roadmap on the >>> website once per release (-bugfix releases), i.e., every three month. >>> We can use the wiki to collect and draft the roadmap for the next >>> update. >>> >>> Best, Fabian >>> >>> >>> Am Do., 14. Feb. 2019 um 11:03 Uhr schrieb Jeff Zhang <zjf...@gmail.com >>> >: >>> >>>> Hi Stephan, >>>> >>>> Thanks for this proposal. It is a good idea to track the roadmap. One >>>> suggestion is that it might be better to put it into wiki page first. >>>> Because it is easier to update the roadmap on wiki compared to on flink web >>>> site. And I guess we may need to update the roadmap very often at the >>>> beginning as there's so many discussions and proposals in community >>>> recently. We can move it into flink web site later when we feel it could be >>>> nailed down. >>>> >>>> Stephan Ewen <se...@apache.org> 于2019年2月14日周四 下午5:44写道： >>>> >>>>> Thanks Jincheng and Rong Rong! >>>>> >>>>> I am not deciding a roadmap and making a call on what features should >>>>> be developed or not. I was only collecting broader issues that are already >>>>> happening or have an active FLIP/design discussion plus committer support. >>>>> >>>>> Do we have that for the suggested issues as well? If yes , we can add >>>>> them (can you point me to the issue/mail-thread), if not, let's try and >>>>> move the discussion forward and add them to the roadmap overview then. >>>>> >>>>> Best, >>>>> Stephan >>>>> >>>>> >>>>> On Wed, Feb 13, 2019 at 6:47 PM Rong Rong <walter...@gmail.com> wrote: >>>>> >>>>>> Thanks Stephan for the great proposal. >>>>>> >>>>>> This would not only be beneficial for new users but also for >>>>>> contributors to keep track on all upcoming features. >>>>>> >>>>>> I think that better window operator support can also be separately >>>>>> group into its own category, as they affects both future DataStream API >>>>>> and >>>>>> batch stream unification. >>>>>> can we also include: >>>>>> - OVER aggregate for DataStream API separately as @jincheng suggested. >>>>>> - Improving sliding window operator [1] >>>>>> >>>>>> One more additional suggestion, can we also include a more extendable >>>>>> security module [2,3] @shuyi and I are currently working on? >>>>>> This will significantly improve the usability for Flink in corporate >>>>>> environments where proprietary or 3rd-party security integration is >>>>>> needed. >>>>>> >>>>>> Thanks, >>>>>> Rong >>>>>> >>>>>> >>>>>> [1] >>>>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Improvement-to-Flink-Window-Operator-with-Slicing-td25750.html >>>>>> [2] >>>>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-security-improvements-td21068.html >>>>>> [3] >>>>>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-Kerberos-Improvement-td25983.html >>>>>> >>>>>> >>>>>> >>>>>> >>>>>> On Wed, Feb 13, 2019 at 3:39 AM jincheng sun < >>>>>> sunjincheng...@gmail.com> wrote: >>>>>> >>>>>>> Very excited and thank you for launching such a great discussion, >>>>>>> Stephan ! >>>>>>> >>>>>>> Here only a little suggestion that in the Batch Streaming >>>>>>> Unification section, do we need to add an item: >>>>>>> >>>>>>> - Same window operators on bounded/unbounded Table API and >>>>>>> DataStream API >>>>>>> (currently OVER window only exists in SQL/TableAPI, DataStream API >>>>>>> does not yet support) >>>>>>> >>>>>>> Best, >>>>>>> Jincheng >>>>>>> >>>>>>> Stephan Ewen <se...@apache.org> 于2019年2月13日周三 下午7:21写道： >>>>>>> >>>>>>>> Hi all! >>>>>>>> >>>>>>>> Recently several contributors, committers, and users asked about >>>>>>>> making it more visible in which way the project is currently going. >>>>>>>> >>>>>>>> Users and developers can track the direction by following the >>>>>>>> discussion threads and JIRA, but due to the mass of discussions and >>>>>>>> open >>>>>>>> issues, it is very hard to get a good overall picture. >>>>>>>> Especially for new users and contributors, is is very hard to get a >>>>>>>> quick overview of the project direction. >>>>>>>> >>>>>>>> To fix this, I suggest to add a brief roadmap summary to the >>>>>>>> homepage. It is a bit of a commitment to keep that roadmap up to date, >>>>>>>> but >>>>>>>> I think the benefit for users justifies that. >>>>>>>> The Apache Beam project has added such a roadmap [1] >>>>>>>> <https://beam.apache.org/roadmap/>, which was received very well >>>>>>>> by the community, I would suggest to follow a similar structure here. >>>>>>>> >>>>>>>> If the community is in favor of this, I would volunteer to write a >>>>>>>> first version of such a roadmap. The points I would include are below. >>>>>>>> >>>>>>>> Best, >>>>>>>> Stephan >>>>>>>> >>>>>>>> [1] https://beam.apache.org/roadmap/ >>>>>>>> >>>>>>>> ======================================================== >>>>>>>> >>>>>>>> Disclaimer: Apache Flink is not governed or steered by any one >>>>>>>> single entity, but by its community and Project Management Committee >>>>>>>> (PMC). >>>>>>>> This is not a authoritative roadmap in the sense of a plan with a >>>>>>>> specific >>>>>>>> timeline. Instead, we share our vision for the future and major >>>>>>>> initiatives >>>>>>>> that are receiving attention and give users and contributors an >>>>>>>> understanding what they can look forward to. >>>>>>>> >>>>>>>> *Future Role of Table API and DataStream API* >>>>>>>> - Table API becomes first class citizen >>>>>>>> - Table API becomes primary API for analytics use cases >>>>>>>> * Declarative, automatic optimizations >>>>>>>> * No manual control over state and timers >>>>>>>> - DataStream API becomes primary API for applications and data >>>>>>>> pipeline use cases >>>>>>>> * Physical, user controls data types, no magic or optimizer >>>>>>>> * Explicit control over state and time >>>>>>>> >>>>>>>> *Batch Streaming Unification* >>>>>>>> - Table API unification (environments) (FLIP-32) >>>>>>>> - New unified source interface (FLIP-27) >>>>>>>> - Runtime operator unification & code reuse between DataStream / >>>>>>>> Table >>>>>>>> - Extending Table API to make it convenient API for all >>>>>>>> analytical use cases (easier mix in of UDFs) >>>>>>>> - Same join operators on bounded/unbounded Table API and >>>>>>>> DataStream API >>>>>>>> >>>>>>>> *Faster Batch (Bounded Streams)* >>>>>>>> - Much of this comes via Blink contribution/merging >>>>>>>> - Fine-grained Fault Tolerance on bounded data (Table API) >>>>>>>> - Batch Scheduling on bounded data (Table API) >>>>>>>> - External Shuffle Services Support on bounded streams >>>>>>>> - Caching of intermediate results on bounded data (Table API) >>>>>>>> - Extending DataStream API to explicitly model bounded streams >>>>>>>> (API breaking) >>>>>>>> - Add fine fault tolerance, scheduling, caching also to >>>>>>>> DataStream API >>>>>>>> >>>>>>>> *Streaming State Evolution* >>>>>>>> - Let all built-in serializers support stable evolution >>>>>>>> - First class support for other evolvable formats (Protobuf, >>>>>>>> Thrift) >>>>>>>> - Savepoint input/output format to modify / adjust savepoints >>>>>>>> >>>>>>>> *Simpler Event Time Handling* >>>>>>>> - Event Time Alignment in Sources >>>>>>>> - Simpler out-of-the box support in sources >>>>>>>> >>>>>>>> *Checkpointing* >>>>>>>> - Consistency of Side Effects: suspend / end with savepoint >>>>>>>> (FLIP-34) >>>>>>>> - Failed checkpoints explicitly aborted on TaskManagers (not only >>>>>>>> on coordinator) >>>>>>>> >>>>>>>> *Automatic scaling (adjusting parallelism)* >>>>>>>> - Reactive scaling >>>>>>>> - Active scaling policies >>>>>>>> >>>>>>>> *Kubernetes Integration* >>>>>>>> - Active Kubernetes Integration (Flink actively manages >>>>>>>> containers) >>>>>>>> >>>>>>>> *SQL Ecosystem* >>>>>>>> - Extended Metadata Stores / Catalog / Schema Registries support >>>>>>>> - DDL support >>>>>>>> - Integration with Hive Ecosystem >>>>>>>> >>>>>>>> *Simpler Handling of Dependencies* >>>>>>>> - Scala in the APIs, but not in the core (hide in separate class >>>>>>>> loader) >>>>>>>> - Hadoop-free by default >>>>>>>> >>>>>>>> >>>> >>>> -- >>>> Best Regards >>>> >>>> Jeff Zhang >>>> >>>
