Thanks Stephan for the proposal and a big +1 to this! I also think it's a good idea to add a link of discussion/FLIP/JIRA to each item as Zhijiang mentioned above. This would be a great help for keeping track of progress and joining in the discussion easily.
Best, Jark On Fri, 15 Feb 2019 at 11:34, jincheng sun <sunjincheng...@gmail.com> wrote: > Hi Stephan, > > Thanks for the clarification! You are right, we have never initiated a > discussion about supporting OVER Window on DataStream, we can discuss it in > a separate thread. I agree with you add the item after move the discussion > forward. > > +1 for putting the roadmap on the website. > +1 for periodically update the roadmap, as mentioned by Fabian, we can > update it at every feature version release. > > Thanks, > Jincheng > > Stephan Ewen <se...@apache.org> 于2019年2月14日周四 下午5:44写道： > >> Thanks Jincheng and Rong Rong! >> >> I am not deciding a roadmap and making a call on what features should be >> developed or not. I was only collecting broader issues that are already >> happening or have an active FLIP/design discussion plus committer support. >> >> Do we have that for the suggested issues as well? If yes , we can add >> them (can you point me to the issue/mail-thread), if not, let's try and >> move the discussion forward and add them to the roadmap overview then. >> >> Best, >> Stephan >> >> >> On Wed, Feb 13, 2019 at 6:47 PM Rong Rong <walter...@gmail.com> wrote: >> >>> Thanks Stephan for the great proposal. >>> >>> This would not only be beneficial for new users but also for >>> contributors to keep track on all upcoming features. >>> >>> I think that better window operator support can also be separately group >>> into its own category, as they affects both future DataStream API and batch >>> stream unification. >>> can we also include: >>> - OVER aggregate for DataStream API separately as @jincheng suggested. >>> - Improving sliding window operator [1] >>> >>> One more additional suggestion, can we also include a more extendable >>> security module [2,3] @shuyi and I are currently working on? >>> This will significantly improve the usability for Flink in corporate >>> environments where proprietary or 3rd-party security integration is needed. >>> >>> Thanks, >>> Rong >>> >>> >>> [1] >>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Improvement-to-Flink-Window-Operator-with-Slicing-td25750.html >>> [2] >>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-security-improvements-td21068.html >>> [3] >>> http://apache-flink-mailing-list-archive.1008284.n3.nabble.com/DISCUSS-Flink-Kerberos-Improvement-td25983.html >>> >>> >>> >>> >>> On Wed, Feb 13, 2019 at 3:39 AM jincheng sun <sunjincheng...@gmail.com> >>> wrote: >>> >>>> Very excited and thank you for launching such a great discussion, >>>> Stephan ! >>>> >>>> Here only a little suggestion that in the Batch Streaming Unification >>>> section, do we need to add an item: >>>> >>>> - Same window operators on bounded/unbounded Table API and DataStream >>>> API >>>> (currently OVER window only exists in SQL/TableAPI, DataStream API does >>>> not yet support) >>>> >>>> Best, >>>> Jincheng >>>> >>>> Stephan Ewen <se...@apache.org> 于2019年2月13日周三 下午7:21写道： >>>> >>>>> Hi all! >>>>> >>>>> Recently several contributors, committers, and users asked about >>>>> making it more visible in which way the project is currently going. >>>>> >>>>> Users and developers can track the direction by following the >>>>> discussion threads and JIRA, but due to the mass of discussions and open >>>>> issues, it is very hard to get a good overall picture. >>>>> Especially for new users and contributors, is is very hard to get a >>>>> quick overview of the project direction. >>>>> >>>>> To fix this, I suggest to add a brief roadmap summary to the homepage. >>>>> It is a bit of a commitment to keep that roadmap up to date, but I think >>>>> the benefit for users justifies that. >>>>> The Apache Beam project has added such a roadmap [1] >>>>> <https://beam.apache.org/roadmap/>, which was received very well by >>>>> the community, I would suggest to follow a similar structure here. >>>>> >>>>> If the community is in favor of this, I would volunteer to write a >>>>> first version of such a roadmap. The points I would include are below. >>>>> >>>>> Best, >>>>> Stephan >>>>> >>>>> [1] https://beam.apache.org/roadmap/ >>>>> >>>>> ======================================================== >>>>> >>>>> Disclaimer: Apache Flink is not governed or steered by any one single >>>>> entity, but by its community and Project Management Committee (PMC). This >>>>> is not a authoritative roadmap in the sense of a plan with a specific >>>>> timeline. Instead, we share our vision for the future and major >>>>> initiatives >>>>> that are receiving attention and give users and contributors an >>>>> understanding what they can look forward to. >>>>> >>>>> *Future Role of Table API and DataStream API* >>>>> - Table API becomes first class citizen >>>>> - Table API becomes primary API for analytics use cases >>>>> * Declarative, automatic optimizations >>>>> * No manual control over state and timers >>>>> - DataStream API becomes primary API for applications and data >>>>> pipeline use cases >>>>> * Physical, user controls data types, no magic or optimizer >>>>> * Explicit control over state and time >>>>> >>>>> *Batch Streaming Unification* >>>>> - Table API unification (environments) (FLIP-32) >>>>> - New unified source interface (FLIP-27) >>>>> - Runtime operator unification & code reuse between DataStream / >>>>> Table >>>>> - Extending Table API to make it convenient API for all analytical >>>>> use cases (easier mix in of UDFs) >>>>> - Same join operators on bounded/unbounded Table API and DataStream >>>>> API >>>>> >>>>> *Faster Batch (Bounded Streams)* >>>>> - Much of this comes via Blink contribution/merging >>>>> - Fine-grained Fault Tolerance on bounded data (Table API) >>>>> - Batch Scheduling on bounded data (Table API) >>>>> - External Shuffle Services Support on bounded streams >>>>> - Caching of intermediate results on bounded data (Table API) >>>>> - Extending DataStream API to explicitly model bounded streams (API >>>>> breaking) >>>>> - Add fine fault tolerance, scheduling, caching also to DataStream >>>>> API >>>>> >>>>> *Streaming State Evolution* >>>>> - Let all built-in serializers support stable evolution >>>>> - First class support for other evolvable formats (Protobuf, Thrift) >>>>> - Savepoint input/output format to modify / adjust savepoints >>>>> >>>>> *Simpler Event Time Handling* >>>>> - Event Time Alignment in Sources >>>>> - Simpler out-of-the box support in sources >>>>> >>>>> *Checkpointing* >>>>> - Consistency of Side Effects: suspend / end with savepoint (FLIP-34) >>>>> - Failed checkpoints explicitly aborted on TaskManagers (not only on >>>>> coordinator) >>>>> >>>>> *Automatic scaling (adjusting parallelism)* >>>>> - Reactive scaling >>>>> - Active scaling policies >>>>> >>>>> *Kubernetes Integration* >>>>> - Active Kubernetes Integration (Flink actively manages containers) >>>>> >>>>> *SQL Ecosystem* >>>>> - Extended Metadata Stores / Catalog / Schema Registries support >>>>> - DDL support >>>>> - Integration with Hive Ecosystem >>>>> >>>>> *Simpler Handling of Dependencies* >>>>> - Scala in the APIs, but not in the core (hide in separate class >>>>> loader) >>>>> - Hadoop-free by default >>>>> >>>>>
