[ https://issues.apache.org/jira/browse/FLINK-1618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14363036#comment-14363036 ]
ASF GitHub Bot commented on FLINK-1618: --------------------------------------- GitHub user gyfora opened a pull request: https://github.com/apache/flink/pull/485 [FLINK-1618] [streaming] Parallel time reduce This commit introduces a new critical feature for the windowing api, which allows parallel discretization and reduce over Time windows. You can merge this pull request into a Git repository by running: $ git pull https://github.com/mbalassi/flink FLINK-1618 Alternatively you can review and apply these changes as the patch at: https://github.com/apache/flink/pull/485.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit message: This closes #485 ---- commit d52a04f8e66f110406adb8976be00822155009db Author: Gyula Fora <gyf...@apache.org> Date: 2015-03-16T08:52:41Z [FLINK-1618] [streaming] Parallel time reduce ---- > Add parallel time discretisation for time-window transformations > ----------------------------------------------------------------- > > Key: FLINK-1618 > URL: https://issues.apache.org/jira/browse/FLINK-1618 > Project: Flink > Issue Type: Improvement > Components: Streaming > Reporter: Gyula Fora > Assignee: Gyula Fora > > Currently discretizers for all windowing policies including time are executed > with parallelism 1 when they define global windows. (for instance: sum of the > last 10 minutes) > While this is necessary for arbitrary policies like delta based or > user-defined policies. Some discretizers such as Time can be implemented in a > distributed fashion. > Distributed time discretisers (and other types) can be implemented in the > following way: > -The discretisers should create StreamWindow s with incrementally increasing > ID-s starting from the same value so that it is possible to merge them after > the transformation > - The partitioner for each discretizer should send the number of partitions > created to the merger (the merger should be aware of the number of > partitioners present to wait for all the information) > - Based on all the partitioning info the merger can merge the windows > properly afterwards -- This message was sent by Atlassian JIRA (v6.3.4#6332)