[
https://issues.apache.org/jira/browse/FLINK-951?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14040622#comment-14040622
]
ASF GitHub Bot commented on FLINK-951:
--------------------------------------
GitHub user markus-h opened a pull request:
https://github.com/apache/incubator-flink/pull/36
Reworking of Iteration Synchronization, Accumulators and Aggregators
This pullrequest includes:
Iteration superstep synchronization through RPC and JobManager
Unification of Accumulators and Aggregators (removal of former Aggregators)
Adjusted Testcases accordingly
A detailed description can also be found on Jira:
https://issues.apache.org/jira/browse/FLINK-951
Currently the testcase "KMeansIterativeNepheleITCase" sometimes fails. I
think this is connected to this issue:
https://issues.apache.org/jira/browse/FLINK-909 and needs to be resolved in a
following pull request.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/markus-h/incubator-flink
aggregatorsReworkToAccumulatorsRebase4
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-flink/pull/36.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #36
----
commit 9da8aa6442861be17c7658c1dd2c2d9a6943d507
Author: Markus Holzemer <[email protected]>
Date: 2014-06-16T12:56:36Z
Iteration superstep synchronization through RPC and JobManager
Unification of Accumulators and Aggregators (removal of former Aggregators)
Adjusted Testcases accordingly
----
> Reworking of Iteration Synchronization, Accumulators and Aggregators
> --------------------------------------------------------------------
>
> Key: FLINK-951
> URL: https://issues.apache.org/jira/browse/FLINK-951
> Project: Flink
> Issue Type: Improvement
> Components: Optimizer
> Affects Versions: 0.6-incubating
> Reporter: Markus Holzemer
> Assignee: Markus Holzemer
> Labels: refactoring
> Original Estimate: 168h
> Remaining Estimate: 168h
>
> I just realized that there is no real Jira issue for the task I am currently
> working on.
> I am currently reworking a few things regarding Iteration Synchronization,
> Accumulators and Aggregators. Currently the synchronization at the end of one
> superstep is done through channel events. That makes it hard to track the
> current status of iterations. That is why I am changing this synchronization
> to use RPC calls with the JobManager, so that the JobManager manages the
> current status of all iterations.
> Currently we use Accumulators outside of iterations and Aggregators inside of
> iterations. Both have a similiar function, but a bit different interfaces and
> handling. I want to unify these two concepts. I propose that we stick in the
> future to Accumulators only. Aggregators therefore are removed and
> Accumulators are extended to cover the usecases Aggregators were used fore
> before. The switch to RPC for iterations makes it possible to also send the
> current Accumulator values at the end of each superstep, so that the
> JobManager (and thereby the webinterface) will be able to print intermediate
> accumulation results.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
