[
https://issues.apache.org/jira/browse/GIRAPH-273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13445839#comment-13445839
]
Maja Kabiljo commented on GIRAPH-273:
-------------------------------------
One of the applications we are working on is having huge maps as aggregator
values.
This approach is only adding one more connection per worker, and making master
have the same number of connections workers are having. I don't see that as a
significant change, right now we have O(W^2) connections in the system and we
are adding just O(W).
I was planning on adding some additional option (as default) which would be
used in case when we have just a few small aggregators (for example having one
worker own all the aggregators). But for the big aggregators case, I think the
way described above is better than tree approach.
> Aggregators shouldn't use Zookeeper
> -----------------------------------
>
> Key: GIRAPH-273
> URL: https://issues.apache.org/jira/browse/GIRAPH-273
> Project: Giraph
> Issue Type: Improvement
> Reporter: Maja Kabiljo
> Assignee: Maja Kabiljo
>
> We use Zookeeper znodes to transfer aggregated values from workers to master
> and back. Zookeeper is supposed to be used for coordination, and it also has
> a memory limit which prevents users from having aggregators with large value
> objects. These are the reasons why we should implement aggregators gathering
> and distribution in a different way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
