[
https://issues.apache.org/jira/browse/GIRAPH-45?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150755#comment-13150755
]
Jakob Homan commented on GIRAPH-45:
-----------------------------------
@Jake
I'm suggesting that messages should only be spilled to disk on their target
local disk. Spilling to disk would still be the main way to relieve memory
pressure (along with combiners, of course). The question is should you be
responsible to spilling messages that you won't be processing in the next
superstep? If so, we're spending more time writing and reading the messages on
the originator (and then, possibly, writing and reading again on the
destination worker).
bq. But now this is a much more complex computation model: to send out
messages, you typically need to know what you've recieved in the previous step.
I'm afraid I don't follow. This is still consistent with the BSP model. In a
compute iteration, I can send a message to Z and it can either be held until
everyone else is done computing, or immediately sent to the worker responsible
for Z. Either way it gets stored in a container until the next superstep
starts. No actual processing is done on the message itself.
> Improve the way to keep outgoing messages
> -----------------------------------------
>
> Key: GIRAPH-45
> URL: https://issues.apache.org/jira/browse/GIRAPH-45
> Project: Giraph
> Issue Type: Improvement
> Components: bsp
> Reporter: Hyunsik Choi
> Assignee: Hyunsik Choi
>
> As discussed in GIRAPH-12(http://goo.gl/CE32U), I think that there is a
> potential problem to cause out of memory when the rate of message generation
> is higher than the rate of message flush (or network bandwidth).
> To overcome this problem, we need more eager strategy for message flushing or
> some approach to spill messages into disk.
> The below link is Dmitriy's suggestion.
> https://issues.apache.org/jira/browse/GIRAPH-12?focusedCommentId=13116253&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13116253
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
