[
https://issues.apache.org/jira/browse/GIRAPH-300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13434328#comment-13434328
]
Eli Reisman commented on GIRAPH-300:
------------------------------------
Can't wait to try this out, fantastic!
We have not found any hard limits on # of workers == Netty fail here, but when
any one worker for any reason starts getting Netty buffers backed up, thats
when it will happen. Sometimes its during temp partition shuffle on
INPUT_SUPERSTEP, sometimes (more rare) its during computation in a message
intensive algorithm.
I got one run in this morning to try out some code, hope to get more in over
the next few days and this just made my short list. Review forthcoming, looking
forward to it!
> Improve netty reliability with retrying failed connections, tracking
> requests, thread-safe hash partitioning
> ------------------------------------------------------------------------------------------------------------
>
> Key: GIRAPH-300
> URL: https://issues.apache.org/jira/browse/GIRAPH-300
> Project: Giraph
> Issue Type: Improvement
> Reporter: Avery Ching
> Assignee: Avery Ching
> Attachments: GIRAPH-300.patch
>
>
> * Upgrade to the most recent stable version of Netty (3.5.3.Final)
> * Try multiple connection attempts up to n failures
> * Track requests throughout the system by keeping track of the request id and
> then matching the request id to the response (minor refactoring of
> WritableRequest to make requests simpler and support the request id)
> * Improved handling of netty exceptions by dumping the exception stack to
> help debug failures
> * Fixes bug in HashWorkerPartitioner by making partitionList thread-safe
> (this causes divide by zero exceptions in real life)
> Currently, netty connection failures causes issues with more than 75 workers
> in my setup. This allows us to reach over 200+ in a reasonably reliable
> network that doesn't kill connections.
> This code passes the local Hadoop regressions and the single node Hadoop
> instance regressions. It also succeeded on large runs (200+ workers) on a
> real Hadoop cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
