[jira] [Commented] (GIRAPH-37) Implement Netty-backed rpc solution
[ https://issues.apache.org/jira/browse/GIRAPH-37?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13271242#comment-13271242 ] Claudio Martella commented on GIRAPH-37: Hi Avery, this is super impressive, both in terms of architecture change and benchmarking results. Congratulations on the great work. I particularly welcome the per-partition inbox, which is something I was also requiring for out-of-core-messages. I'll try to review this as soon as possible, which is probably going to be during the weekend. Implement Netty-backed rpc solution --- Key: GIRAPH-37 URL: https://issues.apache.org/jira/browse/GIRAPH-37 Project: Giraph Issue Type: New Feature Reporter: Jakob Homan Assignee: Jakob Homan Attachments: GIRAPH-37-wip.patch, GIRAPH-37.patch GIRAPH-12 considered replacing the current Hadoop based rpc method with Netty, but didn't went in another direction. I think there is still value in this approach, and will also look at Finagle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-37) Implement Netty-backed rpc solution
[ https://issues.apache.org/jira/browse/GIRAPH-37?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13271245#comment-13271245 ] Avery Ching commented on GIRAPH-37: --- Thanks Claudio. Here are more results with a scaled up 10 worker setup: Hadoop RPC: hadoop jar ~/giraph-0.2-SNAPSHOT-jar-with-dependencies.jar org.apache.giraph.benchmark.PageRankBenchmark -Dgiraph.useNetty=false -w 10 -V 1000 -s 5 -e 2 -v 12/05/09 02:32:05 INFO mapred.JobClient: Giraph Timers 12/05/09 02:32:05 INFO mapred.JobClient: Total (milliseconds)=149880 12/05/09 02:32:05 INFO mapred.JobClient: Superstep 3 (milliseconds)=21575 12/05/09 02:32:05 INFO mapred.JobClient: Setup (milliseconds)=7428 12/05/09 02:32:05 INFO mapred.JobClient: Shutdown (milliseconds)=174 12/05/09 02:32:05 INFO mapred.JobClient: Vertex input superstep (milliseconds)=39558 12/05/09 02:32:05 INFO mapred.JobClient: Superstep 0 (milliseconds)=16887 12/05/09 02:32:05 INFO mapred.JobClient: Superstep 4 (milliseconds)=18613 12/05/09 02:32:05 INFO mapred.JobClient: Superstep 5 (milliseconds)=3292 12/05/09 02:32:05 INFO mapred.JobClient: Superstep 2 (milliseconds)=21313 12/05/09 02:32:05 INFO mapred.JobClient: Superstep 1 (milliseconds)=21035 Netty: hadoop jar ~/giraph-0.2-SNAPSHOT-jar-with-dependencies.jar org.apache.giraph.benchmark.PageRankBenchmark -Dgiraph.useNetty=true -w 10 -V 1000 -s 5 -e 2 -v 12/05/09 02:35:06 INFO mapred.JobClient: Giraph Timers 12/05/09 02:35:06 INFO mapred.JobClient: Total (milliseconds)=59270 12/05/09 02:35:06 INFO mapred.JobClient: Superstep 3 (milliseconds)=11827 12/05/09 02:35:06 INFO mapred.JobClient: Setup (milliseconds)=3196 12/05/09 02:35:06 INFO mapred.JobClient: Shutdown (milliseconds)=124 12/05/09 02:35:06 INFO mapred.JobClient: Vertex input superstep (milliseconds)=13130 12/05/09 02:35:06 INFO mapred.JobClient: Superstep 0 (milliseconds)=8564 12/05/09 02:35:06 INFO mapred.JobClient: Superstep 4 (milliseconds)=5540 12/05/09 02:35:06 INFO mapred.JobClient: Superstep 5 (milliseconds)=2012 12/05/09 02:35:06 INFO mapred.JobClient: Superstep 2 (milliseconds)=8601 12/05/09 02:35:06 INFO mapred.JobClient: Superstep 1 (milliseconds)=6271 These results are fairly similar to the first set (even though there are more workers). I'm pretty sure we can squeeze more performance from Netty in the future in future patches (i.e. local send optimization is missing, tuning TCP parameters, exposing more knobs to the user, etc.). Implement Netty-backed rpc solution --- Key: GIRAPH-37 URL: https://issues.apache.org/jira/browse/GIRAPH-37 Project: Giraph Issue Type: New Feature Reporter: Jakob Homan Assignee: Jakob Homan Attachments: GIRAPH-37-wip.patch, GIRAPH-37.patch GIRAPH-12 considered replacing the current Hadoop based rpc method with Netty, but didn't went in another direction. I think there is still value in this approach, and will also look at Finagle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-37) Implement Netty-backed rpc solution
[ https://issues.apache.org/jira/browse/GIRAPH-37?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13269304#comment-13269304 ] Bo Wang commented on GIRAPH-37: --- This is really good news. RPC doesn't seems very scalable. Look forward to the Netty implementation. Implement Netty-backed rpc solution --- Key: GIRAPH-37 URL: https://issues.apache.org/jira/browse/GIRAPH-37 Project: Giraph Issue Type: New Feature Reporter: Jakob Homan Assignee: Jakob Homan Attachments: GIRAPH-37-wip.patch GIRAPH-12 considered replacing the current Hadoop based rpc method with Netty, but didn't went in another direction. I think there is still value in this approach, and will also look at Finagle. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-37) Implement Netty-backed rpc solution
[ https://issues.apache.org/jira/browse/GIRAPH-37?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13107455#comment-13107455 ] Jakob Homan commented on GIRAPH-37: --- I'll take a look at Finagle. Security probably won't be in the first version, but as a veteran of the Hadoop Security Wars myself, I'll be sure it can be supported. Implement Netty-backed rpc solution --- Key: GIRAPH-37 URL: https://issues.apache.org/jira/browse/GIRAPH-37 Project: Giraph Issue Type: New Feature Reporter: Jakob Homan Assignee: Jakob Homan GIRAPH-12 considered replacing the current Hadoop based rpc method with Netty, but didn't went in another direction. I think there is still value in this approach, and will also look at Finagle. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [jira] [Commented] (GIRAPH-37) Implement Netty-backed rpc solution
Note that finagle is not thrift specific. It's rpc protocol agnostic. We can make a finagle-hadooprpc connector. Granted, the thrift implementation is pretty hardened. Actually the fact that finagle is independent of rpc frework may be another reason to use it -- flip between hadooprpc and thrift depending on whether you want performance or security. On Sep 18, 2011, at 8:08 AM, Jakob Homan (JIRA) j...@apache.org wrote: [ https://issues.apache.org/jira/browse/GIRAPH-37?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13107455#comment-13107455 ] Jakob Homan commented on GIRAPH-37: --- I'll take a look at Finagle. Security probably won't be in the first version, but as a veteran of the Hadoop Security Wars myself, I'll be sure it can be supported. Implement Netty-backed rpc solution --- Key: GIRAPH-37 URL: https://issues.apache.org/jira/browse/GIRAPH-37 Project: Giraph Issue Type: New Feature Reporter: Jakob Homan Assignee: Jakob Homan GIRAPH-12 considered replacing the current Hadoop based rpc method with Netty, but didn't went in another direction. I think there is still value in this approach, and will also look at Finagle. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-37) Implement Netty-backed rpc solution
[ https://issues.apache.org/jira/browse/GIRAPH-37?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13107165#comment-13107165 ] Jake Mannix commented on GIRAPH-37: --- We should make sure we don't all work on the same thing (note the discussion at the end of GIRAPH-12) - two at a time might be fine, but half of the developers all on RPC might be excessive. Do you want to take this one? I was going to go in and try and implement a Finagle-based solution, as it's already an async RPC-system on top of Netty, but if you're already going to look at this, I can drop what I was doing and work on something else. Implement Netty-backed rpc solution --- Key: GIRAPH-37 URL: https://issues.apache.org/jira/browse/GIRAPH-37 Project: Giraph Issue Type: New Feature Reporter: Jakob Homan Assignee: Jakob Homan GIRAPH-12 considered replacing the current Hadoop based rpc method with Netty, but didn't went in another direction. I think there is still value in this approach, and will also look at Finagle. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
