[jira] [Updated] (GIRAPH-441) Keep track of connected channels in NettyServer
[ https://issues.apache.org/jira/browse/GIRAPH-441?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo updated GIRAPH-441: Attachment: GIRAPH-441.patch > Keep track of connected channels in NettyServer > --- > > Key: GIRAPH-441 > URL: https://issues.apache.org/jira/browse/GIRAPH-441 > Project: Giraph > Issue Type: Bug > Reporter: Maja Kabiljo > Assignee: Maja Kabiljo > Attachments: GIRAPH-441.patch > > > We are still getting a timeout in NettyServer.stop(). We are not calling > close on all the connected channels since we are not keeping track of them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (GIRAPH-441) Keep track of connected channels in NettyServer
Maja Kabiljo created GIRAPH-441: --- Summary: Keep track of connected channels in NettyServer Key: GIRAPH-441 URL: https://issues.apache.org/jira/browse/GIRAPH-441 Project: Giraph Issue Type: Bug Reporter: Maja Kabiljo Assignee: Maja Kabiljo We are still getting a timeout in NettyServer.stop(). We are not calling close on all the connected channels since we are not keeping track of them. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-440) ProgressableUtils - TimeoutException from future.get shouldn't be rethrown
[ https://issues.apache.org/jira/browse/GIRAPH-440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13507702#comment-13507702 ] Maja Kabiljo commented on GIRAPH-440: - Thanks Avery > ProgressableUtils - TimeoutException from future.get shouldn't be rethrown > -- > > Key: GIRAPH-440 > URL: https://issues.apache.org/jira/browse/GIRAPH-440 > Project: Giraph > Issue Type: Bug > Reporter: Maja Kabiljo >Assignee: Maja Kabiljo > Attachments: GIRAPH-440.patch > > > Introduced by GIRAPH-437 - when Future.get() throws TimeoutException we > should catch it and continue waiting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-440) ProgressableUtils - TimeoutException from future.get shouldn't be rethrown
[ https://issues.apache.org/jira/browse/GIRAPH-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo updated GIRAPH-440: Attachment: GIRAPH-440.patch > ProgressableUtils - TimeoutException from future.get shouldn't be rethrown > -- > > Key: GIRAPH-440 > URL: https://issues.apache.org/jira/browse/GIRAPH-440 > Project: Giraph > Issue Type: Bug > Reporter: Maja Kabiljo > Assignee: Maja Kabiljo > Attachments: GIRAPH-440.patch > > > Introduced by GIRAPH-437 - when Future.get() throws TimeoutException we > should catch it and continue waiting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (GIRAPH-440) ProgressableUtils - TimeoutException from future.get shouldn't be rethrown
Maja Kabiljo created GIRAPH-440: --- Summary: ProgressableUtils - TimeoutException from future.get shouldn't be rethrown Key: GIRAPH-440 URL: https://issues.apache.org/jira/browse/GIRAPH-440 Project: Giraph Issue Type: Bug Reporter: Maja Kabiljo Introduced by GIRAPH-437 - when Future.get() throws TimeoutException we should catch it and continue waiting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (GIRAPH-440) ProgressableUtils - TimeoutException from future.get shouldn't be rethrown
[ https://issues.apache.org/jira/browse/GIRAPH-440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo reassigned GIRAPH-440: --- Assignee: Maja Kabiljo > ProgressableUtils - TimeoutException from future.get shouldn't be rethrown > -- > > Key: GIRAPH-440 > URL: https://issues.apache.org/jira/browse/GIRAPH-440 > Project: Giraph > Issue Type: Bug > Reporter: Maja Kabiljo > Assignee: Maja Kabiljo > > Introduced by GIRAPH-437 - when Future.get() throws TimeoutException we > should catch it and continue waiting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: GIRAPH-437: Missing progress calls when stopping Netty server
> On Nov. 30, 2012, 1:15 a.m., Avery Ching wrote: > > +1, looks good to me. I like the Waitable interface. Thanks for the review, Avery! - Maja --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/8286/#review13885 --- On Nov. 29, 2012, 11:59 p.m., Maja Kabiljo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/8286/ > --- > > (Updated Nov. 29, 2012, 11:59 p.m.) > > > Review request for giraph. > > > Description > --- > > At the end of a long running job I got an exception about not reporting > progress. The last log line was: "stop: Halting netty server", so I suspect > it's because awaitUninterruptibly() call there. > > Changing awaitUninterruptibly to periodical calls of await. Added a log line > to the end of NettyServer.stop so if it happens again we can be sure where > the problem is. I also refactored ProgressableUtils so new cases where we > need to wait for something will be easier to write. > > > This addresses bug GIRAPH-437. > https://issues.apache.org/jira/browse/GIRAPH-437 > > > Diffs > - > > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyMasterServer.java > 1415035 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyServer.java > 1415035 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerServer.java > 1415035 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspServiceMaster.java > 1415035 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/ProgressableUtils.java > 1415035 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/ConnectionTest.java > 1415035 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestFailureTest.java > 1415035 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestTest.java > 1415035 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/SaslConnectionTest.java > 1415035 > > Diff: https://reviews.apache.org/r/8286/diff/ > > > Testing > --- > > mvn verify > > > Thanks, > > Maja Kabiljo > >
Re: Review Request: GIRAPH-437: Missing progress calls when stopping Netty server
> On Nov. 30, 2012, 1:21 a.m., Nitay Joffe wrote: > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/ProgressableUtils.java, > > line 155 > > <https://reviews.apache.org/r/8286/diff/1/?file=231725#file231725line155> > > > > Why not just java Future? > > http://docs.oracle.com/javase/1.5.0/docs/api/java/util/concurrent/Future.html It has some methods we don't need and doesn't have the timeout result. So we'll have to extend it anyway. It's also nicer this way for WaitableWithoutResult. - Maja --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/8286/#review13886 ------- On Nov. 29, 2012, 11:59 p.m., Maja Kabiljo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/8286/ > --- > > (Updated Nov. 29, 2012, 11:59 p.m.) > > > Review request for giraph. > > > Description > --- > > At the end of a long running job I got an exception about not reporting > progress. The last log line was: "stop: Halting netty server", so I suspect > it's because awaitUninterruptibly() call there. > > Changing awaitUninterruptibly to periodical calls of await. Added a log line > to the end of NettyServer.stop so if it happens again we can be sure where > the problem is. I also refactored ProgressableUtils so new cases where we > need to wait for something will be easier to write. > > > This addresses bug GIRAPH-437. > https://issues.apache.org/jira/browse/GIRAPH-437 > > > Diffs > - > > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyMasterServer.java > 1415035 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyServer.java > 1415035 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerServer.java > 1415035 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspServiceMaster.java > 1415035 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/ProgressableUtils.java > 1415035 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/ConnectionTest.java > 1415035 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestFailureTest.java > 1415035 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestTest.java > 1415035 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/SaslConnectionTest.java > 1415035 > > Diff: https://reviews.apache.org/r/8286/diff/ > > > Testing > --- > > mvn verify > > > Thanks, > > Maja Kabiljo > >
[jira] [Commented] (GIRAPH-437) Missing progress calls when stopping Netty server
[ https://issues.apache.org/jira/browse/GIRAPH-437?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506960#comment-13506960 ] Maja Kabiljo commented on GIRAPH-437: - Sure, here it is: https://reviews.apache.org/r/8286/ > Missing progress calls when stopping Netty server > - > > Key: GIRAPH-437 > URL: https://issues.apache.org/jira/browse/GIRAPH-437 > Project: Giraph > Issue Type: Improvement > Reporter: Maja Kabiljo > Assignee: Maja Kabiljo > Attachments: GIRAPH-437.patch > > > At the end of a long running job I got an exception about not reporting > progress. The last log line was: "stop: Halting netty server", so I suspect > it's because awaitUninterruptibly() call there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request: GIRAPH-437: Missing progress calls when stopping Netty server
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/8286/ --- Review request for giraph. Description --- At the end of a long running job I got an exception about not reporting progress. The last log line was: "stop: Halting netty server", so I suspect it's because awaitUninterruptibly() call there. Changing awaitUninterruptibly to periodical calls of await. Added a log line to the end of NettyServer.stop so if it happens again we can be sure where the problem is. I also refactored ProgressableUtils so new cases where we need to wait for something will be easier to write. This addresses bug GIRAPH-437. https://issues.apache.org/jira/browse/GIRAPH-437 Diffs - http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyMasterServer.java 1415035 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyServer.java 1415035 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerServer.java 1415035 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspServiceMaster.java 1415035 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/ProgressableUtils.java 1415035 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/ConnectionTest.java 1415035 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestFailureTest.java 1415035 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestTest.java 1415035 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/SaslConnectionTest.java 1415035 Diff: https://reviews.apache.org/r/8286/diff/ Testing --- mvn verify Thanks, Maja Kabiljo
[jira] [Commented] (GIRAPH-424) Fix hashCode modulo computation
[ https://issues.apache.org/jira/browse/GIRAPH-424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13506746#comment-13506746 ] Maja Kabiljo commented on GIRAPH-424: - vertexId.hashCode() can return Integer.MIN_VALUE and Math.abs(Integer.MIN_VALUE) returns Integer.MIN_VALUE. When we do that % something, we get a negative value. > Fix hashCode modulo computation > --- > > Key: GIRAPH-424 > URL: https://issues.apache.org/jira/browse/GIRAPH-424 > Project: Giraph > Issue Type: Bug > Reporter: Maja Kabiljo > Assignee: Maja Kabiljo > Attachments: GIRAPH-424.patch > > > hashCode() can return Integer.MIN_VALUE, and on several places we do > Math.abs(hashCode), expecting to get a positive value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (GIRAPH-424) Fix hashCode modulo computation
[ https://issues.apache.org/jira/browse/GIRAPH-424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo reassigned GIRAPH-424: --- Assignee: Maja Kabiljo > Fix hashCode modulo computation > --- > > Key: GIRAPH-424 > URL: https://issues.apache.org/jira/browse/GIRAPH-424 > Project: Giraph > Issue Type: Bug > Reporter: Maja Kabiljo > Assignee: Maja Kabiljo > Attachments: GIRAPH-424.patch > > > hashCode() can return Integer.MIN_VALUE, and on several places we do > Math.abs(hashCode), expecting to get a positive value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (GIRAPH-437) Missing progress calls when stopping Netty server
Maja Kabiljo created GIRAPH-437: --- Summary: Missing progress calls when stopping Netty server Key: GIRAPH-437 URL: https://issues.apache.org/jira/browse/GIRAPH-437 Project: Giraph Issue Type: Improvement Reporter: Maja Kabiljo Attachments: GIRAPH-437.patch At the end of a long running job I got an exception about not reporting progress. The last log line was: "stop: Halting netty server", so I suspect it's because awaitUninterruptibly() call there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (GIRAPH-437) Missing progress calls when stopping Netty server
[ https://issues.apache.org/jira/browse/GIRAPH-437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo reassigned GIRAPH-437: --- Assignee: Maja Kabiljo > Missing progress calls when stopping Netty server > - > > Key: GIRAPH-437 > URL: https://issues.apache.org/jira/browse/GIRAPH-437 > Project: Giraph > Issue Type: Improvement > Reporter: Maja Kabiljo > Assignee: Maja Kabiljo > Attachments: GIRAPH-437.patch > > > At the end of a long running job I got an exception about not reporting > progress. The last log line was: "stop: Halting netty server", so I suspect > it's because awaitUninterruptibly() call there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-437) Missing progress calls when stopping Netty server
[ https://issues.apache.org/jira/browse/GIRAPH-437?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo updated GIRAPH-437: Attachment: GIRAPH-437.patch Changing awaitUninterruptibly to periodical calls of await. Added a log line to the end of NettyServer.stop so if it happens again we can be sure where the problem is. I also refactored ProgressableUtils so new cases where we need to wait for something will be easier to write. Passes mvn verify. > Missing progress calls when stopping Netty server > - > > Key: GIRAPH-437 > URL: https://issues.apache.org/jira/browse/GIRAPH-437 > Project: Giraph > Issue Type: Improvement > Reporter: Maja Kabiljo > Assignee: Maja Kabiljo > Attachments: GIRAPH-437.patch > > > At the end of a long running job I got an exception about not reporting > progress. The last log line was: "stop: Halting netty server", so I suspect > it's because awaitUninterruptibly() call there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: GIRAPH-435: Serialize server messages for memory and less GC
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/8140/#review13777 --- Thank you Avery, I'm +1 on this now. http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerServer.java <https://reviews.apache.org/r/8140/#comment29453> Change to ByteArrayMessagesPerVertexStore - Maja Kabiljo On Nov. 27, 2012, 7:52 a.m., Avery Ching wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/8140/ > --- > > (Updated Nov. 27, 2012, 7:52 a.m.) > > > Review request for giraph. > > > Description > --- > > This patch serializes messages on the server into byte[] and also reduces the > amount of objects being created by copying a ByteArrayVertexIdMessages object > to a MessageStore and using a representative message when doing computation > of the vertices. > > - Added new helper classes ByteArrayIterable, ByteArrayIterator, > RepresentativeByteArrayIterable, and RepresentativeByteArrayIterator to help > create objects only when necessary > - Removed all collection methods from SimpleMessageStore and its children > - ByteArrayMessagesPerVertexStore stores messages on the server in serialized > byte[] through ExtendedDataOutput (replaces > CollectionOfMessagesPerVertexStore, which used objects) > - Iterators from ByteArrayVertexIdMessages are specialized to reduce the > number of objects created (only when necessary) > - Allow the choice of message serialization with encoded message size (better > for complex message objects) or just the message serialized (better for > simple messages) > > Results: 10 workers, 2.5 B edges, 10 m vertices, page rank > > When not using the combiner, compute time on superstep 0 dropped from 12.05 > -> 10.11, other supersteps improved accordingly as well. Free memory after > superstep 2 improved from 26727.61 M -> 53102.54 M. GC time went down from > 1,213.502 -> 518.597 (more than half). Overall application time dropped from > 353.085 -> 244.434 seconds. > > When using the combiner, there are still some modest gains due to the > reduction in objects created (35 -> 29 seconds for superstep 0). > > Trunk (no combiner): > > INFO2012-11-16 17:57:35,036 [compute-23] > org.apache.giraph.graph.ComputeCallable - call: Computation took 12.054391 > secs for 1 partitions on superstep 0. Flushing started > INFO2012-11-16 17:57:35,081 [main] > org.apache.giraph.graph.BspServiceWorker - finishSuperstep: Waiting on all > requests, superstep 0 totalMem = 81728.6875M, maxMem = 81728.6875M, freeMem = > 61302.61231994629M > INFO2012-11-16 17:59:04,322 [compute-13] > org.apache.giraph.graph.ComputeCallable - call: Computation took 12.514889 > secs for 1 partitions on superstep 1. Flushing started > INFO2012-11-16 17:59:04,369 [main] > org.apache.giraph.graph.BspServiceWorker - finishSuperstep: Waiting on all > requests, superstep 1 totalMem = 81728.6875M, maxMem = 81728.6875M, freeMem = > 34318.67655181885M > INFO2012-11-16 18:00:29,559 [compute-10] > org.apache.giraph.graph.ComputeCallable - call: Computation took 11.869164 > secs for 1 partitions on superstep 2. Flushing started > INFO2012-11-16 18:00:29,602 [main] > org.apache.giraph.graph.BspServiceWorker - finishSuperstep: Waiting on all > requests, superstep 2 totalMem = 81728.6875M, maxMem = 81728.6875M, freeMem = > 26727.610847473145M > INFO2012-11-16 18:01:45,788 [compute-5] > org.apache.giraph.graph.ComputeCallable - call: Computation took 1.5657606 > secs for 1 partitions on superstep 3. Flushing started > INFO2012-11-16 18:01:45,789 [main] > org.apache.giraph.graph.BspServiceWorker - finishSuperstep: Waiting on all > requests, superstep 3 totalMem = 81728.6875M, maxMem = 81728.6875M, freeMem = > 38338.96612548828M > > Giraph Timers Total (milliseconds)353,085 0 353,085 > Superstep 3 (milliseconds) 4,694 0 4,694 > Setup (milliseconds)2,354 0 2,354 > Vertex input superstep (milliseconds) 71,891 0 71,891 > Shutdown (milliseconds) 15,104 0 15,104 > Superstep 0 (milliseconds) 85,516 0 85,516 > Superstep 2 (milliseconds) 87,720 0 87,720 > Superstep 1 (milliseconds) 85,803 0 85,803 > Max he
Re: Review Request: GIRAPH-435: Serialize server messages for memory and less GC
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/8140/#review13679 --- Thanks for fixing it! This is for some future patch, but just to put it out there. Having messages in serialized format can help us better tune the moment when we want to go out-of-core - now we actually know how much bytes are the messages in memory taking. So we don't need to use the number of messages as the limit anymore. Maybe Alessandro can tell more, but I guess the similar thing can be applied to partitions, when we use the serialized format. http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/ByteArrayMessagesPerVertexStore.java <https://reviews.apache.org/r/8140/#comment29309> Can we use getExtendedDataOutput here also? http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/DiskBackedMessageStore.java <https://reviews.apache.org/r/8140/#comment29315> Here we are keeping the message store locked while we do disk i/o. Can we do better? http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/CollectionUtils.java <https://reviews.apache.org/r/8140/#comment29319> Probably not important, but this won't work correctly if we have duplicates in the iterable. Maybe just add a comment. - Maja Kabiljo On Nov. 21, 2012, 10:01 p.m., Avery Ching wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/8140/ > --- > > (Updated Nov. 21, 2012, 10:01 p.m.) > > > Review request for giraph. > > > Description > --- > > This patch serializes messages on the server into byte[] and also reduces the > amount of objects being created by copying a ByteArrayVertexIdMessages object > to a MessageStore and using a representative message when doing computation > of the vertices. > > - Added new helper classes ByteArrayIterable, ByteArrayIterator, > RepresentativeByteArrayIterable, and RepresentativeByteArrayIterator to help > create objects only when necessary > - Removed all collection methods from SimpleMessageStore and its children > - ByteArrayMessagesPerVertexStore stores messages on the server in serialized > byte[] through ExtendedDataOutput (replaces > CollectionOfMessagesPerVertexStore, which used objects) > - Iterators from ByteArrayVertexIdMessages are specialized to reduce the > number of objects created (only when necessary) > - Allow the choice of message serialization with encoded message size (better > for complex message objects) or just the message serialized (better for > simple messages) > > Results: 10 workers, 2.5 B edges, 10 m vertices, page rank > > When not using the combiner, compute time on superstep 0 dropped from 12.05 > -> 10.11, other supersteps improved accordingly as well. Free memory after > superstep 2 improved from 26727.61 M -> 53102.54 M. GC time went down from > 1,213.502 -> 518.597 (more than half). Overall application time dropped from > 353.085 -> 244.434 seconds. > > When using the combiner, there are still some modest gains due to the > reduction in objects created (35 -> 29 seconds for superstep 0). > > Trunk (no combiner): > > INFO2012-11-16 17:57:35,036 [compute-23] > org.apache.giraph.graph.ComputeCallable - call: Computation took 12.054391 > secs for 1 partitions on superstep 0. Flushing started > INFO2012-11-16 17:57:35,081 [main] > org.apache.giraph.graph.BspServiceWorker - finishSuperstep: Waiting on all > requests, superstep 0 totalMem = 81728.6875M, maxMem = 81728.6875M, freeMem = > 61302.61231994629M > INFO2012-11-16 17:59:04,322 [compute-13] > org.apache.giraph.graph.ComputeCallable - call: Computation took 12.514889 > secs for 1 partitions on superstep 1. Flushing started > INFO2012-11-16 17:59:04,369 [main] > org.apache.giraph.graph.BspServiceWorker - finishSuperstep: Waiting on all > requests, superstep 1 totalMem = 81728.6875M, maxMem = 81728.6875M, freeMem = > 34318.67655181885M > INFO2012-11-16 18:00:29,559 [compute-10] > org.apache.giraph.graph.ComputeCallable - call: Computation took 11.869164 > secs for 1 partitions on superstep 2. Flushing started > INFO2012-11-16 18:00:29,602 [main] > org.apache.giraph.graph.BspServiceWorker - finishSuperstep: Waiting on all > requests, superstep 2 totalMem = 81728.6875M, maxMem = 81728.6875M, freeMem = > 26727.610847473145M > INFO2012-11-16
Re: Review Request: GIRAPH-435: Serialize server messages for memory and less GC
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/8140/#review13630 --- Great results! I haven't gone through the whole patch yet, I'll do it tonight/tomorrow, but here are some comments about the part I looked at. http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/ByteArrayMessagesPerVertexStore.java <https://reviews.apache.org/r/8140/#comment29261> Please create a function to do this http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/DiskBackedMessageStore.java <https://reviews.apache.org/r/8140/#comment29266> We can't do this. The whole out-of-core messaging gets broken, but the tests pass because we haven't gone out-of-core (which happened because my other comments). The whole idea of out-of-core messaging was to write them to the file in the order of ids so we could read the files sequentially later. For the future, probably we should add to the test to somehow ensure we are actually going out-of-core. http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/DiskBackedMessageStore.java <https://reviews.apache.org/r/8140/#comment29264> We have to update numberOfMessagesInMemory here. http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/DiskBackedMessageStoreByPartition.java <https://reviews.apache.org/r/8140/#comment29265> Call checkMemory() in the end http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/SequentialFileMessageStore.java <https://reviews.apache.org/r/8140/#comment29258> Write the number of messages? http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/ComputeCallable.java <https://reviews.apache.org/r/8140/#comment29225> You can use Iterables.isEmpty - Maja Kabiljo On Nov. 20, 2012, 10:18 a.m., Avery Ching wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/8140/ > --- > > (Updated Nov. 20, 2012, 10:18 a.m.) > > > Review request for giraph. > > > Description > --- > > This patch serializes messages on the server into byte[] and also reduces the > amount of objects being created by copying a ByteArrayVertexIdMessages object > to a MessageStore and using a representative message when doing computation > of the vertices. > > - Added new helper classes ByteArrayIterable, ByteArrayIterator, > RepresentativeByteArrayIterable, and RepresentativeByteArrayIterator to help > create objects only when necessary > - Removed all collection methods from SimpleMessageStore and its children > - ByteArrayMessagesPerVertexStore stores messages on the server in serialized > byte[] through ExtendedDataOutput (replaces > CollectionOfMessagesPerVertexStore, which used objects) > - Iterators from ByteArrayVertexIdMessages are specialized to reduce the > number of objects created (only when necessary) > - Allow the choice of message serialization with encoded message size (better > for complex message objects) or just the message serialized (better for > simple messages) > > Results: 10 workers, 2.5 B edges, 10 m vertices, page rank > > When not using the combiner, compute time on superstep 0 dropped from 12.05 > -> 10.11, other supersteps improved accordingly as well. Free memory after > superstep 2 improved from 26727.61 M -> 53102.54 M. GC time went down from > 1,213.502 -> 518.597 (more than half). Overall application time dropped from > 353.085 -> 244.434 seconds. > > When using the combiner, there are still some modest gains due to the > reduction in objects created (35 -> 29 seconds for superstep 0). > > Trunk (no combiner): > > INFO2012-11-16 17:57:35,036 [compute-23] > org.apache.giraph.graph.ComputeCallable - call: Computation took 12.054391 > secs for 1 partitions on superstep 0. Flushing started > INFO2012-11-16 17:57:35,081 [main] > org.apache.giraph.graph.BspServiceWorker - finishSuperstep: Waiting on all > requests, superstep 0 totalMem = 81728.6875M, maxMem = 81728.6875M, freeMem = > 61302.61231994629M > INFO2012-11-16 17:59:04,322 [compute-13] > org.apache.giraph.graph.ComputeCallable - call: Computation took 12.514889 > secs for 1 partitions on superstep 1. Flushing started > INFO2012-11-16 17:59:04,369 [main] > org.apache.giraph.graph.BspServ
Re: Review Request: GIRAPH-421: Aggregate metrics up to master
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/8042/#review13602 --- Looks good now. - Maja Kabiljo On Nov. 20, 2012, 12:08 a.m., Nitay Joffe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/8042/ > --- > > (Updated Nov. 20, 2012, 12:08 a.m.) > > > Review request for giraph. > > > Description > --- > > https://issues.apache.org/jira/browse/GIRAPH-421 > > The workers send their metrics to the master by writing them to the ZK node > that is already used for FinishedSuperstepStats. > The master reads each worker's metrics, aggregates, and prints an overall > summary. > > Worker: > https://gist.github.com/f0b6dce82a793173866a > Prints a summary at the end of each superstep. At the end of the job it does > a raw dump of all of the metrics. > > Master: > https://gist.github.com/99453c87bd33488588d4 > At end of each superstep it gathers the metrics and computes mean, min, and > max. Anything else we should compute? > There is also support for the master to track and print its own metrics, but > this is not really used right now. > > As a part of this diff I also cleaned some things up, namely: > - Just one option now to use, called giraph.metrics.enable, which toggles > everything. When false (default), no work is done. When enabled, every worker > and the master tracks metrics. They are aggregated and printed as in examples > below. > - Changed metrics that are often small values like "time to first message" > and "waiting time" to be in microseconds instead of milliseconds. > > > Diffs > - > > /trunk/.reviewboardrc 1411478 > /trunk/giraph/src/main/java/org/apache/giraph/GiraphConfiguration.java > 1411478 > /trunk/giraph/src/main/java/org/apache/giraph/comm/netty/ByteCounter.java > 1411478 > > /trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java > 1411478 > > /trunk/giraph/src/main/java/org/apache/giraph/comm/netty/handler/RequestDecoder.java > 1411478 > > /trunk/giraph/src/main/java/org/apache/giraph/comm/netty/handler/RequestEncoder.java > 1411478 > > /trunk/giraph/src/main/java/org/apache/giraph/comm/netty/handler/RequestInfo.java > 1411478 > > /trunk/giraph/src/main/java/org/apache/giraph/comm/netty/handler/RequestServerHandler.java > 1411478 > /trunk/giraph/src/main/java/org/apache/giraph/graph/BspService.java 1411478 > /trunk/giraph/src/main/java/org/apache/giraph/graph/BspServiceMaster.java > 1411478 > /trunk/giraph/src/main/java/org/apache/giraph/graph/BspServiceWorker.java > 1411478 > /trunk/giraph/src/main/java/org/apache/giraph/graph/ComputeCallable.java > 1411478 > > /trunk/giraph/src/main/java/org/apache/giraph/graph/EdgeInputSplitsCallable.java > 1411478 > /trunk/giraph/src/main/java/org/apache/giraph/graph/GraphMapper.java > 1411478 > > /trunk/giraph/src/main/java/org/apache/giraph/graph/InputSplitsCallable.java > 1411478 > > /trunk/giraph/src/main/java/org/apache/giraph/graph/VertexInputSplitsCallable.java > 1411478 > /trunk/giraph/src/main/java/org/apache/giraph/metrics/AggregatedMetric.java > PRE-CREATION > > /trunk/giraph/src/main/java/org/apache/giraph/metrics/AggregatedMetrics.java > PRE-CREATION > /trunk/giraph/src/main/java/org/apache/giraph/metrics/GiraphMetrics.java > 1411478 > > /trunk/giraph/src/main/java/org/apache/giraph/metrics/GiraphMetricsRegistry.java > 1411478 > /trunk/giraph/src/main/java/org/apache/giraph/metrics/GiraphTimer.java > PRE-CREATION > > /trunk/giraph/src/main/java/org/apache/giraph/metrics/GiraphTimerContext.java > PRE-CREATION > /trunk/giraph/src/main/java/org/apache/giraph/metrics/LongAndTimeUnit.java > PRE-CREATION > /trunk/giraph/src/main/java/org/apache/giraph/metrics/MetricGroup.java > 1411478 > > /trunk/giraph/src/main/java/org/apache/giraph/metrics/MetricsRegistryDebugger.java > PRE-CREATION > > /trunk/giraph/src/main/java/org/apache/giraph/metrics/SuperstepMetricsRegistry.java > 1411478 > /trunk/giraph/src/main/java/org/apache/giraph/metrics/ValueGauge.java > 1411478 > > /trunk/giraph/src/main/java/org/apache/giraph/metrics/ValueWithHostname.java > PRE-CREATION > > /trunk/giraph/src/main/java/org/apache/giraph/metrics/WorkerSuperstepMetrics.java > PRE-CREATIO
[jira] [Updated] (GIRAPH-429) Number of input split threads set to 1 less than necessary
[ https://issues.apache.org/jira/browse/GIRAPH-429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo updated GIRAPH-429: Attachment: GIRAPH-429.patch > Number of input split threads set to 1 less than necessary > -- > > Key: GIRAPH-429 > URL: https://issues.apache.org/jira/browse/GIRAPH-429 > Project: Giraph > Issue Type: Bug > Reporter: Maja Kabiljo > Assignee: Maja Kabiljo >Priority: Minor > Attachments: GIRAPH-429.patch > > > If number of workers is not a factor of number of input splits, we are > decreasing the number of input split threads more than we should. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (GIRAPH-429) Number of input split threads set to 1 less than necessary
Maja Kabiljo created GIRAPH-429: --- Summary: Number of input split threads set to 1 less than necessary Key: GIRAPH-429 URL: https://issues.apache.org/jira/browse/GIRAPH-429 Project: Giraph Issue Type: Bug Reporter: Maja Kabiljo Assignee: Maja Kabiljo Priority: Minor Attachments: GIRAPH-429.patch If number of workers is not a factor of number of input splits, we are decreasing the number of input split threads more than we should. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: GIRAPH-421: Aggregate metrics up to master
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/8042/#review13535 --- This is a very useful thing to have, thanks Nitay. Is it possible to keep the timing logic together? (this mostly exists from the previous metrics patch, but I didn't take a look at it back then) I think that ideally we would have some timer class, which would keep the time unit used and its name within itself. It would also have some functions like start() and stop(), first of which would be saving the current time, and the second would be calculating the time elapsed. I think that's nicer than having time measurement code everywhere. What do you think? /trunk/giraph/src/main/java/org/apache/giraph/graph/GraphMapper.java <https://reviews.apache.org/r/8042/#comment29021> Revisit the comments - Maja Kabiljo On Nov. 16, 2012, 8:18 p.m., Nitay Joffe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/8042/ > --- > > (Updated Nov. 16, 2012, 8:18 p.m.) > > > Review request for giraph. > > > Description > --- > > GIRAPH-421: Aggregate metrics up to master > > > Diffs > - > > /trunk/giraph/src/main/java/org/apache/giraph/GiraphConfiguration.java > 1409973 > /trunk/giraph/src/main/java/org/apache/giraph/graph/BspService.java 1409973 > /trunk/giraph/src/main/java/org/apache/giraph/graph/BspServiceMaster.java > 1409973 > /trunk/giraph/src/main/java/org/apache/giraph/graph/BspServiceWorker.java > 1409973 > /trunk/giraph/src/main/java/org/apache/giraph/graph/GraphMapper.java > 1409973 > /trunk/giraph/src/main/java/org/apache/giraph/metrics/AggregatedMetric.java > PRE-CREATION > > /trunk/giraph/src/main/java/org/apache/giraph/metrics/AggregatedMetrics.java > PRE-CREATION > /trunk/giraph/src/main/java/org/apache/giraph/metrics/GiraphMetrics.java > 1409973 > > /trunk/giraph/src/main/java/org/apache/giraph/metrics/GiraphMetricsRegistry.java > 1409973 > > /trunk/giraph/src/main/java/org/apache/giraph/metrics/MetricsRegistryDebugger.java > PRE-CREATION > > /trunk/giraph/src/main/java/org/apache/giraph/metrics/SuperstepMetricsRegistry.java > 1409973 > > /trunk/giraph/src/main/java/org/apache/giraph/metrics/ValueWithHostname.java > PRE-CREATION > > /trunk/giraph/src/main/java/org/apache/giraph/metrics/WorkerSuperstepMetrics.java > PRE-CREATION > /trunk/giraph/src/main/java/org/apache/giraph/utils/FakeTime.java 1409973 > /trunk/giraph/src/main/java/org/apache/giraph/utils/SystemTime.java 1409973 > /trunk/giraph/src/main/java/org/apache/giraph/utils/Time.java 1409973 > /trunk/giraph/src/main/java/org/apache/giraph/utils/Times.java 1409973 > /trunk/giraph/src/main/java/org/apache/giraph/utils/WritableUtils.java > 1409973 > > Diff: https://reviews.apache.org/r/8042/diff/ > > > Testing > --- > > > Thanks, > > Nitay Joffe > >
Re: Review Request: GIRAPH-417: Serialize the graph/message cache into byte[] for improving memory usage and compute speed
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7990/#review13481 --- Ship it! Great work, +1. Looking forward to see further improvements on this field! - Maja Kabiljo On Nov. 14, 2012, 3:06 a.m., Avery Ching wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/7990/ > --- > > (Updated Nov. 14, 2012, 3:06 a.m.) > > > Review request for giraph. > > > Description > --- > > Our entire graph is currently stored as Java objects in memory. I added an > option to keep only a representative vertex that serializes/deserializes on > the fly and should be used with the new ByteArrayPartition. In conjunction > with a serialized client-side message cache, memory usage then loading > shrinks to almost 1/10 of trunk and loads the input splits almost 3x faster > (see input superstep times below). I added a serializer based on Sun's > unsafe methods that enables this memory savings with a very small performance > hit (maybe a few 1-5% slower). Compared to trunk, when serializing the > messages with our faster serializer, compute time improves significantly as > well against trunk (16.7 -> 12.31 for 2.5B edges, 2.97 -> 1.61 for 250M > edges). There are still further improvements to be made on the server side > where we still store our messages in-memory. I (or someone else) can do that > in a later patch. This also significantly reduces GC time, as there are less > objects to GC. > > - Improves byte[] serialization signficantly > -- Added ExtendedDataInput/ExtendedDataOutput interfaces to allow for some > additional methods needed for byte[] serialization/deserialization > -- Add ExtendedByteArrayDataInput/ExtendedByteArrayDataoutput to > serialize/deserialize Writables to a byte[] > -- Added DynamicChannelBufferOutputStream/DynamicChannelBufferInputStream to > serialize/deserialize Writables to a DynamicChannelBuffer > > - Gives you the choice of partition implementation (SimplePartition (default) > or ByteArrayPartition -> (serialized vertices)) > -- Added a new method to Partition called saveVertex(), which also the > serialization back into the ByteArrayPartition or does nothing when using > SimplePartition > - Gives you the choice of unsafe serialization (using Sun's unsafe class - > default) or regular serialization > - Serializes the messages on the client cache into byte[] (saves memory and > also serializes faster) > -- Created new ByteArrayVertexIdMessageCollection to support the serialized > messages > -- SendVertexRequest now sends Partition objects rather than collections > - Adds 2 more options in PageRankBenchmark to try out RepresentationVertex or > RepresentationVertex with unsafe serialization > - Fixed a bug in LongDoubleFloatDoubleVertex's readFields when edges aren't > cleared before deserializing > > - Added new unittests > -- Replaced TestEdgeListVertex with TestMutableVertex to test all our generic > MutableVertex implementations > --- Added more serialization tests of different serialization > -- TestPartitionStores has more testing of unsafe > serialization/deserialization > > > This addresses bug GIRAPH-417. > https://issues.apache.org/jira/browse/GIRAPH-417 > > > Diffs > - > > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/GiraphConfiguration.java > 1408926 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/ImmutableClassesGiraphConfiguration.java > 1408926 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/benchmark/PageRankBenchmark.java > 1408926 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/benchmark/RepresentativeVertexPageRankBenchmark.java > PRE-CREATION > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/SendMessageCache.java > 1408926 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/SendMutationsCache.java > 1408926 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/SendPartitionCache.java > 1408926 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/VertexIdMessageCollection.java > 1408926 > > http://svn.apache.org/repos/asf/gira
Re: Review Request: GIRAPH-417: Serialize the graph/message cache into byte[] for improving memory usage and compute speed
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7990/#review13308 --- Thanks, Avery, this is a lot of great work. The results seem good, and the memory should really go down when this is extended to messages. We do trade off some speed for the memory, which is why I'm not sure we should have ByteArrayPartition as default. For these Input/Output classes, a significant part of the code is copied from somewhere else, right? If so, I think it would be good to mention it in the comment somewhere. http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/benchmark/RepresentativePageRankBenchmark.java <https://reviews.apache.org/r/7990/#comment28569> Rename to RepresentativeVertexPageRankBenchmark http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/SendMessageCache.java <https://reviews.apache.org/r/7990/#comment28741> You can delete VertexIdMessageCollection class, right? http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/UnsafeByteArrayInputStream.java <https://reviews.apache.org/r/7990/#comment28648> Can you create a separate function to check the remaining bytes, and call it everywhere? http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/WritableUtils.java <https://reviews.apache.org/r/7990/#comment28632> Why do we always write -1 here and ignore the first int value when reading? - Maja Kabiljo On Nov. 14, 2012, 12:43 a.m., Avery Ching wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/7990/ > --- > > (Updated Nov. 14, 2012, 12:43 a.m.) > > > Review request for giraph. > > > Description > --- > > Our entire graph is currently stored as Java objects in memory. I added an > option to keep only a representative vertex that serializes/deserializes on > the fly and should be used with the new ByteArrayPartition. In conjunction > with a serialized client-side message cache, memory usage then loading > shrinks to almost 1/10 of trunk and loads the input splits almost 3x faster > (see input superstep times below). I added a serializer based on Sun's > unsafe methods that enables this memory savings with a very small performance > hit (maybe a few 1-5% slower). Compared to trunk, when serializing the > messages with our faster serializer, compute time improves significantly as > well against trunk (16.7 -> 12.31 for 2.5B edges, 2.97 -> 1.61 for 250M > edges). There are still further improvements to be made on the server side > where we still store our messages in-memory. I (or someone else) can do that > in a later patch. This also significantly reduces GC time, as there are less > objects to GC. > > - Improves byte[] serialization signficantly > -- Added ExtendedDataInput/ExtendedDataOutput interfaces to allow for some > additional methods needed for byte[] serialization/deserialization > -- Add ExtendedByteArrayDataInput/ExtendedByteArrayDataoutput to > serialize/deserialize Writables to a byte[] > -- Added DynamicChannelBufferOutputStream/DynamicChannelBufferInputStream to > serialize/deserialize Writables to a DynamicChannelBuffer > > - Gives you the choice of partition implementation (SimplePartition (default) > or ByteArrayPartition -> (serialized vertices)) > -- Added a new method to Partition called saveVertex(), which also the > serialization back into the ByteArrayPartition or does nothing when using > SimplePartition > - Gives you the choice of unsafe serialization (using Sun's unsafe class - > default) or regular serialization > - Serializes the messages on the client cache into byte[] (saves memory and > also serializes faster) > -- Created new ByteArrayVertexIdMessageCollection to support the serialized > messages > -- SendVertexRequest now sends Partition objects rather than collections > - Adds 2 more options in PageRankBenchmark to try out RepresentationVertex or > RepresentationVertex with unsafe serialization > - Fixed a bug in LongDoubleFloatDoubleVertex's readFields when edges aren't > cleared before deserializing > > - Added new unittests > -- Replaced TestEdgeListVertex with TestMutableVertex to test all our generic > MutableVertex implementations > --- Added more serialization tests of different serialization > -- TestPartitionStores has more testing of unsafe > serialization/deserialization > > >
[jira] [Updated] (GIRAPH-424) Fix hashCode modulo computation
[ https://issues.apache.org/jira/browse/GIRAPH-424?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo updated GIRAPH-424: Attachment: GIRAPH-424.patch > Fix hashCode modulo computation > --- > > Key: GIRAPH-424 > URL: https://issues.apache.org/jira/browse/GIRAPH-424 > Project: Giraph > Issue Type: Bug > Reporter: Maja Kabiljo > Attachments: GIRAPH-424.patch > > > hashCode() can return Integer.MIN_VALUE, and on several places we do > Math.abs(hashCode), expecting to get a positive value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (GIRAPH-424) Fix hashCode modulo computation
Maja Kabiljo created GIRAPH-424: --- Summary: Fix hashCode modulo computation Key: GIRAPH-424 URL: https://issues.apache.org/jira/browse/GIRAPH-424 Project: Giraph Issue Type: Bug Reporter: Maja Kabiljo hashCode() can return Integer.MIN_VALUE, and on several places we do Math.abs(hashCode), expecting to get a positive value. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-386) ClassCastException when giraph.SplitMasterWorker=false
[
https://issues.apache.org/jira/browse/GIRAPH-386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maja Kabiljo updated GIRAPH-386:
Attachment: GIRAPH-386.diff
Rebased.
> ClassCastException when giraph.SplitMasterWorker=false
> --
>
> Key: GIRAPH-386
> URL: https://issues.apache.org/jira/browse/GIRAPH-386
> Project: Giraph
> Issue Type: Bug
>Reporter: Jaeho Shin
> Assignee: Maja Kabiljo
> Attachments: GIRAPH-386.diff, GIRAPH-386.diff, GIRAPH-386.patch,
> GIRAPH-386.patch
>
>
> Using -Dgiraph.SplitMasterWorker=false with a recent trunk (r1401165) caused
> the machine who is playing the master role (showing itself as {{ALL}} from
> the task tracker page) to throw ClassCastException (SendVertexRequest ->
> MasterRequest) from MasterRequestServerHandler class. I'm trying to use as
> many machines as possible for actual computation (can't afford to waste one
> for master+ZK). This worked fine with r1388628 (roughly a month ago), so a
> recent change must've broken something. Here's the relevant log I captured:
> {code}
> 2012-10-24 23:08:02,152 WARN
> org.apache.giraph.comm.netty.handler.RequestServerHandler: exceptionCaught:
> Channel failed with remote address /10.x.y.z:41780
> java.lang.ClassCastException:
> org.apache.giraph.comm.requests.SendVertexRequest cannot be cast to
> org.apache.giraph.comm.requests.MasterRequest
> at
> org.apache.giraph.comm.netty.handler.MasterRequestServerHandler.processRequest(MasterRequestServerHandler.java:25)
> at
> org.apache.giraph.comm.netty.handler.RequestServerHandler.messageReceived(RequestServerHandler.java:100)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:71)
> at
> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:45)
> at
> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:69)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: Create MasterInfo and WorkerInfo, fix giraph.SplitMasterWorker=false
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7911/ --- (Updated Nov. 13, 2012, 6:54 p.m.) Review request for giraph. Changes --- Rebased. Description --- What I first did for GIRAPH-386 didn't quite work once the master communication was turned on. We need to get task ids from TaskInfo to other places also (like request/response handlers, NettyClient etc) so all request tracking would work correctly. This addresses bug GIRAPH-386. https://issues.apache.org/jira/browse/GIRAPH-386 Diffs (updated) - http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/bsp/CentralizedServiceMaster.java 1408885 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java 1408885 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/MasterClientServer.java 1408885 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/WorkerClient.java 1408885 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/WorkerServer.java 1408885 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyClient.java 1408885 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyMasterClient.java 1408885 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyMasterClientServer.java 1408885 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyMasterServer.java 1408885 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyServer.java 1408885 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClient.java 1408885 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerServer.java 1408885 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/handler/ClientRequestId.java 1408885 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/handler/MasterRequestServerHandler.java 1408885 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/handler/RequestServerHandler.java 1408885 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/handler/WorkerRequestServerHandler.java 1408885 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/AddressesAndPartitionsWritable.java 1408885 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspService.java 1408885 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspServiceMaster.java 1408885 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspServiceWorker.java 1408885 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/MasterAggregatorHandler.java 1408885 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/MasterInfo.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/TaskInfo.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/WorkerInfo.java 1408885 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/zk/ZooKeeperManager.java 1408885 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/ConnectionTest.java 1408885 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestFailureTest.java 1408885 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestTest.java 1408885 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/SaslConnectionTest.java 1408885 Diff: https://reviews.apache.org/r/7911/diff/ Testing --- mvn clean verify Run benchmarks on cluster with both values for giraph.splitMasterWorker Thanks, Maja Kabiljo
[jira] [Commented] (GIRAPH-423) Allow overriding addEdge
[ https://issues.apache.org/jira/browse/GIRAPH-423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13496383#comment-13496383 ] Maja Kabiljo commented on GIRAPH-423: - +1, thanks Alessandro! > Allow overriding addEdge > > > Key: GIRAPH-423 > URL: https://issues.apache.org/jira/browse/GIRAPH-423 > Project: Giraph > Issue Type: Bug >Reporter: Alessandro Presta >Assignee: Alessandro Presta >Priority: Minor > Attachments: GIRAPH-423.patch > > > addEdge() is final in some MutableVertex implementations, whereas other > similar methods (like removeEdge()) aren't. > Making it non-final enables some interesting customizations, for example > summing the weights of duplicate edges instead of keeping the oldest one. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: Create BinaryCombiner ands specialized message store for it
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7975/ --- (Updated Nov. 13, 2012, 4:40 a.m.) Review request for giraph. Changes --- Avery's comments Description --- Current combiner interface is very general, but also doesn't provide the best performance. All the combiners we currently have are binary combiners, i.e. they can combine two messages into one. Having a lists around this simple concept makes it slower and requires more object creations. Adding BinaryCombiner, and a specialized message store which will be used with it. This message store has only one message per vertex instead of having a collection per vertex. This addresses bug GIRAPH-414. https://issues.apache.org/jira/browse/GIRAPH-414 Diffs (updated) - http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/GiraphConfiguration.java 1408573 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/GiraphRunner.java 1408573 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/ImmutableClassesGiraphConfiguration.java 1408573 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/benchmark/PageRankBenchmark.java 1408573 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/CollectionOfMessagesPerVertexStore.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/DiskBackedMessageStore.java 1408573 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/OneMessagePerVertexStore.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/SimpleMessageStore.java 1408573 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerServer.java 1408573 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/DoubleSumCombiner.java 1408573 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/MinimumDoubleCombiner.java 1408573 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/MinimumIntCombiner.java 1408573 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/SimpleSumCombiner.java 1408573 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspUtils.java 1408573 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/Combiner.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/GiraphTypeValidator.java 1408573 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/VertexCombiner.java 1408573 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/InternalVertexRunner.java 1408573 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/TestVertexTypes.java 1408573 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/ConnectionTest.java 1408573 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestFailureTest.java 1408573 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestTest.java 1408573 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/SaslConnectionTest.java 1408573 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/TestMessageStores.java 1408573 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/examples/MinimumIntCombinerTest.java 1408573 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/utils/MockUtils.java 1408573 Diff: https://reviews.apache.org/r/7975/diff/ Testing --- mvn verify PageRankBenchmark 20m vertices, 100 edges per vertex, 20 workers 1 compute thread, superstep time 55s->45s 6 compute threads, superstep time 28s->15s 12 compute threads, 1 netty server thread, superstep time 185s->112s Our internal application Similar speedup as Page Rank Thanks, Maja Kabiljo
[jira] [Updated] (GIRAPH-415) Refactor / cleanup Hadoop Counters
[ https://issues.apache.org/jira/browse/GIRAPH-415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo updated GIRAPH-415: Attachment: GIRAPH-415.patch > Refactor / cleanup Hadoop Counters > -- > > Key: GIRAPH-415 > URL: https://issues.apache.org/jira/browse/GIRAPH-415 > Project: Giraph > Issue Type: Improvement >Reporter: Nitay Joffe >Assignee: Nitay Joffe > Attachments: GIRAPH-415-2.patch, GIRAPH-415.patch, GIRAPH-415.patch, > GIRAPH-415.patch > > > https://reviews.apache.org/r/7980 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-415) Refactor / cleanup Hadoop Counters
[ https://issues.apache.org/jira/browse/GIRAPH-415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo updated GIRAPH-415: Attachment: GIRAPH-415.patch There was an '@Override' missing in Nitay's patch. > Refactor / cleanup Hadoop Counters > -- > > Key: GIRAPH-415 > URL: https://issues.apache.org/jira/browse/GIRAPH-415 > Project: Giraph > Issue Type: Improvement >Reporter: Nitay Joffe >Assignee: Nitay Joffe > Attachments: GIRAPH-415-2.patch, GIRAPH-415.patch, GIRAPH-415.patch > > > https://reviews.apache.org/r/7980 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: GIRAPH-415: Refactor / cleanup Hadoop Counters
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7980/#review13368 --- Ship it! +1, thanks Nitay. - Maja Kabiljo On Nov. 12, 2012, 6:59 p.m., Nitay Joffe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/7980/ > --- > > (Updated Nov. 12, 2012, 6:59 p.m.) > > > Review request for giraph. > > > Description > --- > > GIRAPH-415: Refactor / cleanup Hadoop Counters > > > Diffs > - > > giraph/src/main/java/org/apache/giraph/counters/GiraphHadoopCounter.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/counters/GiraphStats.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/counters/GiraphTimers.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/counters/HadoopCountersBase.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/counters/package-info.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/graph/BspServiceMaster.java 7d5dabb > giraph/src/main/java/org/apache/giraph/graph/EdgeListVertex.java 8801dde > giraph/src/main/java/org/apache/giraph/graph/HashMapVertex.java 26b14ca > giraph/src/main/java/org/apache/giraph/graph/IntIntNullIntVertex.java > a23b87f > > giraph/src/main/java/org/apache/giraph/graph/LongDoubleFloatDoubleVertex.java > 1db6e97 > giraph/src/main/java/org/apache/giraph/graph/MasterThread.java 5c9a72d > giraph/src/main/java/org/apache/giraph/graph/SimpleMutableVertex.java > 85af970 > giraph/src/main/java/org/apache/giraph/graph/SimpleVertex.java ed6759a > giraph/src/main/java/org/apache/giraph/graph/Vertex.java 8789ee5 > giraph/src/main/java/org/apache/giraph/metrics/EmptyMetricsRegistry.java > 262db29 > giraph/src/main/java/org/apache/giraph/metrics/GiraphMetricsRegistry.java > eac9f72 > giraph/src/main/java/org/apache/giraph/metrics/NoOpMetricsRegistry.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/metrics/package-info.java f7529f5 > > Diff: https://reviews.apache.org/r/7980/diff/ > > > Testing > --- > > > Thanks, > > Nitay Joffe > >
Re: Review Request: GIRAPH-415: Refactor / cleanup Hadoop Counters
> On Nov. 9, 2012, 7:03 p.m., Maja Kabiljo wrote: > > giraph/src/main/java/org/apache/giraph/graph/BspServiceMaster.java, line 769 > > <https://reviews.apache.org/r/7980/diff/1/?file=187462#file187462line769> > > > > You can use setValue here > > Nitay Joffe wrote: > That makes the logic different than before though? I mean it was using > increment() before too, so this have a different effect? The way it was before isn't correct, right? It just worked when we restart the application because the counter value is zero... - Maja --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7980/#review13296 --- On Nov. 9, 2012, 12:07 a.m., Nitay Joffe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/7980/ > --- > > (Updated Nov. 9, 2012, 12:07 a.m.) > > > Review request for giraph. > > > Description > --- > > GIRAPH-415: Refactor / cleanup Hadoop Counters > > > Diffs > - > > giraph/src/main/java/org/apache/giraph/counters/GiraphHadoopCounter.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/counters/GiraphStats.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/counters/GiraphTimers.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/counters/HadoopCountersBase.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/counters/package-info.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/graph/BspServiceMaster.java > 7d5dabb9df619b82dd42ab631b3c244fbdd9ddcf > giraph/src/main/java/org/apache/giraph/graph/EdgeListVertex.java > 8801dde5102ffca397678b0dcc3dd7c2932d22d7 > giraph/src/main/java/org/apache/giraph/graph/HashMapVertex.java > 26b14caed23706b12f21142862bbd15fe3720bd6 > giraph/src/main/java/org/apache/giraph/graph/IntIntNullIntVertex.java > a23b87f7e43d6a3b78516cbb5f1dd512dca84a8d > > giraph/src/main/java/org/apache/giraph/graph/LongDoubleFloatDoubleVertex.java > 1db6e977eb4a8102351a908dc1a113935002e61e > giraph/src/main/java/org/apache/giraph/graph/MasterThread.java > 5c9a72d5edbaafddca1f5050131a1657e15f5ac2 > giraph/src/main/java/org/apache/giraph/graph/SimpleMutableVertex.java > 85af9700e76a3ef32191fe0dc5631f430b13de42 > giraph/src/main/java/org/apache/giraph/graph/SimpleVertex.java > ed6759a884553fa15ee477e19e483e1d210ba7be > giraph/src/main/java/org/apache/giraph/graph/Vertex.java > 8789ee5c8f170b6d598f0b3605b1f6f36227 > giraph/src/main/java/org/apache/giraph/metrics/EmptyMetricsRegistry.java > 262db29bbd4797e470f426999b85f2c1a7d1dee3 > giraph/src/main/java/org/apache/giraph/metrics/GiraphMetricsRegistry.java > eac9f7295d968c3b30c40b89944e8f9f8549cc70 > giraph/src/main/java/org/apache/giraph/metrics/NoOpMetricsRegistry.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/metrics/package-info.java > f7529f52527e8e4253f579bd6d01865aa980926b > > Diff: https://reviews.apache.org/r/7980/diff/ > > > Testing > --- > > > Thanks, > > Nitay Joffe > >
Re: Review Request: Create BinaryCombiner ands specialized message store for it
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7975/ --- (Updated Nov. 12, 2012, 6:39 p.m.) Review request for giraph. Changes --- This is with just one combiner type. Description --- Current combiner interface is very general, but also doesn't provide the best performance. All the combiners we currently have are binary combiners, i.e. they can combine two messages into one. Having a lists around this simple concept makes it slower and requires more object creations. Adding BinaryCombiner, and a specialized message store which will be used with it. This message store has only one message per vertex instead of having a collection per vertex. This addresses bug GIRAPH-414. https://issues.apache.org/jira/browse/GIRAPH-414 Diffs (updated) - http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/GiraphConfiguration.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/GiraphRunner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/ImmutableClassesGiraphConfiguration.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/benchmark/PageRankBenchmark.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/CollectionOfMessagesPerVertexStore.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/DiskBackedMessageStore.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/OneMessagePerVertexStore.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/SimpleMessageStore.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerServer.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/DoubleSumCombiner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/MinimumDoubleCombiner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/MinimumIntCombiner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/SimpleSumCombiner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspUtils.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/Combiner.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/GiraphTypeValidator.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/VertexCombiner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/InternalVertexRunner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/TestVertexTypes.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/ConnectionTest.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestFailureTest.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestTest.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/SaslConnectionTest.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/TestMessageStores.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/examples/MinimumIntCombinerTest.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/utils/MockUtils.java 1406748 Diff: https://reviews.apache.org/r/7975/diff/ Testing --- mvn verify PageRankBenchmark 20m vertices, 100 edges per vertex, 20 workers 1 compute thread, superstep time 55s->45s 6 compute threads, superstep time 28s->15s 12 compute threads, 1 netty server thread, superstep time 185s->112s Our internal application Similar speedup as Page Rank Thanks, Maja Kabiljo
[jira] [Updated] (GIRAPH-414) Create BinaryCombiner ands specialized message store for it
[ https://issues.apache.org/jira/browse/GIRAPH-414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo updated GIRAPH-414: Attachment: GIRAPH-414.patch Thanks Avery, forgot to post it here... > Create BinaryCombiner ands specialized message store for it > --- > > Key: GIRAPH-414 > URL: https://issues.apache.org/jira/browse/GIRAPH-414 > Project: Giraph > Issue Type: Improvement > Reporter: Maja Kabiljo > Assignee: Maja Kabiljo > Attachments: GIRAPH-414.patch > > > Current combiner interface is very general, but also doesn't provide the best > performance. All the combiners we currently have are binary combiners, i.e. > they can combine two messages into one. Having a lists around this simple > concept makes it slower and requires more object creations. > Adding BinaryCombiner, and a specialized message store which will be used > with it. This message store has only one message per vertex instead of having > a collection per vertex. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: Create BinaryCombiner ands specialized message store for it
> On Nov. 9, 2012, 1:59 a.m., Alessandro Presta wrote: > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BinaryCombiner.java, > > line 47 > > <https://reviews.apache.org/r/7975/diff/1/?file=187405#file187405line47> > > > > Do we have evidence that this in-place version is much better than the > > obvious M combine(M a, M b)? I understand the idea (minimizing object > > instantiation and garbage-collection), just want to make sure we make > > informed choices when going with a less-intuitive interface. > > > > I'm also suggesting that the vertex index is not needed. I see the > > parallel with the key in a MapReduce Reducer, but I'm not sure how it would > > help here to know just the vertex id but not the value. > > Maja Kabiljo wrote: > I'll run some experiments, there are two steps which we are skipping by > having this interface: object creation, and putting object back to the map. > > Went with vertexId because it was in the original version. Does anyone > have a reason why we might need it there? I tried one benchmark from above: 6 compute threads, superstep time 28s->15s and with M combine(M, M) interface it took 21s per superstep. > On Nov. 9, 2012, 1:59 a.m., Alessandro Presta wrote: > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BinaryCombiner.java, > > line 56 > > <https://reviews.apache.org/r/7975/diff/1/?file=187405#file187405line56> > > > > We could get rid of this method by using the following logic when > > processing messages: > > - if the current message is null, just set it to the incoming one > > - otherwise, combine them > > Maja Kabiljo wrote: > The issue here is that we can't always modify the message which message > store receives, because of local requests and sendMessageToAllEdges which > keeps just one copy of message. > > Alessandro Presta wrote: > Got it. We could make a copy of the first message then. Do we have a way to do this? The only thing which I can think of is to serialize the message, and then create a new one and deserialize, but that's silly :-) - Maja ------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7975/#review13268 --- On Nov. 9, 2012, 10:34 p.m., Maja Kabiljo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/7975/ > --- > > (Updated Nov. 9, 2012, 10:34 p.m.) > > > Review request for giraph. > > > Description > --- > > Current combiner interface is very general, but also doesn't provide the best > performance. All the combiners we currently have are binary combiners, i.e. > they can combine two messages into one. Having a lists around this simple > concept makes it slower and requires more object creations. > Adding BinaryCombiner, and a specialized message store which will be used > with it. This message store has only one message per vertex instead of having > a collection per vertex. > > > This addresses bug GIRAPH-414. > https://issues.apache.org/jira/browse/GIRAPH-414 > > > Diffs > - > > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/GiraphConfiguration.java > 1406748 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/GiraphRunner.java > 1406748 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/ImmutableClassesGiraphConfiguration.java > 1406748 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/benchmark/PageRankBenchmark.java > 1406748 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/CollectionOfMessagesPerVertexStore.java > PRE-CREATION > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/DiskBackedMessageStore.java > 1406748 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/OneMessagePerVertexStore.java > PRE-CREATION > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/SimpleMessageStore.java > 1406748 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/j
Re: Review Request: Create BinaryCombiner ands specialized message store for it
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7975/ --- (Updated Nov. 9, 2012, 10:34 p.m.) Review request for giraph. Changes --- Fixed the comments. Description --- Current combiner interface is very general, but also doesn't provide the best performance. All the combiners we currently have are binary combiners, i.e. they can combine two messages into one. Having a lists around this simple concept makes it slower and requires more object creations. Adding BinaryCombiner, and a specialized message store which will be used with it. This message store has only one message per vertex instead of having a collection per vertex. This addresses bug GIRAPH-414. https://issues.apache.org/jira/browse/GIRAPH-414 Diffs (updated) - http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/GiraphConfiguration.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/GiraphRunner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/ImmutableClassesGiraphConfiguration.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/benchmark/PageRankBenchmark.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/CollectionOfMessagesPerVertexStore.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/DiskBackedMessageStore.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/OneMessagePerVertexStore.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/SimpleMessageStore.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerServer.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/DoubleSumBinaryCombiner.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/DoubleSumCombiner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/MinimumDoubleCombiner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/MinimumIntCombiner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/SimpleSumCombiner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BinaryCombiner.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspUtils.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/Combiner.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/GiraphTypeValidator.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/VertexCombiner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/InternalVertexRunner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/TestVertexTypes.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/ConnectionTest.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestFailureTest.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestTest.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/SaslConnectionTest.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/TestMessageStores.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/examples/MinimumIntCombinerTest.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/utils/MockUtils.java 1406748 Diff: https://reviews.apache.org/r/7975/diff/ Testing --- mvn verify PageRankBenchmark 20m vertices, 100 edges per vertex, 20 workers 1 compute thread, superstep time 55s->45s 6 compute threads, superstep time 28s->15s 12 compute threads, 1 netty server thread, superstep time 185s->112s Our internal application Similar speedup as Page Rank Thanks, Maja Kabiljo
Re: Review Request: GIRAPH-415: Refactor / cleanup Hadoop Counters
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/7980/#review13296
---
Looks good.
giraph/src/main/java/org/apache/giraph/counters/GiraphStats.java
<https://reviews.apache.org/r/7980/#comment28561>
Am I missing something or this is not used?
I understand the idea of it being Iterable, but why don't make
HadoopCountersBase Iterable then?
giraph/src/main/java/org/apache/giraph/graph/BspServiceMaster.java
<https://reviews.apache.org/r/7980/#comment28560>
You can use setValue here
giraph/src/main/java/org/apache/giraph/graph/Vertex.java
<https://reviews.apache.org/r/7980/#comment28559>
Nice refactoring.
- Maja Kabiljo
On Nov. 9, 2012, 12:07 a.m., Nitay Joffe wrote:
>
> ---
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/7980/
> ---
>
> (Updated Nov. 9, 2012, 12:07 a.m.)
>
>
> Review request for giraph.
>
>
> Description
> ---
>
> GIRAPH-415: Refactor / cleanup Hadoop Counters
>
>
> Diffs
> -
>
> giraph/src/main/java/org/apache/giraph/counters/GiraphHadoopCounter.java
> PRE-CREATION
> giraph/src/main/java/org/apache/giraph/counters/GiraphStats.java
> PRE-CREATION
> giraph/src/main/java/org/apache/giraph/counters/GiraphTimers.java
> PRE-CREATION
> giraph/src/main/java/org/apache/giraph/counters/HadoopCountersBase.java
> PRE-CREATION
> giraph/src/main/java/org/apache/giraph/counters/package-info.java
> PRE-CREATION
> giraph/src/main/java/org/apache/giraph/graph/BspServiceMaster.java
> 7d5dabb9df619b82dd42ab631b3c244fbdd9ddcf
> giraph/src/main/java/org/apache/giraph/graph/EdgeListVertex.java
> 8801dde5102ffca397678b0dcc3dd7c2932d22d7
> giraph/src/main/java/org/apache/giraph/graph/HashMapVertex.java
> 26b14caed23706b12f21142862bbd15fe3720bd6
> giraph/src/main/java/org/apache/giraph/graph/IntIntNullIntVertex.java
> a23b87f7e43d6a3b78516cbb5f1dd512dca84a8d
>
> giraph/src/main/java/org/apache/giraph/graph/LongDoubleFloatDoubleVertex.java
> 1db6e977eb4a8102351a908dc1a113935002e61e
> giraph/src/main/java/org/apache/giraph/graph/MasterThread.java
> 5c9a72d5edbaafddca1f5050131a1657e15f5ac2
> giraph/src/main/java/org/apache/giraph/graph/SimpleMutableVertex.java
> 85af9700e76a3ef32191fe0dc5631f430b13de42
> giraph/src/main/java/org/apache/giraph/graph/SimpleVertex.java
> ed6759a884553fa15ee477e19e483e1d210ba7be
> giraph/src/main/java/org/apache/giraph/graph/Vertex.java
> 8789ee5c8f170b6d598f0b3605b1f6f36227
> giraph/src/main/java/org/apache/giraph/metrics/EmptyMetricsRegistry.java
> 262db29bbd4797e470f426999b85f2c1a7d1dee3
> giraph/src/main/java/org/apache/giraph/metrics/GiraphMetricsRegistry.java
> eac9f7295d968c3b30c40b89944e8f9f8549cc70
> giraph/src/main/java/org/apache/giraph/metrics/NoOpMetricsRegistry.java
> PRE-CREATION
> giraph/src/main/java/org/apache/giraph/metrics/package-info.java
> f7529f52527e8e4253f579bd6d01865aa980926b
>
> Diff: https://reviews.apache.org/r/7980/diff/
>
>
> Testing
> ---
>
>
> Thanks,
>
> Nitay Joffe
>
>
Re: Review Request: GIRAPH-416: MasterObserver for user post-application customization
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7981/#review13283 --- Ship it! Thanks Nitay, +1. - Maja Kabiljo On Nov. 9, 2012, 3:09 a.m., Nitay Joffe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/7981/ > --- > > (Updated Nov. 9, 2012, 3:09 a.m.) > > > Review request for giraph. > > > Description > --- > > GIRAPH-416: MasterObserver for user post-application customization > > > Diffs > - > > giraph/src/main/java/org/apache/giraph/GiraphConfiguration.java 496bce7 > > giraph/src/main/java/org/apache/giraph/ImmutableClassesGiraphConfiguration.java > bb6a739 > giraph/src/main/java/org/apache/giraph/bsp/CentralizedServiceMaster.java > 688ce43 > giraph/src/main/java/org/apache/giraph/graph/BspServiceMaster.java 7d5dabb > giraph/src/main/java/org/apache/giraph/graph/IntNullNullNullVertex.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/graph/MasterThread.java 5c9a72d > > giraph/src/main/java/org/apache/giraph/io/IntNullNullNullTextInputFormat.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/master/DefaultMasterObserver.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/master/MasterObserver.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/master/package-info.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/utils/InternalVertexRunner.java > b891690 > giraph/src/test/java/org/apache/giraph/TestGiraphConfiguration.java > PRE-CREATION > giraph/src/test/java/org/apache/giraph/TestMasterObserver.java PRE-CREATION > > Diff: https://reviews.apache.org/r/7981/diff/ > > > Testing > --- > > > Thanks, > > Nitay Joffe > >
Re: Review Request: Create BinaryCombiner ands specialized message store for it
> On Nov. 9, 2012, 1:59 a.m., Alessandro Presta wrote: > > The results look good. How does the no-combiner version fare in these same > > benchmarks? > > > > Frankly, at this point I would just go for one combiner interface (the > > binary one). > > No need to use it in the disk-backed message store (if we have one message > > per vertex, there is no point in spilling to disk). > > > > The only use I can think of for the Iterable->Iterable form is when you're > > doing some sort of top-k computation, but I'm not sure if we would reap any > > benefits when the graph is sparse. > > > > Regarding the interface, I think two sensible options are: > > 1) M combine(M a, M b) > > 2) M combine(Iterable messages) > > > > Why 2)? We may find that when processing a list of incoming messages, > > calling combine() for each of them has an overhead, whereas a loop inside > > combine() would be faster. > > However, if we also use the binary combiner on the sender (which may be a > > good idea if it's fast enough), all the requests will have at most one > > message per vertex, so then 1) would be the way to go (assuming we call the > > combiner each time we process a request). The way messaging is implemented right now, we don't have a list of messages per vertex. Please take a look at SendWorkerMessagesRequest. So there is no iterative combiner calls. This was discussed before, we get the chance to use the combiner on the client side very rarely, so benefit of having just lists instead of maps on the client side is much bigger then the one we could get from combiner. > On Nov. 9, 2012, 1:59 a.m., Alessandro Presta wrote: > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BinaryCombiner.java, > > line 56 > > <https://reviews.apache.org/r/7975/diff/1/?file=187405#file187405line56> > > > > We could get rid of this method by using the following logic when > > processing messages: > > - if the current message is null, just set it to the incoming one > > - otherwise, combine them The issue here is that we can't always modify the message which message store receives, because of local requests and sendMessageToAllEdges which keeps just one copy of message. > On Nov. 9, 2012, 1:59 a.m., Alessandro Presta wrote: > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BinaryCombiner.java, > > line 47 > > <https://reviews.apache.org/r/7975/diff/1/?file=187405#file187405line47> > > > > Do we have evidence that this in-place version is much better than the > > obvious M combine(M a, M b)? I understand the idea (minimizing object > > instantiation and garbage-collection), just want to make sure we make > > informed choices when going with a less-intuitive interface. > > > > I'm also suggesting that the vertex index is not needed. I see the > > parallel with the key in a MapReduce Reducer, but I'm not sure how it would > > help here to know just the vertex id but not the value. I'll run some experiments, there are two steps which we are skipping by having this interface: object creation, and putting object back to the map. Went with vertexId because it was in the original version. Does anyone have a reason why we might need it there? > On Nov. 9, 2012, 1:59 a.m., Alessandro Presta wrote: > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/OneMessagePerVertexStore.java, > > line 35 > > <https://reviews.apache.org/r/7975/diff/1/?file=187397#file187397line35> > > > > I find it confusing that we talk about "message store per vertex" when > > it isn't really a MessageStore. > > Also, I think here you mean "a single message" instead of "a collection > > of messages". I'll revisit the comment > On Nov. 9, 2012, 1:59 a.m., Alessandro Presta wrote: > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/ImmutableClassesGiraphConfiguration.java, > > line 92 > > <https://reviews.apache.org/r/7975/diff/1/?file=187393#file187393line92> > > > > Shouldn't you change the "vertex combiner" terminology to "multi > > combiner" across the board? As I said in my comment, I haven't cleaned everything up because I am not sure what is the best way to deal with two combiner interfaces. Please take a look at the updated diff I posted. - Maja --- This is an automatica
Re: Review Request: GIRAPH-416: MasterObserver for user post-application customization
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7981/#review13273 --- Nice work! And thanks for splitting it up. giraph/src/main/java/org/apache/giraph/ImmutableClassesGiraphConfiguration.java <https://reviews.apache.org/r/7981/#comment28531> Formatting giraph/src/main/java/org/apache/giraph/master/MasterObserver.java <https://reviews.apache.org/r/7981/#comment28529> How about adding preApplication, pre/postSuperstep, like in WorkerContext? A lso, I think MasterObserver should at least be Configurable, or have a GraphState in there, so we could access application state from implementing classes. - Maja Kabiljo On Nov. 9, 2012, 12:29 a.m., Nitay Joffe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/7981/ > --- > > (Updated Nov. 9, 2012, 12:29 a.m.) > > > Review request for giraph. > > > Description > --- > > GIRAPH-416: MasterObserver for user post-application customization > > > Diffs > - > > giraph/src/main/java/org/apache/giraph/GiraphConfiguration.java > 496bce7e13cd282337d8bfd8526a25657316ea33 > > giraph/src/main/java/org/apache/giraph/ImmutableClassesGiraphConfiguration.java > bb6a739749d6110faf0a39f1b3bfee769bea28a2 > giraph/src/main/java/org/apache/giraph/bsp/CentralizedServiceMaster.java > 688ce4389a9b5ab6ed3d124af92936286563f21d > giraph/src/main/java/org/apache/giraph/graph/BspServiceMaster.java > 7d5dabb9df619b82dd42ab631b3c244fbdd9ddcf > giraph/src/main/java/org/apache/giraph/graph/IntNullNullNullVertex.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/graph/MasterThread.java > 5c9a72d5edbaafddca1f5050131a1657e15f5ac2 > > giraph/src/main/java/org/apache/giraph/io/IntNullNullNullTextInputFormat.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/master/MasterObserver.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/master/package-info.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/utils/InternalVertexRunner.java > b891690474ba2b64ed2778c332a7855565a64e5b > giraph/src/test/java/org/apache/giraph/TestGiraphConfiguration.java > PRE-CREATION > giraph/src/test/java/org/apache/giraph/TestMasterObserver.java PRE-CREATION > > Diff: https://reviews.apache.org/r/7981/diff/ > > > Testing > --- > > > Thanks, > > Nitay Joffe > >
Re: Review Request: Create BinaryCombiner ands specialized message store for it
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7975/ --- (Updated Nov. 9, 2012, 1:44 a.m.) Review request for giraph. Changes --- Here is what I had in mind with creating an empty Combiner. Description --- Current combiner interface is very general, but also doesn't provide the best performance. All the combiners we currently have are binary combiners, i.e. they can combine two messages into one. Having a lists around this simple concept makes it slower and requires more object creations. Adding BinaryCombiner, and a specialized message store which will be used with it. This message store has only one message per vertex instead of having a collection per vertex. This addresses bug GIRAPH-414. https://issues.apache.org/jira/browse/GIRAPH-414 Diffs (updated) - http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/GiraphConfiguration.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/GiraphRunner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/ImmutableClassesGiraphConfiguration.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/benchmark/PageRankBenchmark.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/CollectionOfMessagesPerVertexStore.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/DiskBackedMessageStore.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/OneMessagePerVertexStore.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/SimpleMessageStore.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerServer.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/DoubleSumBinaryCombiner.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/DoubleSumCombiner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/MinimumDoubleCombiner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/MinimumIntCombiner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/SimpleSumCombiner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BinaryCombiner.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspUtils.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/Combiner.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/GiraphTypeValidator.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/VertexCombiner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/InternalVertexRunner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/TestVertexTypes.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/ConnectionTest.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestFailureTest.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestTest.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/SaslConnectionTest.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/TestMessageStores.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/examples/MinimumIntCombinerTest.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/utils/MockUtils.java 1406748 Diff: https://reviews.apache.org/r/7975/diff/ Testing --- mvn verify PageRankBenchmark 20m vertices, 100 edges per vertex, 20 workers 1 compute thread, superstep time 55s->45s 6 compute threads, superstep time 28s->15s 12 compute threads, 1 netty server thread, superstep time 185s->112s Our internal application Similar speedup as Page Rank Thanks, Maja Kabiljo
Re: Review Request: Create BinaryCombiner ands specialized message store for it
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7975/#review13263 --- This patch is not finished, one thing I wanted to hear your input on is what's the best way to deal with having two different combiner interfaces. I did it in a way that user just sets giraph.combinerClass to any of these types, but this creates an issue on a few places. For example we can't call stuff like conf.getClass(...) without knowing the interface of the combiner in advance. We can also have two different parameters to set, say giraph.binaryCombinerClass and giraph.multiCombinerClass, but having one parameter seems nicer. Also places like InternalVertexRunner would need to change signatures to accept two kinds of combiner. Does it makes sense to have some empty Combiner interface, and then BinaryCombiner and MultiCombiner both extend it, and we can use that interface on these problematic places? I am also going to change all current combiners to BinaryCombiners. - Maja Kabiljo On Nov. 8, 2012, 10:56 p.m., Maja Kabiljo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/7975/ > --- > > (Updated Nov. 8, 2012, 10:56 p.m.) > > > Review request for giraph. > > > Description > --- > > Current combiner interface is very general, but also doesn't provide the best > performance. All the combiners we currently have are binary combiners, i.e. > they can combine two messages into one. Having a lists around this simple > concept makes it slower and requires more object creations. > Adding BinaryCombiner, and a specialized message store which will be used > with it. This message store has only one message per vertex instead of having > a collection per vertex. > > > This addresses bug GIRAPH-414. > https://issues.apache.org/jira/browse/GIRAPH-414 > > > Diffs > - > > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/GiraphConfiguration.java > 1406748 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/GiraphRunner.java > 1406748 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/ImmutableClassesGiraphConfiguration.java > 1406748 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/benchmark/PageRankBenchmark.java > 1406748 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/CollectionOfMessagesPerVertexStore.java > PRE-CREATION > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/DiskBackedMessageStore.java > 1406748 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/OneMessagePerVertexStore.java > PRE-CREATION > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/SimpleMessageStore.java > 1406748 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerServer.java > 1406748 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/DoubleSumBinaryCombiner.java > PRE-CREATION > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/DoubleSumCombiner.java > 1406748 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/MinimumDoubleCombiner.java > 1406748 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/MinimumIntCombiner.java > 1406748 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/SimpleSumCombiner.java > 1406748 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BinaryCombiner.java > PRE-CREATION > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspUtils.java > 1406748 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/GiraphTypeValidator.java > 1406748 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/VertexCombiner.java > 1406748 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/InternalVert
Review Request: Create BinaryCombiner ands specialized message store for it
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7975/ --- Review request for giraph. Description --- Current combiner interface is very general, but also doesn't provide the best performance. All the combiners we currently have are binary combiners, i.e. they can combine two messages into one. Having a lists around this simple concept makes it slower and requires more object creations. Adding BinaryCombiner, and a specialized message store which will be used with it. This message store has only one message per vertex instead of having a collection per vertex. This addresses bug GIRAPH-414. https://issues.apache.org/jira/browse/GIRAPH-414 Diffs - http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/GiraphConfiguration.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/GiraphRunner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/ImmutableClassesGiraphConfiguration.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/benchmark/PageRankBenchmark.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/CollectionOfMessagesPerVertexStore.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/DiskBackedMessageStore.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/OneMessagePerVertexStore.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/SimpleMessageStore.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerServer.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/DoubleSumBinaryCombiner.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/DoubleSumCombiner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/MinimumDoubleCombiner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/MinimumIntCombiner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/SimpleSumCombiner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BinaryCombiner.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspUtils.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/GiraphTypeValidator.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/VertexCombiner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/InternalVertexRunner.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/TestVertexTypes.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/ConnectionTest.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestFailureTest.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestTest.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/SaslConnectionTest.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/TestMessageStores.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/examples/MinimumIntCombinerTest.java 1406748 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/utils/MockUtils.java 1406748 Diff: https://reviews.apache.org/r/7975/diff/ Testing --- mvn verify PageRankBenchmark 20m vertices, 100 edges per vertex, 20 workers 1 compute thread, superstep time 55s->45s 6 compute threads, superstep time 28s->15s 12 compute threads, 1 netty server thread, superstep time 185s->112s Our internal application Similar speedup as Page Rank Thanks, Maja Kabiljo
[jira] [Created] (GIRAPH-414) Create BinaryCombiner ands specialized message store for it
Maja Kabiljo created GIRAPH-414: --- Summary: Create BinaryCombiner ands specialized message store for it Key: GIRAPH-414 URL: https://issues.apache.org/jira/browse/GIRAPH-414 Project: Giraph Issue Type: Improvement Reporter: Maja Kabiljo Assignee: Maja Kabiljo Current combiner interface is very general, but also doesn't provide the best performance. All the combiners we currently have are binary combiners, i.e. they can combine two messages into one. Having a lists around this simple concept makes it slower and requires more object creations. Adding BinaryCombiner, and a specialized message store which will be used with it. This message store has only one message per vertex instead of having a collection per vertex. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-386) ClassCastException when giraph.SplitMasterWorker=false
[
https://issues.apache.org/jira/browse/GIRAPH-386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maja Kabiljo updated GIRAPH-386:
Attachment: GIRAPH-386.diff
https://reviews.apache.org/r/7911/
Eugene, can you please check if this fixes GIRAPH-362 also?
> ClassCastException when giraph.SplitMasterWorker=false
> --
>
> Key: GIRAPH-386
> URL: https://issues.apache.org/jira/browse/GIRAPH-386
> Project: Giraph
> Issue Type: Bug
>Reporter: Jaeho Shin
> Assignee: Maja Kabiljo
> Attachments: GIRAPH-386.diff, GIRAPH-386.patch, GIRAPH-386.patch
>
>
> Using -Dgiraph.SplitMasterWorker=false with a recent trunk (r1401165) caused
> the machine who is playing the master role (showing itself as {{ALL}} from
> the task tracker page) to throw ClassCastException (SendVertexRequest ->
> MasterRequest) from MasterRequestServerHandler class. I'm trying to use as
> many machines as possible for actual computation (can't afford to waste one
> for master+ZK). This worked fine with r1388628 (roughly a month ago), so a
> recent change must've broken something. Here's the relevant log I captured:
> {code}
> 2012-10-24 23:08:02,152 WARN
> org.apache.giraph.comm.netty.handler.RequestServerHandler: exceptionCaught:
> Channel failed with remote address /10.x.y.z:41780
> java.lang.ClassCastException:
> org.apache.giraph.comm.requests.SendVertexRequest cannot be cast to
> org.apache.giraph.comm.requests.MasterRequest
> at
> org.apache.giraph.comm.netty.handler.MasterRequestServerHandler.processRequest(MasterRequestServerHandler.java:25)
> at
> org.apache.giraph.comm.netty.handler.RequestServerHandler.messageReceived(RequestServerHandler.java:100)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:71)
> at
> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:45)
> at
> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:69)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request: Create MasterInfo and WorkerInfo, fix giraph.SplitMasterWorker=false
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7911/ --- Review request for giraph. Summary (updated) - Create MasterInfo and WorkerInfo, fix giraph.SplitMasterWorker=false Description (updated) --- What I first did for GIRAPH-386 didn't quite work once the master communication was turned on. We need to get task ids from TaskInfo to other places also (like request/response handlers, NettyClient etc) so all request tracking would work correctly. This addresses bug GIRAPH-386. https://issues.apache.org/jira/browse/GIRAPH-386 Diffs (updated) - http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/bsp/CentralizedServiceMaster.java 1406415 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java 1406415 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/MasterClientServer.java 1406415 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/WorkerClient.java 1406415 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/WorkerServer.java 1406415 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyClient.java 1406415 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyMasterClient.java 1406415 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyMasterClientServer.java 1406415 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyMasterServer.java 1406415 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyServer.java 1406415 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClient.java 1406415 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerServer.java 1406415 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/handler/ClientRequestId.java 1406415 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/handler/MasterRequestServerHandler.java 1406415 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/handler/RequestServerHandler.java 1406415 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/handler/WorkerRequestServerHandler.java 1406415 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/AddressesAndPartitionsWritable.java 1406415 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspService.java 1406415 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspServiceMaster.java 1406415 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspServiceWorker.java 1406415 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/MasterAggregatorHandler.java 1406415 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/MasterInfo.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/TaskInfo.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/WorkerInfo.java 1406415 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/zk/ZooKeeperManager.java 1406415 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/ConnectionTest.java 1406415 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestFailureTest.java 1406415 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestTest.java 1406415 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/SaslConnectionTest.java 1406415 Diff: https://reviews.apache.org/r/7911/diff/ Testing (updated) --- mvn clean verify Run benchmarks on cluster with both values for giraph.splitMasterWorker Thanks, Maja Kabiljo
Re: Review Request: More SendMessageCache improvements
Hi Eli, I'm not sure which checks are you talking about. Was the change you are referring to committed? I don't have the logs from the previous change anymore, but as we discussed on the bigger graphs there is practically no client side combining, and therefore no vertexId duplications. So the memory could only decrease because we are using lists instead of maps. I looked at the logs for this change, and the memory usage did decrease by about 5%. Not sure what happened with memory when multithreading was added, maybe Avery has more information about it. Maja From: Eli Reisman mailto:apache.mail...@gmail.com>> Date: Tuesday, November 6, 2012 10:44 AM To: "dev@giraph.apache.org<mailto:dev@giraph.apache.org>" mailto:dev@giraph.apache.org>>, Maja Kabiljo mailto:majakabi...@fb.com>> Subject: Re: Review Request: More SendMessageCache improvements Hey out of curiosity as I have not been able to run any cluster jobs myself on the new trunk for some time now: How much more memory does a job use now compared to before the recent changes? Has the new multithreading, lists of messages, etc. made a noticable dent in the heap? I realize heap is a worthwhile tradeoff for speed for you guys either way, just curious. On Tue, Nov 6, 2012 at 10:38 AM, Eli Reisman mailto:apache.mail...@gmail.com>> wrote: Hey Maja, you know when I redid these I was trying to minimize the message duplication as you know, and so I checked for existing references before putting a new (duplicate) copy of the same VertexId or Message (etc) into the maps. This slowed things down a tad (not too bad for us) but saved us some memory (less with more frequent flushing) I think the effects of this stuff must be more visible to you guys because you don't get any speed benefit from the increased parallelism of lots of workers as I was getting. Maybe the multithreading has helped make up for that for you guys? But anyway my point is: it could be the maps aren't the slowdown, but all the checking I had in there. So if you ever hit a situation where the maps seem like a better solution, try them out again (but without the checking) and see if the speed comes back? Nice work! On Mon, Nov 5, 2012 at 2:59 PM, Maja Kabiljo mailto:majakabi...@fb.com>> wrote: --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7883/ --- Review request for giraph. Description --- Having a lot of maps in SendMessageCache still makes it slow, so here is another step towards making it faster. Here are the results on PageRankBenchmark: 10m vertices, 100 edges per vertex, 10 workers 1 thread: Total superstep time: 54s -> 35s 20m vertices, 100 edges per vertex, 12 workers 4 threads: Computation time: 26s -> 17s Also tested on one of our real applications, speedup was a bit smaller, about 20-25%. This addresses bug GIRAPH-404. https://issues.apache.org/jira/browse/GIRAPH-404 Diffs - http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java 1405925 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/SendMessageCache.java 1405925 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/VertexIdMessageCollection.java 1405925 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/DiskBackedMessageStoreByPartition.java 1405925 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/SimpleMessageStore.java 1405925 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java 1405925 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/requests/SendWorkerMessagesRequest.java 1405925 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspServiceWorker.java 1405925 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/PairList.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/PairListWritable.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestFailureTest.java 1405925 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestTest.java 1405925 Diff: https://reviews.apache.org/r/7883/diff/ Testing --- mvn clean verify, pseudo-distributed tests PageRankBenchmark (results above) Tried it out with a real application Thanks, Maja Kabiljo
[jira] [Commented] (GIRAPH-409) Refactor / cleanups
[
https://issues.apache.org/jira/browse/GIRAPH-409?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491670#comment-13491670
]
Maja Kabiljo commented on GIRAPH-409:
-
Great ideas, Nitay!
1,2 - Definitely, our packaging is very bad. Maybe graph could be split by
functionality, i.e. vertices/aggregators/input etc.
4 - Not used, also we can remove MasterClientServer, and just use separate
classes on master also.
5,6 - Agreed, it should be cleaned up.
7 - Looking forward to see what you have in mind for this.
10 - Moving test classes was mentioned some time ago
(http://mail-archives.apache.org/mod_mbox/giraph-dev/201208.mbox/%3c5018efd7.7060...@apache.org%3E),
but I wasn't successful in doing it. Please go ahead!
> Refactor / cleanups
> ---
>
> Key: GIRAPH-409
> URL: https://issues.apache.org/jira/browse/GIRAPH-409
> Project: Giraph
> Issue Type: Improvement
>Reporter: Nitay Joffe
>Assignee: Nitay Joffe
>Priority: Minor
>
> Some general thoughts I've jotted down while going through the code. Writing
> them here to start tracking progress for them.
> 1. Refactor giraph.graph to giraph.master, giraph.worker. The whole
> giraph.graph package name is bad in general I think.
> 2. Cleanup giraph.utils. For example move timers stuff to giraph.time.
> 3. Change module names to be more maven-esque, that is something like
> giraph-root, giraph-core, giraph-formats.
> 4. Remove WorkerClientServer. Is this needed anymore?
> 5. Cleanup MasterThread#run: long convoluted method.
> 6. Cleanup BspService#process: lots of duplication. Use a vector of events or
> something.
> 7. Cleanup Vertex class: seems to me it has too many methods and should be a
> simpler interface (maybe even eventually an actual interface! not an abstract
> class). Add something like a Vertexes/Vertices class with helper methods that
> use can use.
> 8. {Master,Worker}Observer. Discussed elsewhere already. ALlow users to
> easily plug in code at various points in the system. Essentially a cleaner
> implementation of e.g. WorkerContext
> 9. Cleanup GraphMapper. I don't see why we even call a map() method seeing as
> we are overriding run(). We are clearly not particularly "mapreduce-y" so we
> should make it our entry point more clear than a map(). Also I think we
> should have something like a WorkerThread similar to MasterThread and clean
> up all of this to just creare whichever threads the node is assigned roles
> of.
> 10. Move examples and anything else not needed for a giraph library out into
> it's own package (like giraph-examples)?
> If someone +1s the ideas I'll work up some patches. Feel free to add other
> cleanup things here as well.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (GIRAPH-404) More SendMessageCache improvements
[ https://issues.apache.org/jira/browse/GIRAPH-404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo resolved GIRAPH-404. - Resolution: Fixed > More SendMessageCache improvements > -- > > Key: GIRAPH-404 > URL: https://issues.apache.org/jira/browse/GIRAPH-404 > Project: Giraph > Issue Type: Improvement > Reporter: Maja Kabiljo > Assignee: Maja Kabiljo > Attachments: GIRAPH-404.patch > > > Having a lot of maps in SendMessageCache still makes it slow, so here is > another step towards making it faster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: More SendMessageCache improvements
> On Nov. 5, 2012, 11:20 p.m., Avery Ching wrote: > > +1! This is pretty clean. Thanks for the review Avery, committing. - Maja --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7883/#review13115 --- On Nov. 5, 2012, 10:59 p.m., Maja Kabiljo wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/7883/ > --- > > (Updated Nov. 5, 2012, 10:59 p.m.) > > > Review request for giraph. > > > Description > --- > > Having a lot of maps in SendMessageCache still makes it slow, so here is > another step towards making it faster. > > Here are the results on PageRankBenchmark: > 10m vertices, 100 edges per vertex, 10 workers > 1 thread: Total superstep time: 54s -> 35s > 20m vertices, 100 edges per vertex, 12 workers > 4 threads: Computation time: 26s -> 17s > > Also tested on one of our real applications, speedup was a bit smaller, about > 20-25%. > > > This addresses bug GIRAPH-404. > https://issues.apache.org/jira/browse/GIRAPH-404 > > > Diffs > - > > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java > 1405925 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/SendMessageCache.java > 1405925 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/VertexIdMessageCollection.java > 1405925 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/DiskBackedMessageStoreByPartition.java > 1405925 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/SimpleMessageStore.java > 1405925 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java > 1405925 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/requests/SendWorkerMessagesRequest.java > 1405925 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspServiceWorker.java > 1405925 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/PairList.java > PRE-CREATION > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/PairListWritable.java > PRE-CREATION > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestFailureTest.java > 1405925 > > http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestTest.java > 1405925 > > Diff: https://reviews.apache.org/r/7883/diff/ > > > Testing > --- > > mvn clean verify, pseudo-distributed tests > PageRankBenchmark (results above) > Tried it out with a real application > > > Thanks, > > Maja Kabiljo > >
[jira] [Resolved] (GIRAPH-412) Checkstyle error from Giraph-403
[ https://issues.apache.org/jira/browse/GIRAPH-412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo resolved GIRAPH-412. - Resolution: Fixed > Checkstyle error from Giraph-403 > > > Key: GIRAPH-412 > URL: https://issues.apache.org/jira/browse/GIRAPH-412 > Project: Giraph > Issue Type: Bug > Reporter: Maja Kabiljo > Assignee: Maja Kabiljo > Attachments: GIRAPH-412.patch > > > I didn't try to run mvn verify before committing GIRAPH-403. It complained > about: > "The double-checked locking idiom is broken and should be avoided." > The reason why double-locking is bad doesn't apply here - it applies to cases > where we actually use the value of a variable which we are creating in this > way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-412) Checkstyle error from Giraph-403
[ https://issues.apache.org/jira/browse/GIRAPH-412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo updated GIRAPH-412: Attachment: GIRAPH-412.patch > Checkstyle error from Giraph-403 > > > Key: GIRAPH-412 > URL: https://issues.apache.org/jira/browse/GIRAPH-412 > Project: Giraph > Issue Type: Bug > Reporter: Maja Kabiljo > Attachments: GIRAPH-412.patch > > > I didn't try to run mvn verify before committing GIRAPH-403. It complained > about: > "The double-checked locking idiom is broken and should be avoided." > The reason why double-locking is bad doesn't apply here - it applies to cases > where we actually use the value of a variable which we are creating in this > way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (GIRAPH-412) Checkstyle error from Giraph-403
[ https://issues.apache.org/jira/browse/GIRAPH-412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo reassigned GIRAPH-412: --- Assignee: Maja Kabiljo > Checkstyle error from Giraph-403 > > > Key: GIRAPH-412 > URL: https://issues.apache.org/jira/browse/GIRAPH-412 > Project: Giraph > Issue Type: Bug > Reporter: Maja Kabiljo > Assignee: Maja Kabiljo > Attachments: GIRAPH-412.patch > > > I didn't try to run mvn verify before committing GIRAPH-403. It complained > about: > "The double-checked locking idiom is broken and should be avoided." > The reason why double-locking is bad doesn't apply here - it applies to cases > where we actually use the value of a variable which we are creating in this > way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (GIRAPH-412) Checkstyle error from Giraph-403
Maja Kabiljo created GIRAPH-412: --- Summary: Checkstyle error from Giraph-403 Key: GIRAPH-412 URL: https://issues.apache.org/jira/browse/GIRAPH-412 Project: Giraph Issue Type: Bug Reporter: Maja Kabiljo Attachments: GIRAPH-412.patch I didn't try to run mvn verify before committing GIRAPH-403. It complained about: "The double-checked locking idiom is broken and should be avoided." The reason why double-locking is bad doesn't apply here - it applies to cases where we actually use the value of a variable which we are creating in this way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-403) GraphMapper.notiftySentMessages need to be thread-safe
[ https://issues.apache.org/jira/browse/GIRAPH-403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491210#comment-13491210 ] Maja Kabiljo commented on GIRAPH-403: - +1, comitting. Nitay, can you please also take a quick look through the rest of metrics stuff, just to make sure everything that should be thread-safe is? > GraphMapper.notiftySentMessages need to be thread-safe > -- > > Key: GIRAPH-403 > URL: https://issues.apache.org/jira/browse/GIRAPH-403 > Project: Giraph > Issue Type: Bug > Reporter: Maja Kabiljo >Assignee: Nitay Joffe > Attachments: GIRAPH-403-2.patch, GIRAPH-403.patch > > > GraphMapper.notifySentMessages() is not thread-safe. Here is the error I got: > Caused by: java.lang.NullPointerException > at > org.apache.giraph.graph.GraphMapper.notifySentMessages(GraphMapper.java:454) > at org.apache.giraph.graph.Vertex.sendMessage(Vertex.java:225) > at org.apache.giraph.graph.Vertex.sendMessageToAllEdges(Vertex.java:266) > at > org.apache.giraph.benchmark.PageRankComputation.computePageRank(PageRankComputation.java:59) > at > org.apache.giraph.benchmark.EdgeListVertexPageRankBenchmark.compute(EdgeListVertexPageRankBenchmark.java:36) > at > org.apache.giraph.graph.ComputeCallable.computePartition(ComputeCallable.java:194) > at > org.apache.giraph.graph.ComputeCallable.call(ComputeCallable.java:139) > at org.apache.giraph.graph.ComputeCallable.call(ComputeCallable.java:60) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:722) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (GIRAPH-397) We should have copies of aggregators per thread to avoid synchronizing on aggregate()
[ https://issues.apache.org/jira/browse/GIRAPH-397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo resolved GIRAPH-397. - Resolution: Fixed > We should have copies of aggregators per thread to avoid synchronizing on > aggregate() > - > > Key: GIRAPH-397 > URL: https://issues.apache.org/jira/browse/GIRAPH-397 > Project: Giraph > Issue Type: Improvement > Reporter: Maja Kabiljo >Assignee: Maja Kabiljo >Priority: Minor > Attachments: GIRAPH-397.patch, GIRAPH-397.patch > > > Fixing one of TODOs from GIRAPH-273. Adding copies of aggregators for each > thread allows us not to have to synchronize on each aggregate call. > Aggregated values from each thread can be aggregated together after > computation is finished. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (GIRAPH-406) Enforce partition ids in [0, n-1]
[ https://issues.apache.org/jira/browse/GIRAPH-406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo resolved GIRAPH-406. - Resolution: Fixed > Enforce partition ids in [0, n-1] > - > > Key: GIRAPH-406 > URL: https://issues.apache.org/jira/browse/GIRAPH-406 > Project: Giraph > Issue Type: Improvement > Reporter: Maja Kabiljo > Assignee: Maja Kabiljo > Attachments: GIRAPH-406.patch > > > Throw an informative exception when partition ids are not numbers from [0, > numberOfPartitions - 1]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-406) Enforce partition ids in [0, n-1]
[ https://issues.apache.org/jira/browse/GIRAPH-406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491197#comment-13491197 ] Maja Kabiljo commented on GIRAPH-406: - It passed now (https://builds.apache.org/job/Giraph-trunk-Commit/271/). Wonder what happened last time... > Enforce partition ids in [0, n-1] > - > > Key: GIRAPH-406 > URL: https://issues.apache.org/jira/browse/GIRAPH-406 > Project: Giraph > Issue Type: Improvement > Reporter: Maja Kabiljo > Assignee: Maja Kabiljo > Attachments: GIRAPH-406.patch > > > Throw an informative exception when partition ids are not numbers from [0, > numberOfPartitions - 1]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-406) Enforce partition ids in [0, n-1]
[ https://issues.apache.org/jira/browse/GIRAPH-406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491178#comment-13491178 ] Maja Kabiljo commented on GIRAPH-406: - I don't know why this happened, for me these tests pass. Can someone else please try it? > Enforce partition ids in [0, n-1] > - > > Key: GIRAPH-406 > URL: https://issues.apache.org/jira/browse/GIRAPH-406 > Project: Giraph > Issue Type: Improvement > Reporter: Maja Kabiljo >Assignee: Maja Kabiljo > Attachments: GIRAPH-406.patch > > > Throw an informative exception when partition ids are not numbers from [0, > numberOfPartitions - 1]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-403) GraphMapper.notiftySentMessages need to be thread-safe
[
https://issues.apache.org/jira/browse/GIRAPH-403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491171#comment-13491171
]
Maja Kabiljo commented on GIRAPH-403:
-
This means we are synchronizing on every sendMessage call? How about checking
whether it's null before synchronizing? Something like:
{code}
if (timeToFirstMessageContext != null) {
synchronized (notifySentMsgLock) {
if (timeToFirstMessageContext != null) {
...
}
}
}
{code}
> GraphMapper.notiftySentMessages need to be thread-safe
> --
>
> Key: GIRAPH-403
> URL: https://issues.apache.org/jira/browse/GIRAPH-403
> Project: Giraph
> Issue Type: Bug
> Reporter: Maja Kabiljo
>Assignee: Nitay Joffe
> Attachments: GIRAPH-403.patch
>
>
> GraphMapper.notifySentMessages() is not thread-safe. Here is the error I got:
> Caused by: java.lang.NullPointerException
> at
> org.apache.giraph.graph.GraphMapper.notifySentMessages(GraphMapper.java:454)
> at org.apache.giraph.graph.Vertex.sendMessage(Vertex.java:225)
> at org.apache.giraph.graph.Vertex.sendMessageToAllEdges(Vertex.java:266)
> at
> org.apache.giraph.benchmark.PageRankComputation.computePageRank(PageRankComputation.java:59)
> at
> org.apache.giraph.benchmark.EdgeListVertexPageRankBenchmark.compute(EdgeListVertexPageRankBenchmark.java:36)
> at
> org.apache.giraph.graph.ComputeCallable.computePartition(ComputeCallable.java:194)
> at
> org.apache.giraph.graph.ComputeCallable.call(ComputeCallable.java:139)
> at org.apache.giraph.graph.ComputeCallable.call(ComputeCallable.java:60)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-411) Pseudo distributed test failures
[
https://issues.apache.org/jira/browse/GIRAPH-411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491113#comment-13491113
]
Maja Kabiljo commented on GIRAPH-411:
-
TestAutoCheckpoint error:
{code}
java.lang.RuntimeException: restartFromCheckpoint: KeeperException
at
org.apache.giraph.graph.BspServiceMaster.restartFromCheckpoint(BspServiceMaster.java:1095)
at org.apache.giraph.graph.MasterThread.run(MasterThread.java:141)
Caused by: org.apache.zookeeper.KeeperException$NoNodeException:
KeeperErrorCode = NoNode for
/_hadoopBsp/job_201211051724_0001/_edgeInputSplitDir
at org.apache.zookeeper.KeeperException.create(KeeperException.java:102)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.delete(ZooKeeper.java:728)
at org.apache.giraph.zk.ZooKeeperExt.deleteExt(ZooKeeperExt.java:307)
at
org.apache.giraph.graph.BspServiceMaster.restartFromCheckpoint(BspServiceMaster.java:1089)
{code}
TestJsonBase64Format:
{code}
Caused by: java.io.FileNotFoundException:
/var/folders/nk/xr3j7m_n0w92zk5xwwwk25nmpbq0f4/T/_giraphTests/testContinue/_logs
(No such file or directory)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.(FileInputStream.java:120)
at
org.apache.hadoop.fs.RawLocalFileSystem$TrackingFileInputStream.(RawLocalFileSystem.java:71)
at
org.apache.hadoop.fs.RawLocalFileSystem$LocalFSFileInputStream.(RawLocalFileSystem.java:107)
at
org.apache.hadoop.fs.RawLocalFileSystem.open(RawLocalFileSystem.java:177)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.(ChecksumFileSystem.java:126)
at
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:283)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:400)
at
org.apache.hadoop.mapreduce.lib.input.LineRecordReader.initialize(LineRecordReader.java:67)
at
org.apache.giraph.io.TextVertexInputFormat$TextVertexReader.initialize(TextVertexInputFormat.java:106)
at
org.apache.giraph.io.JsonBase64VertexInputFormat$JsonBase64VertexReader.initialize(JsonBase64VertexInputFormat.java:71)
at
org.apache.giraph.graph.VertexInputSplitsCallable.readInputSplit(VertexInputSplitsCallable.java:125)
at
org.apache.giraph.graph.InputSplitsCallable.loadInputSplit(InputSplitsCallable.java:334)
at
org.apache.giraph.graph.InputSplitsCallable.call(InputSplitsCallable.java:161)
{code}
> Pseudo distributed test failures
>
>
> Key: GIRAPH-411
> URL: https://issues.apache.org/jira/browse/GIRAPH-411
> Project: Giraph
> Issue Type: Bug
> Reporter: Maja Kabiljo
>
> We have two tests failing in pseudo-distributed mode again:
> TestAutoCheckpoint
> TestJsonBase64Format
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (GIRAPH-411) Pseudo distributed test failures
Maja Kabiljo created GIRAPH-411: --- Summary: Pseudo distributed test failures Key: GIRAPH-411 URL: https://issues.apache.org/jira/browse/GIRAPH-411 Project: Giraph Issue Type: Bug Reporter: Maja Kabiljo We have two tests failing in pseudo-distributed mode again: TestAutoCheckpoint TestJsonBase64Format -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-406) Enforce partition ids in [0, n-1]
[ https://issues.apache.org/jira/browse/GIRAPH-406?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo updated GIRAPH-406: Attachment: GIRAPH-406.patch > Enforce partition ids in [0, n-1] > - > > Key: GIRAPH-406 > URL: https://issues.apache.org/jira/browse/GIRAPH-406 > Project: Giraph > Issue Type: Improvement > Reporter: Maja Kabiljo > Assignee: Maja Kabiljo > Attachments: GIRAPH-406.patch > > > Throw an informative exception when partition ids are not numbers from [0, > numberOfPartitions - 1]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (GIRAPH-406) Enforce partition ids in [0, n-1]
Maja Kabiljo created GIRAPH-406: --- Summary: Enforce partition ids in [0, n-1] Key: GIRAPH-406 URL: https://issues.apache.org/jira/browse/GIRAPH-406 Project: Giraph Issue Type: Improvement Reporter: Maja Kabiljo Assignee: Maja Kabiljo Throw an informative exception when partition ids are not numbers from [0, numberOfPartitions - 1]. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-404) More SendMessageCache improvements
[ https://issues.apache.org/jira/browse/GIRAPH-404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491014#comment-13491014 ] Maja Kabiljo commented on GIRAPH-404: - https://reviews.apache.org/r/7883/ > More SendMessageCache improvements > -- > > Key: GIRAPH-404 > URL: https://issues.apache.org/jira/browse/GIRAPH-404 > Project: Giraph > Issue Type: Improvement > Reporter: Maja Kabiljo > Assignee: Maja Kabiljo > Attachments: GIRAPH-404.patch > > > Having a lot of maps in SendMessageCache still makes it slow, so here is > another step towards making it faster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: More SendMessageCache improvements
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7883/ --- (Updated Nov. 5, 2012, 10:59 p.m.) Review request for giraph. Description --- Having a lot of maps in SendMessageCache still makes it slow, so here is another step towards making it faster. Here are the results on PageRankBenchmark: 10m vertices, 100 edges per vertex, 10 workers 1 thread: Total superstep time: 54s -> 35s 20m vertices, 100 edges per vertex, 12 workers 4 threads: Computation time: 26s -> 17s Also tested on one of our real applications, speedup was a bit smaller, about 20-25%. This addresses bug GIRAPH-404. https://issues.apache.org/jira/browse/GIRAPH-404 Diffs - http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java 1405925 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/SendMessageCache.java 1405925 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/VertexIdMessageCollection.java 1405925 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/DiskBackedMessageStoreByPartition.java 1405925 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/SimpleMessageStore.java 1405925 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java 1405925 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/requests/SendWorkerMessagesRequest.java 1405925 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspServiceWorker.java 1405925 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/PairList.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/PairListWritable.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestFailureTest.java 1405925 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestTest.java 1405925 Diff: https://reviews.apache.org/r/7883/diff/ Testing --- mvn clean verify, pseudo-distributed tests PageRankBenchmark (results above) Tried it out with a real application Thanks, Maja Kabiljo
Review Request: More SendMessageCache improvements
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7883/ --- Review request for giraph. Description --- Having a lot of maps in SendMessageCache still makes it slow, so here is another step towards making it faster. Here are the results on PageRankBenchmark: 10m vertices, 100 edges per vertex, 10 workers 1 thread: Total superstep time: 54s -> 35s 20m vertices, 100 edges per vertex, 12 workers 4 threads: Computation time: 26s -> 17s Also tested on one of our real applications, speedup was a bit smaller, about 20-25%. This addresses bug GIRAPH-404. https://issues.apache.org/jira/browse/GIRAPH-404 Diffs - http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/bsp/CentralizedServiceWorker.java 1405925 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/SendMessageCache.java 1405925 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/VertexIdMessageCollection.java 1405925 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/DiskBackedMessageStoreByPartition.java 1405925 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/SimpleMessageStore.java 1405925 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java 1405925 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/requests/SendWorkerMessagesRequest.java 1405925 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspServiceWorker.java 1405925 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/PairList.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/PairListWritable.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestFailureTest.java 1405925 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestTest.java 1405925 Diff: https://reviews.apache.org/r/7883/diff/ Testing --- mvn clean verify, pseudo-distributed tests PageRankBenchmark (results above) Tried it out with a real application Thanks, Maja Kabiljo
[jira] [Commented] (GIRAPH-404) More SendMessageCache improvements
[ https://issues.apache.org/jira/browse/GIRAPH-404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13491010#comment-13491010 ] Maja Kabiljo commented on GIRAPH-404: - "not going to work well" means we are always going to have an array of length maxPartitionId (it's still going to work correctly, we'll just have an array with lots of empty space). > More SendMessageCache improvements > -- > > Key: GIRAPH-404 > URL: https://issues.apache.org/jira/browse/GIRAPH-404 > Project: Giraph > Issue Type: Improvement > Reporter: Maja Kabiljo >Assignee: Maja Kabiljo > Attachments: GIRAPH-404.patch > > > Having a lot of maps in SendMessageCache still makes it slow, so here is > another step towards making it faster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-404) More SendMessageCache improvements
[ https://issues.apache.org/jira/browse/GIRAPH-404?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo updated GIRAPH-404: Attachment: GIRAPH-404.patch Here are the results on PageRankBenchmark: 10m vertices, 100 edges per vertex, 10 workers 1 thread: Total superstep time: 54s -> 35s 20m vertices, 100 edges per vertex, 12 workers 4 threads: Computation time: 26s -> 17s Also tested on one of our real applications, speedup was a bit smaller, about 20-25%. Here I assume that partition ids are consecutive numbers, or at least very close to that, otherwise this is not going to work well. I don't think that's required by giraph right now, but I don't see a reason why it wouldn't be. What do you think? If there is a reason not to require it, we can keep two implementations of SendMessageCache. > More SendMessageCache improvements > -- > > Key: GIRAPH-404 > URL: https://issues.apache.org/jira/browse/GIRAPH-404 > Project: Giraph > Issue Type: Improvement >Reporter: Maja Kabiljo >Assignee: Maja Kabiljo > Attachments: GIRAPH-404.patch > > > Having a lot of maps in SendMessageCache still makes it slow, so here is > another step towards making it faster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (GIRAPH-404) More SendMessageCache improvements
Maja Kabiljo created GIRAPH-404: --- Summary: More SendMessageCache improvements Key: GIRAPH-404 URL: https://issues.apache.org/jira/browse/GIRAPH-404 Project: Giraph Issue Type: Improvement Reporter: Maja Kabiljo Assignee: Maja Kabiljo Having a lot of maps in SendMessageCache still makes it slow, so here is another step towards making it faster. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (GIRAPH-403) GraphMapper.notiftySentMessages need to be thread-safe
Maja Kabiljo created GIRAPH-403: --- Summary: GraphMapper.notiftySentMessages need to be thread-safe Key: GIRAPH-403 URL: https://issues.apache.org/jira/browse/GIRAPH-403 Project: Giraph Issue Type: Bug Reporter: Maja Kabiljo Assignee: Nitay Joffe GraphMapper.notifySentMessages() is not thread-safe. Here is the error I got: Caused by: java.lang.NullPointerException at org.apache.giraph.graph.GraphMapper.notifySentMessages(GraphMapper.java:454) at org.apache.giraph.graph.Vertex.sendMessage(Vertex.java:225) at org.apache.giraph.graph.Vertex.sendMessageToAllEdges(Vertex.java:266) at org.apache.giraph.benchmark.PageRankComputation.computePageRank(PageRankComputation.java:59) at org.apache.giraph.benchmark.EdgeListVertexPageRankBenchmark.compute(EdgeListVertexPageRankBenchmark.java:36) at org.apache.giraph.graph.ComputeCallable.computePartition(ComputeCallable.java:194) at org.apache.giraph.graph.ComputeCallable.call(ComputeCallable.java:139) at org.apache.giraph.graph.ComputeCallable.call(ComputeCallable.java:60) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-397) We should have copies of aggregators per thread to avoid synchronizing on aggregate()
[ https://issues.apache.org/jira/browse/GIRAPH-397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo updated GIRAPH-397: Attachment: GIRAPH-397.patch Eli, thanks for looking! I don't think it makes a big difference, since there are just a few extra empty maps, but ok, I turned it off by default. That's the only change in the patch. Will commit by the end of the day if nobody objects. > We should have copies of aggregators per thread to avoid synchronizing on > aggregate() > - > > Key: GIRAPH-397 > URL: https://issues.apache.org/jira/browse/GIRAPH-397 > Project: Giraph > Issue Type: Improvement >Reporter: Maja Kabiljo >Assignee: Maja Kabiljo >Priority: Minor > Attachments: GIRAPH-397.patch, GIRAPH-397.patch > > > Fixing one of TODOs from GIRAPH-273. Adding copies of aggregators for each > thread allows us not to have to synchronize on each aggregate call. > Aggregated values from each thread can be aggregated together after > computation is finished. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: GIRAPH-390: MasterCompute postApplication callback.
> On Nov. 2, 2012, 4:36 a.m., Alessandro Presta wrote: > > I'm not sure how I feel about using MasterCompute for something other than > > computation. I guess it gets the job done but I wonder if we should have a > > better place for this sort of code (like a MasterContext or whatever). It's > > just a matter of separating the roles. > > Other than that, the code looks good. > > Nitay Joffe wrote: > Actually I started it exactly that way but Maja said it's easiser / > cleaner to just add the method to MasterCompute rather than create a separate > MasterContext object. > > Alessandro Presta wrote: > The reason why this is problematic is that MasterCompute is supposed to > be used for defining an algorithm (together with Vertex). Say an application > uses MC to check some termination condition or working on aggregators. > If you also want to add stats the way we're doing it, you need to either > modify the same MC (mixing algorithm code with instrumentation) or make it > subclass a different MC. If you then want to plug that functionality on/off, > there is no seamless way. > I advocate for a clear separation of algorithm code and optional > logging/infrastructure code. > I agree it's slightly easier this way, but I wouldn't say it's cleaner. We have Vertex computation per vertex and WorkerContext per worker, i.e. many vertices. On master you have just one master.compute, that's why I don't see the clear distinction between MasterCompute and MasterContext, it seems redundant having both. But if you feel it would add value, go ahead. When I suggested using master.compute for everything there were no arguments against it, yours sounds reasonable. - Maja --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7762/#review13024 --- On Nov. 2, 2012, 5:53 p.m., Nitay Joffe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/7762/ > --- > > (Updated Nov. 2, 2012, 5:53 p.m.) > > > Review request for giraph. > > > Description > --- > > GIRAPH-390: MasterCompute postApplication callback. > > > Diffs > - > > giraph/src/main/java/org/apache/giraph/bsp/CentralizedServiceMaster.java > 69751086ade95c4d645feafa5131587bb9f18206 > giraph/src/main/java/org/apache/giraph/counters/GiraphHadoopCounter.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/counters/GiraphStats.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/counters/GiraphTimers.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/counters/HadoopCountersBase.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/counters/package-info.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/graph/BspServiceMaster.java > 98a0bd2c6dc9602ea17612d4f68461d5fa43fb00 > giraph/src/main/java/org/apache/giraph/graph/MasterCompute.java > 9de23ad4a7781d6342339f26f44fb1c1d9bce2b0 > giraph/src/main/java/org/apache/giraph/graph/MasterThread.java > 85842d0ab59330d95b843c440f2d5705b41e0d1b > giraph/src/main/java/org/apache/giraph/metrics/EmptyMetricsRegistry.java > 262db29bbd4797e470f426999b85f2c1a7d1dee3 > giraph/src/main/java/org/apache/giraph/metrics/GiraphMetrics.java > ea1cc1ef69aaa2c786b99de51cd30e39ee9f1620 > giraph/src/main/java/org/apache/giraph/metrics/NoOpMetricsRegistry.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/metrics/package-info.java > f7529f52527e8e4253f579bd6d01865aa980926b > > Diff: https://reviews.apache.org/r/7762/diff/ > > > Testing > --- > > > Thanks, > > Nitay Joffe > >
[jira] [Commented] (GIRAPH-398) Missing a dependency?
[ https://issues.apache.org/jira/browse/GIRAPH-398?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13489008#comment-13489008 ] Maja Kabiljo commented on GIRAPH-398: - +1, works now. Thanks, Nitay! > Missing a dependency? > - > > Key: GIRAPH-398 > URL: https://issues.apache.org/jira/browse/GIRAPH-398 > Project: Giraph > Issue Type: Bug > Reporter: Maja Kabiljo >Assignee: Nitay Joffe > Attachments: GIRAPH-398.patch > > > Here is the error I got: > 2012-10-31 12:24:45,404 FATAL org.apache.hadoop.mapred.Child: Error running > child : java.lang.NoSuchMethodError: > org.slf4j.impl.StaticLoggerBinder.getSingleton()Lorg/slf4j/impl/StaticLoggerBinder; > at org.slf4j.LoggerFactory.bind(LoggerFactory.java:128) > at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:108) > at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:279) > at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:252) > at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:265) > at > com.yammer.metrics.reporting.JmxReporter.(JmxReporter.java:19) > at org.apache.giraph.metrics.GiraphMetrics.init(GiraphMetrics.java:86) > at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:313) > at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:664) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369) > at org.apache.hadoop.mapred.Child$4.run(Child.java:259) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) > at org.apache.hadoop.mapred.Child.main(Child.java:253) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (GIRAPH-398) Missing a dependency?
Maja Kabiljo created GIRAPH-398: --- Summary: Missing a dependency? Key: GIRAPH-398 URL: https://issues.apache.org/jira/browse/GIRAPH-398 Project: Giraph Issue Type: Bug Reporter: Maja Kabiljo Assignee: Nitay Joffe Here is the error I got: 2012-10-31 12:24:45,404 FATAL org.apache.hadoop.mapred.Child: Error running child : java.lang.NoSuchMethodError: org.slf4j.impl.StaticLoggerBinder.getSingleton()Lorg/slf4j/impl/StaticLoggerBinder; at org.slf4j.LoggerFactory.bind(LoggerFactory.java:128) at org.slf4j.LoggerFactory.performInitialization(LoggerFactory.java:108) at org.slf4j.LoggerFactory.getILoggerFactory(LoggerFactory.java:279) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:252) at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:265) at com.yammer.metrics.reporting.JmxReporter.(JmxReporter.java:19) at org.apache.giraph.metrics.GiraphMetrics.init(GiraphMetrics.java:86) at org.apache.giraph.graph.GraphMapper.setup(GraphMapper.java:313) at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:664) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369) at org.apache.hadoop.mapred.Child$4.run(Child.java:259) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059) at org.apache.hadoop.mapred.Child.main(Child.java:253) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-397) We should have copies of aggregators per thread to avoid synchronizing on aggregate()
[ https://issues.apache.org/jira/browse/GIRAPH-397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo updated GIRAPH-397: Attachment: GIRAPH-397.patch I added this as an option (default to use it), for one of our applications with huge aggregators we don't want to use it since we don't want to spend that much memory on having a lot of copies of aggregators. Passes verify and AggregatorsBenchmark on cluster. > We should have copies of aggregators per thread to avoid synchronizing on > aggregate() > - > > Key: GIRAPH-397 > URL: https://issues.apache.org/jira/browse/GIRAPH-397 > Project: Giraph > Issue Type: Improvement >Reporter: Maja Kabiljo >Assignee: Maja Kabiljo >Priority: Minor > Attachments: GIRAPH-397.patch > > > Fixing one of TODOs from GIRAPH-273. Adding copies of aggregators for each > thread allows us not to have to synchronize on each aggregate call. > Aggregated values from each thread can be aggregated together after > computation is finished. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (GIRAPH-397) We should have copies of aggregators per thread to avoid synchronizing on aggregate()
Maja Kabiljo created GIRAPH-397: --- Summary: We should have copies of aggregators per thread to avoid synchronizing on aggregate() Key: GIRAPH-397 URL: https://issues.apache.org/jira/browse/GIRAPH-397 Project: Giraph Issue Type: Improvement Reporter: Maja Kabiljo Assignee: Maja Kabiljo Priority: Minor Fixing one of TODOs from GIRAPH-273. Adding copies of aggregators for each thread allows us not to have to synchronize on each aggregate call. Aggregated values from each thread can be aggregated together after computation is finished. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-395) No need to make HashWorkerPartitioner thread-safe.
[ https://issues.apache.org/jira/browse/GIRAPH-395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13488348#comment-13488348 ] Maja Kabiljo commented on GIRAPH-395: - Good catch, Avery! Any reason why getPartitionOwners is synchronized and HashRangeWorkerPartitioner uses that synchronized call? > No need to make HashWorkerPartitioner thread-safe. > -- > > Key: GIRAPH-395 > URL: https://issues.apache.org/jira/browse/GIRAPH-395 > Project: Giraph > Issue Type: Improvement >Reporter: Avery Ching >Assignee: Avery Ching > Attachments: GIRAPH-395.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: GIRAPH-390: MasterCompute postApplication callback.
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7762/#review12888 --- Thanks Nitay, this is a nice feature to have. I like that you pulled out counters to separate classes. But why do you use static instances of these classes, can't you have them as fields of BspServiceMaster/MasterThread? Also, it would be nicer if postApplication wasn't called from inside cleanup. - Maja Kabiljo On Oct. 29, 2012, 5:20 p.m., Nitay Joffe wrote: > > --- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/7762/ > --- > > (Updated Oct. 29, 2012, 5:20 p.m.) > > > Review request for giraph. > > > Description > --- > > GIRAPH-390: MasterCompute postApplication callback. > > > Diffs > - > > giraph/src/main/java/org/apache/giraph/graph/BspServiceMaster.java > d68debc41b44e20c0dcb5907d1a1b398b759b149 > giraph/src/main/java/org/apache/giraph/graph/MasterCompute.java > 9de23ad4a7781d6342339f26f44fb1c1d9bce2b0 > giraph/src/main/java/org/apache/giraph/graph/MasterThread.java > 85842d0ab59330d95b843c440f2d5705b41e0d1b > giraph/src/main/java/org/apache/giraph/metrics/GiraphHadoopCounter.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/metrics/GiraphStats.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/metrics/GiraphTimers.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/metrics/HadoopCountersBase.java > PRE-CREATION > giraph/src/main/java/org/apache/giraph/metrics/package-info.java > PRE-CREATION > > Diff: https://reviews.apache.org/r/7762/diff/ > > > Testing > --- > > > Thanks, > > Nitay Joffe > >
[jira] [Commented] (GIRAPH-388) Improve the way we keep outgoing messages
[ https://issues.apache.org/jira/browse/GIRAPH-388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486367#comment-13486367 ] Maja Kabiljo commented on GIRAPH-388: - My mistake, I meant GIRAPH-322, not 314... The changes I have in mind don't try to improve the case when we have the same message sent to many vertices, as you are trying to do there (but they don't hurt it either). That's something we might want to have as an option. Though in a real social graph application, and running on a large number of workers, the question is how big is the chance that several destination vertices end up on the same worker. I did test this with one of our internal applications also, and the speedup was similar to the one reported. I agree that benchmarks are not good enough to test all improvements, but in some cases it's a good guideline. > Improve the way we keep outgoing messages > - > > Key: GIRAPH-388 > URL: https://issues.apache.org/jira/browse/GIRAPH-388 > Project: Giraph > Issue Type: Improvement > Reporter: Maja Kabiljo >Assignee: Maja Kabiljo > Attachments: GIRAPH-388.patch > > > As per discussion on GIRAPH-357, in standard application chances that we get > to use client-side combiner are very low. I experimented with benefits which > we can get from not having the client-side combiner at all. It turns out that > having a lot of maps in SendMessageCache, and then collection inside each of > them, really hurts the performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-388) Improve the way we keep outgoing messages
[ https://issues.apache.org/jira/browse/GIRAPH-388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486248#comment-13486248 ] Maja Kabiljo commented on GIRAPH-388: - Thanks for the review, Avery! > Improve the way we keep outgoing messages > - > > Key: GIRAPH-388 > URL: https://issues.apache.org/jira/browse/GIRAPH-388 > Project: Giraph > Issue Type: Improvement > Reporter: Maja Kabiljo > Assignee: Maja Kabiljo > Attachments: GIRAPH-388.patch > > > As per discussion on GIRAPH-357, in standard application chances that we get > to use client-side combiner are very low. I experimented with benefits which > we can get from not having the client-side combiner at all. It turns out that > having a lot of maps in SendMessageCache, and then collection inside each of > them, really hurts the performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-388) Improve the way we keep outgoing messages
[ https://issues.apache.org/jira/browse/GIRAPH-388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486247#comment-13486247 ] Maja Kabiljo commented on GIRAPH-388: - Eli, thank you for your comments. Because of message flushing, it rarely happens that we have more than one message for single vertex in one request (which was also the reason why GIRAPH-357 was an improvement). So having same vertexId several times in VertexIdMessageCollection is very rare. From my experiments it turns out that in applications like Page Rank, the most of the time goes on creating and querying all messages structures, both on sender and receiver side (in SendMessageCache, SimpleMessageStore, SendWorkerMessagesRequest). I have a few more changes there which I'll be posting soon which further improve performance. As for the problem which you are addressing in GIRAPH-314, I think we won't be able to have one implementation which has the best performance in all the cases. Overhead of having mappings and other stuff around will be too expensive for another kind of application. So we'll probably end up having something like that as an option. Would like to hear your thoughts on this. > Improve the way we keep outgoing messages > - > > Key: GIRAPH-388 > URL: https://issues.apache.org/jira/browse/GIRAPH-388 > Project: Giraph > Issue Type: Improvement >Reporter: Maja Kabiljo >Assignee: Maja Kabiljo > Attachments: GIRAPH-388.patch > > > As per discussion on GIRAPH-357, in standard application chances that we get > to use client-side combiner are very low. I experimented with benefits which > we can get from not having the client-side combiner at all. It turns out that > having a lot of maps in SendMessageCache, and then collection inside each of > them, really hurts the performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request: Improve the way we keep outgoing messages
--- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/7764/ --- Review request for giraph. Description --- As per discussion on GIRAPH-357, in standard application chances that we get to use client-side combiner are very low. I experimented with benefits which we can get from not having the client-side combiner at all. It turns out that having a lot of maps in SendMessageCache, and then collection inside each of them, really hurts the performance. This addresses bug GIRAPH-388. https://issues.apache.org/jira/browse/GIRAPH-388 Diffs - http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/SendMessageCache.java 1402328 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/VertexIdMessageCollection.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/DiskBackedMessageStoreByPartition.java 1402328 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/MessageStoreByPartition.java 1402328 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/messages/SimpleMessageStore.java 1402328 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java 1402328 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/requests/SendWorkerMessagesRequest.java 1402328 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestFailureTest.java 1402328 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/comm/RequestTest.java 1402328 Diff: https://reviews.apache.org/r/7764/diff/ Testing --- Single node and pseudo-distributed tests. Benchmark results: https://issues.apache.org/jira/browse/GIRAPH-388?focusedCommentId=13484591 Thanks, Maja Kabiljo
[jira] [Commented] (GIRAPH-388) Improve the way we keep outgoing messages
[ https://issues.apache.org/jira/browse/GIRAPH-388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13486168#comment-13486168 ] Maja Kabiljo commented on GIRAPH-388: - Eli, I'm not removing the information about partition id. I'm changing the way messages inside one partition are stored. Review board: https://reviews.apache.org/r/7764/ > Improve the way we keep outgoing messages > - > > Key: GIRAPH-388 > URL: https://issues.apache.org/jira/browse/GIRAPH-388 > Project: Giraph > Issue Type: Improvement >Reporter: Maja Kabiljo > Attachments: GIRAPH-388.patch > > > As per discussion on GIRAPH-357, in standard application chances that we get > to use client-side combiner are very low. I experimented with benefits which > we can get from not having the client-side combiner at all. It turns out that > having a lot of maps in SendMessageCache, and then collection inside each of > them, really hurts the performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-386) ClassCastException when giraph.SplitMasterWorker=false
[
https://issues.apache.org/jira/browse/GIRAPH-386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maja Kabiljo updated GIRAPH-386:
Attachment: GIRAPH-386.patch
Thanks, Avery. I created TaskInfo and then we have MasterInfo and WorkerInfo
extending it.
> ClassCastException when giraph.SplitMasterWorker=false
> --
>
> Key: GIRAPH-386
> URL: https://issues.apache.org/jira/browse/GIRAPH-386
> Project: Giraph
> Issue Type: Bug
>Reporter: Jaeho Shin
> Assignee: Maja Kabiljo
> Attachments: GIRAPH-386.patch, GIRAPH-386.patch
>
>
> Using -Dgiraph.SplitMasterWorker=false with a recent trunk (r1401165) caused
> the machine who is playing the master role (showing itself as {{ALL}} from
> the task tracker page) to throw ClassCastException (SendVertexRequest ->
> MasterRequest) from MasterRequestServerHandler class. I'm trying to use as
> many machines as possible for actual computation (can't afford to waste one
> for master+ZK). This worked fine with r1388628 (roughly a month ago), so a
> recent change must've broken something. Here's the relevant log I captured:
> {code}
> 2012-10-24 23:08:02,152 WARN
> org.apache.giraph.comm.netty.handler.RequestServerHandler: exceptionCaught:
> Channel failed with remote address /10.x.y.z:41780
> java.lang.ClassCastException:
> org.apache.giraph.comm.requests.SendVertexRequest cannot be cast to
> org.apache.giraph.comm.requests.MasterRequest
> at
> org.apache.giraph.comm.netty.handler.MasterRequestServerHandler.processRequest(MasterRequestServerHandler.java:25)
> at
> org.apache.giraph.comm.netty.handler.RequestServerHandler.messageReceived(RequestServerHandler.java:100)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:71)
> at
> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:45)
> at
> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:69)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-388) Improve the way we keep outgoing messages
[ https://issues.apache.org/jira/browse/GIRAPH-388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo updated GIRAPH-388: Attachment: GIRAPH-388.patch By replacing per partition storage with two simple lists here are the improvements I got: PageRankBenchmark, 30m vertices, 100 edges, 40 workers Single computation thread Average computation time from 80s -> 48s Total superstep time 85s -> 56s 10 computation threads Average computation time 14s -> 8s Total superstep time 21s -> 17s I can fix the part from DiskBackedMessageStoreByPartition not to have to copy values to the map, by adding more methods to message store interfaces. But I think those will need to be revised soon anyway because of this and upcoming changes. This removes client-side combiner completely, does anyone have an application which will suffer because of it? If needed, we can have two implementations of something like VertexIdMessageCollection, one of which will still allow combiner to be used. > Improve the way we keep outgoing messages > - > > Key: GIRAPH-388 > URL: https://issues.apache.org/jira/browse/GIRAPH-388 > Project: Giraph > Issue Type: Improvement >Reporter: Maja Kabiljo > Attachments: GIRAPH-388.patch > > > As per discussion on GIRAPH-357, in standard application chances that we get > to use client-side combiner are very low. I experimented with benefits which > we can get from not having the client-side combiner at all. It turns out that > having a lot of maps in SendMessageCache, and then collection inside each of > them, really hurts the performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (GIRAPH-388) Improve the way we keep outgoing messages
Maja Kabiljo created GIRAPH-388: --- Summary: Improve the way we keep outgoing messages Key: GIRAPH-388 URL: https://issues.apache.org/jira/browse/GIRAPH-388 Project: Giraph Issue Type: Improvement Reporter: Maja Kabiljo Attachments: GIRAPH-388.patch As per discussion on GIRAPH-357, in standard application chances that we get to use client-side combiner are very low. I experimented with benefits which we can get from not having the client-side combiner at all. It turns out that having a lot of maps in SendMessageCache, and then collection inside each of them, really hurts the performance. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-273) Aggregators shouldn't use Zookeeper
[ https://issues.apache.org/jira/browse/GIRAPH-273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo updated GIRAPH-273: Attachment: GIRAPH-273.diff > Aggregators shouldn't use Zookeeper > --- > > Key: GIRAPH-273 > URL: https://issues.apache.org/jira/browse/GIRAPH-273 > Project: Giraph > Issue Type: Improvement > Reporter: Maja Kabiljo > Assignee: Maja Kabiljo > Attachments: GIRAPH-273.diff > > > We use Zookeeper znodes to transfer aggregated values from workers to master > and back. Zookeeper is supposed to be used for coordination, and it also has > a memory limit which prevents users from having aggregators with large value > objects. These are the reasons why we should implement aggregators gathering > and distribution in a different way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: Review Request: Remove aggregator handling from Zookeeper
/SendAggregatedValueCache.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/aggregators/SendAggregatorCache.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/aggregators/WorkerAggregatorRequestProcessor.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/aggregators/package-info.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyMasterClient.java 1402331 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyMasterClientServer.java 1402331 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyMasterServer.java 1402331 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerAggregatorRequestProcessor.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/handler/MasterRequestServerHandler.java 1402331 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/requests/ByteArrayRequest.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/requests/MasterRequest.java 1402331 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/requests/RequestType.java 1402331 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/requests/SendAggregatorsToMasterRequest.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/requests/SendAggregatorsToOwnerRequest.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/requests/SendAggregatorsToWorkerRequest.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/requests/SendWorkerAggregatorsRequest.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/AggregatorsTestVertex.java 1402331 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/AggregatorHandler.java 1402331 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspService.java 1402331 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspServiceMaster.java 1402331 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspServiceWorker.java 1402331 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/GraphMapper.java 1402331 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/MasterAggregatorHandler.java 1402331 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/WorkerAggregatorHandler.java 1402331 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/ExpectedBarrier.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/graph/TestAggregatorsHandling.java 1402331 Diff: https://reviews.apache.org/r/7673/diff/ Testing --- mvn clean verify, tests in pseudo-distributed mode. AggregatorsBenchmark (which also checks for correctness) on various amount of aggregators and wokrers. Tested on fb application which uses a lot of big aggregators, also tested it with multithreading. Thanks, Maja Kabiljo
[jira] [Updated] (GIRAPH-386) ClassCastException when giraph.SplitMasterWorker=false
[
https://issues.apache.org/jira/browse/GIRAPH-386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maja Kabiljo updated GIRAPH-386:
Attachment: GIRAPH-386.patch
I'm fixing this for now by artificially setting the task id from master's
WorkerInfo to -1. Since we use that taskid only for the communication purposes
that's fine, and all task ids are >=0.
If someone has a cleaner solution in mind I'd be happy to hear it, or when I
think of something better I'll make a change. But for now, this fixes the issue.
> ClassCastException when giraph.SplitMasterWorker=false
> --
>
> Key: GIRAPH-386
> URL: https://issues.apache.org/jira/browse/GIRAPH-386
> Project: Giraph
> Issue Type: Bug
> Reporter: Jaeho Shin
>Assignee: Maja Kabiljo
> Attachments: GIRAPH-386.patch
>
>
> Using -Dgiraph.SplitMasterWorker=false with a recent trunk (r1401165) caused
> the machine who is playing the master role (showing itself as {{ALL}} from
> the task tracker page) to throw ClassCastException (SendVertexRequest ->
> MasterRequest) from MasterRequestServerHandler class. I'm trying to use as
> many machines as possible for actual computation (can't afford to waste one
> for master+ZK). This worked fine with r1388628 (roughly a month ago), so a
> recent change must've broken something. Here's the relevant log I captured:
> {code}
> 2012-10-24 23:08:02,152 WARN
> org.apache.giraph.comm.netty.handler.RequestServerHandler: exceptionCaught:
> Channel failed with remote address /10.x.y.z:41780
> java.lang.ClassCastException:
> org.apache.giraph.comm.requests.SendVertexRequest cannot be cast to
> org.apache.giraph.comm.requests.MasterRequest
> at
> org.apache.giraph.comm.netty.handler.MasterRequestServerHandler.processRequest(MasterRequestServerHandler.java:25)
> at
> org.apache.giraph.comm.netty.handler.RequestServerHandler.messageReceived(RequestServerHandler.java:100)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:71)
> at
> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:45)
> at
> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:69)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-386) ClassCastException when giraph.SplitMasterWorker=false
[
https://issues.apache.org/jira/browse/GIRAPH-386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13483779#comment-13483779
]
Maja Kabiljo commented on GIRAPH-386:
-
I see, this is because we added mapping from taskid to InetSocketAddress to
NettyClient. And in mode with master and worker on the same node, we have two
addresses corresponding to the same taskid. Let me think what's the best way to
fix this. Just today we were talking about having our own ids for workers
because of some other optimization, maybe that can also be used to fix this
problem.
> ClassCastException when giraph.SplitMasterWorker=false
> --
>
> Key: GIRAPH-386
> URL: https://issues.apache.org/jira/browse/GIRAPH-386
> Project: Giraph
> Issue Type: Bug
>Reporter: Jaeho Shin
>Assignee: Maja Kabiljo
>
> Using -Dgiraph.SplitMasterWorker=false with a recent trunk (r1401165) caused
> the machine who is playing the master role (showing itself as {{ALL}} from
> the task tracker page) to throw ClassCastException (SendVertexRequest ->
> MasterRequest) from MasterRequestServerHandler class. I'm trying to use as
> many machines as possible for actual computation (can't afford to waste one
> for master+ZK). This worked fine with r1388628 (roughly a month ago), so a
> recent change must've broken something. Here's the relevant log I captured:
> {code}
> 2012-10-24 23:08:02,152 WARN
> org.apache.giraph.comm.netty.handler.RequestServerHandler: exceptionCaught:
> Channel failed with remote address /10.x.y.z:41780
> java.lang.ClassCastException:
> org.apache.giraph.comm.requests.SendVertexRequest cannot be cast to
> org.apache.giraph.comm.requests.MasterRequest
> at
> org.apache.giraph.comm.netty.handler.MasterRequestServerHandler.processRequest(MasterRequestServerHandler.java:25)
> at
> org.apache.giraph.comm.netty.handler.RequestServerHandler.messageReceived(RequestServerHandler.java:100)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:71)
> at
> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:45)
> at
> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:69)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (GIRAPH-386) ClassCastException when giraph.SplitMasterWorker=false
[
https://issues.apache.org/jira/browse/GIRAPH-386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Maja Kabiljo reassigned GIRAPH-386:
---
Assignee: Maja Kabiljo
> ClassCastException when giraph.SplitMasterWorker=false
> --
>
> Key: GIRAPH-386
> URL: https://issues.apache.org/jira/browse/GIRAPH-386
> Project: Giraph
> Issue Type: Bug
>Reporter: Jaeho Shin
> Assignee: Maja Kabiljo
>
> Using -Dgiraph.SplitMasterWorker=false with a recent trunk (r1401165) caused
> the machine who is playing the master role (showing itself as {{ALL}} from
> the task tracker page) to throw ClassCastException (SendVertexRequest ->
> MasterRequest) from MasterRequestServerHandler class. I'm trying to use as
> many machines as possible for actual computation (can't afford to waste one
> for master+ZK). This worked fine with r1388628 (roughly a month ago), so a
> recent change must've broken something. Here's the relevant log I captured:
> {code}
> 2012-10-24 23:08:02,152 WARN
> org.apache.giraph.comm.netty.handler.RequestServerHandler: exceptionCaught:
> Channel failed with remote address /10.x.y.z:41780
> java.lang.ClassCastException:
> org.apache.giraph.comm.requests.SendVertexRequest cannot be cast to
> org.apache.giraph.comm.requests.MasterRequest
> at
> org.apache.giraph.comm.netty.handler.MasterRequestServerHandler.processRequest(MasterRequestServerHandler.java:25)
> at
> org.apache.giraph.comm.netty.handler.RequestServerHandler.messageReceived(RequestServerHandler.java:100)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:71)
> at
> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:45)
> at
> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:69)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-386) ClassCastException when giraph.SplitMasterWorker=false
[
https://issues.apache.org/jira/browse/GIRAPH-386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13483756#comment-13483756
]
Maja Kabiljo commented on GIRAPH-386:
-
Thanks for reporting it, Jaeho. I'm not sure why would that happen - current
trunk doesn't even use any MasterRequests, but I'll investigate.
> ClassCastException when giraph.SplitMasterWorker=false
> --
>
> Key: GIRAPH-386
> URL: https://issues.apache.org/jira/browse/GIRAPH-386
> Project: Giraph
> Issue Type: Bug
>Reporter: Jaeho Shin
>
> Using -Dgiraph.SplitMasterWorker=false with a recent trunk (r1401165) caused
> the machine who is playing the master role (showing itself as {{ALL}} from
> the task tracker page) to throw ClassCastException (SendVertexRequest ->
> MasterRequest) from MasterRequestServerHandler class. I'm trying to use as
> many machines as possible for actual computation (can't afford to waste one
> for master+ZK). This worked fine with r1388628 (roughly a month ago), so a
> recent change must've broken something. Here's the relevant log I captured:
> {code}
> 2012-10-24 23:08:02,152 WARN
> org.apache.giraph.comm.netty.handler.RequestServerHandler: exceptionCaught:
> Channel failed with remote address /10.x.y.z:41780
> java.lang.ClassCastException:
> org.apache.giraph.comm.requests.SendVertexRequest cannot be cast to
> org.apache.giraph.comm.requests.MasterRequest
> at
> org.apache.giraph.comm.netty.handler.MasterRequestServerHandler.processRequest(MasterRequestServerHandler.java:25)
> at
> org.apache.giraph.comm.netty.handler.RequestServerHandler.messageReceived(RequestServerHandler.java:100)
> at
> org.jboss.netty.channel.Channels.fireMessageReceived(Channels.java:296)
> at
> org.jboss.netty.handler.codec.oneone.OneToOneDecoder.handleUpstream(OneToOneDecoder.java:71)
> at
> org.jboss.netty.handler.execution.ChannelUpstreamEventRunnable.doRun(ChannelUpstreamEventRunnable.java:45)
> at
> org.jboss.netty.handler.execution.ChannelEventRunnable.run(ChannelEventRunnable.java:69)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (GIRAPH-273) Aggregators shouldn't use Zookeeper
[ https://issues.apache.org/jira/browse/GIRAPH-273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13480609#comment-13480609 ] Maja Kabiljo commented on GIRAPH-273: - https://reviews.apache.org/r/7673/ > Aggregators shouldn't use Zookeeper > --- > > Key: GIRAPH-273 > URL: https://issues.apache.org/jira/browse/GIRAPH-273 > Project: Giraph > Issue Type: Improvement > Reporter: Maja Kabiljo >Assignee: Maja Kabiljo > > We use Zookeeper znodes to transfer aggregated values from workers to master > and back. Zookeeper is supposed to be used for coordination, and it also has > a memory limit which prevents users from having aggregators with large value > objects. These are the reasons why we should implement aggregators gathering > and distribution in a different way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Review Request: Remove aggregator handling from Zookeeper
rc/main/java/org/apache/giraph/comm/aggregators/WorkerAggregatorRequestProcessor.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/aggregators/package-info.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyMasterClient.java 1400335 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyMasterClientServer.java 1400335 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyMasterServer.java 1400335 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/NettyWorkerAggregatorRequestProcessor.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/netty/handler/MasterRequestServerHandler.java 1400335 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/requests/ByteArrayRequest.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/requests/MasterRequest.java 1400335 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/requests/RequestType.java 1400335 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/requests/SendAggregatorsToMasterRequest.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/requests/SendAggregatorsToOwnerRequest.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/requests/SendAggregatorsToWorkerRequest.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/comm/requests/SendWorkerAggregatorsRequest.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/examples/AggregatorsTestVertex.java 1400335 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/AggregatorHandler.java 1400335 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspService.java 1400335 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspServiceMaster.java 1400335 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/BspServiceWorker.java 1400335 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/GraphMapper.java 1400335 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/MasterAggregatorHandler.java 1400335 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/graph/WorkerAggregatorHandler.java 1400335 http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/main/java/org/apache/giraph/utils/ExpectedBarrier.java PRE-CREATION http://svn.apache.org/repos/asf/giraph/trunk/giraph/src/test/java/org/apache/giraph/graph/TestAggregatorsHandling.java 1400335 Diff: https://reviews.apache.org/r/7673/diff/ Testing --- mvn clean verify, tests in pseudo-distributed mode. AggregatorsBenchmark (which also checks for correctness) on various amount of aggregators and wokrers. Tested on fb application which uses a lot of big aggregators, also tested it with multithreading. Thanks, Maja Kabiljo
[jira] [Commented] (GIRAPH-379) HiveGiraphRunner should have a skipOutput option for testing
[ https://issues.apache.org/jira/browse/GIRAPH-379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478756#comment-13478756 ] Maja Kabiljo commented on GIRAPH-379: - Good idea, +1. > HiveGiraphRunner should have a skipOutput option for testing > > > Key: GIRAPH-379 > URL: https://issues.apache.org/jira/browse/GIRAPH-379 > Project: Giraph > Issue Type: Improvement >Reporter: Avery Ching >Assignee: Avery Ching > Attachments: GIRAPH-379.patch > > > It is useful to be able to skip the output when using HiveGiraphRunner when > doing testing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (GIRAPH-380) Hadoop_non_secure is broken
[ https://issues.apache.org/jira/browse/GIRAPH-380?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maja Kabiljo updated GIRAPH-380: Attachment: GIRAPH-380.patch > Hadoop_non_secure is broken > --- > > Key: GIRAPH-380 > URL: https://issues.apache.org/jira/browse/GIRAPH-380 > Project: Giraph > Issue Type: Bug > Reporter: Maja Kabiljo > Assignee: Maja Kabiljo > Attachments: GIRAPH-380.patch > > > This is my bad, from GIRAPH-372, I didn't change openConnections call in > munged part of the code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (GIRAPH-380) Hadoop_non_secure is broken
Maja Kabiljo created GIRAPH-380: --- Summary: Hadoop_non_secure is broken Key: GIRAPH-380 URL: https://issues.apache.org/jira/browse/GIRAPH-380 Project: Giraph Issue Type: Bug Reporter: Maja Kabiljo Assignee: Maja Kabiljo This is my bad, from GIRAPH-372, I didn't change openConnections call in munged part of the code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
