----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/21006/ -----------------------------------------------------------
Review request for giraph, Avery Ching and Maja Kabiljo. Repository: giraph-git Description ------- Currently writableRequests are deserialized from network buffers into a bytearray based datastructure (vertexIdData - ByteArrayVertexIdEdges, ByteArrayVertexIdMessages, etc.) and then immediately stored into MessageStore, EdgeStore, etc. In this diff we ensure that we can directly read from network buffers (while storing in MessageStore for example) without creating an (unnecessary intermediate ByteArray based) vertexIdData object. Diffs ----- giraph-core/src/main/java/org/apache/giraph/comm/SendEdgeCache.java 8350a55 giraph-core/src/main/java/org/apache/giraph/comm/SendMessageCache.java 24848db giraph-core/src/main/java/org/apache/giraph/comm/SendMessageToAllCache.java 54234c5 giraph-core/src/main/java/org/apache/giraph/comm/SendVertexIdDataCache.java afce3ba giraph-core/src/main/java/org/apache/giraph/comm/messages/ByteArrayMessagesPerVertexStore.java 2381078 giraph-core/src/main/java/org/apache/giraph/comm/messages/MessageStore.java 2af7642 giraph-core/src/main/java/org/apache/giraph/comm/messages/MessagesIterable.java 3b22ab3 giraph-core/src/main/java/org/apache/giraph/comm/messages/OneMessagePerVertexStore.java acf68ea giraph-core/src/main/java/org/apache/giraph/comm/messages/out_of_core/DiskBackedMessageStore.java 346e3b3 giraph-core/src/main/java/org/apache/giraph/comm/messages/primitives/IntByteArrayMessageStore.java cc14c6d giraph-core/src/main/java/org/apache/giraph/comm/messages/primitives/IntFloatMessageStore.java 3318610 giraph-core/src/main/java/org/apache/giraph/comm/messages/primitives/LongByteArrayMessageStore.java 9e4325f giraph-core/src/main/java/org/apache/giraph/comm/messages/primitives/LongDoubleMessageStore.java 76d9ffa giraph-core/src/main/java/org/apache/giraph/comm/netty/InboundByteCounter.java bcc888d giraph-core/src/main/java/org/apache/giraph/comm/netty/NettyWorkerClientRequestProcessor.java 0166713 giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/RequestDecoder.java 98a61e6 giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/RequestEncoder.java d379eda giraph-core/src/main/java/org/apache/giraph/comm/netty/handler/RequestServerHandler.java 601cd2f giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerDataRequest.java 4f80224 giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerEdgesRequest.java 793768a giraph-core/src/main/java/org/apache/giraph/comm/requests/SendWorkerMessagesRequest.java 3ac0962 giraph-core/src/main/java/org/apache/giraph/comm/requests/WritableRequest.java 181e681 giraph-core/src/main/java/org/apache/giraph/conf/GiraphConfiguration.java 2862c3e giraph-core/src/main/java/org/apache/giraph/conf/GiraphConstants.java e257b4a giraph-core/src/main/java/org/apache/giraph/conf/ImmutableClassesGiraphConfiguration.java 2e8c935 giraph-core/src/main/java/org/apache/giraph/edge/EdgeStore.java dd8f2a3 giraph-core/src/main/java/org/apache/giraph/utils/AbstractVertexIdData.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayIterable.java d14172e giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayIterator.java 28b2dc8 giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdData.java 5c56038 giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdEdges.java 762802b giraph-core/src/main/java/org/apache/giraph/utils/ByteArrayVertexIdMessages.java 0ac8fdf giraph-core/src/main/java/org/apache/giraph/utils/ByteBufDataInput.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/utils/ByteBufDataOutput.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/utils/ByteBufVertexIdData.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/utils/ByteBufVertexIdEdges.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/utils/ByteBufVertexIdMessages.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/utils/ByteStructIterable.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/utils/ByteStructIterator.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/utils/ExtendedByteArrayDataInput.java 0ecea77 giraph-core/src/main/java/org/apache/giraph/utils/ExtendedByteArrayDataOutput.java 0ff366d giraph-core/src/main/java/org/apache/giraph/utils/ExtendedByteBufDataInput.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/utils/ExtendedByteBufDataOutput.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/utils/ExtendedDataOutput.java 54ef514 giraph-core/src/main/java/org/apache/giraph/utils/Extractable.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/utils/RepresentativeByteArrayIterable.java 2c24e89 giraph-core/src/main/java/org/apache/giraph/utils/RepresentativeByteArrayIterator.java d36c94f giraph-core/src/main/java/org/apache/giraph/utils/RepresentativeByteStructIterable.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/utils/RepresentativeByteStructIterator.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/utils/RequestUtils.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/utils/UnsafeArrayReads.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/utils/UnsafeBufferReads.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/utils/UnsafeByteArrayInputStream.java 20ed92b giraph-core/src/main/java/org/apache/giraph/utils/UnsafeByteArrayOutputStream.java 4b413da giraph-core/src/main/java/org/apache/giraph/utils/UnsafeByteBufDataOutput.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/utils/UnsafeDirectByteBufDataInput.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/utils/UnsafeHeapByteBufDataInput.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/utils/UnsafeReads.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/utils/VerboseByteArrayMessageWrite.java 8673732 giraph-core/src/main/java/org/apache/giraph/utils/VerboseByteStructMessageWrite.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/utils/VertexIdData.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/utils/VertexIdDataIterator.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/utils/VertexIdEdgeIterator.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/utils/VertexIdEdges.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/utils/VertexIdIterator.java bad11d6 giraph-core/src/main/java/org/apache/giraph/utils/VertexIdMessageBytesIterator.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/utils/VertexIdMessageIterator.java PRE-CREATION giraph-core/src/main/java/org/apache/giraph/utils/VertexIdMessages.java PRE-CREATION giraph-core/src/test/java/org/apache/giraph/comm/RequestFailureTest.java 236bc88 giraph-core/src/test/java/org/apache/giraph/comm/RequestTest.java fcdfa5c giraph-core/src/test/java/org/apache/giraph/utils/MockUtils.java 97e88f8 Diff: https://reviews.apache.org/r/21006/diff/ Testing ------- ran test jobs on cluster with option giraph.requestUtils.receiveBufs=false as well as giraph.requestUtils.receiveBufs=true mvn clean verify with hadoop_1 and hadoop_2 profiles for giraph-core Thanks, Pavan Kumar Athivarapu
