[
https://issues.apache.org/jira/browse/GIRAPH-244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420991#comment-13420991
]
Alessandro Presta commented on GIRAPH-244:
------------------------------------------
Here's a PageRank benchmark run with 1M vertices, 100 edges/vertex, 30
supersteps and 10 workers:
trunk + GIRAPH-244:
{code}
12/07/23 14:27:19 INFO mapred.JobClient: Giraph Stats
12/07/23 14:27:19 INFO mapred.JobClient: Aggregate edges=100000000
12/07/23 14:27:19 INFO mapred.JobClient: Superstep=31
12/07/23 14:27:19 INFO mapred.JobClient: Current workers=10
12/07/23 14:27:19 INFO mapred.JobClient: Last checkpointed superstep=0
12/07/23 14:27:19 INFO mapred.JobClient: Current master task partition=9
12/07/23 14:27:19 INFO mapred.JobClient: Sent messages=0
12/07/23 14:27:19 INFO mapred.JobClient: Aggregate finished vertices=1000000
12/07/23 14:27:19 INFO mapred.JobClient: Aggregate vertices=1000000
12/07/23 14:27:19 INFO mapred.JobClient: FileSystemCounters
12/07/23 14:27:19 INFO mapred.JobClient: HDFS_FILES_CREATED=24
12/07/23 14:27:19 INFO mapred.JobClient: Map-Reduce Framework
12/07/23 14:27:19 INFO mapred.JobClient: Map input records=11
12/07/23 14:27:19 INFO mapred.JobClient: Total physical memory in
bytes=295136768000
12/07/23 14:27:19 INFO mapred.JobClient: Spilled Records=0
12/07/23 14:27:19 INFO mapred.JobClient: MAP_TASK_WALLCLOCK=5577508
12/07/23 14:27:19 INFO mapred.JobClient: Total cumulative CPU
milliseconds=28367700
12/07/23 14:27:19 INFO mapred.JobClient: Total virtual memory in
bytes=953247084544
12/07/23 14:27:19 INFO mapred.JobClient: Map output records=0
real 8m43.003s
user 0m6.677s
sys 0m0.877s
{code}
trunk:
{code}
12/07/23 14:36:47 INFO mapred.JobClient: Giraph Stats
12/07/23 14:36:47 INFO mapred.JobClient: Aggregate edges=100000000
12/07/23 14:36:47 INFO mapred.JobClient: Superstep=31
12/07/23 14:36:47 INFO mapred.JobClient: Current workers=10
12/07/23 14:36:47 INFO mapred.JobClient: Last checkpointed superstep=0
12/07/23 14:36:47 INFO mapred.JobClient: Current master task partition=0
12/07/23 14:36:47 INFO mapred.JobClient: Sent messages=0
12/07/23 14:36:47 INFO mapred.JobClient: Aggregate finished vertices=1000000
12/07/23 14:36:47 INFO mapred.JobClient: Aggregate vertices=1000000
12/07/23 14:36:47 INFO mapred.JobClient: FileSystemCounters
12/07/23 14:36:47 INFO mapred.JobClient: HDFS_FILES_CREATED=24
12/07/23 14:36:47 INFO mapred.JobClient: Map-Reduce Framework
12/07/23 14:36:47 INFO mapred.JobClient: Map input records=11
12/07/23 14:36:47 INFO mapred.JobClient: Total physical memory in
bytes=288768761856
12/07/23 14:36:47 INFO mapred.JobClient: Spilled Records=0
12/07/23 14:36:47 INFO mapred.JobClient: MAP_TASK_WALLCLOCK=5754578
12/07/23 14:36:47 INFO mapred.JobClient: Total cumulative CPU
milliseconds=28970270
12/07/23 14:36:47 INFO mapred.JobClient: Total virtual memory in
bytes=953337098240
12/07/23 14:36:47 INFO mapred.JobClient: Map output records=0
real 8m54.799s
user 0m6.436s
sys 0m0.790s
{code}
They're pretty close in both memory usage and running time.
I'm going to run some other benchmarks tomorrow to see how things change e.g.
increasing the number of edges per vertex, since we're using a different
iteration strategy.
> Vertex API redesign
> -------------------
>
> Key: GIRAPH-244
> URL: https://issues.apache.org/jira/browse/GIRAPH-244
> Project: Giraph
> Issue Type: Improvement
> Reporter: Alessandro Presta
> Assignee: Alessandro Presta
> Attachments: GIRAPH-244.patch, GIRAPH-244.patch, GIRAPH-244.patch,
> GIRAPH-244.patch, GIRAPH-244.patch, GIRAPH-244.patch, GIRAPH-244.patch,
> GIRAPH-244.patch, GIRAPH-244.patch, GIRAPH-244.patch, GIRAPH-244.patch,
> GIRAPH-244.patch
>
>
> This is an effort to rationalize the Giraph API. I've put together a few
> issues that we've talked about lately. I'm focusing on making Giraph
> development even more intuitive and less error-prone, and fixing a few
> potential sources of bugs.
> I'm sorry this is a big patch, but most of those issues are intertwined and I
> think this might be easier to review and integrate.
> Here's an account of the changes:
> Vertex API:
> - Renamed BasicVertex to Vertex (as I understand, we used to have both and
> then Vertex was removed).
> - Switched to Iterables instead of Iterators for both edges and messages.
> This makes code more concise for both implementors (no need to call
> .iterator() on collections) and users (can use foreach syntax). See also
> GIRAPH-221.
> - Added SimpleVertex and SimpleMutableVertex classes, where there are no edge
> values and the iterable to be implemented is getNeighbors(). We don’t have
> multiple inheritance, so the only way I could think of was to have
> SimpleVertex extend Vertex, SimpleMutableVertex extend MutableVertex, and
> duplicate the code for the edges iterables.
> Also, due to type erasure, one still has to deal with Edge objects in
> SimpleMutableVertex#initialize. Overall I think this is still an improvement
> over the current situation.
> - Added id and value field to the base Vertex class. All other classes were
> either writing the same boilerplate again and again, or using primitive
> fields and then creating Writables on the fly (inefficient; there was even a
> TODO about that). If there are any actually useful customizations here, I’ve
> yet to see them.
> Also removed redundant “Vertex” from getters/setters (compare vertex.getId()
> with vertex.getVertexId()).
> - Made halt a private field, and added a wakeUp() method to re-activate a
> vertex. isHalted()/voteToHalt()/wakeUp() are just more semantically-charged
> getter/setters.
> - Renamed number of vertices/edges in graph to getTotalNum*. The previous
> naming (getNumEdges) was arguably confusing. If this one sucks too, please
> suggest a better one.
> - Default implementations of hasEdge(), getEdgeValue(), getNumEdges(),
> readFields(), write(), toString(): the implementor can still optimize when
> there is a good opportunity. Currently we are duplicating a lot of code (see
> GIRAPH-238) and potentially introducing bugs (see GIRAPH-239).
> HashMapVertex:
> - Switched representation from Map<I, Edge<I, E>> to Map<I, E> (GIRAPH-242)
> - Only override methods that can be optimized.
> EdgeListVertex:
> - Switched representation from two sorted lists to one list of Edge<I, E>
> (see GIRAPH-243). Mainly this makes iteration over edges (target id and
> value) linear instead of O(n log n). Mutations are still slow and should
> generally be discouraged.
> - Only override methods that can be optimized.
> Small nits:
> - Our code conventions say we should try to avoid abbreviations, so I
> eliminated a few (req -> request, msg -> message).
> - Unilaterally refer to the endpoint of an edge as targetVertex (before we
> had a mix of destVertex and targetVertex).
> - You will notice some rearranged imports. That’s just my IDE trying to be
> helpful (see GIRAPH-230).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
