Hi Denis,
OOC + Checkpointing is one of the TODO's in the code base at the moment.
Unfortunately, OOC cannot be used in combination with checkpointing at the
moment. We are planning to make these two work with each other in the
future.
Best,
Hassan
On Tue, Dec 6, 2016 at 1:43 AM, Denis Dudinski
Hi Denis,
Thanks for trying out the new OOC design. As you mentioned, by using
message combiner we only keep one message for each vertex at all time. That
means the storage needed for messages is #ofVertices*(SizeOfVertexId +
SizeOfOneMessage). We studied several applications that use message
comb
sStaticGraph" issue, but I think it worth mentioning
> to see the picture as a whole.
>
> Are there any other tests/information I am able to execute/check to
> help to pinpoint "isStaticGraph" problem?
>
> Best Regards,
> Denis Dudinski
>
>
> 201
Hi Denis,
Thanks for bringing up the issue. In the previous conversation thread, the
similar problem is reported even with a simpler example connected component
calculation. Although, back then, we were developing other
performance-critical components of OOC.
Let's debug this issue together to ma
partitioning the input file
> for paralell proccesing, but i wasnt' sure.
>
> I will try with only two threads in same worker.
>
> Maybe using giraph.userPartitionCount will help? Or those partitions are
> destinated to computation steps only?
>
> Bye!
> Jose
>
>
It seems that you are reading the data from a single file stored on a local
machine with multiple threads. Having multiple threads accessing the disk
causes IO interference which in turn reduces the IO performance. If you are
reading from a single file on a local machine with 8 threads, the results
Hi Ramesh,
First of all, it seems that you don't want the unsafe-serialization for
some reason, and that makes the out-of-core to be very slow.
(De)Serialization is one of the performance critical components in the new
out-of-core design.
Second, you are using the out-of-core flags based on the d
java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2016-05-15 05:34:31,125 ERROR [main]
> o
Hi Ramesh!
Thanks for bringing this up, and thanks for trying out the new out-of-core
mechanism. The new out-of-core mechanism has not been integrated with
checkpointing yet. This is part of an ongoing project, and we should have
the integration within a few weeks. In the meantime, you can try
out
Anirudh,
1) AFAIK, the load balancing mechanism is not implemented in Giraph.
Although, the mechanism for partition migration is implemented. You may
want to use that mechanism to implement your own load-balancer insider the
framework. You can take a look at BspServiceWorker#exchangeVertexPartitio
8 16
> 24
> 32superstep 15101 3642 31632867 3611
> 3435 (ms)
>
>
> As you see, this is the same story! while no I/O is considered.
>
>
>
> Regards,
> Karos
>
>
> On Mon, Dec 14, 2015 at 9:29 PM, Hassan Eslam
Hi,
Here is my take on it:
It may be a good idea to isolate the time spent in compute supersteps. In
order to do that, you can look at per-superstep timing metrics and
aggregate the times for all supersteps except input (-1) and output (last)
superstep. This eliminates the time for IO and turns yo
OOC features (both for graph and messages) are under active development at
the moment. The previous version (1.0.0 and 1.1.0) did have several issues
(mostly performance issues and also falling short interacting with
stop-the-world GC pauses while OOC tries to spill data to disk). A new
mechanism f
13 matches
Mail list logo
