Hopefully you are using combiners and also re-using objects. This can
keep memory usage much lower. Also implementing your own OutEdges can
make it much more efficient.
How much memory do you have?
Avery
On 9/26/13 12:51 PM, Yingyi Bu wrote:
>> I think you may have added the same vertex 2x?
I ran the job over roughly half of the graph and saw this. However
the input is not a connected components such that there might be
target vertex ids which do not exist.
When I ran the job over the entire graph, I cannot see this but the
job fails with exceeding GC limit (trying out-of-core now).
Yingyi
On Thu, Sep 26, 2013 at 12:05 PM, Avery Ching <ach...@apache.org
<mailto:ach...@apache.org>> wrote:
I think you may have added the same vertex 2x? That being said, I
don't see why the code is this way. It should be fine. We should
file a JIRA.
On 9/26/13 11:02 AM, Yingyi Bu wrote:
Thanks, Lukas!
I think the reason of this exception is that I run the job over
part of the graph where some target ids do not exist.
Yingyi
On Thu, Sep 26, 2013 at 1:13 AM, Lukas Nalezenec
<lukas.naleze...@firma.seznam.cz
<mailto:lukas.naleze...@firma.seznam.cz>> wrote:
Hi,
Do you use partition balancing ?
Lukas
On 09/26/13 05:16, Yingyi Bu wrote:
Hi,
I got this exception when I ran a Giraph-1.0.0 PageRank job over a 60
machine cluster with 28GB input data. But I got this exception:
java.lang.IllegalStateException: run: Caught an unrecoverable exception
resolveMutations: Already has missing vertex on this worker for 20464109
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:102)
at
org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:763)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:369)
at org.apache.hadoop.mapred.Child$4.run(Child.java:259)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
at org.apache.hadoop.mapred.Child.main(Child.java:253)
Caused by: java.lang.IllegalStateException: resolveMutations: Already
has missing vertex on this worker for 20464109
at
org.apache.giraph.comm.netty.NettyWorkerServer.resolveMutations(NettyWorkerServer.java:184)
at
org.apache.giraph.comm.netty.NettyWorkerServer.prepareSuperstep(NettyWorkerServer.java:152)
at
org.apache.giraph.worker.BspServiceWorker.startSuperstep(BspServiceWorker.java:677)
at
org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java:249)
at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:92)
... 7 more
Does anyone know what is the possible cause of this exception?
Thanks!
Yingyi
