how to build a stable version of giraph
hi i am using giraph on Hortonworks Data platform Hadoop 2.2 while building on branch Trunk i am unable to run simple shortestpath example, my issue is similar to exactly same as this one, http://mail-archives.apache.org/mod_mbox/giraph-user/201401.mbox/browser i am thinking there might be some borken things on trunk, so i looked at other branches and tags where they are 1 year old. i want to try reverting to a stable giraph. can someone specify a commit id to revert to a stable version of giraph. thanks, yeshwanth
Re: Superstep duration increases
I'd start by taking HBase out of the equation. On Thu, May 8, 2014 at 1:46 PM, Pascal Jäger wrote: > Hi all, > > I have implemented a label propagation algorithm to find clusters in a > graph. > I just realized that the time the algorithm takes for one superstep is > increasing and I don’t know why. > > The graph is static and the number of messages is the same throughout all > supersteps. > During every superstep each node sends its label to its neighbors which > then calculate their label based on the received messages and then again > send their label. > At the end of each superstep each node writes a nodeID - label pair to an > HBase table. > > Do you have any general hints where I can look at? > > I absolutely have no clue where to start > > Thanks for your help! > > Regards > > Pascal > > -- Claudio Martella
Re: issues while running giraph on Hadoop 2.2
I have the same issue running giraph built with -Phadoop_yarn -Dhadoop.version=2.2.0 on Hortonworks. I get classnotfoundexceptions, looks like the GiraphRunner may not be able to load the classes correctly while running on Yarn. So if anybody has had success please share your experience, wondering what I have been doing wrong. Exception in thread "main" java.lang.ClassNotFoundException: org.apache.giraph.examples.SimpleShortestPathsComputation Regards, - kiru From: yeshwanth kumar To: "user@giraph.apache.org" Sent: Thursday, May 8, 2014 4:47 AM Subject: issues while running giraph on Hadoop 2.2 hi i am running Hortonworks hadoop 2.2 trying to run giraph shortest path example. i am getting this exception $ hadoop jar giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar org.apache.giraph.GiraphRunner org.apache.giraph.examples.SimpleShortestPathsComputation -vif org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip /user/hdfs/input/tiny.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op /user/hdfs/output/shortestpaths -w 1 14/05/07 23:45:42 INFO utils.ConfigurationUtils: No edge input format specified. Ensure your InputFormat does not require one. 14/05/07 23:45:42 INFO utils.ConfigurationUtils: No edge output format specified. Ensure your OutputFormat does not require one. 14/05/07 23:45:42 INFO yarn.GiraphYarnClient: Final output path is: hdfs://server02.infnet:8020/user/hdfs/output/shortestpaths 14/05/07 23:45:42 INFO yarn.GiraphYarnClient: Running Client 14/05/07 23:45:42 INFO client.RMProxy: Connecting to ResourceManager at server02.infnet/192.168.1.85:8050 14/05/07 23:45:42 INFO yarn.GiraphYarnClient: Defaulting per-task heap size to 1024MB. Exception in thread "main" java.lang.IllegalStateException: Giraph's estimated cluster heap 2048MB ask is greater than the current available cluster heap of 0MB. Aborting Job. at org.apache.giraph.yarn.GiraphYarnClient.checkPerNodeResourcesAvailable(GiraphYarnClient.java:230) at org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:124) at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:96) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70) at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84) at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:126) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) i am missing some configuration, can someone point me in the right direction. thanks, Yeshwanth
Re: Error while executing large graph
Hi Thanks for the replay . I am running this example in a cluster of 5 machines each machine is having 16 GB of ram.The java heap size is set as 2000mb and java.child.options is set with 2000mb and each machine has 4 cores and total number of map instance is set as 3. So for each slave machine 10 gb will be used. My input data is of 1gb size In this scenario how can out of memory error occur .Please clarify Regards Arun On Thu, May 15, 2014 at 12:43 AM, Avery Ching wrote: > I think this is the key message. > > > 0 out of 196 partitions computed; min free memory on worker 6 - 0.81MB, > average 11.56MB > > Having less than 1 MB free won't work. Your workers are likely OOM, > killing the job. Can you get more memory for your job? > > > On 5/14/14, 3:13 AM, Arun Kumar wrote: > > Hi when i run giraph job against a data of 1 gb i am getting the below > exception after some times can somebody tell me what is the issue? > 14/05/14 01:54:01 INFO job.JobProgressTracker: Data from 14 workers - > Compute superstep 2: 0 out of 4847571 vertices computed; 0 out of 196 > partitions computed; min free memory on worker 6 - 0.81MB, average 11.56MB > 14/05/14 01:54:03 INFO zookeeper.ClientCnxn: Unable to read additional > data from server sessionid 0x145f9cff031000f, likely server has closed > socket, closing socket connection and attempting reconnect > 14/05/14 01:54:04 INFO zookeeper.ClientCnxn: Opening socket connection to > server mercado-12.hpl.hp.com/15.25.119.147:22181. Will not attempt to > authenticate using SASL (unknown error) > 14/05/14 01:54:04 WARN zookeeper.ClientCnxn: Session 0x145f9cff031000f for > server null, unexpected error, closing socket connection and attempting > reconnect > java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) > at > org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068) > 14/05/14 01:54:06 INFO zookeeper.ClientCnxn: Opening socket connection to > server mercado-12.hpl.hp.com/15.25.119.147:22181. Will not attempt to > authenticate using SASL (unknown error) > 14/05/14 01:54:06 WARN zookeeper.ClientCnxn: Session 0x145f9cff031000f for > server null, unexpected error, closing socket connection and attempting > reconnect > java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) > at > org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068) > 14/05/14 01:54:06 WARN zk.ZooKeeperExt: exists: Connection loss on attempt > 0, waiting 5000 msecs before retrying. > org.apache.zookeeper.KeeperException$ConnectionLossException: > KeeperErrorCode = ConnectionLoss for > /_hadoopBsp/job_201405140108_0003/_workerProgresses > at org.apache.zookeeper.KeeperException.create(KeeperException.java:99) > at org.apache.zookeeper.KeeperException.create(KeeperException.java:51) > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1041) > at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:1069) > at org.apache.giraph.zk.ZooKeeperExt.exists(ZooKeeperExt.java:360) > at > org.apache.giraph.job.JobProgressTracker$2.run(JobProgressTracker.java:87) > at java.lang.Thread.run(Thread.java:745) > 14/05/14 01:54:08 INFO zookeeper.ClientCnxn: Opening socket connection to > server mercado-12.hpl.hp.com/15.25.119.147:22181. Will not attempt to > authenticate using SASL (unknown error) > 14/05/14 01:54:08 WARN zookeeper.ClientCnxn: Session 0x145f9cff031000f for > server null, unexpected error, closing socket connection and attempting > reconnect > java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) > at > org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350) > at org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068) > 14/05/14 01:54:09 INFO mapred.JobClient: map 93% reduce 0% > 14/05/14 01:54:10 INFO zookeeper.ClientCnxn: Opening socket connection to > server mercado-12.hpl.hp.com/15.25.119.147:22181. Will not attempt to > authenticate using SASL (unknown error) > 14/05/14 01:54:10 WARN zookeeper.ClientCnxn: Session 0x145f9cff031000f for > server null, unexpected error, closing socket connection and attempting > reconnect > java.net.ConnectException: Connection refused > at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method) > at > sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739) > at > org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO
Re: resolveMutations error
Hi,
Not throwing the exception in resolveMutations worked for me. Is it
possible to get some understanding of what effects it would have if any
because the results are working as expected.
for (I vertexId : destinations) {
if (partition.getVertex(vertexId) == null) {
if (!resolveVertexIndices.put(partitionId, vertexId)) {
//throw new IllegalStateException(
//"resolveMutations: Already has missing vertex on this "
+
//"worker for " + vertexId);
}
}
}
On Wed, May 14, 2014 at 7:23 AM, Jyotirmoy Sundi wrote:
> Hi Folks,
>
> I am seeing this error recently in jobs, can you please throw some
> light on what resolveMutations in NettyWorkerServer.java tries to achieve ?
>
> *Trace:*
>
> Caused by: java.lang.IllegalStateException: resolveMutations: Already has
> missing vertex on this worker for 8345381246748292335
> at
> org.apache.giraph.comm.netty.NettyWorkerServer.resolveMutations(NettyWorkerServer.java:184)
> at
> org.apache.giraph.comm.netty.NettyWorkerServer.prepareSuperstep(NettyWorkerServer.java:152)
> at
> org.apache.giraph.worker.BspServiceWorker.startSuperstep(BspServiceWorker.java:677)
> at
> org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java:249)
> at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:92)
>
>
> --
> Best Regards,
> Jyotirmoy Sundi
>
>
--
Best Regards,
Jyotirmoy Sundi
