Re: Problem occured when running job with 1 worker.
Hi Jyoti; I assume this is the log of master vertex. It seems like master can not reach a worker for some reason. Did you also check the worker vertex's log? Maybe you can share it too. Sertug On 20-01-2014 09:22, Jyoti Yadav wrote: *h.master.MasterThread: masterThread: Master algorithm failed with ArrayIndexOutOfBoundsException java.lang.ArrayIndexOutOfBoundsException: -1*
Re: Null Pointer Exception when running a job
Hello Artie; I also had such problems when working on large datasets. It probably needs a better hadoop/mapreduce configuration. For example did you try increasing the memory of task tracker child processes in mapred-site.xml such as: property namemapred.child.java.opts/name value-Xmx3000m/value /property Regards; Sertug On 08-11-2013 01:26, Artie Pesh-Imam wrote: Hey It does work against a smaller data set and Im running 1.0.0 for hadoop 2. -- Artie Pesh-Imam Software Engineer @ Tapad 201-253-9647 From: Mohammad Islam Mohammad Islam mailto:misla...@yahoo.com Reply: user@giraph.apache.org user@giraph.apache.org mailto:user@giraph.apache.org, Mohammad Islam misla...@yahoo.com mailto:misla...@yahoo.com Date: November 7, 2013 at 6:20:53 PM To: user@giraph.apache.org user@giraph.apache.org mailto:user@giraph.apache.org Subject: Re: Null Pointer Exception when running a job Hi Artie, Hopefully someone would give you a concrete answer. My rambling comments: Did you try with a smaller number of vertices? I want to know if the problem is due to increased load. What version of Giraph are you using? Regards, Mohammad On Thursday, November 7, 2013 2:10 PM, Artie Pesh-Imam artie.pesh-i...@tapad.com wrote: Please let me know if any additional context would be useful. With no detail in the error message, I have no idea what the root issue could be. -- Artie Pesh-Imam Software Engineer @ Tapad 201-253-9647 From: Artie Pesh-Imam Artie Pesh-Imam mailto:artie.pesh-i...@tapad.com Reply: Artie Pesh-Imam artie.pesh-i...@tapad.com mailto:artie.pesh-i...@tapad.com Date: November 7, 2013 at 4:25:30 PM To: user@giraph.apache.org user@giraph.apache.org mailto:user@giraph.apache.org Subject: Null Pointer Exception when running a job Hi all, Im running a simple connected components algorithm against a graph of almost 200 million vertices. I have pretty much no guess on what this issue could be: java.lang.IllegalStateException: run: Caught an unrecoverable exception null at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:102) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:672) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:330) at org.apache.hadoop.mapred.Child$4.run(Child.java:268) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:415) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.mapred.Child.main(Child.java:262) Caused by: java.lang.NullPointerException at org.apache.giraph.comm.SendCache.removeWorkerData(SendCache.java:157) at org.apache.giraph.comm.SendCache.removeAllData(SendCache.java:171) at org.apache.giraph.comm.SendMessageCache.removeAllMessages(SendMessageCache.java:92) at org.apache.giraph.comm.netty.NettyWorkerClientRequestProcessor.flush(NettyWorkerClientRequestProcessor.java:410) at org.apache.giraph.worker.BspServiceWorker.sendWorkerPartitions(BspServiceWorker.java:1316) at org.apache.giraph.worker.BspServiceWorker.exchangeVertexPartitions(BspServiceWorker.java:1361) at org.apache.giraph.graph.GraphTaskManager.execute(GraphTaskManager.java:255) at org.apache.giraph.graph.GraphMapper.run(GraphMapper.java:92) -- Artie Pesh-Imam Software Engineer @ Tapad 201-253-9647
Some of the workers hanging while others finish
Hi all; I have a 10 machine cluster; 8 Tasktrackers, 1 namenode and 1 Jobtracker. All of them have 8GB memory. When I run a giraph job with 16 workers on a hbase table with 15 millions of rows (12GB of data) my job does not finish due to 2 (sometimes 3) workers stuck at superstep -1. Below you can see the logs of these workers. Any helps/ideas? Thanks Sertug *_syslog logs_* 2013-11-08 19:00:06,576 INFO org.apache.hadoop.util.NativeCodeLoader: Loaded the native-hadoop library 2013-11-08 19:00:07,449 INFO org.apache.zookeeper.ZooKeeper: Client environment:zookeeper.version=3.3.3-1073969, built on 02/23/2011 22:27 GMT 2013-11-08 19:00:07,449 INFO org.apache.zookeeper.ZooKeeper: Client environment:host.name=host-10-6-149-104 2013-11-08 19:00:07,449 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.version=1.7.0_21 2013-11-08 19:00:07,449 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.vendor=Oracle Corporation 2013-11-08 19:00:07,449 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.home=/usr/java/jdk1.7.0_21/jre 2013-11-08 19:00:07,449 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.class.path=/hadoop/mapred/taskTracker/root/jobcache/job_201311081157_0003/jars/classes:/hadoop/mapred/taskTracker/root/jobcache/job_201311081157_0003/jars:/hadoop/mapred/taskTracker/root/jobcache/job_201311081157_0003/attempt_201311081157_0003_m_09_0/work:/usr/local/hadoop-1.2.1/libexec/../conf:/usr/java/default/lib/tools.jar:/usr/local/hadoop-1.2.1/libexec/..:/usr/local/hadoop-1.2.1/libexec/../hadoop-core-1.2.1.jar:/usr/local/hadoop-1.2.1/libexec/../lib/asm-3.2.jar:/usr/local/hadoop-1.2.1/libexec/../lib/aspectjrt-1.6.11.jar:/usr/local/hadoop-1.2.1/libexec/../lib/aspectjtools-1.6.11.jar:/usr/local/hadoop-1.2.1/libexec/../lib/commons-beanutils-1.7.0.jar:/usr/local/hadoop-1.2.1/libexec/../lib/commons-beanutils-core-1.8.0.jar:/usr/local/hadoop-1.2.1/libexec/../lib/commons-cli-1.2.jar:/usr/local/hadoop-1.2.1/libexec/../lib/commons-codec-1.4.jar:/usr/local/hadoop-1.2.1/libexec/../lib/commons-collections-3.2.1.jar:/usr/local/hadoop-1.2.1/libexec/../lib/commons-configuration-1.6.jar:/usr/local/hadoop-1.2.1/libexec/../lib/commons-daemon-1.0.1.jar:/usr/local/hadoop-1.2.1/libexec/../lib/commons-digester-1.8.jar:/usr/local/hadoop-1.2.1/libexec/../lib/commons-el-1.0.jar:/usr/local/hadoop-1.2.1/libexec/../lib/commons-httpclient-3.0.1.jar:/usr/local/hadoop-1.2.1/libexec/../lib/commons-io-2.1.jar:/usr/local/hadoop-1.2.1/libexec/../lib/commons-lang-2.4.jar:/usr/local/hadoop-1.2.1/libexec/../lib/commons-logging-1.1.1.jar:/usr/local/hadoop-1.2.1/libexec/../lib/commons-logging-api-1.0.4.jar:/usr/local/hadoop-1.2.1/libexec/../lib/commons-math-2.1.jar:/usr/local/hadoop-1.2.1/libexec/../lib/commons-net-3.1.jar:/usr/local/hadoop-1.2.1/libexec/../lib/core-3.1.1.jar:/usr/local/hadoop-1.2.1/libexec/../lib/giraph-nutch-1.1.0-SNAPSHOT-for-hadoop-0.20.203.0-jar-with-dependencies.jar:/usr/local/hadoop-1.2.1/libexec/../lib/hadoop-capacity-scheduler-1.2.1.jar:/usr/local/hadoop-1.2.1/libexec/../lib/hadoop-fairscheduler-1.2.1.jar:/usr/local/hadoop-1.2.1/libexec/../lib/hadoop-thriftfs-1.2.1.jar:/usr/local/hadoop-1.2.1/libexec/../lib/hsqldb-1.8.0.10.jar:/usr/local/hadoop-1.2.1/libexec/../lib/jackson-core-asl-1.8.8.jar:/usr/local/hadoop-1.2.1/libexec/../lib/jackson-mapper-asl-1.8.8.jar:/usr/local/hadoop-1.2.1/libexec/../lib/jasper-compiler-5.5.12.jar:/usr/local/hadoop-1.2.1/libexec/../lib/jasper-runtime-5.5.12.jar:/usr/local/hadoop-1.2.1/libexec/../lib/jdeb-0.8.jar:/usr/local/hadoop-1.2.1/libexec/../lib/jersey-core-1.8.jar:/usr/local/hadoop-1.2.1/libexec/../lib/jersey-json-1.8.jar:/usr/local/hadoop-1.2.1/libexec/../lib/jersey-server-1.8.jar:/usr/local/hadoop-1.2.1/libexec/../lib/jets3t-0.6.1.jar:/usr/local/hadoop-1.2.1/libexec/../lib/jetty-6.1.26.jar:/usr/local/hadoop-1.2.1/libexec/../lib/jetty-util-6.1.26.jar:/usr/local/hadoop-1.2.1/libexec/../lib/jsch-0.1.42.jar:/usr/local/hadoop-1.2.1/libexec/../lib/junit-4.5.jar:/usr/local/hadoop-1.2.1/libexec/../lib/kfs-0.2.2.jar:/usr/local/hadoop-1.2.1/libexec/../lib/log4j-1.2.15.jar:/usr/local/hadoop-1.2.1/libexec/../lib/mockito-all-1.8.5.jar:/usr/local/hadoop-1.2.1/libexec/../lib/oro-2.0.8.jar:/usr/local/hadoop-1.2.1/libexec/../lib/servlet-api-2.5-20081211.jar:/usr/local/hadoop-1.2.1/libexec/../lib/slf4j-api-1.4.3.jar:/usr/local/hadoop-1.2.1/libexec/../lib/slf4j-log4j12-1.4.3.jar:/usr/local/hadoop-1.2.1/libexec/../lib/xmlenc-0.52.jar:/usr/local/hadoop-1.2.1/libexec/../lib/jsp-2.1/jsp-2.1.jar:/usr/local/hadoop-1.2.1/libexec/../lib/jsp-2.1/jsp-api-2.1.jar 2013-11-08 19:00:07,449 INFO org.apache.zookeeper.ZooKeeper: Client environment:java.library.path=/usr/local/hadoop-1.2.1/libexec/../lib/native/Linux-amd64-64:/hadoop/mapred/taskTracker/root/jobcache/job_201311081157_0003/attempt_201311081157_0003_m_09_0/work 2013-11-08 19:00:07,449 INFO org.apache.zookeeper.ZooKeeper: Client
