Hi, in the mean time I found the reason why it hangs for long time here: base path checking .
It scan all the directory tree starting from current directory, if you run it into your home (and you have a lot of files and directories) it takes ages to complete. I continue to got the same error. Regards, Alessandro Il giorno 05/nov/2014, alle ore 18:33, Jing Fan <fanjin...@gmail.com> ha scritto: > I run with: > > hadoop jar > giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar > org.apache.giraph.GiraphRunner > org.apache.giraph.examples.SimpleShortestPathsComputation -vif > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip > /user/jing/tiny_graph.txt -vof > org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op > /user/jing/shortestpaths -w 1 -yj > giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar > > On Wed, Nov 5, 2014 at 11:25 AM, Alessandro Negro <alenegr...@yahoo.it> wrote: > This is my complete logs with the command I run: > > > ale:~ />$HADOOP_INSTALL/bin/hadoop jar > /Users/ale/extprj/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar > org.apache.giraph.GiraphRunner > org.apache.giraph.examples.SimpleShortestPathsComputation -vif > org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip > /tiny_graph.txt -vof org.apache.giraph.io.formats.IdWithValueTextOutputFormat > -op /shortestpaths -w 1 > 2014-11-05 13:39:30.235 java[3147:1903] Unable to load realm info from > SCDynamicStore > 14/11/05 13:39:30 WARN util.NativeCodeLoader: Unable to load native-hadoop > library for your platform... using builtin-java classes where applicable > 14/11/05 13:39:30 INFO utils.ConfigurationUtils: No edge input format > specified. Ensure your InputFormat does not require one. > 14/11/05 13:39:30 INFO utils.ConfigurationUtils: No edge output format > specified. Ensure your OutputFormat does not require one. > 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Final output path is: > hdfs://hadoop-master:9000/shortestpaths > 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Running Client > 14/11/05 13:39:31 INFO client.RMProxy: Connecting to ResourceManager at > /0.0.0.0:8032 > 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Got node report from ASM for, > nodeId=192.168.0.105:50658, nodeAddress 192.168.0.105:8042, nodeRackName > /default-rack, nodeNumContainers 0 > 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Defaulting per-task heap size > to 1024MB. > 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Obtained new Application ID: > application_1415189069846_0004 > 14/11/05 13:39:31 INFO Configuration.deprecation: mapred.job.id is > deprecated. Instead, use mapreduce.job.id > 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Set the environment for the > application master > 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: Environment for AM > :{CLASSPATH=${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*} > 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: buildLocalResourceMap .... > 14/11/05 13:39:31 INFO yarn.YarnUtils: Registered file in LocalResources :: > hdfs://hadoop-master:9000/user/ale/giraph_yarn_jar_cache/application_1415189069846_0004/giraph-conf.xml > 14/11/05 13:39:31 INFO yarn.GiraphYarnClient: LIB JARS : > 14/11/05 13:39:31 INFO yarn.YarnUtils: Class path name . > 14/11/05 13:39:31 INFO yarn.YarnUtils: base path checking . > 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name null > 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking null > 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/etc/hadoop > 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/etc/hadoop > 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib/* > 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/common/lib > 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/hadoop/common/* > 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/common > 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs > 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs > 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib/* > 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/lib > 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs/* > 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/hdfs > 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib/* > 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/lib > 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/hadoop/yarn/* > 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/yarn > 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib/* > 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/lib > 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name > /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce/* > 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking > /opt/yarn/hadoop-2.5.1/share/hadoop/mapreduce > 14/11/05 14:42:56 INFO yarn.YarnUtils: Class path name > /contrib/capacity-scheduler/*.jar > 14/11/05 14:42:56 INFO yarn.YarnUtils: base path checking > /contrib/capacity-scheduler > 14/11/05 14:42:56 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext > for GiraphApplicationMaster launch container is populated. > 14/11/05 14:42:56 INFO yarn.GiraphYarnClient: Submitting application to ASM > 14/11/05 14:42:57 INFO impl.YarnClientImpl: Submitted application > application_1415189069846_0004 > 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: Got new appId after submission > :application_1415189069846_0004 > 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: GiraphApplicationMaster > container request was submitted to ResourceManager for job: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation > 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0,82 secs > 14/11/05 14:42:57 INFO yarn.GiraphYarnClient: > appattempt_1415189069846_0004_000001, State: ACCEPTED, Containers used: 1 > 14/11/05 14:43:01 ERROR yarn.GiraphYarnClient: Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED > state, diagnostics show: Application application_1415189069846_0004 failed 2 > times due to AM Container for appattempt_1415189069846_0004_000002 exited > with exitCode: 1 due to: Exception from container-launch: ExitCodeException > exitCode=1: > ExitCodeException exitCode=1: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) > at org.apache.hadoop.util.Shell.run(Shell.java:455) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) > at > org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:744) > > > Container exited with a non-zero exit code 1 > .Failing this attempt.. Failing the application. > 14/11/05 14:43:01 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed > cache directory for Giraph job. > 14/11/05 14:43:01 INFO yarn.GiraphYarnClient: Completed Giraph: > org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total > running time: 0 minutes, 3 seconds. > > > Il giorno 05/nov/2014, alle ore 18:17, Tripti Singh <tri...@yahoo-inc.com> ha > scritto: > >> I faced the similar problem where the workflow just hangs. >> Can u see in ur logs if there is a task with id 0. >> I made a local fix and I could give u the patch if u see a similar problem >> and ur workflow is unable to find the zookeeper manager file and repeats the >> following message: >> For task xyz, got file 'null' >> >> Thanks, >> Tripti >> Sent from my iPhone >> >> On 05-Nov-2014, at 10:40 pm, "Alessandro Negro" <alenegr...@yahoo.it> wrote: >> >>> Hi, >>> have you seen the hadoop logs? >>> Could you write here how are you running the command? >>> >>> Thanks, >>> Alessandro >>> >>> Il giorno 05/nov/2014, alle ore 18:04, Jing Fan <fanjin...@gmail.com> ha >>> scritto: >>> >>>> Yeah, I add -yj option. The result is the same. Still error here. >>>> >>>> On Wed, Nov 5, 2014 at 10:52 AM, Alessandro Negro <alenegr...@yahoo.it> >>>> wrote: >>>> Hi Jing, >>>> I’m facing the same problem. Do you check the hadoop logs? In my case I >>>> got: >>>> Could not find or load main class >>>> org.apache.giraph.yarn.GiraphApplicationMaster >>>> Moreover I don’t know why my application hang for a long time here: >>>> >>>> 14/11/05 17:11:17 INFO yarn.YarnUtils: base path checking . >>>> >>>> And then continue failing with your error into the output and the other >>>> into the hadoop logs. >>>> >>>> Did you test adding this? >>>> >>>> -yj >>>> /Users/ale/extprj/giraph/giraph-core/target/giraph-1.1.0-SNAPSHOT-for-hadoop-2.5.1-jar-with-dependencies.jar >>>> >>>> Thanks, >>>> Alessandro >>>> >>>> >>>> Il giorno 05/nov/2014, alle ore 17:32, Jing Fan <fanjin...@gmail.com> ha >>>> scritto: >>>> >>>>> Hi, >>>>> >>>>> I install the Giraph with Yarn. >>>>> But when I run the program with following command: >>>>> >>>>> hadoop jar >>>>> giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.5.0-cdh5.2.0-jar-with-dependencies.jar >>>>> org.apache.giraph.GiraphRunner >>>>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif >>>>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat >>>>> -vip /user/j/tiny_graph.txt -vof >>>>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op >>>>> /user/j/shortestpaths -w 1 >>>>> >>>>> The program exited with error: >>>>> >>>>> 14/11/05 10:25:14 INFO yarn.GiraphYarnClient: Giraph: >>>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.83 >>>>> secs >>>>> 14/11/05 10:25:14 INFO yarn.GiraphYarnClient: >>>>> appattempt_1415203231170_0002_000001, State: ACCEPTED, Containers used: 0 >>>>> 14/11/05 10:25:18 INFO yarn.GiraphYarnClient: Giraph: >>>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 4.84 >>>>> secs >>>>> 14/11/05 10:25:18 INFO yarn.GiraphYarnClient: >>>>> appattempt_1415203231170_0002_000001, State: ACCEPTED, Containers used: 0 >>>>> 14/11/05 10:25:22 INFO yarn.GiraphYarnClient: Giraph: >>>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 8.85 >>>>> secs >>>>> 14/11/05 10:25:22 INFO yarn.GiraphYarnClient: >>>>> appattempt_1415203231170_0002_000002, State: ACCEPTED, Containers used: 0 >>>>> 14/11/05 10:25:26 INFO yarn.GiraphYarnClient: Giraph: >>>>> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 12.86 >>>>> secs >>>>> 14/11/05 10:25:26 INFO yarn.GiraphYarnClient: >>>>> appattempt_1415203231170_0002_000002, State: ACCEPTED, Containers used: 0 >>>>> 14/11/05 10:25:29 ERROR yarn.GiraphYarnClient: Giraph: >>>>> org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED >>>>> state, diagnostics show: Application application_1415203231170_0002 >>>>> failed 2 times due to AM Container for >>>>> appattempt_1415203231170_0002_000002 exited with exitCode: 1 due to: >>>>> Exception from container-launch. >>>>> Container id: container_1415203231170_0002_02_000001 >>>>> Exit code: 1 >>>>> Stack trace: ExitCodeException exitCode=1: >>>>> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538) >>>>> at org.apache.hadoop.util.Shell.run(Shell.java:455) >>>>> at >>>>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:702) >>>>> at >>>>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:196) >>>>> at >>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:299) >>>>> at >>>>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) >>>>> at java.util.concurrent.FutureTask.run(FutureTask.java:262) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) >>>>> at >>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) >>>>> at java.lang.Thread.run(Thread.java:745) >>>>> >>>>> >>>>> Anyone can help? >>>>> >>>> >>>> >>> > >