@Alexander Sirotin : Thanks for your reply. I am really sorry I haven't
faced this problem after I executed it. Rather I am now having a different
issue with Containers :
*Result : *
14/08/12 15:17:51 INFO yarn.GiraphYarnClient: ApplicationSumbissionContext
for GiraphApplicationMaster launch container is populated.
14/08/12 15:17:51 INFO yarn.GiraphYarnClient: Submitting application to ASM
14/08/12 15:17:52 INFO impl.YarnClientImpl: Submitted application
application_1407836750214_0001
14/08/12 15:17:52 INFO yarn.GiraphYarnClient: Got new appId after
submission :application_1407836750214_0001
14/08/12 15:17:52 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
container request was submitted to ResourceManager for job: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation
14/08/12 15:17:52 INFO yarn.GiraphYarnClient: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.99
secs
14/08/12 15:17:52 INFO yarn.GiraphYarnClient:
appattempt_1407836750214_0001_000001, State: ACCEPTED, Containers used: 1
14/08/12 15:17:56 INFO yarn.GiraphYarnClient: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 5.01
secs
14/08/12 15:17:56 INFO yarn.GiraphYarnClient:
appattempt_1407836750214_0001_000002, State: ACCEPTED, Containers used: 0
14/08/12 15:18:00* ERROR yarn.GiraphYarnClient: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
state, diagnostics show: Application application_1407836750214_0001 failed
2 times due to AM Container for appattempt_1407836750214_0001_000002 exited
with exitCode: 1 due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException: *
*org.apache.hadoop.util.Shell$ExitCodeException:*
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application.
14/08/12 15:18:00 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed
cache directory for Giraph job.
14/08/12 15:18:00 INFO yarn.GiraphYarnClient: Completed Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total
running time: 0 minutes, 7 seconds.
I have also checked my *yarn-site.xml* file and updated with the following
property - value pairs inside configuration:
<property>
<name>yarn.nodemanager.aux-services.mapreduce.shuffle.class</name>
<value>org.apache.hadoop.mapred.ShuffleHandler</value>
</property>
<property>
<name>yarn.application.classpath</name>
<value>
%HADOOP_HOME%\etc\hadoop,
%HADOOP_HOME%\share\hadoop\common\*,
%HADOOP_HOME%\share\hadoop\common\lib\*,
%HADOOP_HOME%\share\hadoop\hdfs\*,
%HADOOP_HOME%\share\hadoop\hdfs\lib\*,
%HADOOP_HOME%\share\hadoop\mapreduce\*,
%HADOOP_HOME%\share\hadoop\mapreduce\lib\*,
%HADOOP_HOME%\share\hadoop\yarn\*,
%HADOOP_HOME%\share\hadoop\yarn\lib\*
</value>
</property>
Regards,
Vikalp Handa
On Tue, Aug 12, 2014 at 12:29 PM, Alexander Sirotin <sirot...@web.de> wrote:
> Hi,
>
> you can increase the Heapsize to 2048 Mb by setting it into
> etc/hadoop/hadoop-env.sh. Search for "export HADOOP_HEAPSIZE="
>
> But if I fix this, the TaskRunner and GiraphApplicationMaster will not
> stop to end. This is what I am seeing from the Client everytime repeating:
>
> 14/08/12 08:51:06 INFO yarn.GiraphYarnClient: Giraph:
> org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 1494,96
> secs
> 14/08/12 08:51:06 INFO yarn.GiraphYarnClient:
> appattempt_1407762054368_0017_000001, State: RUNNING, Containers used: 2
>
> The gam-stderr.log (GiraphApplicationMaster) showed the following:
>
> aphYarnTask 1407762054368 17 2 1
> |- 13562 11012 13562 13562 (bash) 0 0 18870272 435 /bin/bash -c
> java -Xmx1024M -Xms1024M -cp
> .:${CLASSPATH}:./*:$HADOOP_CONF_DIR:$HADOOP_COMMON_HOME/share/hadoop/common/*:$HADOOP_COMMON_HOME/share/hadoop/common/lib/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/*:$HADOOP_HDFS_HOME/share/hadoop/hdfs/lib/*:$HADOOP_YARN_HOME/share/hadoop/yarn/*:$HADOOP_YARN_HOME/share/hadoop/yarn/lib/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/*:$HADOOP_MAPRED_HOME/share/hadoop/mapreduce/lib/*:./*:/usr/local/hadoop/etc/hadoop:/usr/local/hadoop/share/hadoop/common/*:/usr/local/hadoop/share/hadoop/common/lib/*:/usr/local/hadoop/share/hadoop/hdfs/*:/usr/local/hadoop/share/hadoop/hdfs/lib/*:/usr/local/hadoop/share/hadoop/yarn/*:/usr/local/hadoop/share/hadoop/yarn/lib/*:/usr/local/hadoop/share/hadoop/mapreduce/*:/usr/local/hadoop/share/hadoop/mapreduce/lib/*
> org.apache.giraph.yarn.GiraphYarnTask 1407762054368 17 2 1
> 1>/usr/local/hadoop/logs/userlogs/application_1407762054368_0017/container_1407762054368_0017_01_000002/task-2-stdout.log
> 2>/usr/local/hadoop/logs/userlogs/application_1407762054368_0017/container_1407762054368_0017_01_000002/task-2-stderr.log
>
> Container killed on request. Exit code is 143
> Container exited with a non-zero exit code 143
>
> 14/08/12 08:26:23 INFO yarn.GiraphApplicationMaster: After completion of
> one conatiner. current status is: completedCount :1 containersToLaunch :2
> successfulCount :0 failedCount :1
>
>
> And the GiraphYarnTask also repeats the following:
>
> 14/08/12 08:56:15 WARN zookeeper.ClientCnxn: Session 0x147c8e553a10001 for
> server null, unexpected error, closing socket connection and attempting
> reconnect
> java.net.ConnectException: Verbindungsaufbau abgelehnt
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
> 14/08/12 08:56:16 INFO zookeeper.ClientCnxn: Opening socket connection to
> server blabla:22181. Will not attempt to authenticate using SASL (unknown
> error)
> 14/08/12 08:56:16 WARN zookeeper.ClientCnxn: Session 0x147c8e553a10001 for
> server null, unexpected error, closing socket connection and attempting
> reconnect
> java.net.ConnectException: Verbindungsaufbau abgelehnt
> at sun.nio.ch.SocketChannelImpl.checkConnect(Native Method)
> at
> sun.nio.ch.SocketChannelImpl.finishConnect(SocketChannelImpl.java:739)
> at
> org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:350)
> at
> org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
>
> Do you have an idea, how I can fix this? With ZooKeeper, I am also newbie
>
> Thx
>
>
>
>
>
> On 12.08.2014 08:14, Vikalp Handa wrote:
>
> Thanks a lot... I have build successfully even without running mvn clean
> :P
> Now while executing the example code I am facing a different problem of
> Cluster Heap Size.
>
> *How can I update this cluster heap size?*
>
> *Command: *hadoop jar
> /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.4.0-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner
> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
> -vip /user/hadoop/input/tiny_graph.txt -vof
> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
> /user/hadoop/outputpaths -yh 2048 -w 1
>
> *Result:*
>
> 14/08/12 11:37:15 WARN util.NativeCodeLoader: Unable to load
> native-hadoop library for your platform... using builtin-java classes where
> applicable
> 14/08/12 11:37:15 INFO utils.ConfigurationUtils: No edge input format
> specified. Ensure your InputFormat does not require one.
> 14/08/12 11:37:15 INFO utils.ConfigurationUtils: No edge output format
> specified. Ensure your OutputFormat does not require one.
> 14/08/12 11:37:15 INFO yarn.GiraphYarnClient: Final output path is:
> hdfs://localhost:9000/user/hadoop/outputpaths
> 14/08/12 11:37:15 INFO yarn.GiraphYarnClient: Running Client
> 14/08/12 11:37:15 INFO client.RMProxy: Connecting to ResourceManager at /
> 0.0.0.0:8032
> Exception in thread "main" java.lang.IllegalStateException: Giraph's
> estimated cluster heap 4096MB ask is greater than the current available
> cluster heap of 0MB. Aborting Job.
> at
> org.apache.giraph.yarn.GiraphYarnClient.checkPerNodeResourcesAvailable(GiraphYarnClient.java:230)
> at
> org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:124)
> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:96)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:126)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:483)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>
>
> Regards,
> Vikalp Handa
>
>
> On Tue, Aug 12, 2014 at 11:18 AM, 이성일 <si503....@samsung.com> wrote:
>
>> You just run 'mvn clean' before that command :)
>>
>> Best Regards, Seong-il
>> ------- Original Message -------
>> Sender : Vikalp Handa<handa.vik...@gmail.com>
>> Date : 2014-08-12 14:35 (GMT+09:00)
>> Title : Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x
>>
>> Thanks for your replies. Do I need to uninstall anything before running
>> the above mvn command ?
>>
>> Regards,
>> Vikalp Handa
>>
>> On Tue, Aug 12, 2014 at 10:26 AM, 이성일 <si503....@samsung.com> wrote:
>> Hi, Vikalp
>> I think you complied as wrong hadoop version.
>> Try below command
>> If you use yarn
>> mvn -Phadoop_yarn -Dhadoop.version=2.4.0 -DskipTests package
>> or if you use just hadoop2
>> mvn -Phadoop_2 -Dhadoop.version=2.4.0 -DskipTests package
>> Best Regards, Seong-il
>>
>> ------- Original Message -------
>> Sender : Vikalp Handa<handa.vik...@gmail.com>
>> Date : 2014-08-12 13:40 (GMT+09:00)
>> Title : ShortestPath Code execution on Hadoop 2.4.0 Inbox x
>>
>> Hi everyone,
>> I am new to Apache Giraph and would like execute Shortestpath and
>> PageRank example code on Hadoop 2.4.0 single node cluster (my machine)
>> having Centos 6.5.
>> I have successfully build Giraph on my machine but unable to execute
>> ShortestPath code.
>> Please let me know if there are any dependencies to be resolved before
>> code execution.
>>
>> P.S. :
>> Command used:
>> hadoop jar
>> /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar
>> org.apache.giraph.GiraphRunner
>> org.apache.giraph.examples.SimpleShortestPathsComputation -vif
>> org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
>> -vip /user/hadoop/input/tiny_graph.txt -vof
>> org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
>> /user/hduser/output/shortestpaths -w 1 -ca giraph.SplitMasterWorker=false
>>
>> Execution Result:
>> 14/08/11 18:48:37 WARN util.NativeCodeLoader: Unable to load
>> native-hadoop library for your platform... using builtin-java classes where
>> applicable
>> 14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge input format
>> specified. Ensure your InputFormat does not require one.
>> 14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge output format
>> specified. Ensure your OutputFormat does not require one.
>> 14/08/11 18:48:40 INFO utils.ConfigurationUtils: Setting custom argument
>> [giraph.SplitMasterWorker] to [false] in GiraphConfiguration
>> 14/08/11 18:48:40 INFO Configuration.deprecation:
>> mapreduce.job.counters.limit is deprecated. Instead, use
>> mapreduce.job.counters.max
>> 14/08/11 18:48:40 INFO Configuration.deprecation:
>> mapred.job.map.memory.mb is deprecated. Instead, use mapreduce.map.memory.mb
>> 14/08/11 18:48:40 INFO Configuration.deprecation:
>> mapred.job.reduce.memory.mb is deprecated. Instead, use
>> mapreduce.reduce.memory.mb
>> 14/08/11 18:48:40 INFO Configuration.deprecation:
>> mapred.map.tasks.speculative.execution is deprecated. Instead, use
>> mapreduce.map.speculative
>> 14/08/11 18:48:40 INFO Configuration.deprecation:
>> mapreduce.user.classpath.first is deprecated. Instead, use
>> mapreduce.job.user.classpath.first
>> 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.map.max.attempts
>> is deprecated. Instead, use mapreduce.map.maxattempts
>> 14/08/11 18:48:40 INFO job.GiraphJob: run: Since checkpointing is
>> disabled (default), do not allow any task retries (setting
>> mapred.map.max.attempts = 0, old value = 4)
>> 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.tracker is
>> deprecated. Instead, use mapreduce.jobtracker.address
>> 14/08/11 18:48:41 INFO Configuration.deprecation: session.id is
>> deprecated. Instead, use dfs.metrics.session-id
>> 14/08/11 18:48:41 INFO jvm.JvmMetrics: Initializing JVM Metrics with
>> processName=JobTracker, sessionId=
>> Exception in thread "main" java.lang.IncompatibleClassChangeError: Found
>> interface org.apache.hadoop.mapreduce.JobContext, but class was expected
>> at
>> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:44)
>> at
>> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:458)
>> at
>> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:343)
>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
>> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:422)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
>> at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
>> at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250)
>> at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
>> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
>> at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>> at java.lang.reflect.Method.invoke(Method.java:483)
>> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
>> Regards,
>> Vikalp Handa
>>
>
>
>
