Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x

2014-08-20 Thread Alexander Sirotin 

Hi Vikalp,

finally I got to run Giraph successfully :-)

Following steps:
#Preparation

 *

   su

 *

   rm -rf /usr/local/giraph; mkdir /usr/local/giraph

 *

   chown hadoop:hadoop /usr/local/giraph

 *

   mkdir ~/setup; cd ~/setup

#Sourcecode (Version 1.1.0)

 *

   git clone https://git-wip-us.apache.org/repos/asf/giraph.git

 *

   mv giraph giraph_src; cd giraph_src

 *

   mvn clean

 *

   mvn -Phadoop_yarn -Dhadoop.version=2.4.0 -DskipTests package

#Installation (little ugly)

 *

   rm -rf myGiraph; mkdir -p myGiraph/lib; mkdir myGiraph/source

 *

   find . -name \*.jar -exec cp {} myGiraph/lib \;

 *

   cp -r conf myGiraph

 *

   cp -r bin myGiraph

 *

   cp CH* CO* RE* LI* NO* myGiraph

 *

   cd myGiraph

 *

   mv lib/*source*.jar source

 *

   mv lib/giraph*.jar .

 *

   rename 's/-SNAPSHOT-for//' *.jar

 *

   rm -rf ../myGiraph; cd ..; mv myGiraph ..

 *

   cd ../myGiraph; rm -rf /usr/local/giraph/*; cp -r * /usr/local/giraph

Now you have to put a variable into hadoop-env.sh (at the end is ok):

 o

   /MY_LIBS=$HADOOP_HOME/share/myLib/*.jar/

 o

   /export HADOOP_TASKTRACKER_OPTS=-classpath ${MY_LIBS}
   ${HADOOP_TASKTRACKER_OPTS}/

The folder myLib you have to create and put the Giraph-Jar:

 * mkdir /usr/local/hadoop/share/myLib

 *

   cp
   
/usr/local/giraph/giraph-examples-1.1.0-hadoop-2.4.0-jar-with-dependencies.jar
   /usr/local/hadoop/share/myLib

#Configuration
Put
property
namemapred.map.tasks/name
value4/value
/property

into mapred-site.xml

 *

   echo export PATH=\$PATH:/usr/local/giraph/bin  ~/.bashrc

 *

   echo export GIRAPH_HOME=/usr/local/giraph  ~/.bashrc

 *

   source ~/.bashrc

Now you can test:
#creating testdata

 *

   rm -rf ~/giraph; mkdir ~/giraph; cd ~/giraph

 * cat  EOF  tiny_graph.txt
 * [0,0,[[1,1],[3,3]]]
   [1,0,[[0,1],[2,2],[3,1]]]
   [2,0,[[1,2],[4,4]]]
   [3,0,[[0,3],[1,1],[4,4]]]
   [4,0,[[3,4],[2,4]]]
 * EOF

#Run example-Jar

 *

   jar=giraph-examples-1.1.0-hadoop-2.4.0-jar-with-dependencies.jar

 *

   runner=org.apache.giraph.GiraphRunner

 *

   computation=org.apache.giraph.examples.SimpleShortestPathsComputation

 *

   
informat=org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat

 *

   outformat=org.apache.giraph.io.formats.IdWithValueTextOutputFormat

 *

   hdfs dfs -rm -r shortestpaths

 *

   yarn jar $GIRAPH_HOME/$jar $runner -Dgiraph.yarn.task.heap.mb=3000
   $computation -vif $informat -vip tiny_graph.txt -vof $outformat -op
   shortestpaths -w 1 -yj $jar

 *

   hdfs dfs -cat shortestpaths/*


Finished!! Hope it functions on your site.

Best regards,
Alexander




On 12.08.2014 15:52, Vikalp Handa wrote:

Hi Alexander,

I looked into my gam-stderr.log file and found *Error: Could not find 
or load main class org.apache.giraph.yarn.GiraphApplicationMaster*
As you have already mentioned in your reply about that for loop in 
*hadoop-env.sh. *
So can you please tell me what is *mylib *and***share/myLib/giraph** ? 
*Because I am having only doc and hadoop directories 
inside  $HADOOP_HOME/share/


Also how to get *giraph-1.1.0-hadoop-2.4.0.jar 
*and*giraph-examples-1.1.0-hadoop-*2.4.0.jar (without the 
dependencies) as I am having only 
giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.4.0-jar-with-dependencies.jar 
inside giraph-examples/target/munged/.

Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x

2014-08-12 Thread Alexander Sirotin 
 
available cluster heap of 0MB. Aborting Job.
at 
org.apache.giraph.yarn.GiraphYarnClient.checkPerNodeResourcesAvailable(GiraphYarnClient.java:230)
at 
org.apache.giraph.yarn.GiraphYarnClient.run(GiraphYarnClient.java:124)

at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:96)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:126)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)

at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)


Regards,
Vikalp Handa


On Tue, Aug 12, 2014 at 11:18 AM, 이성일 si503@samsung.com 
mailto:si503@samsung.com wrote:


You just run 'mvn clean' before that command :)

Best Regards, Seong-il
--- Original Message ---
Sender : Vikalp Handahanda.vik...@gmail.com
mailto:handa.vik...@gmail.com
Date : 2014-08-12 14:35 (GMT+09:00)
Title : Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x

Thanks for your replies. Do I need to uninstall anything before
running the above mvn command ?

Regards,
Vikalp Handa

On Tue, Aug 12, 2014 at 10:26 AM, 이성일 si503@samsung.com
mailto:si503@samsung.com wrote:
Hi, Vikalp
I think you complied as wrong hadoop version.
Try below command
If you use yarn
mvn -Phadoop_yarn -Dhadoop.version=2.4.0 -DskipTests package
or if you use just hadoop2
mvn -Phadoop_2  -Dhadoop.version=2.4.0 -DskipTests package
Best Regards, Seong-il

--- Original Message ---
Sender : Vikalp Handahanda.vik...@gmail.com
mailto:handa.vik...@gmail.com
Date : 2014-08-12 13:40 (GMT+09:00)
Title : ShortestPath Code execution on Hadoop 2.4.0 Inbox x

Hi everyone,
I am new to Apache Giraph and would like execute Shortestpath and
PageRank example code on Hadoop 2.4.0 single node cluster (my
machine) having Centos 6.5.
I have successfully build Giraph on my machine but unable to
execute ShortestPath code.
Please let me know if there are any dependencies to be resolved
before code execution.

P.S. :
Command used:
hadoop jar

/usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner
org.apache.giraph.examples.SimpleShortestPathsComputation -vif
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
-vip /user/hadoop/input/tiny_graph.txt -vof
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
/user/hduser/output/shortestpaths -w 1  -ca
giraph.SplitMasterWorker=false

Execution Result:
14/08/11 18:48:37 WARN util.NativeCodeLoader: Unable to load
native-hadoop library for your platform... using builtin-java
classes where applicable
14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge input
format specified. Ensure your InputFormat does not require one.
14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge output
format specified. Ensure your OutputFormat does not require one.
14/08/11 18:48:40 INFO utils.ConfigurationUtils: Setting custom
argument [giraph.SplitMasterWorker] to [false] in GiraphConfiguration
14/08/11 18:48:40 INFO Configuration.deprecation:
mapreduce.job.counters.limit is deprecated. Instead, use
mapreduce.job.counters.max
14/08/11 18:48:40 INFO Configuration.deprecation:
mapred.job.map.memory.mb is deprecated. Instead, use
mapreduce.map.memory.mb
14/08/11 18:48:40 INFO Configuration.deprecation:
mapred.job.reduce.memory.mb is deprecated. Instead, use
mapreduce.reduce.memory.mb
14/08/11 18:48:40 INFO Configuration.deprecation:
mapred.map.tasks.speculative.execution is deprecated. Instead, use
mapreduce.map.speculative
14/08/11 18:48:40 INFO Configuration.deprecation:
mapreduce.user.classpath.first is deprecated. Instead, use
mapreduce.job.user.classpath.first
14/08/11 18:48:40 INFO Configuration.deprecation:
mapred.map.max.attempts is deprecated. Instead, use
mapreduce.map.maxattempts
14/08/11 18:48:40 INFO job.GiraphJob: run: Since checkpointing is
disabled (default), do not allow any task retries (setting
mapred.map.max.attempts = 0, old value = 4)
14/08/11 18:48:40 INFO Configuration.deprecation:
mapred.job.tracker is deprecated. Instead, use
mapreduce.jobtracker.address
14/08/11 18:48:41 INFO Configuration.deprecation: session.id
http://session.id is deprecated. Instead, use dfs.metrics.session-id
14/08/11 18:48:41 INFO jvm.JvmMetrics: Initializing JVM

Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x

2014-08-12 Thread Vikalp Handa 
 : Vikalp Handahanda.vik...@gmail.com
  Date : 2014-08-12 14:35 (GMT+09:00)
 Title : Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x

 Thanks for your replies. Do I need to uninstall anything before running
 the above mvn command ?

 Regards,
 Vikalp Handa

 On Tue, Aug 12, 2014 at 10:26 AM, 이성일 si503@samsung.com wrote:
 Hi, Vikalp
 I think you complied as wrong hadoop version.
 Try below command
 If you use yarn
 mvn -Phadoop_yarn -Dhadoop.version=2.4.0 -DskipTests package
 or if you use just hadoop2
 mvn -Phadoop_2  -Dhadoop.version=2.4.0 -DskipTests package
 Best Regards, Seong-il

 --- Original Message ---
 Sender : Vikalp Handahanda.vik...@gmail.com
 Date : 2014-08-12 13:40 (GMT+09:00)
 Title : ShortestPath Code execution on Hadoop 2.4.0 Inbox x

 Hi everyone,
 I am new to Apache Giraph and would like execute Shortestpath and
 PageRank example code on Hadoop 2.4.0 single node cluster (my machine)
 having Centos 6.5.
 I have successfully build Giraph on my machine but unable to execute
 ShortestPath code.
 Please let me know if there are any dependencies to be resolved before
 code execution.

 P.S. :
 Command used:
 hadoop jar
 /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar
 org.apache.giraph.GiraphRunner
 org.apache.giraph.examples.SimpleShortestPathsComputation -vif
 org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
 -vip /user/hadoop/input/tiny_graph.txt -vof
 org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
 /user/hduser/output/shortestpaths -w 1  -ca giraph.SplitMasterWorker=false

 Execution Result:
 14/08/11 18:48:37 WARN util.NativeCodeLoader: Unable to load
 native-hadoop library for your platform... using builtin-java classes where
 applicable
 14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge input format
 specified. Ensure your InputFormat does not require one.
 14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge output format
 specified. Ensure your OutputFormat does not require one.
 14/08/11 18:48:40 INFO utils.ConfigurationUtils: Setting custom argument
 [giraph.SplitMasterWorker] to [false] in GiraphConfiguration
 14/08/11 18:48:40 INFO Configuration.deprecation:
 mapreduce.job.counters.limit is deprecated. Instead, use
 mapreduce.job.counters.max
 14/08/11 18:48:40 INFO Configuration.deprecation:
 mapred.job.map.memory.mb is deprecated. Instead, use mapreduce.map.memory.mb
 14/08/11 18:48:40 INFO Configuration.deprecation:
 mapred.job.reduce.memory.mb is deprecated. Instead, use
 mapreduce.reduce.memory.mb
 14/08/11 18:48:40 INFO Configuration.deprecation:
 mapred.map.tasks.speculative.execution is deprecated. Instead, use
 mapreduce.map.speculative
 14/08/11 18:48:40 INFO Configuration.deprecation:
 mapreduce.user.classpath.first is deprecated. Instead, use
 mapreduce.job.user.classpath.first
 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.map.max.attempts
 is deprecated. Instead, use mapreduce.map.maxattempts
 14/08/11 18:48:40 INFO job.GiraphJob: run: Since checkpointing is
 disabled (default), do not allow any task retries (setting
 mapred.map.max.attempts = 0, old value = 4)
 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.tracker is
 deprecated. Instead, use mapreduce.jobtracker.address
 14/08/11 18:48:41 INFO Configuration.deprecation: session.id is
 deprecated. Instead, use dfs.metrics.session-id
 14/08/11 18:48:41 INFO jvm.JvmMetrics: Initializing JVM Metrics with
 processName=JobTracker, sessionId=
 Exception in thread main java.lang.IncompatibleClassChangeError: Found
 interface org.apache.hadoop.mapreduce.JobContext, but class was expected
 at
 org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:44)
 at
 org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:458)
 at
 org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:343)
 at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
 at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
 at java.security.AccessController.doPrivileged(Native Method)
 at javax.security.auth.Subject.doAs(Subject.java:422)
 at
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
 at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
 at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250)
 at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
 at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
 at
 sun.reflect.DelegatingMethodAccessorImpl.invoke

Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x

2014-08-12 Thread Alexander Sirotin 

Hello Vikalp,

you are showing the output of the client. The errors I ve showed were 
coming from the ResourceManager, the GiraphApplicationMaster 
(gam-stderr.log) and the GiraphYarnTask (TaskRunner). Checkout these 
logfiles on your systems.


In case you get an error-message class GiraphApplicationMaster not 
found: I solved it by adding a for-loop in hadoop-env.sh:


for f in `find $HADOOP_HOME/share/myLib/ -name \*.jar`; do
  if [ $HADOOP_CLASSPATH ]; then
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f
  else
export HADOOP_CLASSPATH=$f
  fi
done

In share/myLib/giraph I have put giraph-1.1.0-hadoop-2.4.0.jar and 
giraph-examples-1.1.0-hadoop-2.4.0.jar (without the dependencies). 
Additionally I copied every jar from /usr/local/giraph/lib also to this 
folder.


Then you can restart the cluster and every instance should know giraph.

After this it should function, if you have enough memory. I get then a 
message, that the GiraphApplicationMaster dont have enough memory and 
then get killed. But before this, it has started the TaskRunner, which 
keeps the whole job endlessly alive.


Sincerely,



On 12.08.2014 11:55, Vikalp Handa wrote:
@Alexander Sirotin : Thanks for your reply. I am really sorry I 
haven't faced this problem after I executed it. Rather I am now having 
a different issue with Containers :

*
*
*Result : *
14/08/12 15:17:51 INFO yarn.GiraphYarnClient: 
ApplicationSumbissionContext for GiraphApplicationMaster launch 
container is populated.
14/08/12 15:17:51 INFO yarn.GiraphYarnClient: Submitting application 
to ASM
14/08/12 15:17:52 INFO impl.YarnClientImpl: Submitted application 
application_1407836750214_0001
14/08/12 15:17:52 INFO yarn.GiraphYarnClient: Got new appId after 
submission :application_1407836750214_0001
14/08/12 15:17:52 INFO yarn.GiraphYarnClient: GiraphApplicationMaster 
container request was submitted to ResourceManager for job: Giraph: 
org.apache.giraph.examples.SimpleShortestPathsComputation
14/08/12 15:17:52 INFO yarn.GiraphYarnClient: Giraph: 
org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 
0.99 secs
14/08/12 15:17:52 INFO yarn.GiraphYarnClient: 
appattempt_1407836750214_0001_01, State: ACCEPTED, Containers used: 1
14/08/12 15:17:56 INFO yarn.GiraphYarnClient: Giraph: 
org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 
5.01 secs
14/08/12 15:17:56 INFO yarn.GiraphYarnClient: 
appattempt_1407836750214_0001_02, State: ACCEPTED, Containers used: 0
14/08/12 15:18:00*ERROR yarn.GiraphYarnClient: Giraph: 
org.apache.giraph.examples.SimpleShortestPathsComputation reports 
FAILED state, diagnostics show: Application 
application_1407836750214_0001 failed 2 times due to AM Container for 
appattempt_1407836750214_0001_02 exited with  exitCode: 1 due to: 
Exception from container-launch: 
org.apache.hadoop.util.Shell$ExitCodeException: *

*org.apache.hadoop.util.Shell$ExitCodeException:*
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at org.apache.hadoop.util.Shell.run(Shell.java:418)
at 
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
at 
org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
at 
org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)

at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)

at java.lang.Thread.run(Thread.java:745)


Container exited with a non-zero exit code 1
.Failing this attempt.. Failing the application.
14/08/12 15:18:00 INFO yarn.GiraphYarnClient: Cleaning up HDFS 
distributed cache directory for Giraph job.
14/08/12 15:18:00 INFO yarn.GiraphYarnClient: Completed Giraph: 
org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, 
total running time: 0 minutes, 7 seconds.


I have also checked my *yarn-site.xml* file and updated with the 
following property - value pairs inside configuration:


|property|||
|name||yarn.nodemanager.aux-services.mapreduce.shuffle.class/||name|||
|value||org.apache.hadoop.mapred.ShuffleHandler/||value|||
|||/||property|||
|property|||
|name||yarn.application.classpath/||name|||
|value|||
|||%HADOOP_HOME%\etc\hadoop,|
|||%HADOOP_HOME%\share\hadoop\common\*,|
|||%HADOOP_HOME%\share\hadoop\common\lib\*,|
|||%HADOOP_HOME%\share\hadoop\hdfs\*,|
|||%HADOOP_HOME%\share\hadoop\hdfs\lib\*,|
|||%HADOOP_HOME%\share\hadoop\mapreduce\*,|
|||%HADOOP_HOME%\share\hadoop\mapreduce\lib\*,|
|||%HADOOP_HOME%\share\hadoop\yarn\*,|
|||%HADOOP_HOME%\share\hadoop\yarn\lib\*|
|||/||value|||
|||/||property|||
*|
|*


Regards,
Vikalp Handa

Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x

2014-08-12 Thread Vikalp Handa 
Hi Alexander,

I looked into my gam-stderr.log file and found *Error: Could not find or
load main class org.apache.giraph.yarn.GiraphApplicationMaster*
As you have already mentioned in your reply about that for loop in
*hadoop-env.sh. *
So can you please tell me what is *mylib *and *share/myLib/giraph** ? *Because
I am having only doc and hadoop directories inside  $HADOOP_HOME/share/

Also how to get *giraph-1.1.0-hadoop-2.4.0.jar *and*
giraph-examples-1.1.0-hadoop-*2.4.0.jar (without the dependencies) as I am
having only 
giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.4.0-jar-with-dependencies.jar
inside giraph-examples/target/munged/.




Regards,
Vikalp Handa


On Tue, Aug 12, 2014 at 4:25 PM, Alexander Sirotin sirot...@web.de wrote:

  Hello Vikalp,

 you are showing the output of the client. The errors I ve showed were
 coming from the ResourceManager, the GiraphApplicationMaster
 (gam-stderr.log) and the GiraphYarnTask (TaskRunner). Checkout these
 logfiles on your systems.

 In case you get an error-message class GiraphApplicationMaster not
 found: I solved it by adding a for-loop in hadoop-env.sh:

 for f in `find $HADOOP_HOME/share/myLib/ -name \*.jar`; do
   if [ $HADOOP_CLASSPATH ]; then
 export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f
   else
 export HADOOP_CLASSPATH=$f
   fi
 done

 In share/myLib/giraph I have put giraph-1.1.0-hadoop-2.4.0.jar and
 giraph-examples-1.1.0-hadoop-2.4.0.jar (without the dependencies).
 Additionally I copied every jar from /usr/local/giraph/lib also to this
 folder.

 Then you can restart the cluster and every instance should know giraph.

 After this it should function, if you have enough memory. I get then a
 message, that the GiraphApplicationMaster dont have enough memory and then
 get killed. But before this, it has started the TaskRunner, which keeps the
 whole job endlessly alive.

 Sincerely,




 On 12.08.2014 11:55, Vikalp Handa wrote:

 @Alexander Sirotin : Thanks for your reply. I am really sorry I haven't
 faced this problem after I executed it. Rather I am now having a different
 issue with Containers :

  *Result : *
  14/08/12 15:17:51 INFO yarn.GiraphYarnClient:
 ApplicationSumbissionContext for GiraphApplicationMaster launch container
 is populated.
 14/08/12 15:17:51 INFO yarn.GiraphYarnClient: Submitting application to ASM
 14/08/12 15:17:52 INFO impl.YarnClientImpl: Submitted application
 application_1407836750214_0001
 14/08/12 15:17:52 INFO yarn.GiraphYarnClient: Got new appId after
 submission :application_1407836750214_0001
 14/08/12 15:17:52 INFO yarn.GiraphYarnClient: GiraphApplicationMaster
 container request was submitted to ResourceManager for job: Giraph:
 org.apache.giraph.examples.SimpleShortestPathsComputation
 14/08/12 15:17:52 INFO yarn.GiraphYarnClient: Giraph:
 org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 0.99
 secs
 14/08/12 15:17:52 INFO yarn.GiraphYarnClient:
 appattempt_1407836750214_0001_01, State: ACCEPTED, Containers used: 1
 14/08/12 15:17:56 INFO yarn.GiraphYarnClient: Giraph:
 org.apache.giraph.examples.SimpleShortestPathsComputation, Elapsed: 5.01
 secs
 14/08/12 15:17:56 INFO yarn.GiraphYarnClient:
 appattempt_1407836750214_0001_02, State: ACCEPTED, Containers used: 0
 14/08/12 15:18:00* ERROR yarn.GiraphYarnClient: Giraph:
 org.apache.giraph.examples.SimpleShortestPathsComputation reports FAILED
 state, diagnostics show: Application application_1407836750214_0001 failed
 2 times due to AM Container for appattempt_1407836750214_0001_02 exited
 with  exitCode: 1 due to: Exception from container-launch:
 org.apache.hadoop.util.Shell$ExitCodeException: *
 *org.apache.hadoop.util.Shell$ExitCodeException:*
  at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
  at org.apache.hadoop.util.Shell.run(Shell.java:418)
  at
 org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650)
  at
 org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:195)
  at
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300)
  at
 org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81)
  at java.util.concurrent.FutureTask.run(FutureTask.java:266)
  at
 java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
  at
 java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
  at java.lang.Thread.run(Thread.java:745)


  Container exited with a non-zero exit code 1
 .Failing this attempt.. Failing the application.
 14/08/12 15:18:00 INFO yarn.GiraphYarnClient: Cleaning up HDFS distributed
 cache directory for Giraph job.
 14/08/12 15:18:00 INFO yarn.GiraphYarnClient: Completed Giraph:
 org.apache.giraph.examples.SimpleShortestPathsComputation: FAILED, total
 running time: 0 minutes, 7 seconds.

  I have also checked my *yarn-site.xml* file and updated with the
 following

Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x

2014-08-12 Thread Alexander Sirotin 

Hi Vikalp,

in $HADOOP_HOME/share there you will find Jars. But also Hadoop itself 
uses this place. Important is also there the subdirectory 
hadoop/mapreduce, where Jars will be used for TaskRunner and 
ApplicationMasters. If you would put your giraph-Jar there, the 
ApplicationMaster should find it, but the bad thing is, it is not your 
directory ;-) and 2nd in the Jar with dependencies, there are Classes 
included, which has Hadoop already and this caused errors on my systems. 
Therefore I created myLib by my own and put only additional Jars into it.


In hadoop-env.sh you can search for a loop-command 'for f in ' This 
loop I just copied and modified for my own usage. I am sorry I am not an 
expert, but I do not want to waste so much time on it and made this 
workaround :-P


If I start to run my system successfully ( I also using Hadoop 2.4 with 
Giraph 1.1.0), then I will let you know how.


Sincerely

On 12.08.2014 15:52, Vikalp Handa wrote:

Hi Alexander,

I looked into my gam-stderr.log file and found *Error: Could not find 
or load main class org.apache.giraph.yarn.GiraphApplicationMaster*
As you have already mentioned in your reply about that for loop in 
*hadoop-env.sh. *
So can you please tell me what is *mylib *and***share/myLib/giraph** ? 
*Because I am having only doc and hadoop directories 
inside  $HADOOP_HOME/share/


Also how to get *giraph-1.1.0-hadoop-2.4.0.jar 
*and*giraph-examples-1.1.0-hadoop-*2.4.0.jar (without the 
dependencies) as I am having only 
giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.4.0-jar-with-dependencies.jar 
inside giraph-examples/target/munged/.





Regards,
Vikalp Handa


On Tue, Aug 12, 2014 at 4:25 PM, Alexander Sirotin sirot...@web.de 
mailto:sirot...@web.de wrote:


Hello Vikalp,

you are showing the output of the client. The errors I ve showed
were coming from the ResourceManager, the GiraphApplicationMaster
(gam-stderr.log) and the GiraphYarnTask (TaskRunner). Checkout
these logfiles on your systems.

In case you get an error-message class GiraphApplicationMaster
not found: I solved it by adding a for-loop in hadoop-env.sh:

for f in `find $HADOOP_HOME/share/myLib/ -name \*.jar`; do
  if [ $HADOOP_CLASSPATH ]; then
export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f
  else
export HADOOP_CLASSPATH=$f
  fi
done

In share/myLib/giraph I have put giraph-1.1.0-hadoop-2.4.0.jar and
giraph-examples-1.1.0-hadoop-2.4.0.jar (without the dependencies).
Additionally I copied every jar from /usr/local/giraph/lib also to
this folder.

Then you can restart the cluster and every instance should know
giraph.

After this it should function, if you have enough memory. I get
then a message, that the GiraphApplicationMaster dont have enough
memory and then get killed. But before this, it has started the
TaskRunner, which keeps the whole job endlessly alive.

Sincerely,




On 12.08.2014 11:55, Vikalp Handa wrote:

@Alexander Sirotin : Thanks for your reply. I am really sorry I
haven't faced this problem after I executed it. Rather I am now
having a different issue with Containers :
*
*
*Result : *
14/08/12 15:17:51 INFO yarn.GiraphYarnClient:
ApplicationSumbissionContext for GiraphApplicationMaster launch
container is populated.
14/08/12 15:17:51 INFO yarn.GiraphYarnClient: Submitting
application to ASM
14/08/12 15:17:52 INFO impl.YarnClientImpl: Submitted application
application_1407836750214_0001
14/08/12 15:17:52 INFO yarn.GiraphYarnClient: Got new appId after
submission :application_1407836750214_0001
14/08/12 15:17:52 INFO yarn.GiraphYarnClient:
GiraphApplicationMaster container request was submitted to
ResourceManager for job: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation
14/08/12 15:17:52 INFO yarn.GiraphYarnClient: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation,
Elapsed: 0.99 secs
14/08/12 15:17:52 INFO yarn.GiraphYarnClient:
appattempt_1407836750214_0001_01, State: ACCEPTED, Containers
used: 1
14/08/12 15:17:56 INFO yarn.GiraphYarnClient: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation,
Elapsed: 5.01 secs
14/08/12 15:17:56 INFO yarn.GiraphYarnClient:
appattempt_1407836750214_0001_02, State: ACCEPTED, Containers
used: 0
14/08/12 15:18:00*ERROR yarn.GiraphYarnClient: Giraph:
org.apache.giraph.examples.SimpleShortestPathsComputation reports
FAILED state, diagnostics show: Application
application_1407836750214_0001 failed 2 times due to AM Container
for appattempt_1407836750214_0001_02 exited with  exitCode: 1
due to: Exception from container-launch:
org.apache.hadoop.util.Shell$ExitCodeException: *
*org.apache.hadoop.util.Shell$ExitCodeException:*
at org.apache.hadoop.util.Shell.runCommand(Shell.java:505)
at

Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x

2014-08-11 Thread Sundara Raghavan Sankaran 
Hi,

You built Giraph for the Hadoop version 1.2.1 which is evident from your
command line
*hadoop jar
/usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar
*

You have the build Giraph against the Hadoop version you'll be using. If
you are using Hadoop 2.4.0, the command to build would be
*mvn -Phadoop_2 -Dhadoop.version=2.4.0 -DskipTests ...*
or
*mvn -Phadoop_yarn -Dhadoop.version=2.4.0 -DskipTests ...*

You can find this in the *README* file from Giraph trunk

Hope this helps.



On Tue, Aug 12, 2014 at 10:10 AM, Vikalp Handa handa.vik...@gmail.com
wrote:

 Hi everyone,

 I am new to Apache Giraph and would like execute Shortestpath and PageRank
 example code on *Hadoop 2.4.0 single node cluster* (my machine) having
 Centos 6.5.

 I have successfully build Giraph on my machine but unable to execute
 ShortestPath code.

 Please let me know if there are any dependencies to be resolved before
 code execution.

  *P.S. :*
 *Command used:*
 hadoop jar
 /usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar
 org.apache.giraph.GiraphRunner
 org.apache.giraph.examples.SimpleShortestPathsComputation -vif
 org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat
 -vip /user/hadoop/input/tiny_graph.txt -vof
 org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op
 /user/hduser/output/shortestpaths -w 1  -ca giraph.SplitMasterWorker=false

 *Execution Result:*
 14/08/11 18:48:37 WARN util.NativeCodeLoader: Unable to load native-hadoop
 library for your platform... using builtin-java classes where applicable
 14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge input format
 specified. Ensure your InputFormat does not require one.
 14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge output format
 specified. Ensure your OutputFormat does not require one.
 14/08/11 18:48:40 INFO utils.ConfigurationUtils: Setting custom argument
 [giraph.SplitMasterWorker] to [false] in GiraphConfiguration
 14/08/11 18:48:40 INFO Configuration.deprecation:
 mapreduce.job.counters.limit is deprecated. Instead, use
 mapreduce.job.counters.max
 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.map.memory.mb
 is deprecated. Instead, use mapreduce.map.memory.mb
 14/08/11 18:48:40 INFO Configuration.deprecation:
 mapred.job.reduce.memory.mb is deprecated. Instead, use
 mapreduce.reduce.memory.mb
 14/08/11 18:48:40 INFO Configuration.deprecation:
 mapred.map.tasks.speculative.execution is deprecated. Instead, use
 mapreduce.map.speculative
 14/08/11 18:48:40 INFO Configuration.deprecation:
 mapreduce.user.classpath.first is deprecated. Instead, use
 mapreduce.job.user.classpath.first
 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.map.max.attempts
 is deprecated. Instead, use mapreduce.map.maxattempts
 14/08/11 18:48:40 INFO job.GiraphJob: run: Since checkpointing is disabled
 (default), do not allow any task retries (setting mapred.map.max.attempts =
 0, old value = 4)
 14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.tracker is
 deprecated. Instead, use mapreduce.jobtracker.address
 14/08/11 18:48:41 INFO Configuration.deprecation: session.id is
 deprecated. Instead, use dfs.metrics.session-id
 14/08/11 18:48:41 INFO jvm.JvmMetrics: Initializing JVM Metrics with
 processName=JobTracker, sessionId=
 Exception in thread main java.lang.IncompatibleClassChangeError: Found
 interface org.apache.hadoop.mapreduce.JobContext, but class was expected
  at
 org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:44)
 at
 org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:458)
  at
 org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:343)
 at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
  at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
 at java.security.AccessController.doPrivileged(Native Method)
  at javax.security.auth.Subject.doAs(Subject.java:422)
 at
 org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
  at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
 at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250)
  at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
 at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
  at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
 at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
 at
 sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
  at
 sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at java.lang.reflect.Method.invoke(Method.java:483)
  at org.apache.hadoop.util.RunJar.main(RunJar.java:212)




 Regards,
 Vikalp Handa

Re: Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x

2014-08-11 Thread 이성일 
You just run 'mvn clean' before that command :)

Best Regards, Seong-il
--- Original Message ---
Sender : Vikalp Handahanda.vik...@gmail.com
Date : 2014-08-12 14:35 (GMT+09:00)
Title : Re: ShortestPath Code execution on Hadoop 2.4.0 Inbox x
 
Thanks for your replies. Do I need to uninstall anything before running the 
above mvn command ?

Regards,
Vikalp Handa

On Tue, Aug 12, 2014 at 10:26 AM, 이성일 si503@samsung.com wrote:
Hi, Vikalp
I think you complied as wrong hadoop version.
Try below command
If you use yarn
mvn -Phadoop_yarn -Dhadoop.version=2.4.0 -DskipTests package
or if you use just hadoop2
mvn -Phadoop_2  -Dhadoop.version=2.4.0 -DskipTests package
Best Regards, Seong-il

--- Original Message ---
Sender : Vikalp Handahanda.vik...@gmail.com
Date : 2014-08-12 13:40 (GMT+09:00)
Title : ShortestPath Code execution on Hadoop 2.4.0 Inbox x

Hi everyone,
I am new to Apache Giraph and would like execute Shortestpath and PageRank 
example code on Hadoop 2.4.0 single node cluster (my machine) having Centos 6.5.
I have successfully build Giraph on my machine but unable to execute 
ShortestPath code.
Please let me know if there are any dependencies to be resolved before code 
execution.

P.S. :
Command used:
hadoop jar 
/usr/local/giraph/giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-1.2.1-jar-with-dependencies.jar
 org.apache.giraph.GiraphRunner 
org.apache.giraph.examples.SimpleShortestPathsComputation -vif 
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip 
/user/hadoop/input/tiny_graph.txt -vof 
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op 
/user/hduser/output/shortestpaths -w 1  -ca giraph.SplitMasterWorker=false

Execution Result:
14/08/11 18:48:37 WARN util.NativeCodeLoader: Unable to load native-hadoop 
library for your platform... using builtin-java classes where applicable
14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge input format 
specified. Ensure your InputFormat does not require one.
14/08/11 18:48:40 INFO utils.ConfigurationUtils: No edge output format 
specified. Ensure your OutputFormat does not require one.
14/08/11 18:48:40 INFO utils.ConfigurationUtils: Setting custom argument 
[giraph.SplitMasterWorker] to [false] in GiraphConfiguration
14/08/11 18:48:40 INFO Configuration.deprecation: mapreduce.job.counters.limit 
is deprecated. Instead, use mapreduce.job.counters.max
14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.map.memory.mb is 
deprecated. Instead, use mapreduce.map.memory.mb
14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.reduce.memory.mb 
is deprecated. Instead, use mapreduce.reduce.memory.mb
14/08/11 18:48:40 INFO Configuration.deprecation: 
mapred.map.tasks.speculative.execution is deprecated. Instead, use 
mapreduce.map.speculative
14/08/11 18:48:40 INFO Configuration.deprecation: 
mapreduce.user.classpath.first is deprecated. Instead, use 
mapreduce.job.user.classpath.first
14/08/11 18:48:40 INFO Configuration.deprecation: mapred.map.max.attempts is 
deprecated. Instead, use mapreduce.map.maxattempts
14/08/11 18:48:40 INFO job.GiraphJob: run: Since checkpointing is disabled 
(default), do not allow any task retries (setting mapred.map.max.attempts = 0, 
old value = 4)
14/08/11 18:48:40 INFO Configuration.deprecation: mapred.job.tracker is 
deprecated. Instead, use mapreduce.jobtracker.address
14/08/11 18:48:41 INFO Configuration.deprecation: session.id is deprecated. 
Instead, use dfs.metrics.session-id
14/08/11 18:48:41 INFO jvm.JvmMetrics: Initializing JVM Metrics with 
processName=JobTracker, sessionId=
Exception in thread main java.lang.IncompatibleClassChangeError: Found 
interface org.apache.hadoop.mapreduce.JobContext, but class was expected
at 
org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:44)
at 
org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:458)
at 
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:343)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at 
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1548)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at org.apache.giraph.job.GiraphJob.run(GiraphJob.java:250)
at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:94)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:124)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62