Giraph keeps trying to connect to 9000 on Hadoop 2.2.0/YARN

2014-06-01 Thread John Yost 
Hi Everyone,

Not sure why, but Giraph tries to connect to port 9000:

java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 to
localhost:9000 failed on connection exception: java.net.ConnectException:
Connection refused; For more details see:
http://wiki.apache.org/hadoop/ConnectionRefused
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

I set the following in the Giraph configuration:

  GiraphConstants.IS_PURE_YARN_JOB.set(conf,true);
  conf.set(giraph.useNetty,true);
  conf.set(giraph.zkList,localhost.localdomain);
  conf.set(fs.defaultFS,hdfs://localhost.localdomain:8020)
  conf.set(mapreduce.job.tracker,localhost.localdomain:54311);
  conf.set(mapreduce.framework.name,yarn);
  conf.set(yarn.resourcemanager.address,localhost.localdomain:8032);

I built Giraph as follows:

mvn -DskipTests=true -Dhadoop.version=2.2.0 -Phadoop_yarn clean install

Any ideas as to why Giraph attempts to connect to 9000 instead of 8020?

--John

Re: Giraph keeps trying to connect to 9000 on Hadoop 2.2.0/YARN

2014-06-01 Thread Avery Ching 
Giraph should just pick up your cluster's HDFS configuration.  Can you 
check your hadoop *.xml files?


On 6/1/14, 3:34 AM, John Yost wrote:

Hi Everyone,

Not sure why, but Giraph tries to connect to port 9000:

java.net.ConnectException: Call From localhost.localdomain/127.0.0.1 
http://127.0.0.1 to localhost:9000 failed on connection exception: 
java.net.ConnectException: Connection refused; For more details see: 
http://wiki.apache.org/hadoop/ConnectionRefused

at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)

I set the following in the Giraph configuration:

 GiraphConstants.IS_PURE_YARN_JOB.set(conf,true);
 conf.set(giraph.useNetty,true);
 conf.set(giraph.zkList,localhost.localdomain);
conf.set(fs.defaultFS,hdfs://localhost.localdomain:8020)
 conf.set(mapreduce.job.tracker,localhost.localdomain:54311);
 conf.set(mapreduce.framework.name 
http://mapreduce.framework.name,yarn);

 conf.set(yarn.resourcemanager.address,localhost.localdomain:8032);

I built Giraph as follows:

mvn -DskipTests=true -Dhadoop.version=2.2.0 -Phadoop_yarn clean install

Any ideas as to why Giraph attempts to connect to 9000 instead of 8020?

--John

Re: issues while running giraph on Hadoop 2.2

2014-06-01 Thread Kiru Pakkirisamy 
Hi Eli,
Thanks for the response.
I get this error now -

(for this cmd - 
hadoop jar giraph-examples-1.1.0-SNAPSHOT.jar org.apache.giraph.GiraphRunner 
org.apache.giraph.examples.SimpleShortestPathsComputation -vif 
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip 
/user/hdfs/input/tiny.txt -vof 
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op 
/user/hdfs/output/shortest -w 1 -ca giraph.SplitMasterWorker=false -ca 
giraph.zkList=localhost:2181 -yj 
giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
 
)

14/06/01 14:25:20 INFO utils.ConfigurationUtils: No edge input format 
specified. Ensure your InputFormat does not require one.
14/06/01 14:25:20 INFO utils.ConfigurationUtils: No edge output format 
specified. Ensure your OutputFormat does not require one.
14/06/01 14:25:20 INFO utils.ConfigurationUtils: Setting custom argument 
[giraph.SplitMasterWorker] to [false] in GiraphConfiguration
14/06/01 14:25:20 INFO utils.ConfigurationUtils: Setting custom argument 
[giraph.zkList] to [localhost:2181] in GiraphConfiguration
Exception in thread main java.lang.ClassNotFoundException: 
org.apache.giraph.examples.SimpleShortestPathsComputation
at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.Class.forName0(Native Method)
at java.lang.Class.forName(Class.java:190)
at 
org.apache.giraph.utils.ConfigurationUtils.handleComputationClass(ConfigurationUtils.java:482)
at 
org.apache.giraph.utils.ConfigurationUtils.populateGiraphConfiguration(ConfigurationUtils.java:465)
at 
org.apache.giraph.utils.ConfigurationUtils.parseArgs(ConfigurationUtils.java:207)
at org.apache.giraph.GiraphRunner.run(GiraphRunner.java:74)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.giraph.GiraphRunner.main(GiraphRunner.java:127)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)

 
Regards,
- kiru


Kiru Pakkirisamy | webcloudtech.wordpress.com



 From: Eli Reisman apache.mail...@gmail.com
To: user@giraph.apache.org; Kiru Pakkirisamy kirupakkiris...@yahoo.com 
Sent: Tuesday, May 27, 2014 2:27 PM
Subject: Re: issues while running giraph on Hadoop 2.2
 


Hi, see here (from your stack trace):


14/05/07 23:45:42 INFO yarn.GiraphYarnClient: Defaulting per-task heap size to 
1024MB.
Exception in thread main java.lang.IllegalStateException: Giraph's estimated 
cluster heap 2048MB ask is greater than the current available cluster heap of 
0MB. Aborting Job.

You'll need to adjust the min/max heap allocations allowed on your YARN cluster 
if possible. Alternately you can alter the defaults set up in the Giraph-Yarn 
client code and rebuild Giraph with those set differently.



On Thu, May 15, 2014 at 7:56 PM, Kiru Pakkirisamy kirupakkiris...@yahoo.com 
wrote:

I have the same issue running giraph built with -Phadoop_yarn 
-Dhadoop.version=2.2.0 on Hortonworks.
I get classnotfoundexceptions, looks like the GiraphRunner may not be able to 
load the classes correctly while running on Yarn.
So if anybody has had success please share your experience, wondering what I 
have been doing wrong.


Exception in thread main java.lang.ClassNotFoundException: 
org.apache.giraph.examples.SimpleShortestPathsComputation


 
Regards,
- kiru



 From: yeshwanth kumar yeshwant...@gmail.com
To: user@giraph.apache.org user@giraph.apache.org 
Sent: Thursday, May 8, 2014 4:47 AM
Subject: issues while running giraph on Hadoop 2.2
 


hi i am running Hortonworks hadoop 2.2
trying to run giraph shortest path example.
i am getting this exception






$ hadoop jar 
giraph-examples/target/giraph-examples-1.1.0-SNAPSHOT-for-hadoop-2.2.0-jar-with-dependencies.jar
 org.apache.giraph.GiraphRunner 
org.apache.giraph.examples.SimpleShortestPathsComputation -vif 
org.apache.giraph.io.formats.JsonLongDoubleFloatDoubleVertexInputFormat -vip 
/user/hdfs/input/tiny.txt -vof 
org.apache.giraph.io.formats.IdWithValueTextOutputFormat -op 
/user/hdfs/output/shortestpaths -w 1
14/05/07 23:45:42 INFO utils.ConfigurationUtils: No edge input format 
specified. Ensure your InputFormat does not require one.
14/05/07 23:45:42 INFO utils.ConfigurationUtils: No edge output format 
specified. Ensure your