Hi, YARN module is still a alpha version, you can't run a graph job on top of Hadoop 2.0. Please use Hadoop 1.0 and Hama 0.6.1 at this time. We'll try to support Hadoop 2.0 soon.
On Wed, May 15, 2013 at 8:13 PM, Rubén González <[email protected]> wrote: > Hi all. > > I'm trying to run a graph hama process on top of YARN. I suppose that I > have something wrong but I don't know what :( > > I didn't found a lot of documentation about it, in fact I only have seen > this link > > http://wiki.apache.org/hama/GettingStartedYARN > > So with this link and, the package of org.apache.hama.graph.GraphJob I > built my job configuration: > > val conf = new HamaConfiguration() > conf.addResource("/etc/hadoop/conf/yarn-site.xml") > conf.set("yarn.resourcemanager.address", "0.0.0.0:8040") > conf.set("yarn.ipc.rpc.class", > "org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC") > > val job = new YARNBSPJob(conf) > > > job.setBspClass(classOf[GraphJobRunner[LongWritable,LongWritable,LongWritable]]) > //this.setBspClass(GraphJobRunner.class) > job.setJarByClass(classOf[UserCompanyMain]) > //this.setJarByClass(exampleClass) > > conf.setClass(GraphJob.VERTEX_CLASS_ATTR, classOf[UserCompanyVertex], > classOf[Writable]) > conf.setClass(GraphJob.VERTEX_ID_CLASS_ATTR, classOf[LongWritable], > classOf[Writable]) //this.setVertexIDClass(Text.class) > conf.setClass(GraphJob.VERTEX_VALUE_CLASS_ATTR, classOf[LongWritable], > classOf[Writable]) //this.setVertexValueClass(IntWritable.class); > conf.setClass(GraphJob.VERTEX_EDGE_VALUE_CLASS_ATTR, > classOf[LongWritable], classOf[Writable]) > //this.setEdgeValueClass(IntWritable.class) > > > job.setJobName("social_company_network") > job.setMemoryUsedPerTaskInMb(50) > job.setNumBspTask(2) > > // rest of configuration (input, reader etc…) > > job.waitForCompletion(false) > > > > (Note: it is scala code for internal reason but I have not problems with it > in other map/reduce jobs on top of YARN) > > > The code that I run is > > -bash-4.1$ yarn jar /tmp/hdp-companies_2.10-0.1.0_SNAPSHOT.jar > com.hdp.ds.hama.YarnGraph > 13/05/15 05:40:27 INFO bsp.YARNBSPJob: Connecting to ResourceManager at / > 0.0.0.0:8040 > 13/05/15 05:40:28 INFO bsp.YARNBSPJob: Starting job... > 13/05/15 05:40:30 WARN conf.Configuration: user.name is deprecated. > Instead, use mapreduce.job.user.name > 13/05/15 05:40:30 INFO bsp.YARNBSPJobClient: Submitting job... > Exception in thread "main" java.lang.reflect.UndeclaredThrowableException > at > org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:128) > at > org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getNewApplication(ClientRMProtocolPBClientImpl.java:162) > at org.apache.hama.bsp.YARNBSPJobClient.launchJob(YARNBSPJobClient.java:82) > at org.apache.hama.bsp.BSPJobClient.submitJobInternal(BSPJobClient.java:369) > at org.apache.hama.bsp.YARNBSPJob.submit(YARNBSPJob.java:80) > at org.apache.hama.bsp.YARNBSPJob.waitForCompletion(YARNBSPJob.java:95) > at com.hdp.ds.hama.YarnGraph$.main(YarnSerializePrinting.scala:50) > at com.hdp.ds.hama.YarnGraph.main(YarnSerializePrinting.scala) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.util.RunJar.main(RunJar.java:212) > Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): > Unknown protocol: org.apache.hadoop.yarn.api.ClientRMProtocolPB > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.getProtocolImpl(ProtobufRpcEngine.java:398) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:436) > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1735) > at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1731) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1441) > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1729) > > at org.apache.hadoop.ipc.Client.call(Client.java:1235) > at > org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202) > at $Proxy7.getNewApplication(Unknown Source) > at > org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getNewApplication(ClientRMProtocolPBClientImpl.java:159) > ... 11 more > > > I know that the problem is the protocol, and it is because I added the > configuration in the HamaConfiguration, but it is not the solution > conf.set("yarn.ipc.rpc.class", > "org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC") > > > These are the version that I have installed: > > Hadoop -> 2.0.3.22-alpha > hama -> 0.6.1 > > > Thanks for your help > rgc -- Best Regards, Edward J. Yoon @eddieyoon
