Hi all. I'm trying to run a graph hama process on top of YARN. I suppose that I have something wrong but I don't know what :(
I didn't found a lot of documentation about it, in fact I only have seen this link http://wiki.apache.org/hama/GettingStartedYARN So with this link and, the package of org.apache.hama.graph.GraphJob I built my job configuration: val conf = new HamaConfiguration() conf.addResource("/etc/hadoop/conf/yarn-site.xml") conf.set("yarn.resourcemanager.address", "0.0.0.0:8040") conf.set("yarn.ipc.rpc.class", "org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC") val job = new YARNBSPJob(conf) job.setBspClass(classOf[GraphJobRunner[LongWritable,LongWritable,LongWritable]]) //this.setBspClass(GraphJobRunner.class) job.setJarByClass(classOf[UserCompanyMain]) //this.setJarByClass(exampleClass) conf.setClass(GraphJob.VERTEX_CLASS_ATTR, classOf[UserCompanyVertex], classOf[Writable]) conf.setClass(GraphJob.VERTEX_ID_CLASS_ATTR, classOf[LongWritable], classOf[Writable]) //this.setVertexIDClass(Text.class) conf.setClass(GraphJob.VERTEX_VALUE_CLASS_ATTR, classOf[LongWritable], classOf[Writable]) //this.setVertexValueClass(IntWritable.class); conf.setClass(GraphJob.VERTEX_EDGE_VALUE_CLASS_ATTR, classOf[LongWritable], classOf[Writable]) //this.setEdgeValueClass(IntWritable.class) job.setJobName("social_company_network") job.setMemoryUsedPerTaskInMb(50) job.setNumBspTask(2) // rest of configuration (input, reader etc…) job.waitForCompletion(false) (Note: it is scala code for internal reason but I have not problems with it in other map/reduce jobs on top of YARN) The code that I run is -bash-4.1$ yarn jar /tmp/hdp-companies_2.10-0.1.0_SNAPSHOT.jar com.hdp.ds.hama.YarnGraph 13/05/15 05:40:27 INFO bsp.YARNBSPJob: Connecting to ResourceManager at / 0.0.0.0:8040 13/05/15 05:40:28 INFO bsp.YARNBSPJob: Starting job... 13/05/15 05:40:30 WARN conf.Configuration: user.name is deprecated. Instead, use mapreduce.job.user.name 13/05/15 05:40:30 INFO bsp.YARNBSPJobClient: Submitting job... Exception in thread "main" java.lang.reflect.UndeclaredThrowableException at org.apache.hadoop.yarn.exceptions.impl.pb.YarnRemoteExceptionPBImpl.unwrapAndThrowException(YarnRemoteExceptionPBImpl.java:128) at org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getNewApplication(ClientRMProtocolPBClientImpl.java:162) at org.apache.hama.bsp.YARNBSPJobClient.launchJob(YARNBSPJobClient.java:82) at org.apache.hama.bsp.BSPJobClient.submitJobInternal(BSPJobClient.java:369) at org.apache.hama.bsp.YARNBSPJob.submit(YARNBSPJob.java:80) at org.apache.hama.bsp.YARNBSPJob.waitForCompletion(YARNBSPJob.java:95) at com.hdp.ds.hama.YarnGraph$.main(YarnSerializePrinting.scala:50) at com.hdp.ds.hama.YarnGraph.main(YarnSerializePrinting.scala) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.util.RunJar.main(RunJar.java:212) Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): Unknown protocol: org.apache.hadoop.yarn.api.ClientRMProtocolPB at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.getProtocolImpl(ProtobufRpcEngine.java:398) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:436) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1014) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1735) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1731) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1441) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1729) at org.apache.hadoop.ipc.Client.call(Client.java:1235) at org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:202) at $Proxy7.getNewApplication(Unknown Source) at org.apache.hadoop.yarn.api.impl.pb.client.ClientRMProtocolPBClientImpl.getNewApplication(ClientRMProtocolPBClientImpl.java:159) ... 11 more I know that the problem is the protocol, and it is because I added the configuration in the HamaConfiguration, but it is not the solution conf.set("yarn.ipc.rpc.class", "org.apache.hadoop.yarn.ipc.HadoopYarnProtoRPC") These are the version that I have installed: Hadoop -> 2.0.3.22-alpha hama -> 0.6.1 Thanks for your help rgc
