Dear All,
we are trying to run Giraph 1.1.0 RC against a distribution of Hadoop
2.2.0 that uses Jetty 7 in place of Jetty 6. When we launch our
application on our cluster we get the following error:
2014-07-02 08:14:23,307 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for
class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
2014-07-02 08:14:23,551 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
hadoop-metrics2.properties
2014-07-02 08:14:23,619 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot
period at 10 second(s).
2014-07-02 08:14:23,619 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics
system started
2014-07-02 08:14:23,630 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token
for job_1403542528912_0632 to jobTokenSecretManager
2014-07-02 08:14:23,726 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing
job_1403542528912_0632 because: not enabled; too much RAM;
2014-07-02 08:14:23,742 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job
job_1403542528912_0632 = 0. Number of splits = 1
2014-07-02 08:14:23,742 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces
for job job_1403542528912_0632 = 0
2014-07-02 08:14:23,742 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl:
job_1403542528912_0632Job Transitioned from NEW to INITED
2014-07-02 08:14:23,744 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching
normal, non-uberized, multi-container job job_1403542528912_0632.
2014-07-02 08:14:23,782 INFO [Socket Reader #1 for port 7041]
org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 7041
2014-07-02 08:14:23,804 INFO [main]
org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
2014-07-02 08:14:23,805 INFO [IPC Server Responder]
org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2014-07-02 08:14:23,805 INFO [IPC Server listener on 7041]
org.apache.hadoop.ipc.Server: IPC Server listener on 7041: starting
2014-07-02 08:14:23,805 INFO [main]
org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated
MRClientService at node0654/10.1.33.14:7041
2014-07-02 08:14:23,858 FATAL [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.NoSuchMethodError:
org.apache.hadoop.http.HttpServer.<init>(Ljava/lang/String;Ljava/lang/String;IZLorg/apache/hadoop/conf/Configuration;Lorg/apache/hadoop/security/authorize/AccessControlList;Lorg/mortbay/jetty/Connector;[Ljava/lang/String;)V
at
org.apache.hadoop.yarn.webapp.WebApps$Builder$2.<init>(WebApps.java:222)
at
org.apache.hadoop.yarn.webapp.WebApps$Builder.start(WebApps.java:219)
at
org.apache.hadoop.mapreduce.v2.app.client.MRClientService.serviceStart(MRClientService.java:136)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1058)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1445)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1491)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1441)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1374)
The error is caused by the fact that in our distribution of Hadoop the
HTTPServer constructor is expecting a parameter of type
org.eclipse.jetty.Connector, but is receiving a parameter of type
org.mortbay.jetty.Connector. Indeed, one big difference of Jetty 7 wrt
Jetty 6 is the different naming of packages.
Similar errors arise when the application is launched in pure yarn mode
(containers just fail).
We examined the code of Giraph 1.1.0, but we actually did not find any
place where a Collector object is created and/or Jetty is really used.
Do you have any idea on how to solve this issue? Please, observe that
we are forced to use this Hadoop distribution and cannot switch to a
plain Hadoop 2.2.0 distribution with Jetty 6.
Thanks in advance.
Carlo
--
"Humanity is overrated", Gregory House, MD
