I have been searching for a handle on a problem without very little clues. Any
help pointing me to the right direction will be huge.
I have not received any input form the Cloudera google groups. Perhaps this is
more Yarn based and I am hoping I have more luck here.
Any help is greatly appreciated.
I am running a Hadoop cluster using CDH5.3. I also have a client machine with a
standalone one node setup (VM).
All environments are running CentOS 6.6.
I have submitted some Java mapreduce jobs locally on both the cluster and the
standalone environment with successfully completions.
I can submit a remote HDFS job from client to cluster using -conf
hadoop-cluster.xml (see below) and get data back from the cluster with no
problem.
When submitted remotely the mapreduce jobs remotely, I get an AM error:
AM fails the job with the error:
SecretManager$InvalidToken: appattempt_1424003606313_0001_000002 not
found in AMRMTokenSecretManager
I searched /var/log/secure on the client and cluster with no unusual messages.
Here is the contents of hadoop-cluster.xml:
<?xml version="1.0" encoding="UTF-8"?>
<!--generated by Roland-->
<configuration>
<property>
<name>fs.defaultFS</name>
<value>hdfs://mycluser:8020</value>
</property>
<property>
<name>mapreduce.jobtracker.address</name>
<value>hdfs://mycluster:8032</value>
</property>
<property>
<name>yarn.resourcemanager.address</name>
<value>hdfs://mycluster:8032</value>
</property>
Here is the output from the job log on the cluster:
2015-02-15 07:51:06,544 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for
application appattempt_1424003606313_0001_000002
2015-02-15 07:51:06,949 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.require.client.cert;
Ignoring.
2015-02-15 07:51:06,952 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-15 07:51:06,952 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.client.conf;
Ignoring.
2015-02-15 07:51:06,954 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
hadoop.ssl.keystores.factory.class; Ignoring.
2015-02-15 07:51:06,957 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.server.conf;
Ignoring.
2015-02-15 07:51:06,973 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-15 07:51:07,241 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2015-02-15 07:51:07,241 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN,
Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@33be1aa0)
2015-02-15 07:51:07,332 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Using mapred newApiCommitter.
2015-02-15 07:51:07,627 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.require.client.cert;
Ignoring.
2015-02-15 07:51:07,632 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-15 07:51:07,632 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.client.conf;
Ignoring.
2015-02-15 07:51:07,639 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
hadoop.ssl.keystores.factory.class; Ignoring.
2015-02-15 07:51:07,645 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.server.conf;
Ignoring.
2015-02-15 07:51:07,663 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-15 07:51:08,237 WARN [main] org.apache.hadoop.util.NativeCodeLoader:
Unable to load native-hadoop library for your platform... using builtin-java
classes where applicable
2015-02-15 07:51:08,429 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config
null
2015-02-15 07:51:08,499 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter is
org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
2015-02-15 07:51:08,526 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.jobhistory.EventType for class
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
2015-02-15 07:51:08,527 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
2015-02-15 07:51:08,561 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
2015-02-15 07:51:08,562 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskAttemptEventDispatcher
2015-02-15 07:51:08,566 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType for class
org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
2015-02-15 07:51:08,568 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$SpeculatorEventDispatcher
2015-02-15 07:51:08,568 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter
2015-02-15 07:51:08,570 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType for
class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerLauncherRouter
2015-02-15 07:51:08,599 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Recovery is enabled. Will try
to recover from previous life on best effort basis.
2015-02-15 07:51:08,642 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Previous history file is at
hdfs://mycluster.com:8020/user/cloudera/.staging/job_1424003606313_0001/job_1424003606313_0001_1.jhist
<http://mycluster.com:8020/user/cloudera/.staging/job_1424003606313_0001/job_1424003606313_0001_1.jhist2015-02-15>
2015-02-15
<http://mycluster.com:8020/user/cloudera/.staging/job_1424003606313_0001/job_1424003606313_0001_1.jhist2015-02-15>
07:51:09,147 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Read
completed tasks from history 0
2015-02-15 07:51:09,193 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher: Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type for class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobFinishEventHandler
2015-02-15 07:51:09,222 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsConfig: loaded properties from
hadoop-metrics2.properties
2015-02-15 07:51:09,277 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period at
10 second(s).
2015-02-15 07:51:09,277 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MRAppMaster metrics system
started
2015-02-15 07:51:09,286 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Adding job token for
job_1424003606313_0001 to jobTokenSecretManager
2015-02-15 07:51:09,306 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Not uberizing
job_1424003606313_0001 because: not enabled; too much RAM;
2015-02-15 07:51:09,324 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Input size for job
job_1424003606313_0001 = 5343207. Number of splits = 5
2015-02-15 07:51:09,325 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: Number of reduces for job
job_1424003606313_0001 = 1
2015-02-15 07:51:09,325 INFO [main]
org.apache.hadoop.mapreduce.v2.app.job.impl.JobImpl: job_1424003606313_0001Job
Transitioned from NEW to INITED
2015-02-15 07:51:09,327 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster: MRAppMaster launching normal,
non-uberized, multi-container job job_1424003606313_0001.
2015-02-15 07:51:09,387 INFO [main] org.apache.hadoop.ipc.CallQueueManager:
Using callQueue class java.util.concurrent.LinkedBlockingQueue
2015-02-15 07:51:09,398 INFO [Socket Reader #1 for port 56348]
org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 56348
2015-02-15 07:51:09,418 INFO [main]
org.apache.hadoop.yarn.factories.impl.pb.RpcServerFactoryPBImpl: Adding
protocol org.apache.hadoop.mapreduce.v2.api.MRClientProtocolPB to the server
2015-02-15 07:51:09,418 INFO [IPC Server Responder]
org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2015-02-15 07:51:09,419 INFO [main]
org.apache.hadoop.mapreduce.v2.app.client.MRClientService: Instantiated
MRClientService at mycluster/mycluster:56348
2015-02-15 07:51:09,425 INFO [IPC Server listener on 56348]
org.apache.hadoop.ipc.Server: IPC Server listener on 56348: starting
2015-02-15 07:51:09,492 INFO [main] org.mortbay.log: Logging to
org.slf4j.impl.Log4jLoggerAdapter(org.mortbay.log) via org.mortbay.log.Slf4jLog
2015-02-15 07:51:09,497 INFO [main] org.apache.hadoop.http.HttpRequestLog: Http
request log for http.requests.mapreduce is not defined
2015-02-15 07:51:09,509 INFO [main] org.apache.hadoop.http.HttpServer2: Added
global filter 'safety'
(class=org.apache.hadoop.http.HttpServer2$QuotingInputFilter)
2015-02-15 07:51:09,514 INFO [main] org.apache.hadoop.http.HttpServer2: Added
filter AM_PROXY_FILTER
(class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context
mapreduce
2015-02-15 07:51:09,514 INFO [main] org.apache.hadoop.http.HttpServer2: Added
filter AM_PROXY_FILTER
(class=org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter) to context
static
2015-02-15 07:51:09,518 INFO [main] org.apache.hadoop.http.HttpServer2: adding
path spec: /mapreduce/*
2015-02-15 07:51:09,518 INFO [main] org.apache.hadoop.http.HttpServer2: adding
path spec: /ws/*
2015-02-15 07:51:09,529 INFO [main] org.apache.hadoop.http.HttpServer2: Jetty
bound to port 34473
2015-02-15 07:51:09,529 INFO [main] org.mortbay.log: jetty-6.1.26.cloudera.4
2015-02-15 07:51:09,561 INFO [main] org.mortbay.log: Extract
jar:file:/opt/cloudera/parcels/CDH-5.3.0-1.cdh5.3.0.p0.30/jars/hadoop-yarn-common-2.5.0-cdh5.3.0.jar!/webapps/mapreduce
to /tmp/Jetty_0_0_0_0_34473_mapreduce____.ezh3w6/webapp
2015-02-15 07:51:09,932 INFO [main] org.mortbay.log: Started
HttpServer2$SelectChannelConnectorWithSafeStartup@0.0.0.0:34473
2015-02-15 07:51:09,932 INFO [main] org.apache.hadoop.yarn.webapp.WebApps: Web
app /mapreduce started at 34473
2015-02-15 07:51:10,425 INFO [main] org.apache.hadoop.yarn.webapp.WebApps:
Registered webapp guice modules
2015-02-15 07:51:10,430 INFO [main] org.apache.hadoop.ipc.CallQueueManager:
Using callQueue class java.util.concurrent.LinkedBlockingQueue
2015-02-15 07:51:10,431 INFO [Socket Reader #1 for port 41190]
org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port 41190
2015-02-15 07:51:10,438 INFO [IPC Server Responder]
org.apache.hadoop.ipc.Server: IPC Server Responder: starting
2015-02-15 07:51:10,438 INFO [IPC Server listener on 41190]
org.apache.hadoop.ipc.Server: IPC Server listener on 41190: starting
2015-02-15 07:51:10,459 INFO [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
nodeBlacklistingEnabled:true
2015-02-15 07:51:10,459 INFO [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
maxTaskFailuresPerNode is 3
2015-02-15 07:51:10,459 INFO [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerRequestor:
blacklistDisablePercent is 33
2015-02-15 07:51:10,576 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.require.client.cert;
Ignoring.
2015-02-15 07:51:10,577 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.retry.interval; Ignoring.
2015-02-15 07:51:10,577 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.client.conf;
Ignoring.
2015-02-15 07:51:10,577 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
hadoop.ssl.keystores.factory.class; Ignoring.
2015-02-15 07:51:10,578 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter: hadoop.ssl.server.conf;
Ignoring.
2015-02-15 07:51:10,592 WARN [main] org.apache.hadoop.conf.Configuration:
job.xml:an attempt to override final parameter:
mapreduce.job.end-notification.max.attempts; Ignoring.
2015-02-15 07:51:10,602 INFO [main] org.apache.hadoop.yarn.client.RMProxy:
Connecting to ResourceManager at quickstart.cloudera/myclient:8030
2015-02-15 07:51:10,749 WARN [main]
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:cloudera (auth:SIMPLE)
cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
appattempt_1424003606313_0001_000002 not found in AMRMTokenSecretManager.
2015-02-15 07:51:10,750 WARN [main] org.apache.hadoop.ipc.Client: Exception
encountered while connecting to the server :
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
appattempt_1424003606313_0001_000002 not found in AMRMTokenSecretManager.
2015-02-15 07:51:10,750 WARN [main]
org.apache.hadoop.security.UserGroupInformation: PriviledgedActionException
as:cloudera (auth:SIMPLE)
cause:org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
appattempt_1424003606313_0001_000002 not found in AMRMTokenSecretManager.
2015-02-15 07:51:10,762 ERROR [main]
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator: Exception while
registering
org.apache.hadoop.security.token.SecretManager$InvalidToken:
appattempt_1424003606313_0001_000002 not found in AMRMTokenSecretManager.
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.yarn.ipc.RPCUtil.instantiateException(RPCUtil.java:53)
at
org.apache.hadoop.yarn.ipc.RPCUtil.unwrapAndThrowException(RPCUtil.java:104)
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:109)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:187)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
at com.sun.proxy.$Proxy36.registerApplicationMaster(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:161)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.serviceStart(RMCommunicator.java:122)
at
org.apache.hadoop.mapreduce.v2.app.rm.RMContainerAllocator.serviceStart(RMContainerAllocator.java:238)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$ContainerAllocatorRouter.serviceStart(MRAppMaster.java:807)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.service.CompositeService.serviceStart(CompositeService.java:120)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1075)
at
org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.run(MRAppMaster.java:1478)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1642)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1474)
at
org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1407)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
appattempt_1424003606313_0001_000002 not found in AMRMTokenSecretManager.
at org.apache.hadoop.ipc.Client.call(Client.java:1411)
at org.apache.hadoop.ipc.Client.call(Client.java:1364)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy35.registerApplicationMaster(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.registerApplicationMaster(ApplicationMasterProtocolPBClientImpl.java:106)
... 22 more
2015-02-15 07:51:10,765 INFO [main] org.apache.hadoop.service.AbstractService:
Service RMCommunicator failed in state STARTED; cause:
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
org.apache.hadoop.security.token.SecretManager$InvalidToken:
appattempt_1424003606313_0001_000002 not found in AMRMTokenSecretManager.
org.apache.hadoop.yarn.exceptions.YarnRuntimeException:
org.apache.hadoop.security.token.SecretManager$InvalidToken:
appattempt_1424003606313_0001_000002 not found in AMRMTokenSecretManager.
at
org.apache.hadoop.mapreduce.v2.app.rm.RMCommunicator.register(RMCommunicator.java:178)
at org.apac
Any help is greatly appreciated.
