Setup
- Hadoop 2.6 with RM HA, Kerberos enabled
- Slider 0.80
- In my slider-client.xml, I have added all RM HA properties, including the
ones mentioned in http://markmail.org/message/wnhpp2zn6ixo65e3.
Following is the issue
* rm1 is active, rm2 is standby
* deploy and start slider application, it runs fine
* restart rm1, rm2 is now active.
* The slider-am now goes from running into "ACCEPTED" mode. It stays there
till rm1 is made active again.
In the slider-am log, it tries to connect to RM2 and connection fails due
to org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[TOKEN]. See detailed log below
It seems it has some token (delegation token?) for RM1 but tries to use
same(?) for RM2 and fails. Am I missing some configuration ???
Thanks,
2016-07-25 19:06:48,088 [AMRM Heartbeater thread] INFO
client.ConfiguredRMFailoverProxyProvider - Failing over to rm2
2016-07-25 19:06:48,088 [AMRM Heartbeater thread] WARN
security.UserGroupInformation - PriviledgedActionException as:abc@XYZ
(auth:KERBEROS) cause:org.apache.hadoop.security.AccessControlException:
Client cannot authenticate via:[TOKEN]
2016-07-25 19:06:48,088 [AMRM Heartbeater thread] WARN ipc.Client -
Exception encountered while connecting to the server :
org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[TOKEN]
2016-07-25 19:06:48,088 [AMRM Heartbeater thread] WARN
security.UserGroupInformation - PriviledgedActionException
as:workdayadmin@BIGDATA (auth:KERBEROS) cause:java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[TOKEN]
2016-07-25 19:06:48,089 [AMRM Heartbeater thread] INFO
retry.RetryInvocationHandler - Exception while invoking allocate of class
ApplicationMasterProtocolPBClientImpl over rm2 after 287 fail over
attempts. Trying to fail over immediately.
java.io.IOException: Failed on local exception: java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[TOKEN]; Host Details : local host is: "<SliderAM
HOST>/<slider AM Host IP>"; destination host is: "<RM2 HOST>":23130;
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772)
at org.apache.hadoop.ipc.Client.call(Client.java:1476)
at org.apache.hadoop.ipc.Client.call(Client.java:1403)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:230)
at com.sun.proxy.$Proxy23.allocate(Unknown Source)
at
org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClientImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77)
at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:497)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:252)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:104)
at com.sun.proxy.$Proxy24.allocate(Unknown Source)
at
org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClientImpl.java:278)
at
org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$HeartbeatThread.run(AMRMClientAsyncImpl.java:224)
Caused by: java.io.IOException:
org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[TOKEN]
at org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:682)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at
org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Client.java:645)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:733)
at
org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:370)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1525)
at org.apache.hadoop.ipc.Client.call(Client.java:1442)
... 12 more
Caused by: org.apache.hadoop.security.AccessControlException: Client cannot
authenticate via:[TOKEN]
at
org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.java:172)
at
org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:396)
at
org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:555)
at
org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:370)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:725)
at org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:721)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
at
org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:720)
... 15 more
2016-07-25 19:06:48,089 [AMRM Heartbeater thread] INFO
client.ConfiguredRMFailoverProxyProvider - Failing over to rm1
