If possible, can you copy the entire content of the directory /etc/hadoop/conf and then set HADOOP_CONF_DIR in slider-env.sh to it. Keep slider-client.xml empty.
Now when you do the same rm1->rm2 and then the reverse failovers, do you see the same behaviors? -Gour On 7/25/16, 2:28 PM, "Manoj Samel" <manojsamelt...@gmail.com> wrote: >Another observation (whatever it is worth) > >If slider app is created and started when rm2 was active, then it seems to >survive switches between rm2 and rm1 (and back). I.e > >* rm2 is active >* create and start slider application >* fail over to rm1. Now the Slider AM keeps running >* fail over to rm2 again. Slider AM still keeps running > >So, it seems if it starts with rm1 active, then the AM goes to "ACCEPTED" >state when RM fails to rm2. If it starts with rm2 active, then it runs >fine >with any switches between rm1 and rm2. > >Any feedback ? > >Thanks, > >Manoj > >On Mon, Jul 25, 2016 at 12:25 PM, Manoj Samel <manojsamelt...@gmail.com> >wrote: > >> Setup >> >> - Hadoop 2.6 with RM HA, Kerberos enabled >> - Slider 0.80 >> - In my slider-client.xml, I have added all RM HA properties, including >> the ones mentioned in http://markmail.org/message/wnhpp2zn6ixo65e3. >> >> Following is the issue >> >> * rm1 is active, rm2 is standby >> * deploy and start slider application, it runs fine >> * restart rm1, rm2 is now active. >> * The slider-am now goes from running into "ACCEPTED" mode. It stays >>there >> till rm1 is made active again. >> >> In the slider-am log, it tries to connect to RM2 and connection fails >>due >> to org.apache.hadoop.security.AccessControlException: Client cannot >> authenticate via:[TOKEN]. See detailed log below >> >> It seems it has some token (delegation token?) for RM1 but tries to use >> same(?) for RM2 and fails. Am I missing some configuration ??? >> >> Thanks, >> >> >> >> 2016-07-25 19:06:48,088 [AMRM Heartbeater thread] INFO >> client.ConfiguredRMFailoverProxyProvider - Failing over to rm2 >> 2016-07-25 19:06:48,088 [AMRM Heartbeater thread] WARN >> security.UserGroupInformation - PriviledgedActionException as:abc@XYZ >> (auth:KERBEROS) cause:org.apache.hadoop.security.AccessControlException: >> Client cannot authenticate via:[TOKEN] >> 2016-07-25 19:06:48,088 [AMRM Heartbeater thread] WARN ipc.Client - >> Exception encountered while connecting to the server : >> org.apache.hadoop.security.AccessControlException: Client cannot >> authenticate via:[TOKEN] >> 2016-07-25 19:06:48,088 [AMRM Heartbeater thread] WARN >> security.UserGroupInformation - PriviledgedActionException as:abc@XYZ >> (auth:KERBEROS) cause:java.io.IOException: >> org.apache.hadoop.security.AccessControlException: Client cannot >> authenticate via:[TOKEN] >> 2016-07-25 19:06:48,089 [AMRM Heartbeater thread] INFO >> retry.RetryInvocationHandler - Exception while invoking allocate of >>class >> ApplicationMasterProtocolPBClientImpl over rm2 after 287 fail over >> attempts. Trying to fail over immediately. >> java.io.IOException: Failed on local exception: java.io.IOException: >> org.apache.hadoop.security.AccessControlException: Client cannot >> authenticate via:[TOKEN]; Host Details : local host is: "<SliderAM >> HOST>/<slider AM Host IP>"; destination host is: "<RM2 HOST>":23130; >> at >>org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:772) >> at org.apache.hadoop.ipc.Client.call(Client.java:1476) >> at org.apache.hadoop.ipc.Client.call(Client.java:1403) >> at >> >>org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine. >>java:230) >> at com.sun.proxy.$Proxy23.allocate(Unknown Source) >> at >> >>org.apache.hadoop.yarn.api.impl.pb.client.ApplicationMasterProtocolPBClie >>ntImpl.allocate(ApplicationMasterProtocolPBClientImpl.java:77) >> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source) >> at >> >>sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorI >>mpl.java:43) >> at java.lang.reflect.Method.invoke(Method.java:497) >> at >> >>org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvoc >>ationHandler.java:252) >> at >> >>org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationH >>andler.java:104) >> at com.sun.proxy.$Proxy24.allocate(Unknown Source) >> at >> >>org.apache.hadoop.yarn.client.api.impl.AMRMClientImpl.allocate(AMRMClient >>Impl.java:278) >> at >> >>org.apache.hadoop.yarn.client.api.async.impl.AMRMClientAsyncImpl$Heartbea >>tThread.run(AMRMClientAsyncImpl.java:224) >> Caused by: java.io.IOException: >> org.apache.hadoop.security.AccessControlException: Client cannot >> authenticate via:[TOKEN] >> at >>org.apache.hadoop.ipc.Client$Connection$1.run(Client.java:682) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:422) >> at >> >>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation >>.java:1671) >> at >> >>org.apache.hadoop.ipc.Client$Connection.handleSaslConnectionFailure(Clien >>t.java:645) >> at >> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:733) >> at >> org.apache.hadoop.ipc.Client$Connection.access$2800(Client.java:370) >> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1525) >> at org.apache.hadoop.ipc.Client.call(Client.java:1442) >> ... 12 more >> Caused by: org.apache.hadoop.security.AccessControlException: Client >> cannot authenticate via:[TOKEN] >> at >> >>org.apache.hadoop.security.SaslRpcClient.selectSaslClient(SaslRpcClient.j >>ava:172) >> at >> >>org.apache.hadoop.security.SaslRpcClient.saslConnect(SaslRpcClient.java:3 >>96) >> at >> >>org.apache.hadoop.ipc.Client$Connection.setupSaslConnection(Client.java:5 >>55) >> at >> org.apache.hadoop.ipc.Client$Connection.access$1800(Client.java:370) >> at >>org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:725) >> at >>org.apache.hadoop.ipc.Client$Connection$2.run(Client.java:721) >> at java.security.AccessController.doPrivileged(Native Method) >> at javax.security.auth.Subject.doAs(Subject.java:422) >> at >> >>org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation >>.java:1671) >> at >> org.apache.hadoop.ipc.Client$Connection.setupIOstreams(Client.java:720) >> ... 15 more >> 2016-07-25 19:06:48,089 [AMRM Heartbeater thread] INFO >> client.ConfiguredRMFailoverProxyProvider - Failing over to rm1 >>
