This issue is resolved after replacing the Java JCE jars on client side as well. I feel its worth documenting in HBase book.
-- Regards, Laxman > -----Original Message----- > From: Laxman [mailto:[email protected]] > Sent: Tuesday, May 22, 2012 3:21 PM > To: [email protected] > Subject: Secure HBase setup > > We got stuck with a problem while verifying client authentication in a > secure HBase cluster. > We are able to start a secure HBase cluster successfully. > > However, clients are not able to establish secure connection with HBase > server successfully. > > Other details: > HBase version: 0.94.0 > Hadoop version: 0.23.1 > Kerberos version: 1.10.1 > Java version: 1.6.0_31, 64 bit > Linux version: SuSE 11.1 [Kernel version : 2.6.32.12-0.7-default x86_64 > GNU/Linux] > > We had gone thru the solutions available @ > http://docs.oracle.com/javase/1.5.0/docs/guide/security/jgss/tutorials/ > Troubleshooting.html > https://ccp.cloudera.com/display/CDHDOC/Appendix+A+- > +Troubleshooting#AppendixA-Troubleshooting- > Problem2%3AJavaisunabletoreadtheKerberoscredentialscachecreatedbyversio > nsofMITKerberos1.8.1orhigher. > > But none of then seems to work. Any clue? > > There are no change in server logs as client is failing is failing even > before it communicates with server. > Exception we are hitting (Client side logs): > > 2012-05-22 09:42:22,627 WARN org.apache.hadoop.ipc.SecureClient: > Exception encountered while connecting to the server : > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to > find any Kerberos tgt)] > 2012-05-22 09:42:22,627 ERROR > org.apache.hadoop.security.UserGroupInformation: > PriviledgedActionException as:testuser (auth:KERBEROS) > cause:java.io.IOException: javax.security.sasl.SaslException: GSS > initiate failed [Caused by GSSException: No valid credentials provided > (Mechanism level: Failed to find any Kerberos tgt)] > 2012-05-22 09:42:22,630 DEBUG org.apache.hadoop.ipc.SecureClient: > closing ipc connection to HOST-10-18-40-19/10.18.40.19:60020: > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to > find any Kerberos tgt)] > java.io.IOException: javax.security.sasl.SaslException: GSS initiate > failed [Caused by GSSException: No valid credentials provided > (Mechanism level: Failed to find any Kerberos tgt)] > at > org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureC > lient.java:227) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati > on.java:1177) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja > va:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso > rImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37) > at org.apache.hadoop.hbase.security.User.call(User.java:586) > at org.apache.hadoop.hbase.security.User.access$700(User.java:50) > at > org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java: > 440) > at > org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.handleSaslCon > nectionFailure(SecureClient.java:194) > at > org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstream > s(SecureClient.java:274) > at > org.apache.hadoop.hbase.ipc.SecureClient.getConnection(SecureClient.jav > a:485) > at > org.apache.hadoop.hbase.ipc.SecureClient.getConnection(SecureClient.jav > a:69) > at > org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:897) > at > org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEng > ine.java:164) > at $Proxy6.getProtocolVersion(Unknown Source) > at > org.apache.hadoop.hbase.ipc.SecureRpcEngine.getProxy(SecureRpcEngine.ja > va:208) > at > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:303) > at > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:280) > at > org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:332) > at > org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:236) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa > tion.getHRegionConnection(HConnectionManager.java:1284) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa > tion.getHRegionConnection(HConnectionManager.java:1240) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa > tion.getHRegionConnection(HConnectionManager.java:1227) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa > tion.locateRegionInMeta(HConnectionManager.java:936) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa > tion.locateRegion(HConnectionManager.java:832) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa > tion.locateRegion(HConnectionManager.java:801) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa > tion.locateRegionInMeta(HConnectionManager.java:933) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa > tion.locateRegion(HConnectionManager.java:836) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa > tion.locateRegion(HConnectionManager.java:801) > at > org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234) > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174) > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133) > at hbase.test.Hbasetest.main(Hbasetest.java:37) > Caused by: javax.security.sasl.SaslException: GSS initiate failed > [Caused by GSSException: No valid credentials provided (Mechanism > level: Failed to find any Kerberos tgt)] > at > com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Cl > ient.java:194) > at > org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSa > slRpcClient.java:138) > at > org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConn > ection(SecureClient.java:176) > at > org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$500(Se > cureClient.java:84) > at > org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureC > lient.java:267) > at > org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureC > lient.java:264) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati > on.java:1177) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja > va:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso > rImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37) > at org.apache.hadoop.hbase.security.User.call(User.java:586) > at org.apache.hadoop.hbase.security.User.access$700(User.java:50) > at > org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java: > 440) > at > org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstream > s(SecureClient.java:263) > ... 23 more > Caused by: GSSException: No valid credentials provided (Mechanism > level: Failed to find any Kerberos tgt) > at > sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredentia > l.java:130) > at > sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFac > tory.java:106) > at > sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFact > ory.java:172) > at > sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.jav > a:209) > at > sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195 > ) > at > sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162 > ) > at > com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Cl > ient.java:175) > ... 40 more > 2012-05-22 09:42:22,636 DEBUG org.apache.hadoop.ipc.SecureClient: IPC > Client (1778276127) connection to HOST-10-18-40-19/10.18.40.19:60020 > from testuser: closed > 2012-05-22 09:42:22,638 DEBUG > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa > tion: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=- > ROOT-,,0.70236052, hostname=HOST-10-18-40-19, port=60020}, attempt=0 of > 120 failed; retrying after sleep of 1000 because: > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to > find any Kerberos tgt)] > 2012-05-22 09:42:22,640 DEBUG > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa > tion: Looked up root region location, > connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectio > nImplementation@6ecf829d; serverName=HOST-10-18-40- > 19,60020,1337574445438 > 2012-05-22 09:42:23,641 DEBUG > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa > tion: Looked up root region location, > connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectio > nImplementation@6ecf829d; serverName=HOST-10-18-40- > 19,60020,1337574445438 > 2012-05-22 09:42:23,642 DEBUG org.apache.hadoop.ipc.SecureClient: RPC > Server Kerberos principal name for > protocol=org.apache.hadoop.hbase.ipc.HRegionInterface is > hbase/[email protected] > > > -- > Regards, > Laxman
