Thanks for taking the time to write back Laxman. I've proposed an
update to the troubleshooting section of the HBase manual, please see
https://issues.apache.org/jira/browse/HBASE-6077.
- Andy
On Tue, May 22, 2012 at 8:59 PM, Laxman <[email protected]> wrote:
> This issue is resolved after replacing the Java JCE jars on client side as
> well.
> I feel its worth documenting in HBase book.
>
> --
> Regards,
> Laxman
>> -----Original Message-----
>> From: Laxman [mailto:[email protected]]
>> Sent: Tuesday, May 22, 2012 3:21 PM
>> To: [email protected]
>> Subject: Secure HBase setup
>>
>> We got stuck with a problem while verifying client authentication in a
>> secure HBase cluster.
>> We are able to start a secure HBase cluster successfully.
>>
>> However, clients are not able to establish secure connection with HBase
>> server successfully.
>>
>> Other details:
>> HBase version: 0.94.0
>> Hadoop version: 0.23.1
>> Kerberos version: 1.10.1
>> Java version: 1.6.0_31, 64 bit
>> Linux version: SuSE 11.1 [Kernel version : 2.6.32.12-0.7-default x86_64
>> GNU/Linux]
>>
>> We had gone thru the solutions available @
>> http://docs.oracle.com/javase/1.5.0/docs/guide/security/jgss/tutorials/
>> Troubleshooting.html
>> https://ccp.cloudera.com/display/CDHDOC/Appendix+A+-
>> +Troubleshooting#AppendixA-Troubleshooting-
>> Problem2%3AJavaisunabletoreadtheKerberoscredentialscachecreatedbyversio
>> nsofMITKerberos1.8.1orhigher.
>>
>> But none of then seems to work. Any clue?
>>
>> There are no change in server logs as client is failing is failing even
>> before it communicates with server.
>> Exception we are hitting (Client side logs):
>>
>> 2012-05-22 09:42:22,627 WARN org.apache.hadoop.ipc.SecureClient:
>> Exception encountered while connecting to the server :
>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>> GSSException: No valid credentials provided (Mechanism level: Failed to
>> find any Kerberos tgt)]
>> 2012-05-22 09:42:22,627 ERROR
>> org.apache.hadoop.security.UserGroupInformation:
>> PriviledgedActionException as:testuser (auth:KERBEROS)
>> cause:java.io.IOException: javax.security.sasl.SaslException: GSS
>> initiate failed [Caused by GSSException: No valid credentials provided
>> (Mechanism level: Failed to find any Kerberos tgt)]
>> 2012-05-22 09:42:22,630 DEBUG org.apache.hadoop.ipc.SecureClient:
>> closing ipc connection to HOST-10-18-40-19/10.18.40.19:60020:
>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>> GSSException: No valid credentials provided (Mechanism level: Failed to
>> find any Kerberos tgt)]
>> java.io.IOException: javax.security.sasl.SaslException: GSS initiate
>> failed [Caused by GSSException: No valid credentials provided
>> (Mechanism level: Failed to find any Kerberos tgt)]
>> at
>> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$1.run(SecureC
>> lient.java:227)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
>> on.java:1177)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja
>> va:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
>> rImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
>> at org.apache.hadoop.hbase.security.User.call(User.java:586)
>> at org.apache.hadoop.hbase.security.User.access$700(User.java:50)
>> at
>> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:
>> 440)
>> at
>> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.handleSaslCon
>> nectionFailure(SecureClient.java:194)
>> at
>> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstream
>> s(SecureClient.java:274)
>> at
>> org.apache.hadoop.hbase.ipc.SecureClient.getConnection(SecureClient.jav
>> a:485)
>> at
>> org.apache.hadoop.hbase.ipc.SecureClient.getConnection(SecureClient.jav
>> a:69)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseClient.call(HBaseClient.java:897)
>> at
>> org.apache.hadoop.hbase.ipc.SecureRpcEngine$Invoker.invoke(SecureRpcEng
>> ine.java:164)
>> at $Proxy6.getProtocolVersion(Unknown Source)
>> at
>> org.apache.hadoop.hbase.ipc.SecureRpcEngine.getProxy(SecureRpcEngine.ja
>> va:208)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:303)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:280)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC.getProxy(HBaseRPC.java:332)
>> at
>> org.apache.hadoop.hbase.ipc.HBaseRPC.waitForProxy(HBaseRPC.java:236)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
>> tion.getHRegionConnection(HConnectionManager.java:1284)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
>> tion.getHRegionConnection(HConnectionManager.java:1240)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
>> tion.getHRegionConnection(HConnectionManager.java:1227)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
>> tion.locateRegionInMeta(HConnectionManager.java:936)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
>> tion.locateRegion(HConnectionManager.java:832)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
>> tion.locateRegion(HConnectionManager.java:801)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
>> tion.locateRegionInMeta(HConnectionManager.java:933)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
>> tion.locateRegion(HConnectionManager.java:836)
>> at
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
>> tion.locateRegion(HConnectionManager.java:801)
>> at
>> org.apache.hadoop.hbase.client.HTable.finishSetup(HTable.java:234)
>> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:174)
>> at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:133)
>> at hbase.test.Hbasetest.main(Hbasetest.java:37)
>> Caused by: javax.security.sasl.SaslException: GSS initiate failed
>> [Caused by GSSException: No valid credentials provided (Mechanism
>> level: Failed to find any Kerberos tgt)]
>> at
>> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Cl
>> ient.java:194)
>> at
>> org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSa
>> slRpcClient.java:138)
>> at
>> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupSaslConn
>> ection(SecureClient.java:176)
>> at
>> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.access$500(Se
>> cureClient.java:84)
>> at
>> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureC
>> lient.java:267)
>> at
>> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection$2.run(SecureC
>> lient.java:264)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at javax.security.auth.Subject.doAs(Subject.java:396)
>> at
>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformati
>> on.java:1177)
>> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> at
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.ja
>> va:39)
>> at
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccesso
>> rImpl.java:25)
>> at java.lang.reflect.Method.invoke(Method.java:597)
>> at org.apache.hadoop.hbase.util.Methods.call(Methods.java:37)
>> at org.apache.hadoop.hbase.security.User.call(User.java:586)
>> at org.apache.hadoop.hbase.security.User.access$700(User.java:50)
>> at
>> org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:
>> 440)
>> at
>> org.apache.hadoop.hbase.ipc.SecureClient$SecureConnection.setupIOstream
>> s(SecureClient.java:263)
>> ... 23 more
>> Caused by: GSSException: No valid credentials provided (Mechanism
>> level: Failed to find any Kerberos tgt)
>> at
>> sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredentia
>> l.java:130)
>> at
>> sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFac
>> tory.java:106)
>> at
>> sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFact
>> ory.java:172)
>> at
>> sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.jav
>> a:209)
>> at
>> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:195
>> )
>> at
>> sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:162
>> )
>> at
>> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Cl
>> ient.java:175)
>> ... 40 more
>> 2012-05-22 09:42:22,636 DEBUG org.apache.hadoop.ipc.SecureClient: IPC
>> Client (1778276127) connection to HOST-10-18-40-19/10.18.40.19:60020
>> from testuser: closed
>> 2012-05-22 09:42:22,638 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
>> tion: locateRegionInMeta parentTable=-ROOT-, metaLocation={region=-
>> ROOT-,,0.70236052, hostname=HOST-10-18-40-19, port=60020}, attempt=0 of
>> 120 failed; retrying after sleep of 1000 because:
>> javax.security.sasl.SaslException: GSS initiate failed [Caused by
>> GSSException: No valid credentials provided (Mechanism level: Failed to
>> find any Kerberos tgt)]
>> 2012-05-22 09:42:22,640 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
>> tion: Looked up root region location,
>> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectio
>> nImplementation@6ecf829d; serverName=HOST-10-18-40-
>> 19,60020,1337574445438
>> 2012-05-22 09:42:23,641 DEBUG
>> org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementa
>> tion: Looked up root region location,
>> connection=org.apache.hadoop.hbase.client.HConnectionManager$HConnectio
>> nImplementation@6ecf829d; serverName=HOST-10-18-40-
>> 19,60020,1337574445438
>> 2012-05-22 09:42:23,642 DEBUG org.apache.hadoop.ipc.SecureClient: RPC
>> Server Kerberos principal name for
>> protocol=org.apache.hadoop.hbase.ipc.HRegionInterface is
>> hbase/[email protected]
>>
>>
>> --
>> Regards,
>> Laxman
>
--
Best regards,
- Andy
Problems worthy of attack prove their worth by hitting back. - Piet
Hein (via Tom White)
