[ https://issues.apache.org/jira/browse/SENTRY-1749?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15997493#comment-15997493 ]
Vamsee Yarlagadda commented on SENTRY-1749: ------------------------------------------- Some analysis on this problem: Looking at the code of Sentry and HiveMetaStoreClient, it looks like this problem has been there for a long time. But it's mostly masked by the restarts we are doing after the clean deployment. So during the course of initial service start, sentry and other services don't have any keytabs active under local unix user and during this time, the log of sentry is cluttered with the above errors. But after we do a restart after deployment completes, the Sentry code picks up the principal that was activated under "sentry" user (which users do it for testing purposes) and thus works properly. *Ideally* the service processes shouldn't depend on the keytabs active on running unix user but rather use the keytabs supplied by startup system in the process directory. Looking at the code of [HMSFollower|https://github.com/apache/sentry/blob/sentry-ha-redesign/sentry-provider/sentry-provider-db/src/main/java/org/apache/sentry/service/thrift/HMSFollower.java] we invoke the classes of Hive - HiveMetaStoreClient which exclusively uses the UserGroupInformation object. And for UserGroupInformation object to pick up the keytab, one should explicitly call the methods like [loginUserFromKeytabAndReturnUGI|https://github.com/apache/hadoop/blob/61858a5c378da75aff9cde84d418af46d718d08b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L1406] or use [getUGIFromSubject|https://github.com/apache/hadoop/blob/61858a5c378da75aff9cde84d418af46d718d08b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L841]. For the fact that HMSFollower already logs into the keytab, we can leverage the method getUGIFromSubject() to make the UserGroupInformation aware of keytab authentication. If UGI object is not made aware of keytab, then the invocation of [UserGroupInformation#getCurrentUser|https://github.com/apache/hadoop/blob/61858a5c378da75aff9cde84d418af46d718d08b/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java#L736-L742] falls back to use Unix user. To avoid this, we need to add extra logic to HMSFollower to make UserGroupInformation aware of the keytab that way it uses it for communication with HMS. > Sentry fails to setup secure connection to HMS if the local running unix user > is missing active tgt > --------------------------------------------------------------------------------------------------- > > Key: SENTRY-1749 > URL: https://issues.apache.org/jira/browse/SENTRY-1749 > Project: Sentry > Issue Type: Bug > Components: Sentry > Affects Versions: sentry-ha-redesign > Reporter: Vamsee Yarlagadda > Assignee: Vamsee Yarlagadda > Priority: Critical > > Sentry fails to establish secure connection to HMS if the local unix user is > missing an active ticket. > {code} > 2017-05-03 18:04:52,279 INFO org.apache.sentry.service.thrift.HMSFollower: > Making a kerberos connection to HMS > 2017-05-03 18:04:52,279 INFO org.apache.sentry.service.thrift.HMSFollower: > Using kerberos principal: sentry/ve1113.halxg.cloudera....@halxg.cloudera.com > 2017-05-03 18:04:52,279 INFO > org.apache.sentry.service.thrift.SentryKerberosContext: Logging in with new > Context > 2017-05-03 18:04:52,283 INFO > org.apache.sentry.service.thrift.SentryKerberosContext: Sentry Ticket renewer > thread started > 2017-05-03 18:04:52,285 INFO hive.metastore: Trying to connect to metastore > with URI thrift://ve1113.halxg.cloudera.com:9083 > 2017-05-03 18:04:52,286 ERROR org.apache.thrift.transport.TSaslTransport: > SASL negotiation failure > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt)] > at > com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) > at > org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) > at > org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) > at > org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:464) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:244) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:187) > at > org.apache.sentry.service.thrift.HMSFollower$1.run(HMSFollower.java:167) > at > org.apache.sentry.service.thrift.HMSFollower$1.run(HMSFollower.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.sentry.service.thrift.HMSFollower.getMetaStoreClient(HMSFollower.java:164) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:204) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt) > at > sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) > at > sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) > at > sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) > at > sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) > at > sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) > at > sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) > at > com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193) > ... 25 more > 2017-05-03 18:04:52,286 WARN hive.metastore: Failed to connect to the > MetaStore Server... > 2017-05-03 18:04:52,286 INFO hive.metastore: Waiting 1 seconds before next > connection attempt. > 2017-05-03 18:04:53,220 INFO > org.apache.sentry.service.thrift.SentryKerberosContext: Sentry Ticket renewer > thread finished > 2017-05-03 18:04:53,286 INFO hive.metastore: Trying to connect to metastore > with URI thrift://ve1113.halxg.cloudera.com:9083 > 2017-05-03 18:04:53,289 ERROR org.apache.thrift.transport.TSaslTransport: > SASL negotiation failure > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt)] > at > com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) > at > org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) > at > org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) > at > org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:464) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:244) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:187) > at > org.apache.sentry.service.thrift.HMSFollower$1.run(HMSFollower.java:167) > at > org.apache.sentry.service.thrift.HMSFollower$1.run(HMSFollower.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.sentry.service.thrift.HMSFollower.getMetaStoreClient(HMSFollower.java:164) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:204) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt) > at > sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) > at > sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) > at > sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) > at > sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) > at > sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) > at > sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) > at > com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193) > ... 25 more > 2017-05-03 18:04:53,290 WARN hive.metastore: Failed to connect to the > MetaStore Server... > 2017-05-03 18:04:53,290 INFO hive.metastore: Waiting 1 seconds before next > connection attempt. > 2017-05-03 18:04:54,290 INFO hive.metastore: Trying to connect to metastore > with URI thrift://ve1113.halxg.cloudera.com:9083 > 2017-05-03 18:04:54,294 ERROR org.apache.thrift.transport.TSaslTransport: > SASL negotiation failure > javax.security.sasl.SaslException: GSS initiate failed [Caused by > GSSException: No valid credentials provided (Mechanism level: Failed to find > any Kerberos tgt)] > at > com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) > at > org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94) > at > org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271) > at > org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:464) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:244) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:187) > at > org.apache.sentry.service.thrift.HMSFollower$1.run(HMSFollower.java:167) > at > org.apache.sentry.service.thrift.HMSFollower$1.run(HMSFollower.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.sentry.service.thrift.HMSFollower.getMetaStoreClient(HMSFollower.java:164) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:204) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: GSSException: No valid credentials provided (Mechanism level: > Failed to find any Kerberos tgt) > at > sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) > at > sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) > at > sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) > at > sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) > at > sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) > at > sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) > at > com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193) > ... 25 more > 2017-05-03 18:04:54,295 WARN hive.metastore: Failed to connect to the > MetaStore Server... > 2017-05-03 18:04:54,295 INFO hive.metastore: Waiting 1 seconds before next > connection attempt. > 2017-05-03 18:04:55,295 ERROR org.apache.sentry.service.thrift.HMSFollower: > Failed to setup secure connection to HMS. > 2017-05-03 18:04:55,296 ERROR org.apache.sentry.service.thrift.HMSFollower: > HMSFollower cannot connect to HMS!! > java.security.PrivilegedActionException: MetaException(message:Could not > connect to meta store using any of the URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failed > at > org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) > at > org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316) > at > org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:464) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:244) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:187) > at > org.apache.sentry.service.thrift.HMSFollower$1.run(HMSFollower.java:167) > at > org.apache.sentry.service.thrift.HMSFollower$1.run(HMSFollower.java:164) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.sentry.service.thrift.HMSFollower.getMetaStoreClient(HMSFollower.java:164) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:204) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > ) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.sentry.service.thrift.HMSFollower.getMetaStoreClient(HMSFollower.java:164) > at > org.apache.sentry.service.thrift.HMSFollower.run(HMSFollower.java:204) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:304) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Caused by: MetaException(message:Could not connect to meta store using any of > the URIs provided. Most recent failure: > org.apache.thrift.transport.TTransportException: GSS initiate failed > at > org.apache.thrift.transport.TSaslTransport.sendAndThrowMessage(TSaslTransport.java:232) > at > org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:316) > at > org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920) > at > org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:464) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:244) > at > org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:187) > at > org.apache.sentry.service.thrift.HMSFollower$1.run(HMSFollower.java:167) > {code} -- This message was sent by Atlassian JIRA (v6.3.15#6346)