[jira] [Updated] (HIVE-17368) DBTokenStore fails to connect in Kerberos enabled remote HMS environment
[
https://issues.apache.org/jira/browse/HIVE-17368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-17368:
---
Resolution: Fixed
Fix Version/s: 2.4.0
3.0.0
Status: Resolved (was: Patch Available)
> DBTokenStore fails to connect in Kerberos enabled remote HMS environment
>
>
> Key: HIVE-17368
> URL: https://issues.apache.org/jira/browse/HIVE-17368
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Fix For: 3.0.0, 2.4.0
>
> Attachments: HIVE-17368.01-branch-2.patch, HIVE-17368.01.patch,
> HIVE-17368.02-branch-2.patch, HIVE-17368.02.patch,
> HIVE-17368.03-branch-2.patch, HIVE-17368.04-branch-2.patch,
> HIVE-17368.05-branch-2.patch, HIVE-17368.06-branch-2.patch
>
>
> In setups where HMS is running as a remote process secured using Kerberos,
> and when {{DBTokenStore}} is configured as the token store, the HS2 Thrift
> API call {{GetDelegationToken}} fail with exception trace seen below. HS2 is
> not able to invoke HMS APIs needed to add/remove/renew tokens from the DB
> since it is possible that the user which is issue the {{GetDelegationToken}}
> is not kerberos enabled.
> Eg. Oozie submits a job on behalf of user "Joe". When Oozie opens a session
> with HS2 it uses Oozie's principal and creates a proxy UGI with Hive. This
> principal can establish a transport authenticated using Kerberos. It stores
> the HMS delegation token string in the sessionConf and sessionToken. Now,
> lets say Oozie issues a {{GetDelegationToken}} which has {{Joe}} as the owner
> and {{oozie}} as the renewer in {{GetDelegationTokenReq}}. This API call
> cannot instantiate a HMSClient and open transport to HMS using the HMSToken
> string available in the sessionConf, since DBTokenStore uses server HiveConf
> instead of sessionConf. It tries to establish transport using Kerberos and it
> fails since user Joe is not Kerberos enabled.
> I see the following exception trace in HS2 logs.
> {noformat}
> 2017-08-21T18:07:19,644 ERROR [HiveServer2-Handler-Pool: Thread-61]
> transport.TSaslTransport: SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> ~[?:1.8.0_121]
> at
> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
> ~[libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_121]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_121]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> [hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:488)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:255)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:70)
> [hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) ~[?:1.8.0_121]
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> [?:1.8.0_121]
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> [?:1.8.0_121]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> [?:1.8.0_121]
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1699)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
>
[jira] [Updated] (HIVE-17368) DBTokenStore fails to connect in Kerberos enabled remote HMS environment
[
https://issues.apache.org/jira/browse/HIVE-17368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-17368:
---
Attachment: HIVE-17368.02.patch
Attaching patch for master branch.
> DBTokenStore fails to connect in Kerberos enabled remote HMS environment
>
>
> Key: HIVE-17368
> URL: https://issues.apache.org/jira/browse/HIVE-17368
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-17368.01-branch-2.patch, HIVE-17368.01.patch,
> HIVE-17368.02-branch-2.patch, HIVE-17368.02.patch,
> HIVE-17368.03-branch-2.patch, HIVE-17368.04-branch-2.patch,
> HIVE-17368.05-branch-2.patch, HIVE-17368.06-branch-2.patch
>
>
> In setups where HMS is running as a remote process secured using Kerberos,
> and when {{DBTokenStore}} is configured as the token store, the HS2 Thrift
> API call {{GetDelegationToken}} fail with exception trace seen below. HS2 is
> not able to invoke HMS APIs needed to add/remove/renew tokens from the DB
> since it is possible that the user which is issue the {{GetDelegationToken}}
> is not kerberos enabled.
> Eg. Oozie submits a job on behalf of user "Joe". When Oozie opens a session
> with HS2 it uses Oozie's principal and creates a proxy UGI with Hive. This
> principal can establish a transport authenticated using Kerberos. It stores
> the HMS delegation token string in the sessionConf and sessionToken. Now,
> lets say Oozie issues a {{GetDelegationToken}} which has {{Joe}} as the owner
> and {{oozie}} as the renewer in {{GetDelegationTokenReq}}. This API call
> cannot instantiate a HMSClient and open transport to HMS using the HMSToken
> string available in the sessionConf, since DBTokenStore uses server HiveConf
> instead of sessionConf. It tries to establish transport using Kerberos and it
> fails since user Joe is not Kerberos enabled.
> I see the following exception trace in HS2 logs.
> {noformat}
> 2017-08-21T18:07:19,644 ERROR [HiveServer2-Handler-Pool: Thread-61]
> transport.TSaslTransport: SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> ~[?:1.8.0_121]
> at
> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
> ~[libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_121]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_121]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> [hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:488)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:255)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:70)
> [hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) ~[?:1.8.0_121]
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> [?:1.8.0_121]
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> [?:1.8.0_121]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> [?:1.8.0_121]
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1699)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:83)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
[jira] [Updated] (HIVE-17368) DBTokenStore fails to connect in Kerberos enabled remote HMS environment
[
https://issues.apache.org/jira/browse/HIVE-17368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-17368:
---
Description:
In setups where HMS is running as a remote process secured using Kerberos, and
when {{DBTokenStore}} is configured as the token store, the HS2 Thrift API call
{{GetDelegationToken}} fail with exception trace seen below. HS2 is not able to
invoke HMS APIs needed to add/remove/renew tokens from the DB since it is
possible that the user which is issue the {{GetDelegationToken}} is not
kerberos enabled.
Eg. Oozie submits a job on behalf of user "Joe". When Oozie opens a session
with HS2 it uses Oozie's principal and creates a proxy UGI with Hive. This
principal can establish a transport authenticated using Kerberos. It stores the
HMS delegation token string in the sessionConf and sessionToken. Now, lets say
Oozie issues a {{GetDelegationToken}} which has {{Joe}} as the owner and
{{oozie}} as the renewer in {{GetDelegationTokenReq}}. This API call cannot
instantiate a HMSClient and open transport to HMS using the HMSToken string
available in the sessionConf, since DBTokenStore uses server HiveConf instead
of sessionConf. It tries to establish transport using Kerberos and it fails
since user Joe is not Kerberos enabled.
I see the following exception trace in HS2 logs.
{noformat}
2017-08-21T18:07:19,644 ERROR [HiveServer2-Handler-Pool: Thread-61]
transport.TSaslTransport: SASL negotiation failure
javax.security.sasl.SaslException: GSS initiate failed
at
com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
~[?:1.8.0_121]
at
org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
~[libthrift-0.9.3.jar:0.9.3]
at
org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
[libthrift-0.9.3.jar:0.9.3]
at
org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
[libthrift-0.9.3.jar:0.9.3]
at
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
[hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
at
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
[hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
at java.security.AccessController.doPrivileged(Native Method)
~[?:1.8.0_121]
at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_121]
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
[hadoop-common-2.7.2.jar:?]
at
org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
[hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:488)
[hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:255)
[hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:70)
[hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
Method) ~[?:1.8.0_121]
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
[?:1.8.0_121]
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
[?:1.8.0_121]
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
[?:1.8.0_121]
at
org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1699)
[hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:83)
[hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133)
[hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
at
org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:104)
[hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
at
org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:3595)
[hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3647)
[hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:3627)
[hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
~[?:1.8.0_121]
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
~[?:1.8.0_121]
at
[jira] [Updated] (HIVE-17368) DBTokenStore fails to connect in Kerberos enabled remote HMS environment
[
https://issues.apache.org/jira/browse/HIVE-17368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-17368:
---
Attachment: HIVE-17368.06-branch-2.patch
> DBTokenStore fails to connect in Kerberos enabled remote HMS environment
>
>
> Key: HIVE-17368
> URL: https://issues.apache.org/jira/browse/HIVE-17368
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-17368.01-branch-2.patch, HIVE-17368.01.patch,
> HIVE-17368.02-branch-2.patch, HIVE-17368.03-branch-2.patch,
> HIVE-17368.04-branch-2.patch, HIVE-17368.05-branch-2.patch,
> HIVE-17368.06-branch-2.patch
>
>
> In setups where HMS is running as a remote process secured using Kerberos,
> and when {{DBTokenStore}} is configured as the token store, the HS2 Thrift
> API calls like {{GetDelegationToken}}, {{CancelDelegationToken}} and
> {{RenewDelegationToken}} fail with exception trace seen below. HS2 is not
> able to invoke HMS APIs needed to add/remove/renew tokens from the DB since
> it is possible that the user which is issue the {{GetDelegationToken}} is not
> kerberos enabled.
> Eg. Oozie submits a job on behalf of user "Joe". When Oozie opens a session
> with HS2 it uses Oozie's principal and creates a proxy UGI with Hive. This
> principal can establish a transport authenticated using Kerberos. It stores
> the HMS delegation token string in the sessionConf and sessionToken. Now,
> lets say Oozie issues a {{GetDelegationToken}} which has {{Joe}} as the owner
> and {{oozie}} as the renewer in {{GetDelegationTokenReq}}. This API call
> cannot instantiate a HMSClient and open transport to HMS using the HMSToken
> string available in the sessionConf, since DBTokenStore uses server HiveConf
> instead of sessionConf. It tries to establish transport using Kerberos and it
> fails since user Joe is not Kerberos enabled.
> I see the following exception trace in HS2 logs.
> {noformat}
> 2017-08-21T18:07:19,644 ERROR [HiveServer2-Handler-Pool: Thread-61]
> transport.TSaslTransport: SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> ~[?:1.8.0_121]
> at
> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
> ~[libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_121]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_121]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> [hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:488)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:255)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:70)
> [hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) ~[?:1.8.0_121]
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> [?:1.8.0_121]
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> [?:1.8.0_121]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> [?:1.8.0_121]
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1699)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:83)
>
[jira] [Updated] (HIVE-17368) DBTokenStore fails to connect in Kerberos enabled remote HMS environment
[
https://issues.apache.org/jira/browse/HIVE-17368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-17368:
---
Attachment: (was: HIVE-17368.06-branch-2.patch)
> DBTokenStore fails to connect in Kerberos enabled remote HMS environment
>
>
> Key: HIVE-17368
> URL: https://issues.apache.org/jira/browse/HIVE-17368
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-17368.01-branch-2.patch, HIVE-17368.01.patch,
> HIVE-17368.02-branch-2.patch, HIVE-17368.03-branch-2.patch,
> HIVE-17368.04-branch-2.patch, HIVE-17368.05-branch-2.patch,
> HIVE-17368.06-branch-2.patch
>
>
> In setups where HMS is running as a remote process secured using Kerberos,
> and when {{DBTokenStore}} is configured as the token store, the HS2 Thrift
> API calls like {{GetDelegationToken}}, {{CancelDelegationToken}} and
> {{RenewDelegationToken}} fail with exception trace seen below. HS2 is not
> able to invoke HMS APIs needed to add/remove/renew tokens from the DB since
> it is possible that the user which is issue the {{GetDelegationToken}} is not
> kerberos enabled.
> Eg. Oozie submits a job on behalf of user "Joe". When Oozie opens a session
> with HS2 it uses Oozie's principal and creates a proxy UGI with Hive. This
> principal can establish a transport authenticated using Kerberos. It stores
> the HMS delegation token string in the sessionConf and sessionToken. Now,
> lets say Oozie issues a {{GetDelegationToken}} which has {{Joe}} as the owner
> and {{oozie}} as the renewer in {{GetDelegationTokenReq}}. This API call
> cannot instantiate a HMSClient and open transport to HMS using the HMSToken
> string available in the sessionConf, since DBTokenStore uses server HiveConf
> instead of sessionConf. It tries to establish transport using Kerberos and it
> fails since user Joe is not Kerberos enabled.
> I see the following exception trace in HS2 logs.
> {noformat}
> 2017-08-21T18:07:19,644 ERROR [HiveServer2-Handler-Pool: Thread-61]
> transport.TSaslTransport: SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> ~[?:1.8.0_121]
> at
> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
> ~[libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_121]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_121]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> [hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:488)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:255)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:70)
> [hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) ~[?:1.8.0_121]
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> [?:1.8.0_121]
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> [?:1.8.0_121]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> [?:1.8.0_121]
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1699)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:83)
>
[jira] [Updated] (HIVE-17368) DBTokenStore fails to connect in Kerberos enabled remote HMS environment
[
https://issues.apache.org/jira/browse/HIVE-17368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-17368:
---
Attachment: HIVE-17368.06-branch-2.patch
Attaching the updated patch which addresses review comments. Also, updates the
realUser value of the token like it was doing previously.
> DBTokenStore fails to connect in Kerberos enabled remote HMS environment
>
>
> Key: HIVE-17368
> URL: https://issues.apache.org/jira/browse/HIVE-17368
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-17368.01-branch-2.patch, HIVE-17368.01.patch,
> HIVE-17368.02-branch-2.patch, HIVE-17368.03-branch-2.patch,
> HIVE-17368.04-branch-2.patch, HIVE-17368.05-branch-2.patch,
> HIVE-17368.06-branch-2.patch
>
>
> In setups where HMS is running as a remote process secured using Kerberos,
> and when {{DBTokenStore}} is configured as the token store, the HS2 Thrift
> API calls like {{GetDelegationToken}}, {{CancelDelegationToken}} and
> {{RenewDelegationToken}} fail with exception trace seen below. HS2 is not
> able to invoke HMS APIs needed to add/remove/renew tokens from the DB since
> it is possible that the user which is issue the {{GetDelegationToken}} is not
> kerberos enabled.
> Eg. Oozie submits a job on behalf of user "Joe". When Oozie opens a session
> with HS2 it uses Oozie's principal and creates a proxy UGI with Hive. This
> principal can establish a transport authenticated using Kerberos. It stores
> the HMS delegation token string in the sessionConf and sessionToken. Now,
> lets say Oozie issues a {{GetDelegationToken}} which has {{Joe}} as the owner
> and {{oozie}} as the renewer in {{GetDelegationTokenReq}}. This API call
> cannot instantiate a HMSClient and open transport to HMS using the HMSToken
> string available in the sessionConf, since DBTokenStore uses server HiveConf
> instead of sessionConf. It tries to establish transport using Kerberos and it
> fails since user Joe is not Kerberos enabled.
> I see the following exception trace in HS2 logs.
> {noformat}
> 2017-08-21T18:07:19,644 ERROR [HiveServer2-Handler-Pool: Thread-61]
> transport.TSaslTransport: SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> ~[?:1.8.0_121]
> at
> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
> ~[libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_121]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_121]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> [hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:488)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:255)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:70)
> [hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) ~[?:1.8.0_121]
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> [?:1.8.0_121]
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> [?:1.8.0_121]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> [?:1.8.0_121]
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1699)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
>
[jira] [Updated] (HIVE-17368) DBTokenStore fails to connect in Kerberos enabled remote HMS environment
[
https://issues.apache.org/jira/browse/HIVE-17368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-17368:
---
Attachment: HIVE-17368.05-branch-2.patch
Following test failures are related. Rest are failing without the patch as well
so I guess they are broken already. I have created sub-tasks under HIVE-17436
Attaching one time with the fix for the failing tests.
{noformat}
org.apache.hive.minikdc.TestJdbcNonKrbSASLWithMiniKdc.testNegativeTokenAuth
(batchId=237)
org.apache.hive.minikdc.TestJdbcWithDBTokenStore.testNegativeTokenAuth
(batchId=237)
org.apache.hive.minikdc.TestJdbcWithMiniKdc.testNegativeTokenAuth (batchId=237)
org.apache.hadoop.hive.thrift.TestHadoopAuthBridge23.testMetastoreProxyUser
(batchId=228)
{noformat}
> DBTokenStore fails to connect in Kerberos enabled remote HMS environment
>
>
> Key: HIVE-17368
> URL: https://issues.apache.org/jira/browse/HIVE-17368
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-17368.01-branch-2.patch, HIVE-17368.01.patch,
> HIVE-17368.02-branch-2.patch, HIVE-17368.03-branch-2.patch,
> HIVE-17368.04-branch-2.patch, HIVE-17368.05-branch-2.patch
>
>
> In setups where HMS is running as a remote process secured using Kerberos,
> and when {{DBTokenStore}} is configured as the token store, the HS2 Thrift
> API calls like {{GetDelegationToken}}, {{CancelDelegationToken}} and
> {{RenewDelegationToken}} fail with exception trace seen below. HS2 is not
> able to invoke HMS APIs needed to add/remove/renew tokens from the DB since
> it is possible that the user which is issue the {{GetDelegationToken}} is not
> kerberos enabled.
> Eg. Oozie submits a job on behalf of user "Joe". When Oozie opens a session
> with HS2 it uses Oozie's principal and creates a proxy UGI with Hive. This
> principal can establish a transport authenticated using Kerberos. It stores
> the HMS delegation token string in the sessionConf and sessionToken. Now,
> lets say Oozie issues a {{GetDelegationToken}} which has {{Joe}} as the owner
> and {{oozie}} as the renewer in {{GetDelegationTokenReq}}. This API call
> cannot instantiate a HMSClient and open transport to HMS using the HMSToken
> string available in the sessionConf, since DBTokenStore uses server HiveConf
> instead of sessionConf. It tries to establish transport using Kerberos and it
> fails since user Joe is not Kerberos enabled.
> I see the following exception trace in HS2 logs.
> {noformat}
> 2017-08-21T18:07:19,644 ERROR [HiveServer2-Handler-Pool: Thread-61]
> transport.TSaslTransport: SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> ~[?:1.8.0_121]
> at
> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
> ~[libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_121]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_121]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> [hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:488)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:255)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:70)
> [hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) ~[?:1.8.0_121]
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>
[jira] [Updated] (HIVE-17368) DBTokenStore fails to connect in Kerberos enabled remote HMS environment
[
https://issues.apache.org/jira/browse/HIVE-17368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-17368:
---
Attachment: HIVE-17368.04-branch-2.patch
Fixed the compilation error.
> DBTokenStore fails to connect in Kerberos enabled remote HMS environment
>
>
> Key: HIVE-17368
> URL: https://issues.apache.org/jira/browse/HIVE-17368
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-17368.01-branch-2.patch, HIVE-17368.01.patch,
> HIVE-17368.02-branch-2.patch, HIVE-17368.03-branch-2.patch,
> HIVE-17368.04-branch-2.patch
>
>
> In setups where HMS is running as a remote process secured using Kerberos,
> and when {{DBTokenStore}} is configured as the token store, the HS2 Thrift
> API calls like {{GetDelegationToken}}, {{CancelDelegationToken}} and
> {{RenewDelegationToken}} fail with exception trace seen below. HS2 is not
> able to invoke HMS APIs needed to add/remove/renew tokens from the DB since
> it is possible that the user which is issue the {{GetDelegationToken}} is not
> kerberos enabled.
> Eg. Oozie submits a job on behalf of user "Joe". When Oozie opens a session
> with HS2 it uses Oozie's principal and creates a proxy UGI with Hive. This
> principal can establish a transport authenticated using Kerberos. It stores
> the HMS delegation token string in the sessionConf and sessionToken. Now,
> lets say Oozie issues a {{GetDelegationToken}} which has {{Joe}} as the owner
> and {{oozie}} as the renewer in {{GetDelegationTokenReq}}. This API call
> cannot instantiate a HMSClient and open transport to HMS using the HMSToken
> string available in the sessionConf, since DBTokenStore uses server HiveConf
> instead of sessionConf. It tries to establish transport using Kerberos and it
> fails since user Joe is not Kerberos enabled.
> I see the following exception trace in HS2 logs.
> {noformat}
> 2017-08-21T18:07:19,644 ERROR [HiveServer2-Handler-Pool: Thread-61]
> transport.TSaslTransport: SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> ~[?:1.8.0_121]
> at
> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
> ~[libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_121]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_121]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> [hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:488)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:255)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:70)
> [hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) ~[?:1.8.0_121]
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> [?:1.8.0_121]
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> [?:1.8.0_121]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> [?:1.8.0_121]
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1699)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:83)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
>
[jira] [Updated] (HIVE-17368) DBTokenStore fails to connect in Kerberos enabled remote HMS environment
[
https://issues.apache.org/jira/browse/HIVE-17368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-17368:
---
Attachment: HIVE-17368.03-branch-2.patch
Attaching the 3rd version of the patch. This removes the doAs to get the
delegationToken. The doAs is not necessary since when impersonation is ON the
current user is already set to impersonated user. Creating a proxy user to get
the delegation might not work in all the cases (eg. when tokenstore is DB)
since it needs to establish a authenticated (Kerberos or using
HMSDelegationToken + Digest) transport to make the connection to HMS.
> DBTokenStore fails to connect in Kerberos enabled remote HMS environment
>
>
> Key: HIVE-17368
> URL: https://issues.apache.org/jira/browse/HIVE-17368
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-17368.01-branch-2.patch, HIVE-17368.01.patch,
> HIVE-17368.02-branch-2.patch, HIVE-17368.03-branch-2.patch
>
>
> In setups where HMS is running as a remote process secured using Kerberos,
> and when {{DBTokenStore}} is configured as the token store, the HS2 Thrift
> API calls like {{GetDelegationToken}}, {{CancelDelegationToken}} and
> {{RenewDelegationToken}} fail with exception trace seen below. HS2 is not
> able to invoke HMS APIs needed to add/remove/renew tokens from the DB since
> it is possible that the user which is issue the {{GetDelegationToken}} is not
> kerberos enabled.
> Eg. Oozie submits a job on behalf of user "Joe". When Oozie opens a session
> with HS2 it uses Oozie's principal and creates a proxy UGI with Hive. This
> principal can establish a transport authenticated using Kerberos. It stores
> the HMS delegation token string in the sessionConf and sessionToken. Now,
> lets say Oozie issues a {{GetDelegationToken}} which has {{Joe}} as the owner
> and {{oozie}} as the renewer in {{GetDelegationTokenReq}}. This API call
> cannot instantiate a HMSClient and open transport to HMS using the HMSToken
> string available in the sessionConf, since DBTokenStore uses server HiveConf
> instead of sessionConf. It tries to establish transport using Kerberos and it
> fails since user Joe is not Kerberos enabled.
> I see the following exception trace in HS2 logs.
> {noformat}
> 2017-08-21T18:07:19,644 ERROR [HiveServer2-Handler-Pool: Thread-61]
> transport.TSaslTransport: SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> ~[?:1.8.0_121]
> at
> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
> ~[libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_121]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_121]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> [hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:488)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:255)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:70)
> [hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) ~[?:1.8.0_121]
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> [?:1.8.0_121]
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> [?:1.8.0_121]
> at
[jira] [Updated] (HIVE-17368) DBTokenStore fails to connect in Kerberos enabled remote HMS environment
[
https://issues.apache.org/jira/browse/HIVE-17368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-17368:
---
Attachment: HIVE-17368.02-branch-2.patch
Attaching the second version of the patch. In the current patch when the
session is closed {{HiveSessionImplWithUGI.close()}} calls {{super.close()}}
which calls SessionState.close(). One of the steps of SessionState.close() is
to {{unCacheDataNucleusClassLoaders}}. This code tries to create a HMS Client
to check if it is localMetastore. Since the HMS delegation token is already
cancelled by this time and the UGI might not open transport to HMS, the
connection will fail and it will log a {{INFO}} level error. I think this check
can be simplified by just using {{HiveConfUtil.isEmbeddedMetaStore}} method
which doesn't need to instantiate a HMS client.
If HMS is remote, this method is will return false and previous behaviour is
maintained. If HMS is embedded, this code will return true and there would be
no need to open the transport. {{ObjectStore.unCacheDataNucleusClassLoaders}}
will execute in the same process.
> DBTokenStore fails to connect in Kerberos enabled remote HMS environment
>
>
> Key: HIVE-17368
> URL: https://issues.apache.org/jira/browse/HIVE-17368
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-17368.01-branch-2.patch, HIVE-17368.01.patch,
> HIVE-17368.02-branch-2.patch
>
>
> In setups where HMS is running as a remote process secured using Kerberos,
> and when {{DBTokenStore}} is configured as the token store, the HS2 Thrift
> API calls like {{GetDelegationToken}}, {{CancelDelegationToken}} and
> {{RenewDelegationToken}} fail with exception trace seen below. HS2 is not
> able to invoke HMS APIs needed to add/remove/renew tokens from the DB since
> it is possible that the user which is issue the {{GetDelegationToken}} is not
> kerberos enabled.
> Eg. Oozie submits a job on behalf of user "Joe". When Oozie opens a session
> with HS2 it uses Oozie's principal and creates a proxy UGI with Hive. This
> principal can establish a transport authenticated using Kerberos. It stores
> the HMS delegation token string in the sessionConf and sessionToken. Now,
> lets say Oozie issues a {{GetDelegationToken}} which has {{Joe}} as the owner
> and {{oozie}} as the renewer in {{GetDelegationTokenReq}}. This API call
> cannot instantiate a HMSClient and open transport to HMS using the HMSToken
> string available in the sessionConf, since DBTokenStore uses server HiveConf
> instead of sessionConf. It tries to establish transport using Kerberos and it
> fails since user Joe is not Kerberos enabled.
> I see the following exception trace in HS2 logs.
> {noformat}
> 2017-08-21T18:07:19,644 ERROR [HiveServer2-Handler-Pool: Thread-61]
> transport.TSaslTransport: SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> ~[?:1.8.0_121]
> at
> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
> ~[libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_121]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_121]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> [hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:488)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:255)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:70)
>
[jira] [Updated] (HIVE-17368) DBTokenStore fails to connect in Kerberos enabled remote HMS environment
[
https://issues.apache.org/jira/browse/HIVE-17368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-17368:
---
Attachment: (was: HIVE-17368-branch-2.01.patch)
> DBTokenStore fails to connect in Kerberos enabled remote HMS environment
>
>
> Key: HIVE-17368
> URL: https://issues.apache.org/jira/browse/HIVE-17368
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-17368.01-branch-2.patch, HIVE-17368.01.patch,
> HIVE-17368.02-branch-2.patch
>
>
> In setups where HMS is running as a remote process secured using Kerberos,
> and when {{DBTokenStore}} is configured as the token store, the HS2 Thrift
> API calls like {{GetDelegationToken}}, {{CancelDelegationToken}} and
> {{RenewDelegationToken}} fail with exception trace seen below. HS2 is not
> able to invoke HMS APIs needed to add/remove/renew tokens from the DB since
> it is possible that the user which is issue the {{GetDelegationToken}} is not
> kerberos enabled.
> Eg. Oozie submits a job on behalf of user "Joe". When Oozie opens a session
> with HS2 it uses Oozie's principal and creates a proxy UGI with Hive. This
> principal can establish a transport authenticated using Kerberos. It stores
> the HMS delegation token string in the sessionConf and sessionToken. Now,
> lets say Oozie issues a {{GetDelegationToken}} which has {{Joe}} as the owner
> and {{oozie}} as the renewer in {{GetDelegationTokenReq}}. This API call
> cannot instantiate a HMSClient and open transport to HMS using the HMSToken
> string available in the sessionConf, since DBTokenStore uses server HiveConf
> instead of sessionConf. It tries to establish transport using Kerberos and it
> fails since user Joe is not Kerberos enabled.
> I see the following exception trace in HS2 logs.
> {noformat}
> 2017-08-21T18:07:19,644 ERROR [HiveServer2-Handler-Pool: Thread-61]
> transport.TSaslTransport: SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> ~[?:1.8.0_121]
> at
> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
> ~[libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_121]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_121]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> [hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:488)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:255)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:70)
> [hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) ~[?:1.8.0_121]
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> [?:1.8.0_121]
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> [?:1.8.0_121]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> [?:1.8.0_121]
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1699)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:83)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
>
[jira] [Updated] (HIVE-17368) DBTokenStore fails to connect in Kerberos enabled remote HMS environment
[
https://issues.apache.org/jira/browse/HIVE-17368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-17368:
---
Attachment: HIVE-17368.01-branch-2.patch
Looks like precommit doesn't like my branch-2 file name. Reattaching with a
different name format
> DBTokenStore fails to connect in Kerberos enabled remote HMS environment
>
>
> Key: HIVE-17368
> URL: https://issues.apache.org/jira/browse/HIVE-17368
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-17368.01-branch-2.patch, HIVE-17368.01.patch,
> HIVE-17368-branch-2.01.patch
>
>
> In setups where HMS is running as a remote process secured using Kerberos,
> and when {{DBTokenStore}} is configured as the token store, the HS2 Thrift
> API calls like {{GetDelegationToken}}, {{CancelDelegationToken}} and
> {{RenewDelegationToken}} fail with exception trace seen below. HS2 is not
> able to invoke HMS APIs needed to add/remove/renew tokens from the DB since
> it is possible that the user which is issue the {{GetDelegationToken}} is not
> kerberos enabled.
> Eg. Oozie submits a job on behalf of user "Joe". When Oozie opens a session
> with HS2 it uses Oozie's principal and creates a proxy UGI with Hive. This
> principal can establish a transport authenticated using Kerberos. It stores
> the HMS delegation token string in the sessionConf and sessionToken. Now,
> lets say Oozie issues a {{GetDelegationToken}} which has {{Joe}} as the owner
> and {{oozie}} as the renewer in {{GetDelegationTokenReq}}. This API call
> cannot instantiate a HMSClient and open transport to HMS using the HMSToken
> string available in the sessionConf, since DBTokenStore uses server HiveConf
> instead of sessionConf. It tries to establish transport using Kerberos and it
> fails since user Joe is not Kerberos enabled.
> I see the following exception trace in HS2 logs.
> {noformat}
> 2017-08-21T18:07:19,644 ERROR [HiveServer2-Handler-Pool: Thread-61]
> transport.TSaslTransport: SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> ~[?:1.8.0_121]
> at
> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
> ~[libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_121]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_121]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> [hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:488)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:255)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:70)
> [hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) ~[?:1.8.0_121]
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> [?:1.8.0_121]
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> [?:1.8.0_121]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> [?:1.8.0_121]
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1699)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:83)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
>
[jira] [Updated] (HIVE-17368) DBTokenStore fails to connect in Kerberos enabled remote HMS environment
[
https://issues.apache.org/jira/browse/HIVE-17368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-17368:
---
Attachment: HIVE-17368-branch-2.01.patch
DBTokenStore and ZKTokenStore is currently broken on master after HIVE-17241.
Attaching branch-2 version of the patch.
> DBTokenStore fails to connect in Kerberos enabled remote HMS environment
>
>
> Key: HIVE-17368
> URL: https://issues.apache.org/jira/browse/HIVE-17368
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-17368.01.patch, HIVE-17368-branch-2.01.patch
>
>
> In setups where HMS is running as a remote process secured using Kerberos,
> and when {{DBTokenStore}} is configured as the token store, the HS2 Thrift
> API calls like {{GetDelegationToken}}, {{CancelDelegationToken}} and
> {{RenewDelegationToken}} fail with exception trace seen below. HS2 is not
> able to invoke HMS APIs needed to add/remove/renew tokens from the DB since
> it is possible that the user which is issue the {{GetDelegationToken}} is not
> kerberos enabled.
> Eg. Oozie submits a job on behalf of user "Joe". When Oozie opens a session
> with HS2 it uses Oozie's principal and creates a proxy UGI with Hive. This
> principal can establish a transport authenticated using Kerberos. It stores
> the HMS delegation token string in the sessionConf and sessionToken. Now,
> lets say Oozie issues a {{GetDelegationToken}} which has {{Joe}} as the owner
> and {{oozie}} as the renewer in {{GetDelegationTokenReq}}. This API call
> cannot instantiate a HMSClient and open transport to HMS using the HMSToken
> string available in the sessionConf, since DBTokenStore uses server HiveConf
> instead of sessionConf. It tries to establish transport using Kerberos and it
> fails since user Joe is not Kerberos enabled.
> I see the following exception trace in HS2 logs.
> {noformat}
> 2017-08-21T18:07:19,644 ERROR [HiveServer2-Handler-Pool: Thread-61]
> transport.TSaslTransport: SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> ~[?:1.8.0_121]
> at
> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
> ~[libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_121]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_121]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> [hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:488)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:255)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:70)
> [hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) ~[?:1.8.0_121]
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> [?:1.8.0_121]
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> [?:1.8.0_121]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> [?:1.8.0_121]
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1699)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:83)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
>
[jira] [Updated] (HIVE-17368) DBTokenStore fails to connect in Kerberos enabled remote HMS environment
[
https://issues.apache.org/jira/browse/HIVE-17368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-17368:
---
Attachment: HIVE-17368.01.patch
Adding the first version of the patch. Modified the existing test
{{TestJdbcWithDBTokenStore}} so that it now uses a secure remote HMS.
> DBTokenStore fails to connect in Kerberos enabled remote HMS environment
>
>
> Key: HIVE-17368
> URL: https://issues.apache.org/jira/browse/HIVE-17368
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-17368.01.patch
>
>
> In setups where HMS is running as a remote process secured using Kerberos,
> and when {{DBTokenStore}} is configured as the token store, the HS2 Thrift
> API calls like {{GetDelegationToken}}, {{CancelDelegationToken}} and
> {{RenewDelegationToken}} fail with exception trace seen below. HS2 is not
> able to invoke HMS APIs needed to add/remove/renew tokens from the DB since
> it is possible that the user which is issue the {{GetDelegationToken}} is not
> kerberos enabled.
> Eg. Oozie submits a job on behalf of user "Joe". When Oozie opens a session
> with HS2 it uses Oozie's principal and creates a proxy UGI with Hive. This
> principal can establish a transport authenticated using Kerberos. It stores
> the HMS delegation token string in the sessionConf and sessionToken. Now,
> lets say Oozie issues a {{GetDelegationToken}} which has {{Joe}} as the owner
> and {{oozie}} as the renewer in {{GetDelegationTokenReq}}. This API call
> cannot instantiate a HMSClient and open transport to HMS using the HMSToken
> string available in the sessionConf, since DBTokenStore uses server HiveConf
> instead of sessionConf. It tries to establish transport using Kerberos and it
> fails since user Joe is not Kerberos enabled.
> I see the following exception trace in HS2 logs.
> {noformat}
> 2017-08-21T18:07:19,644 ERROR [HiveServer2-Handler-Pool: Thread-61]
> transport.TSaslTransport: SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> ~[?:1.8.0_121]
> at
> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
> ~[libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_121]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_121]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> [hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:488)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:255)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:70)
> [hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) ~[?:1.8.0_121]
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> [?:1.8.0_121]
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> [?:1.8.0_121]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> [?:1.8.0_121]
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1699)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:83)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
>
[jira] [Updated] (HIVE-17368) DBTokenStore fails to connect in Kerberos enabled remote HMS environment
[
https://issues.apache.org/jira/browse/HIVE-17368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-17368:
---
Status: Patch Available (was: Open)
> DBTokenStore fails to connect in Kerberos enabled remote HMS environment
>
>
> Key: HIVE-17368
> URL: https://issues.apache.org/jira/browse/HIVE-17368
> Project: Hive
> Issue Type: Bug
>Affects Versions: 2.2.0, 2.1.0, 2.0.0, 1.1.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
> Attachments: HIVE-17368.01.patch
>
>
> In setups where HMS is running as a remote process secured using Kerberos,
> and when {{DBTokenStore}} is configured as the token store, the HS2 Thrift
> API calls like {{GetDelegationToken}}, {{CancelDelegationToken}} and
> {{RenewDelegationToken}} fail with exception trace seen below. HS2 is not
> able to invoke HMS APIs needed to add/remove/renew tokens from the DB since
> it is possible that the user which is issue the {{GetDelegationToken}} is not
> kerberos enabled.
> Eg. Oozie submits a job on behalf of user "Joe". When Oozie opens a session
> with HS2 it uses Oozie's principal and creates a proxy UGI with Hive. This
> principal can establish a transport authenticated using Kerberos. It stores
> the HMS delegation token string in the sessionConf and sessionToken. Now,
> lets say Oozie issues a {{GetDelegationToken}} which has {{Joe}} as the owner
> and {{oozie}} as the renewer in {{GetDelegationTokenReq}}. This API call
> cannot instantiate a HMSClient and open transport to HMS using the HMSToken
> string available in the sessionConf, since DBTokenStore uses server HiveConf
> instead of sessionConf. It tries to establish transport using Kerberos and it
> fails since user Joe is not Kerberos enabled.
> I see the following exception trace in HS2 logs.
> {noformat}
> 2017-08-21T18:07:19,644 ERROR [HiveServer2-Handler-Pool: Thread-61]
> transport.TSaslTransport: SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> ~[?:1.8.0_121]
> at
> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
> ~[libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_121]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_121]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> [hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:488)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:255)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:70)
> [hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) ~[?:1.8.0_121]
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> [?:1.8.0_121]
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> [?:1.8.0_121]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> [?:1.8.0_121]
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1699)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:83)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
>
[jira] [Updated] (HIVE-17368) DBTokenStore fails to connect in Kerberos enabled remote HMS environment
[
https://issues.apache.org/jira/browse/HIVE-17368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vihang Karajgaonkar updated HIVE-17368:
---
Affects Version/s: 1.1.0
2.0.0
2.1.0
2.2.0
> DBTokenStore fails to connect in Kerberos enabled remote HMS environment
>
>
> Key: HIVE-17368
> URL: https://issues.apache.org/jira/browse/HIVE-17368
> Project: Hive
> Issue Type: Bug
>Affects Versions: 1.1.0, 2.0.0, 2.1.0, 2.2.0
>Reporter: Vihang Karajgaonkar
>Assignee: Vihang Karajgaonkar
>
> In setups where HMS is running as a remote process secured using Kerberos,
> and when {{DBTokenStore}} is configured as the token store, the HS2 Thrift
> API calls like {{GetDelegationToken}}, {{CancelDelegationToken}} and
> {{RenewDelegationToken}} fail with exception trace seen below. HS2 is not
> able to invoke HMS APIs needed to add/remove/renew tokens from the DB since
> it is possible that the user which is issue the {{GetDelegationToken}} is not
> kerberos enabled.
> Eg. Oozie submits a job on behalf of user "Joe". When Oozie opens a session
> with HS2 it uses Oozie's principal and creates a proxy UGI with Hive. This
> principal can establish a transport authenticated using Kerberos. It stores
> the HMS delegation token string in the sessionConf and sessionToken. Now,
> lets say Oozie issues a {{GetDelegationToken}} which has {{Joe}} as the owner
> and {{oozie}} as the renewer in {{GetDelegationTokenReq}}. This API call
> cannot instantiate a HMSClient and open transport to HMS using the HMSToken
> string available in the sessionConf, since DBTokenStore uses server HiveConf
> instead of sessionConf. It tries to establish transport using Kerberos and it
> fails since user Joe is not Kerberos enabled.
> I see the following exception trace in HS2 logs.
> {noformat}
> 2017-08-21T18:07:19,644 ERROR [HiveServer2-Handler-Pool: Thread-61]
> transport.TSaslTransport: SASL negotiation failure
> javax.security.sasl.SaslException: GSS initiate failed
> at
> com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:211)
> ~[?:1.8.0_121]
> at
> org.apache.thrift.transport.TSaslClientTransport.handleSaslStartMessage(TSaslClientTransport.java:94)
> ~[libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslTransport.open(TSaslTransport.java:271)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.thrift.transport.TSaslClientTransport.open(TSaslClientTransport.java:37)
> [libthrift-0.9.3.jar:0.9.3]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:52)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport$1.run(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at java.security.AccessController.doPrivileged(Native Method)
> ~[?:1.8.0_121]
> at javax.security.auth.Subject.doAs(Subject.java:422) [?:1.8.0_121]
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> [hadoop-common-2.7.2.jar:?]
> at
> org.apache.hadoop.hive.thrift.client.TUGIAssumingTransport.open(TUGIAssumingTransport.java:49)
> [hive-shims-common-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:488)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(HiveMetaStoreClient.java:255)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.(SessionHiveMetaStoreClient.java:70)
> [hive-exec-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method) ~[?:1.8.0_121]
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> [?:1.8.0_121]
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> [?:1.8.0_121]
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> [?:1.8.0_121]
> at
> org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1699)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.(RetryingMetaStoreClient.java:83)
> [hive-metastore-2.3.0-SNAPSHOT.jar:2.3.0-SNAPSHOT]
> at
> org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:133)
>
