[jira] (HIVE-26522) Test for HIVE-22033 and backport to 3.1 and 2.3
[ https://issues.apache.org/jira/browse/HIVE-26522 ] Pavan Lanka deleted comment on HIVE-26522: was (Author: planka): This is already patched as part of HIVE-22033, I will use this add a test case that covers the renewal testing > Test for HIVE-22033 and backport to 3.1 and 2.3 > --- > > Key: HIVE-26522 > URL: https://issues.apache.org/jira/browse/HIVE-26522 > Project: Hive > Issue Type: Bug > Components: Standalone Metastore >Affects Versions: 2.3.8, 3.1.3 >Reporter: Pavan Lanka >Assignee: Pavan Lanka >Priority: Major > Labels: pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > HIVE-22033 fixes the issue with Hive Delegation tokens so that the renewal > time is effective. > This looks at adding a test for HIVE-22033 and backporting this fix to 3.1 > and 2.3 branches in Hive. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Updated] (HIVE-26522) Test for HIVE-22033 and backport to 3.1 and 2.3
[
https://issues.apache.org/jira/browse/HIVE-26522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pavan Lanka updated HIVE-26522:
---
Description:
HIVE-22033 fixes the issue with Hive Delegation tokens so that the renewal time
is effective.
This looks at adding a test for HIVE-22033 and backporting this fix to 3.1 and
2.3 branches in Hive.
was:
The HMS currently exposes method to renew an obtained delegation token
{code:java}
@Override
public long renewDelegationToken(String tokenStrForm) throws MetaException,
TException {
if (localMetaStore) {
return 0;
}
return client.renew_delegation_token(tokenStrForm);
}{code}
However on the server side, the renewal of the delegation token does not result
in the update of the token information with the updated expiry
{code:java}
@Override
public long renewToken(Token token, String renewer)
throws IOException {
// since renewal is KERBEROS authenticated token may not be cached
final DelegationTokenIdentifier id = getTokenIdentifier(token);
DelegationTokenInformation tokenInfo = this.tokenStore.getToken(id);
if (tokenInfo == null) {
throw new InvalidToken("token does not exist: " + id); // no token found
}
// ensure associated master key is available
if (!super.allKeys.containsKey(id.getMasterKeyId())) {
LOGGER.info("Unknown master key (id={}), (re)loading keys from token
store.",
id.getMasterKeyId());
reloadKeys();
}
// reuse super renewal logic
synchronized (this) {
--> super.currentTokens.put(id, tokenInfo);
try {
--> return super.renewToken(token, renewer);
} finally {
--> super.currentTokens.remove(id);
}
}
} {code}
Here you can see that we populate the `super.currentTokens` perform the renewal
and then remove the token without updating the `tokenStore`
As a result of this even though the call for renewal is successful the renewal
time is not updated for the token and the token is invalidated based on the
initial expiry time i.e based on when the token was created.
> Test for HIVE-22033 and backport to 3.1 and 2.3
> ---
>
> Key: HIVE-26522
> URL: https://issues.apache.org/jira/browse/HIVE-26522
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 2.3.8, 3.1.3
>Reporter: Pavan Lanka
>Assignee: Pavan Lanka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> HIVE-22033 fixes the issue with Hive Delegation tokens so that the renewal
> time is effective.
> This looks at adding a test for HIVE-22033 and backporting this fix to 3.1
> and 2.3 branches in Hive.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26522) Test for HIVE-22033 and backport to 3.1 and 2.3
[
https://issues.apache.org/jira/browse/HIVE-26522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pavan Lanka updated HIVE-26522:
---
Summary: Test for HIVE-22033 and backport to 3.1 and 2.3 (was: Test for )
> Test for HIVE-22033 and backport to 3.1 and 2.3
> ---
>
> Key: HIVE-26522
> URL: https://issues.apache.org/jira/browse/HIVE-26522
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 2.3.8, 3.1.3
>Reporter: Pavan Lanka
>Assignee: Pavan Lanka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The HMS currently exposes method to renew an obtained delegation token
> {code:java}
> @Override
> public long renewDelegationToken(String tokenStrForm) throws MetaException,
> TException {
> if (localMetaStore) {
> return 0;
> }
> return client.renew_delegation_token(tokenStrForm);
> }{code}
> However on the server side, the renewal of the delegation token does not
> result in the update of the token information with the updated expiry
> {code:java}
> @Override
> public long renewToken(Token token, String
> renewer) throws IOException {
> // since renewal is KERBEROS authenticated token may not be cached
> final DelegationTokenIdentifier id = getTokenIdentifier(token);
> DelegationTokenInformation tokenInfo = this.tokenStore.getToken(id);
> if (tokenInfo == null) {
> throw new InvalidToken("token does not exist: " + id); // no token found
> }
> // ensure associated master key is available
> if (!super.allKeys.containsKey(id.getMasterKeyId())) {
> LOGGER.info("Unknown master key (id={}), (re)loading keys from token
> store.",
> id.getMasterKeyId());
> reloadKeys();
> }
> // reuse super renewal logic
> synchronized (this) {
> --> super.currentTokens.put(id, tokenInfo);
> try {
> --> return super.renewToken(token, renewer);
> } finally {
> --> super.currentTokens.remove(id);
> }
> }
> } {code}
> Here you can see that we populate the `super.currentTokens` perform the
> renewal and then remove the token without updating the `tokenStore`
>
> As a result of this even though the call for renewal is successful the
> renewal time is not updated for the token and the token is invalidated based
> on the initial expiry time i.e based on when the token was created.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Updated] (HIVE-26522) Test for
[
https://issues.apache.org/jira/browse/HIVE-26522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pavan Lanka updated HIVE-26522:
---
Summary: Test for (was: Metastore DelegationToken renewal is ineffective)
> Test for
> -
>
> Key: HIVE-26522
> URL: https://issues.apache.org/jira/browse/HIVE-26522
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 2.3.8, 3.1.3
>Reporter: Pavan Lanka
>Assignee: Pavan Lanka
>Priority: Major
> Labels: pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> The HMS currently exposes method to renew an obtained delegation token
> {code:java}
> @Override
> public long renewDelegationToken(String tokenStrForm) throws MetaException,
> TException {
> if (localMetaStore) {
> return 0;
> }
> return client.renew_delegation_token(tokenStrForm);
> }{code}
> However on the server side, the renewal of the delegation token does not
> result in the update of the token information with the updated expiry
> {code:java}
> @Override
> public long renewToken(Token token, String
> renewer) throws IOException {
> // since renewal is KERBEROS authenticated token may not be cached
> final DelegationTokenIdentifier id = getTokenIdentifier(token);
> DelegationTokenInformation tokenInfo = this.tokenStore.getToken(id);
> if (tokenInfo == null) {
> throw new InvalidToken("token does not exist: " + id); // no token found
> }
> // ensure associated master key is available
> if (!super.allKeys.containsKey(id.getMasterKeyId())) {
> LOGGER.info("Unknown master key (id={}), (re)loading keys from token
> store.",
> id.getMasterKeyId());
> reloadKeys();
> }
> // reuse super renewal logic
> synchronized (this) {
> --> super.currentTokens.put(id, tokenInfo);
> try {
> --> return super.renewToken(token, renewer);
> } finally {
> --> super.currentTokens.remove(id);
> }
> }
> } {code}
> Here you can see that we populate the `super.currentTokens` perform the
> renewal and then remove the token without updating the `tokenStore`
>
> As a result of this even though the call for renewal is successful the
> renewal time is not updated for the token and the token is invalidated based
> on the initial expiry time i.e based on when the token was created.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Comment Edited] (HIVE-22033) HiveServer2: fix delegation token renewal
[ https://issues.apache.org/jira/browse/HIVE-22033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17601996#comment-17601996 ] Pavan Lanka edited comment on HIVE-22033 at 9/8/22 8:08 PM: [~szehon]/[~yetanotherion] I have another Jira assuming this was not fixed HIVE-26522 having seen this, I will just use that to add a test for this. Hope that makes sense. Additionally I am hoping that we can bring this to the 3.1 and 2.3 branches. was (Author: planka): [~szehon] I have another Jira assuming this was not fixed HIVE-26522 having seen this, I will just use that to add a test for this. Hope that makes sense. Additionally I am hoping that we can bring this to the 3.1 and 2.3 branches. > HiveServer2: fix delegation token renewal > - > > Key: HIVE-22033 > URL: https://issues.apache.org/jira/browse/HIVE-22033 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.5 >Reporter: Ion Alberdi >Assignee: Ion Alberdi >Priority: Major > Fix For: 4.0.0, 4.0.0-alpha-1 > > Attachments: HIVE-22033.2.patch, HIVE-22033.patch > > > Hello, the issue we faced (and a proposal for a fix) in our hive instances is > depicted at > [https://github.com/criteo-forks/hive/pull/24] > Reading the master branch of the project > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/TokenStoreDelegationTokenSecretManager.java#L147] > I think the same behavior is replicated there. > Long story short, *TokenStoreDelegationTokenSecretManager.renewToken*, does > not update the expiry date of a given token (as it does not get the updated > DelegationTokenInformation from *super.currentTokens*). > This makes any call to renewToken ineffective (the expiry date of the token > is not postponed). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-22033) HiveServer2: fix delegation token renewal
[ https://issues.apache.org/jira/browse/HIVE-22033?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17601996#comment-17601996 ] Pavan Lanka commented on HIVE-22033: [~szehon] I have another Jira assuming this was not fixed HIVE-26522 having seen this, I will just use that to add a test for this. Hope that makes sense. Additionally I am hoping that we can bring this to the 3.1 and 2.3 branches. > HiveServer2: fix delegation token renewal > - > > Key: HIVE-22033 > URL: https://issues.apache.org/jira/browse/HIVE-22033 > Project: Hive > Issue Type: Bug >Affects Versions: 2.3.5 >Reporter: Ion Alberdi >Assignee: Ion Alberdi >Priority: Major > Fix For: 4.0.0, 4.0.0-alpha-1 > > Attachments: HIVE-22033.2.patch, HIVE-22033.patch > > > Hello, the issue we faced (and a proposal for a fix) in our hive instances is > depicted at > [https://github.com/criteo-forks/hive/pull/24] > Reading the master branch of the project > > [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/security/TokenStoreDelegationTokenSecretManager.java#L147] > I think the same behavior is replicated there. > Long story short, *TokenStoreDelegationTokenSecretManager.renewToken*, does > not update the expiry date of a given token (as it does not get the updated > DelegationTokenInformation from *super.currentTokens*). > This makes any call to renewToken ineffective (the expiry date of the token > is not postponed). -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Commented] (HIVE-26522) Metastore DelegationToken renewal is ineffective
[
https://issues.apache.org/jira/browse/HIVE-26522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17601989#comment-17601989
]
Pavan Lanka commented on HIVE-26522:
This is already patched as part of HIVE-22033, I will use this add a test case
that covers the renewal testing
> Metastore DelegationToken renewal is ineffective
>
>
> Key: HIVE-26522
> URL: https://issues.apache.org/jira/browse/HIVE-26522
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 2.3.8, 3.1.3
>Reporter: Pavan Lanka
>Assignee: Pavan Lanka
>Priority: Major
>
> The HMS currently exposes method to renew an obtained delegation token
> {code:java}
> @Override
> public long renewDelegationToken(String tokenStrForm) throws MetaException,
> TException {
> if (localMetaStore) {
> return 0;
> }
> return client.renew_delegation_token(tokenStrForm);
> }{code}
> However on the server side, the renewal of the delegation token does not
> result in the update of the token information with the updated expiry
> {code:java}
> @Override
> public long renewToken(Token token, String
> renewer) throws IOException {
> // since renewal is KERBEROS authenticated token may not be cached
> final DelegationTokenIdentifier id = getTokenIdentifier(token);
> DelegationTokenInformation tokenInfo = this.tokenStore.getToken(id);
> if (tokenInfo == null) {
> throw new InvalidToken("token does not exist: " + id); // no token found
> }
> // ensure associated master key is available
> if (!super.allKeys.containsKey(id.getMasterKeyId())) {
> LOGGER.info("Unknown master key (id={}), (re)loading keys from token
> store.",
> id.getMasterKeyId());
> reloadKeys();
> }
> // reuse super renewal logic
> synchronized (this) {
> --> super.currentTokens.put(id, tokenInfo);
> try {
> --> return super.renewToken(token, renewer);
> } finally {
> --> super.currentTokens.remove(id);
> }
> }
> } {code}
> Here you can see that we populate the `super.currentTokens` perform the
> renewal and then remove the token without updating the `tokenStore`
>
> As a result of this even though the call for renewal is successful the
> renewal time is not updated for the token and the token is invalidated based
> on the initial expiry time i.e based on when the token was created.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Assigned] (HIVE-26522) Metastore DelegationToken renewal is ineffective
[
https://issues.apache.org/jira/browse/HIVE-26522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Pavan Lanka reassigned HIVE-26522:
--
Assignee: Pavan Lanka
> Metastore DelegationToken renewal is ineffective
>
>
> Key: HIVE-26522
> URL: https://issues.apache.org/jira/browse/HIVE-26522
> Project: Hive
> Issue Type: Bug
> Components: Standalone Metastore
>Affects Versions: 2.3.8, 3.1.3
>Reporter: Pavan Lanka
>Assignee: Pavan Lanka
>Priority: Major
>
> The HMS currently exposes method to renew an obtained delegation token
> {code:java}
> @Override
> public long renewDelegationToken(String tokenStrForm) throws MetaException,
> TException {
> if (localMetaStore) {
> return 0;
> }
> return client.renew_delegation_token(tokenStrForm);
> }{code}
> However on the server side, the renewal of the delegation token does not
> result in the update of the token information with the updated expiry
> {code:java}
> @Override
> public long renewToken(Token token, String
> renewer) throws IOException {
> // since renewal is KERBEROS authenticated token may not be cached
> final DelegationTokenIdentifier id = getTokenIdentifier(token);
> DelegationTokenInformation tokenInfo = this.tokenStore.getToken(id);
> if (tokenInfo == null) {
> throw new InvalidToken("token does not exist: " + id); // no token found
> }
> // ensure associated master key is available
> if (!super.allKeys.containsKey(id.getMasterKeyId())) {
> LOGGER.info("Unknown master key (id={}), (re)loading keys from token
> store.",
> id.getMasterKeyId());
> reloadKeys();
> }
> // reuse super renewal logic
> synchronized (this) {
> --> super.currentTokens.put(id, tokenInfo);
> try {
> --> return super.renewToken(token, renewer);
> } finally {
> --> super.currentTokens.remove(id);
> }
> }
> } {code}
> Here you can see that we populate the `super.currentTokens` perform the
> renewal and then remove the token without updating the `tokenStore`
>
> As a result of this even though the call for renewal is successful the
> renewal time is not updated for the token and the token is invalidated based
> on the initial expiry time i.e based on when the token was created.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
