[jira] [Updated] (HADOOP-13487) Hadoop KMS doesn't clean up old delegation tokens stored in Zookeeper
[ https://issues.apache.org/jira/browse/HADOOP-13487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-13487: --- Attachment: HADOOP-13487.04.patch Thanks Eddy! I fixed the typo for checkstyle in patch 4 as we talked offline. Will commit after jenkins come back. > Hadoop KMS doesn't clean up old delegation tokens stored in Zookeeper > - > > Key: HADOOP-13487 > URL: https://issues.apache.org/jira/browse/HADOOP-13487 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.6.0 >Reporter: Alex Ivanov >Assignee: Xiao Chen > Attachments: HADOOP-13487.01.patch, HADOOP-13487.02.patch, > HADOOP-13487.03.patch, HADOOP-13487.04.patch > > > Configuration: > CDH 5.5.1 (Hadoop 2.6+) > KMS configured to store delegation tokens in Zookeeper > DEBUG logging enabled in /etc/hadoop-kms/conf/kms-log4j.properties > Findings: > It seems to me delegation tokens never get cleaned up from Zookeeper past > their renewal date. I can see in the logs that the removal thread is started > with the expected interval: > {code} > 2016-08-11 08:15:24,511 INFO AbstractDelegationTokenSecretManager - Starting > expired delegation token remover thread, tokenRemoverScanInterval=60 min(s) > {code} > However, I don't see any delegation token removals, indicated by the > following log message: > org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager > --> removeStoredToken(TokenIdent ident), line 769 [CDH] > {code} > if (LOG.isDebugEnabled()) { > LOG.debug("Removing ZKDTSMDelegationToken_" > + ident.getSequenceNumber()); > } > {code} > Meanwhile, I see a lot of expired delegation tokens in Zookeeper that don't > get cleaned up. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13487) Hadoop KMS doesn't clean up old delegation tokens stored in Zookeeper
[ https://issues.apache.org/jira/browse/HADOOP-13487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-13487: --- Attachment: HADOOP-13487.03.patch Thanks for looking at this, [~eddyxu]! As explained offline, the call to {{processTokenAddOrUpdate}} or {{processKeyAddOrUpdate}} are different due to how the decoding is done, so a common method still looks a little messy. I don't feel strongly one way or the other. Patch 3 extracts the method, please take a look. > Hadoop KMS doesn't clean up old delegation tokens stored in Zookeeper > - > > Key: HADOOP-13487 > URL: https://issues.apache.org/jira/browse/HADOOP-13487 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.6.0 >Reporter: Alex Ivanov >Assignee: Xiao Chen > Attachments: HADOOP-13487.01.patch, HADOOP-13487.02.patch, > HADOOP-13487.03.patch > > > Configuration: > CDH 5.5.1 (Hadoop 2.6+) > KMS configured to store delegation tokens in Zookeeper > DEBUG logging enabled in /etc/hadoop-kms/conf/kms-log4j.properties > Findings: > It seems to me delegation tokens never get cleaned up from Zookeeper past > their renewal date. I can see in the logs that the removal thread is started > with the expected interval: > {code} > 2016-08-11 08:15:24,511 INFO AbstractDelegationTokenSecretManager - Starting > expired delegation token remover thread, tokenRemoverScanInterval=60 min(s) > {code} > However, I don't see any delegation token removals, indicated by the > following log message: > org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager > --> removeStoredToken(TokenIdent ident), line 769 [CDH] > {code} > if (LOG.isDebugEnabled()) { > LOG.debug("Removing ZKDTSMDelegationToken_" > + ident.getSequenceNumber()); > } > {code} > Meanwhile, I see a lot of expired delegation tokens in Zookeeper that don't > get cleaned up. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13487) Hadoop KMS doesn't clean up old delegation tokens stored in Zookeeper
[ https://issues.apache.org/jira/browse/HADOOP-13487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-13487: --- Attachment: HADOOP-13487.02.patch Patch 2 to fix the javac warning. I've tested this in a cluster with 100k pre-existing token znodes. Startup took about a minute, and the new code took about 1 second. Appreciate any review / comments. > Hadoop KMS doesn't clean up old delegation tokens stored in Zookeeper > - > > Key: HADOOP-13487 > URL: https://issues.apache.org/jira/browse/HADOOP-13487 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.6.0 >Reporter: Alex Ivanov >Assignee: Xiao Chen > Attachments: HADOOP-13487.01.patch, HADOOP-13487.02.patch > > > Configuration: > CDH 5.5.1 (Hadoop 2.6+) > KMS configured to store delegation tokens in Zookeeper > DEBUG logging enabled in /etc/hadoop-kms/conf/kms-log4j.properties > Findings: > It seems to me delegation tokens never get cleaned up from Zookeeper past > their renewal date. I can see in the logs that the removal thread is started > with the expected interval: > {code} > 2016-08-11 08:15:24,511 INFO AbstractDelegationTokenSecretManager - Starting > expired delegation token remover thread, tokenRemoverScanInterval=60 min(s) > {code} > However, I don't see any delegation token removals, indicated by the > following log message: > org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager > --> removeStoredToken(TokenIdent ident), line 769 [CDH] > {code} > if (LOG.isDebugEnabled()) { > LOG.debug("Removing ZKDTSMDelegationToken_" > + ident.getSequenceNumber()); > } > {code} > Meanwhile, I see a lot of expired delegation tokens in Zookeeper that don't > get cleaned up. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13487) Hadoop KMS doesn't clean up old delegation tokens stored in Zookeeper
[ https://issues.apache.org/jira/browse/HADOOP-13487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-13487: --- Attachment: HADOOP-13487.01.patch After further looking into this, I think we can simply go with option #1 above. On thread startup, {{PathChildrenCache}} need to load the znode anyways, which is the most time consuming operation. Patch 1 to express the idea, I will test it in a test cluster and update here. Benchmarked with 100k existing expired znodes, while kms start up takes minutes, the new node running in memory take about 2 seconds, which I think is fine. I intentionally ignored exceptions for compatibility - if the directory contains some znodes that can't be understood by ZKDTSM, KMS should still be able to start and run as normal. > Hadoop KMS doesn't clean up old delegation tokens stored in Zookeeper > - > > Key: HADOOP-13487 > URL: https://issues.apache.org/jira/browse/HADOOP-13487 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.6.0 >Reporter: Alex Ivanov >Assignee: Xiao Chen > Attachments: HADOOP-13487.01.patch > > > Configuration: > CDH 5.5.1 (Hadoop 2.6+) > KMS configured to store delegation tokens in Zookeeper > DEBUG logging enabled in /etc/hadoop-kms/conf/kms-log4j.properties > Findings: > It seems to me delegation tokens never get cleaned up from Zookeeper past > their renewal date. I can see in the logs that the removal thread is started > with the expected interval: > {code} > 2016-08-11 08:15:24,511 INFO AbstractDelegationTokenSecretManager - Starting > expired delegation token remover thread, tokenRemoverScanInterval=60 min(s) > {code} > However, I don't see any delegation token removals, indicated by the > following log message: > org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager > --> removeStoredToken(TokenIdent ident), line 769 [CDH] > {code} > if (LOG.isDebugEnabled()) { > LOG.debug("Removing ZKDTSMDelegationToken_" > + ident.getSequenceNumber()); > } > {code} > Meanwhile, I see a lot of expired delegation tokens in Zookeeper that don't > get cleaned up. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org
[jira] [Updated] (HADOOP-13487) Hadoop KMS doesn't clean up old delegation tokens stored in Zookeeper
[ https://issues.apache.org/jira/browse/HADOOP-13487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiao Chen updated HADOOP-13487: --- Status: Patch Available (was: Open) > Hadoop KMS doesn't clean up old delegation tokens stored in Zookeeper > - > > Key: HADOOP-13487 > URL: https://issues.apache.org/jira/browse/HADOOP-13487 > Project: Hadoop Common > Issue Type: Bug > Components: kms >Affects Versions: 2.6.0 >Reporter: Alex Ivanov >Assignee: Xiao Chen > Attachments: HADOOP-13487.01.patch > > > Configuration: > CDH 5.5.1 (Hadoop 2.6+) > KMS configured to store delegation tokens in Zookeeper > DEBUG logging enabled in /etc/hadoop-kms/conf/kms-log4j.properties > Findings: > It seems to me delegation tokens never get cleaned up from Zookeeper past > their renewal date. I can see in the logs that the removal thread is started > with the expected interval: > {code} > 2016-08-11 08:15:24,511 INFO AbstractDelegationTokenSecretManager - Starting > expired delegation token remover thread, tokenRemoverScanInterval=60 min(s) > {code} > However, I don't see any delegation token removals, indicated by the > following log message: > org.apache.hadoop.security.token.delegation.ZKDelegationTokenSecretManager > --> removeStoredToken(TokenIdent ident), line 769 [CDH] > {code} > if (LOG.isDebugEnabled()) { > LOG.debug("Removing ZKDTSMDelegationToken_" > + ident.getSequenceNumber()); > } > {code} > Meanwhile, I see a lot of expired delegation tokens in Zookeeper that don't > get cleaned up. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: common-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: common-issues-h...@hadoop.apache.org