[jira] [Updated] (HADOOP-11157) ZKDelegationTokenSecretManager never shuts down listenerThreadPool
[ https://issues.apache.org/jira/browse/HADOOP-11157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated HADOOP-11157: Attachment: HADOOP-11157.patch Here's a patch that addresses the issue. I moved the listenerThreadPool to follow the same lifetime as the keyCache/tokenCache. I'm not sure how to add a testcase because it is a thread leak; are there any existing tests that look for thread leaks? > ZKDelegationTokenSecretManager never shuts down listenerThreadPool > -- > > Key: HADOOP-11157 > URL: https://issues.apache.org/jira/browse/HADOOP-11157 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.6.0 >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Attachments: HADOOP-11157.patch > > > I'm trying to integrate Solr with the DelegationTokenAuthenticationFilter and > running into this issue. The solr unit tests look for leaked threads and > when I started using the ZKDelegationTokenSecretManager it started reporting > leaks. Shuting down the listenerThreadPool after the objects that use it > resolves the leak threads errors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11157) ZKDelegationTokenSecretManager never shuts down listenerThreadPool
[ https://issues.apache.org/jira/browse/HADOOP-11157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated HADOOP-11157: Status: Patch Available (was: Open) > ZKDelegationTokenSecretManager never shuts down listenerThreadPool > -- > > Key: HADOOP-11157 > URL: https://issues.apache.org/jira/browse/HADOOP-11157 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.6.0 >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Attachments: HADOOP-11157.patch > > > I'm trying to integrate Solr with the DelegationTokenAuthenticationFilter and > running into this issue. The solr unit tests look for leaked threads and > when I started using the ZKDelegationTokenSecretManager it started reporting > leaks. Shuting down the listenerThreadPool after the objects that use it > resolves the leak threads errors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11157) ZKDelegationTokenSecretManager never shuts down listenerThreadPool
[ https://issues.apache.org/jira/browse/HADOOP-11157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated HADOOP-11157: Attachment: HADOOP-11157.patch Kicking off the job again. > ZKDelegationTokenSecretManager never shuts down listenerThreadPool > -- > > Key: HADOOP-11157 > URL: https://issues.apache.org/jira/browse/HADOOP-11157 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.6.0 >Reporter: Gregory Chanan >Assignee: Gregory Chanan > Attachments: HADOOP-11157.patch, HADOOP-11157.patch > > > I'm trying to integrate Solr with the DelegationTokenAuthenticationFilter and > running into this issue. The solr unit tests look for leaked threads and > when I started using the ZKDelegationTokenSecretManager it started reporting > leaks. Shuting down the listenerThreadPool after the objects that use it > resolves the leak threads errors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11157) ZKDelegationTokenSecretManager never shuts down listenerThreadPool
[ https://issues.apache.org/jira/browse/HADOOP-11157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-11157: - Attachment: HADOOP-11157.2.patch Uploading new patch. Summary of changes : * Changed the listener threadpool to a SingleThreadedExecutor so that ZK watch updates are always processed in the order it is received. * add/update/delete of a token used to updated the localCache first and then make changes in ZK. Changed this to ensure that localCache is always updated as a result of a ZK notification. * Fixed up the stopThreads to call super first * During shutdown of AbstractDTSM, there can be calls to increment Delegation token seqnum or KeyId.. which is a ZK operation now.. this used to result in an InterruptedException which used to be thrown as a RuntimeException. This is now caught and ignored (There is no possibility of an inconsistent state in ZK..) I ran the test around 10 - 20 times in a loop.. Havnt seen an issue yet.. > ZKDelegationTokenSecretManager never shuts down listenerThreadPool > -- > > Key: HADOOP-11157 > URL: https://issues.apache.org/jira/browse/HADOOP-11157 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.6.0 >Reporter: Gregory Chanan >Assignee: Arun Suresh > Attachments: HADOOP-11157.2.patch, HADOOP-11157.patch, > HADOOP-11157.patch > > > I'm trying to integrate Solr with the DelegationTokenAuthenticationFilter and > running into this issue. The solr unit tests look for leaked threads and > when I started using the ZKDelegationTokenSecretManager it started reporting > leaks. Shuting down the listenerThreadPool after the objects that use it > resolves the leak threads errors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11157) ZKDelegationTokenSecretManager never shuts down listenerThreadPool
[ https://issues.apache.org/jira/browse/HADOOP-11157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-11157: - Attachment: HADOOP-11157.3.patch Uploading new patch.. * I removed the Stat checks from the add/remove and delete method. I was thinking this should be required.. If we agree that Each node will receive all node state changes (The Curator framework gurantees that.. you can check the Path Cache code.. looks like on reregistering with ZK after a watch.. it checks to see state change) * I added another test case.. that test node shutdown and startup after sometime etc.. I also ran the test multiple times in a loop.. i dont see any errors.. > ZKDelegationTokenSecretManager never shuts down listenerThreadPool > -- > > Key: HADOOP-11157 > URL: https://issues.apache.org/jira/browse/HADOOP-11157 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.6.0 >Reporter: Gregory Chanan >Assignee: Arun Suresh > Attachments: HADOOP-11157.2.patch, HADOOP-11157.3.patch, > HADOOP-11157.patch, HADOOP-11157.patch > > > I'm trying to integrate Solr with the DelegationTokenAuthenticationFilter and > running into this issue. The solr unit tests look for leaked threads and > when I started using the ZKDelegationTokenSecretManager it started reporting > leaks. Shuting down the listenerThreadPool after the objects that use it > resolves the leak threads errors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11157) ZKDelegationTokenSecretManager never shuts down listenerThreadPool
[
https://issues.apache.org/jira/browse/HADOOP-11157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-11157:
-
Attachment: HADOOP-11157.4.patch
[~gchanan], Uploaded patch addressing some of your comments
wrt. if {{allKeys}} need to be synchronized like {{currentTokens}} :
It doesn't.. Unlike {{currentTokens}}, an update to the {{allKeys}} maps
happens only during a {{rollMasterKeys()}} operation and each ZKDTSM always
rolls over a different set of keys.
> ZKDelegationTokenSecretManager never shuts down listenerThreadPool
> --
>
> Key: HADOOP-11157
> URL: https://issues.apache.org/jira/browse/HADOOP-11157
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.6.0
>Reporter: Gregory Chanan
>Assignee: Arun Suresh
> Attachments: HADOOP-11157.2.patch, HADOOP-11157.3.patch,
> HADOOP-11157.4.patch, HADOOP-11157.patch, HADOOP-11157.patch
>
>
> I'm trying to integrate Solr with the DelegationTokenAuthenticationFilter and
> running into this issue. The solr unit tests look for leaked threads and
> when I started using the ZKDelegationTokenSecretManager it started reporting
> leaks. Shuting down the listenerThreadPool after the objects that use it
> resolves the leak threads errors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11157) ZKDelegationTokenSecretManager never shuts down listenerThreadPool
[ https://issues.apache.org/jira/browse/HADOOP-11157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-11157: - Attachment: HADOOP-11157.5.patch Uploading test case fix.. The reason for your error : Since it is possible that there can be a delay for the cancel token message initiated by one node to reach another node.. The second node can ofcourse verify with ZK directly if the token that needs verification has been cancelled or not before verification but.. that would mean having to make an RPC call for every verification request. Thus, the eventual consistency tradef-off should be acceptable here... In this patch, I have * Updated testcases to handle this (If failure is expected on verification, it retires couple of times till it fails) * All test cases run 10 times (please modify the TEST_RETRIES constant locally for anything more.. I verified locally with 100 retries) > ZKDelegationTokenSecretManager never shuts down listenerThreadPool > -- > > Key: HADOOP-11157 > URL: https://issues.apache.org/jira/browse/HADOOP-11157 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.6.0 >Reporter: Gregory Chanan >Assignee: Arun Suresh > Attachments: HADOOP-11157.2.patch, HADOOP-11157.3.patch, > HADOOP-11157.4.patch, HADOOP-11157.5.patch, HADOOP-11157.patch, > HADOOP-11157.patch > > > I'm trying to integrate Solr with the DelegationTokenAuthenticationFilter and > running into this issue. The solr unit tests look for leaked threads and > when I started using the ZKDelegationTokenSecretManager it started reporting > leaks. Shuting down the listenerThreadPool after the objects that use it > resolves the leak threads errors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11157) ZKDelegationTokenSecretManager never shuts down listenerThreadPool
[
https://issues.apache.org/jira/browse/HADOOP-11157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun Suresh updated HADOOP-11157:
-
Attachment: HADOOP-11157.6.patch
Uploading new patch... added synchronization for {{allKeys}} and reusing
{{getTokenInfo()}} in {{ZKDTSM.cancelToken()}}
> ZKDelegationTokenSecretManager never shuts down listenerThreadPool
> --
>
> Key: HADOOP-11157
> URL: https://issues.apache.org/jira/browse/HADOOP-11157
> Project: Hadoop Common
> Issue Type: Bug
> Components: security
>Affects Versions: 2.6.0
>Reporter: Gregory Chanan
>Assignee: Arun Suresh
> Attachments: HADOOP-11157.2.patch, HADOOP-11157.3.patch,
> HADOOP-11157.4.patch, HADOOP-11157.5.patch, HADOOP-11157.6.patch,
> HADOOP-11157.patch, HADOOP-11157.patch
>
>
> I'm trying to integrate Solr with the DelegationTokenAuthenticationFilter and
> running into this issue. The solr unit tests look for leaked threads and
> when I started using the ZKDelegationTokenSecretManager it started reporting
> leaks. Shuting down the listenerThreadPool after the objects that use it
> resolves the leak threads errors.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HADOOP-11157) ZKDelegationTokenSecretManager never shuts down listenerThreadPool
[ https://issues.apache.org/jira/browse/HADOOP-11157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arun Suresh updated HADOOP-11157: - Attachment: HADOOP-11157.7.patch Updating patch... thanks for the review [~atm] > ZKDelegationTokenSecretManager never shuts down listenerThreadPool > -- > > Key: HADOOP-11157 > URL: https://issues.apache.org/jira/browse/HADOOP-11157 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.6.0 >Reporter: Gregory Chanan >Assignee: Arun Suresh > Attachments: HADOOP-11157.2.patch, HADOOP-11157.3.patch, > HADOOP-11157.4.patch, HADOOP-11157.5.patch, HADOOP-11157.6.patch, > HADOOP-11157.7.patch, HADOOP-11157.patch, HADOOP-11157.patch > > > I'm trying to integrate Solr with the DelegationTokenAuthenticationFilter and > running into this issue. The solr unit tests look for leaked threads and > when I started using the ZKDelegationTokenSecretManager it started reporting > leaks. Shuting down the listenerThreadPool after the objects that use it > resolves the leak threads errors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HADOOP-11157) ZKDelegationTokenSecretManager never shuts down listenerThreadPool
[ https://issues.apache.org/jira/browse/HADOOP-11157?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HADOOP-11157: Resolution: Fixed Fix Version/s: 2.7.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've just committed this to trunk and branch-2. Thanks a lot for the contribution, Arun and thanks also to Greg and Karthik for the great reviews. > ZKDelegationTokenSecretManager never shuts down listenerThreadPool > -- > > Key: HADOOP-11157 > URL: https://issues.apache.org/jira/browse/HADOOP-11157 > Project: Hadoop Common > Issue Type: Bug > Components: security >Affects Versions: 2.6.0 >Reporter: Gregory Chanan >Assignee: Arun Suresh > Fix For: 2.7.0 > > Attachments: HADOOP-11157.2.patch, HADOOP-11157.3.patch, > HADOOP-11157.4.patch, HADOOP-11157.5.patch, HADOOP-11157.6.patch, > HADOOP-11157.7.patch, HADOOP-11157.patch, HADOOP-11157.patch > > > I'm trying to integrate Solr with the DelegationTokenAuthenticationFilter and > running into this issue. The solr unit tests look for leaked threads and > when I started using the ZKDelegationTokenSecretManager it started reporting > leaks. Shuting down the listenerThreadPool after the objects that use it > resolves the leak threads errors. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
