[jira] [Commented] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039863#comment-14039863 ] Hudson commented on HDFS-6222: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1808 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1808/]) HDFS-6222. Remove background token renewer from webhdfs. Contributed by Rushabh Shah and Daryn Sharp. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604300) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenIdentifier.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/SWebHdfsFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java > Remove background token renewer from webhdfs > > > Key: HDFS-6222 > URL: https://issues.apache.org/jira/browse/HDFS-6222 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 3.0.0, 2.5.0 > > Attachments: HDFS-6222.branch-2-v2.patch, > HDFS-6222.branch-2-v3.patch, HDFS-6222.branch-2.patch, > HDFS-6222.branch-2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk-v2.patch, > HDFS-6222.trunk-v3.patch, HDFS-6222.trunk.patch, HDFS-6222.trunk.patch > > > The background token renewer is a source of problems for long-running > daemons. Webhdfs should lazy fetch a new token when it receives an > InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039839#comment-14039839 ] Hudson commented on HDFS-6222: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1781 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1781/]) HDFS-6222. Remove background token renewer from webhdfs. Contributed by Rushabh Shah and Daryn Sharp. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604300) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenIdentifier.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/SWebHdfsFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java > Remove background token renewer from webhdfs > > > Key: HDFS-6222 > URL: https://issues.apache.org/jira/browse/HDFS-6222 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 3.0.0, 2.5.0 > > Attachments: HDFS-6222.branch-2-v2.patch, > HDFS-6222.branch-2-v3.patch, HDFS-6222.branch-2.patch, > HDFS-6222.branch-2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk-v2.patch, > HDFS-6222.trunk-v3.patch, HDFS-6222.trunk.patch, HDFS-6222.trunk.patch > > > The background token renewer is a source of problems for long-running > daemons. Webhdfs should lazy fetch a new token when it receives an > InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039779#comment-14039779 ] Hudson commented on HDFS-6222: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #590 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/590/]) HDFS-6222. Remove background token renewer from webhdfs. Contributed by Rushabh Shah and Daryn Sharp. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604300) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenIdentifier.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/SWebHdfsFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java > Remove background token renewer from webhdfs > > > Key: HDFS-6222 > URL: https://issues.apache.org/jira/browse/HDFS-6222 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 3.0.0, 2.5.0 > > Attachments: HDFS-6222.branch-2-v2.patch, > HDFS-6222.branch-2-v3.patch, HDFS-6222.branch-2.patch, > HDFS-6222.branch-2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk-v2.patch, > HDFS-6222.trunk-v3.patch, HDFS-6222.trunk.patch, HDFS-6222.trunk.patch > > > The background token renewer is a source of problems for long-running > daemons. Webhdfs should lazy fetch a new token when it receives an > InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039624#comment-14039624 ] Hudson commented on HDFS-6222: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5747 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5747/]) HDFS-6222. Remove background token renewer from webhdfs. Contributed by Rushabh Shah and Daryn Sharp. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604300) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenIdentifier.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/SWebHdfsFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java > Remove background token renewer from webhdfs > > > Key: HDFS-6222 > URL: https://issues.apache.org/jira/browse/HDFS-6222 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Fix For: 3.0.0, 2.5.0 > > Attachments: HDFS-6222.branch-2-v2.patch, > HDFS-6222.branch-2-v3.patch, HDFS-6222.branch-2.patch, > HDFS-6222.branch-2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk-v2.patch, > HDFS-6222.trunk-v3.patch, HDFS-6222.trunk.patch, HDFS-6222.trunk.patch > > > The background token renewer is a source of problems for long-running > daemons. Webhdfs should lazy fetch a new token when it receives an > InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039273#comment-14039273 ] Hadoop QA commented on HDFS-6222: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12651716/HDFS-6222.branch-2-v3.patch against trunk revision . {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7191//console This message is automatically generated. > Remove background token renewer from webhdfs > > > Key: HDFS-6222 > URL: https://issues.apache.org/jira/browse/HDFS-6222 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-6222.branch-2-v2.patch, > HDFS-6222.branch-2-v3.patch, HDFS-6222.branch-2.patch, > HDFS-6222.branch-2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk-v2.patch, > HDFS-6222.trunk-v3.patch, HDFS-6222.trunk.patch, HDFS-6222.trunk.patch > > > The background token renewer is a source of problems for long-running > daemons. Webhdfs should lazy fetch a new token when it receives an > InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039068#comment-14039068 ] Rushabh S Shah commented on HDFS-6222: -- The failed test runs fine on my machine. > Remove background token renewer from webhdfs > > > Key: HDFS-6222 > URL: https://issues.apache.org/jira/browse/HDFS-6222 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-6222.branch-2-v2.patch, HDFS-6222.branch-2.patch, > HDFS-6222.branch-2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk-v2.patch, > HDFS-6222.trunk-v3.patch, HDFS-6222.trunk.patch, HDFS-6222.trunk.patch > > > The background token renewer is a source of problems for long-running > daemons. Webhdfs should lazy fetch a new token when it receives an > InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14039047#comment-14039047 ] Hadoop QA commented on HDFS-6222: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12651662/HDFS-6222.trunk-v3.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestBootstrapStandbyWithQJM {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7189//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7189//console This message is automatically generated. > Remove background token renewer from webhdfs > > > Key: HDFS-6222 > URL: https://issues.apache.org/jira/browse/HDFS-6222 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-6222.branch-2-v2.patch, HDFS-6222.branch-2.patch, > HDFS-6222.branch-2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk-v2.patch, > HDFS-6222.trunk-v3.patch, HDFS-6222.trunk.patch, HDFS-6222.trunk.patch > > > The background token renewer is a source of problems for long-running > daemons. Webhdfs should lazy fetch a new token when it receives an > InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038249#comment-14038249 ] Hadoop QA commented on HDFS-6222: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12651547/HDFS-6222.trunk-v2.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7182//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/7182//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7182//console This message is automatically generated. > Remove background token renewer from webhdfs > > > Key: HDFS-6222 > URL: https://issues.apache.org/jira/browse/HDFS-6222 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-6222.branch-2-v2.patch, HDFS-6222.branch-2.patch, > HDFS-6222.branch-2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk-v2.patch, > HDFS-6222.trunk.patch, HDFS-6222.trunk.patch > > > The background token renewer is a source of problems for long-running > daemons. Webhdfs should lazy fetch a new token when it receives an > InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038014#comment-14038014 ] Hadoop QA commented on HDFS-6222: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12651519/HDFS-6222.branch-2-v2.patch against trunk revision . {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7180//console This message is automatically generated. > Remove background token renewer from webhdfs > > > Key: HDFS-6222 > URL: https://issues.apache.org/jira/browse/HDFS-6222 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-6222.branch-2-v2.patch, HDFS-6222.branch-2.patch, > HDFS-6222.branch-2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk.patch, > HDFS-6222.trunk.patch > > > The background token renewer is a source of problems for long-running > daemons. Webhdfs should lazy fetch a new token when it receives an > InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028117#comment-14028117 ] Chris Nauroth commented on HDFS-6222: - Thanks for the response. This all makes sense. bq. A secure client is supposed to be able to talk to an insecure server... Now I get it. Would you mind changing that comment to "security is disabled on server" to make this clearer? Thanks for checking earlier versions for the compatibility concern. bq. TokenAspect is still used by hftp... I forgot about this. I had been looking at trunk, where {{HftpFileSystem}} is gone. bq. I'm not sure how to test swebhdfs... There is an example in {{TestHttpsFileSystem}}, and there are some helper methods that take care of bootstrapping configuration, including SSL certs. > Remove background token renewer from webhdfs > > > Key: HDFS-6222 > URL: https://issues.apache.org/jira/browse/HDFS-6222 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-6222.branch-2.patch, HDFS-6222.branch-2.patch, > HDFS-6222.trunk.patch, HDFS-6222.trunk.patch > > > The background token renewer is a source of problems for long-running > daemons. Webhdfs should lazy fetch a new token when it receives an > InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028080#comment-14028080 ] Daryn Sharp commented on HDFS-6222: --- Good questions. This design is the result of problems encountered by converting a mission critical production system to use webhdfs. We've been internally running in production for months with this change on 0.23, and a sandbox 2.x grid. A few of the issues: The renewer is hardcoded to assume 24h, which isn't a guarantee by any means. The filesystem can go dead for up to a day. Decreasing the token renewal on our QA clusters to 30s to stress token handling obviously didn't work either... We've also encountered class loader leaks. Filesystems would become unusable if the token expired, erroneously cancelled, or transient renewal failures such as during a NN restart. # A secure client is supposed to be able to talk to an insecure server which is why earlier logic had this same behavior. Regarding malformed responses, NPEs used to be generated, not null returns. My earlier work trapped and converted the NPEs, which in this case will trigger the retry loop. Unlike the current implementation, the fs will attempt to re-acquire a token even after one operation fails which prevents the fs from becoming unusable - # Yes, very unfortunate, but I only did it for backwards compatibility with NNs, and also cross-compatibility with DNs that don't munge the token exception. I checked earlier versions and it appears to have always been this way. # True. We've become very performance conscience, but token renewal is infrequent if ever required by non-daemons so I consider the tiny latency worth the robustness. # TokenAspect is still used by hftp or I would have happily removed it... # I think this was covered by other tests. I'll double check and add if necessary. I'm not sure how to test swebhdfs since it requires extra configuration and ssl certs to function... > Remove background token renewer from webhdfs > > > Key: HDFS-6222 > URL: https://issues.apache.org/jira/browse/HDFS-6222 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-6222.branch-2.patch, HDFS-6222.branch-2.patch, > HDFS-6222.trunk.patch, HDFS-6222.trunk.patch > > > The background token renewer is a source of problems for long-running > daemons. Webhdfs should lazy fetch a new token when it receives an > InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14028002#comment-14028002 ] Chris Nauroth commented on HDFS-6222: - Hi, [~daryn]. Can you describe what kinds of problems you've seen "for long-running daemons"? Is there a race condition where the renewer thread might not do the renewal ahead of the client operation that needs it? Something else? A couple of questions/observations: # I have a question on this logic: {code} } else { token = getDelegationToken(null); if (token != null) { LOG.debug("Fetched new token: " + token); } else { // security is disabled canRefreshDelegationToken = false; } } {code} In the second else block, I don't see how we can reach this code path when security is disabled, because it's already guarded by an earlier check of {{canRefreshDelegationToken}}, which is initialized from {{UserGroupInformation#isSecurityEnabled}}. Also, I seem to recall you mentioned to me offline seeing certain failures where a malformed response from the server can cause the JSON parser to return a null object. If that happens, then this logic would turn off future attempts to refresh. Would that effectively leave the file system instance in an unusable state, with the only workaround being to discard the instance and get a new one? # It's unfortunate that we have to parse the exception strings, but I don't see an alternative. Do you know if these exception messages are consistent across prior versions? Is there any chance that a 2.5.0 {{WebHdfsFileSystem}} instance would be incompatible with servers running earlier versions back to 2.2.0? # The latency of token renewal is now synchronous with the client operation. It's probably no big deal if your internal testing on secure clusters didn't show a problem. # Unless I'm misreading, it seems like much of {{TokenAspect}} has become dead code. Is there a need for further clean-up? # Thanks for adding those tests. They're very thorough. Do you think we need to add assertions that we're getting the correct token kind? Is it also worthwhile to add some tests through swebhdfs too, and assert that they get {{SWebHdfsFileSystem#TOKEN_KIND}}? > Remove background token renewer from webhdfs > > > Key: HDFS-6222 > URL: https://issues.apache.org/jira/browse/HDFS-6222 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-6222.branch-2.patch, HDFS-6222.branch-2.patch, > HDFS-6222.trunk.patch, HDFS-6222.trunk.patch > > > The background token renewer is a source of problems for long-running > daemons. Webhdfs should lazy fetch a new token when it receives an > InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14026441#comment-14026441 ] Daryn Sharp commented on HDFS-6222: --- The test failure is unrelated. > Remove background token renewer from webhdfs > > > Key: HDFS-6222 > URL: https://issues.apache.org/jira/browse/HDFS-6222 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-6222.branch-2.patch, HDFS-6222.branch-2.patch, > HDFS-6222.trunk.patch, HDFS-6222.trunk.patch > > > The background token renewer is a source of problems for long-running > daemons. Webhdfs should lazy fetch a new token when it receives an > InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14014058#comment-14014058 ] Hadoop QA commented on HDFS-6222: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12647613/HDFS-6222.trunk.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests in hadoop-hdfs-project/hadoop-hdfs: org.apache.hadoop.hdfs.server.namenode.ha.TestStandbyCheckpoints {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/7014//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7014//console This message is automatically generated. > Remove background token renewer from webhdfs > > > Key: HDFS-6222 > URL: https://issues.apache.org/jira/browse/HDFS-6222 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-6222.branch-2.patch, HDFS-6222.branch-2.patch, > HDFS-6222.trunk.patch, HDFS-6222.trunk.patch > > > The background token renewer is a source of problems for long-running > daemons. Webhdfs should lazy fetch a new token when it receives an > InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14010007#comment-14010007 ] Hadoop QA commented on HDFS-6222: - {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12646923/HDFS-6222.trunk.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. There were no new javadoc warning messages. {color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse. {color:red}-1 findbugs{color}. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in hadoop-hdfs-project/hadoop-hdfs. {color:green}+1 contrib tests{color}. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/6980//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/6980//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/6980//console This message is automatically generated. > Remove background token renewer from webhdfs > > > Key: HDFS-6222 > URL: https://issues.apache.org/jira/browse/HDFS-6222 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-6222.branch-2.patch, HDFS-6222.trunk.patch > > > The background token renewer is a source of problems for long-running > daemons. Webhdfs should lazy fetch a new token when it receives an > InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)