[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Yongjun Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14271633#comment-14271633
 ] 

Yongjun Zhang commented on MAPREDUCE-2764:
--

Hi [~owen.omalley],

All my earlier questions boil down to:

* When do we expect the TRIVIAL_RENEWER to be used?
* Can client (e.g., distcp) somehow control whether to use it (Any suggestion 
on how to let YARN not to renew a token and let the client handle renewal)?

Thanks a lot.



> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0, 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764-trunk.patch, 
> MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764.patch, 
> delegation.patch, token-renew-trunk.patch, token-renew.patch, 
> token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Yongjun Zhang (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14270283#comment-14270283
 ] 

Yongjun Zhang commented on MAPREDUCE-2764:
--

HI [~owen.omalley],

Thanks for your earlier work here. I have some questions, appreciate your time 
to answer, and hope I'm not taxing your memory too much because it has been 
quite long since this jira was resolved.

279   private synchronized TokenRenewer getRenewer() throws IOException {
280 if (renewer != null) {
281   return renewer;
282 }
283 renewer = TRIVIAL_RENEWER;
284 for (TokenRenewer canidate: renewers) {
285   if (canidate.handleKind(this.kind)) {
286 renewer = canidate;
287 return renewer;
288   }
289 }
290 LOG.warn("No TokenRenewer defined for token kind " + this.kind);
291 return renewer;
292   }

If the loop to find a candicate failes, we would be using TRIVIAL_RENEWER, 
whose {{isManaged()}} method returns false. When this happens, it means YARN 
will not try to renew the token. Is this intended?

I have a distcp case that I was hoping I could let distcp to pass an HDFS token 
with null renewer to YARN because I don't want YARN to renew the token. 
However,  looks like the above loop will always find a candicate renewer, thus 
the TRIVIAL_RENEWER will not be used. Is this expected?

Any suggestion on how to let YARN not to renew a token and let the client 
handle renewal? 

I guess the current way how YARN works is to always try to renew the job token 
before scheduling the job, to avoid invalid token to begin with (right?), then 
when the TRIVIAL_RENEWER will be used in the above code? Why the above method 
does return failure if it can't find a matching renewer (instead, it uses 
TRIVIAL_RENEWER)?

Sorry for quite a few questions, thanks again for your time!




> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0, 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764-trunk.patch, 
> MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764.patch, 
> delegation.patch, token-renew-trunk.patch, token-renew.patch, 
> token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138360#comment-13138360
 ] 

Hudson commented on MAPREDUCE-2764:
---

Integrated in Hadoop-Hdfs-trunk #846 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/846/])
Move HADOOP-7668, HADOOP-7509, HADOOP-7745, MAPREDUCE-2764, HADOOP-7766 and 
HADOOP-7721 entries from trunk to 0.23 in CHANGES.txt.

szetszwo : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190132
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt


> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0, 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764-trunk.patch, 
> MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764.patch, 
> delegation.patch, token-renew-trunk.patch, token-renew.patch, 
> token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13138012#comment-13138012
 ] 

Hudson commented on MAPREDUCE-2764:
---

Integrated in Hadoop-Mapreduce-trunk #876 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/876/])
Move HADOOP-7668, HADOOP-7509, HADOOP-7745, MAPREDUCE-2764, HADOOP-7766 and 
HADOOP-7721 entries from trunk to 0.23 in CHANGES.txt.

szetszwo : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190132
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt


> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0, 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764-trunk.patch, 
> MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764.patch, 
> delegation.patch, token-renew-trunk.patch, token-renew.patch, 
> token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137969#comment-13137969
 ] 

Hudson commented on MAPREDUCE-2764:
---

Integrated in Hadoop-Common-trunk-Commit #1182 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1182/])
Move HADOOP-7668, HADOOP-7509, HADOOP-7745, MAPREDUCE-2764, HADOOP-7766 and 
HADOOP-7721 entries from trunk to 0.23 in CHANGES.txt.

szetszwo : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190132
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt


> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0, 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764-trunk.patch, 
> MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764.patch, 
> delegation.patch, token-renew-trunk.patch, token-renew.patch, 
> token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137948#comment-13137948
 ] 

Hudson commented on MAPREDUCE-2764:
---

Integrated in Hadoop-Hdfs-trunk-Commit #1259 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1259/])
Move HADOOP-7668, HADOOP-7509, HADOOP-7745, MAPREDUCE-2764, HADOOP-7766 and 
HADOOP-7721 entries from trunk to 0.23 in CHANGES.txt.

szetszwo : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190132
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt


> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0, 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764-trunk.patch, 
> MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764.patch, 
> delegation.patch, token-renew-trunk.patch, token-renew.patch, 
> token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13137909#comment-13137909
 ] 

Hudson commented on MAPREDUCE-2764:
---

Integrated in Hadoop-Mapreduce-trunk-Commit #1201 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1201/])
Move HADOOP-7668, HADOOP-7509, HADOOP-7745, MAPREDUCE-2764, HADOOP-7766 and 
HADOOP-7721 entries from trunk to 0.23 in CHANGES.txt.

szetszwo : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1190132
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt


> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0, 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764-trunk.patch, 
> MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764.patch, 
> delegation.patch, token-renew-trunk.patch, token-renew.patch, 
> token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13128172#comment-13128172
 ] 

Hudson commented on MAPREDUCE-2764:
---

Integrated in Hadoop-Mapreduce-0.23-Build #52 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/52/])
Merged r1183187 from trunk for MAPREDUCE-2764.

jitendra : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183576
Files : 
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/TokenRenewer.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/Hdfs.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenIdentifier.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestResolveHdfsSymlink.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationToken.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/OfflineEditsViewerHelper.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestDelegationTokenFetcher.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF/services
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Cluster.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/DelegationTokenRenewal.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/JobTokenIdentifier.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/delegation/DelegationTokenIdentifier.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ApplicationTokenIdentifier.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ContainerTokenIdentifier.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/s

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13128154#comment-13128154
 ] 

Hudson commented on MAPREDUCE-2764:
---

Integrated in Hadoop-Hdfs-0.23-Build #40 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/40/])
Merged r1183187 from trunk for MAPREDUCE-2764.

jitendra : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183576
Files : 
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/TokenRenewer.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/Hdfs.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenIdentifier.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestResolveHdfsSymlink.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationToken.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/OfflineEditsViewerHelper.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestDelegationTokenFetcher.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF/services
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Cluster.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/DelegationTokenRenewal.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/JobTokenIdentifier.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/delegation/DelegationTokenIdentifier.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ApplicationTokenIdentifier.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ContainerTokenIdentifier.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/re

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13128030#comment-13128030
 ] 

Hudson commented on MAPREDUCE-2764:
---

Integrated in Hadoop-Mapreduce-0.23-Commit #5 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/5/])
Merged r1183187 from trunk for MAPREDUCE-2764.

jitendra : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183576
Files : 
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/TokenRenewer.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/Hdfs.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenIdentifier.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestResolveHdfsSymlink.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationToken.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/OfflineEditsViewerHelper.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestDelegationTokenFetcher.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF/services
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Cluster.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/DelegationTokenRenewal.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/JobTokenIdentifier.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/delegation/DelegationTokenIdentifier.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ApplicationTokenIdentifier.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ContainerTokenIdentifier.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/s

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13128020#comment-13128020
 ] 

Hudson commented on MAPREDUCE-2764:
---

Integrated in Hadoop-Common-0.23-Commit #3 (See 
[https://builds.apache.org/job/Hadoop-Common-0.23-Commit/3/])
Merged r1183187 from trunk for MAPREDUCE-2764.

jitendra : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183576
Files : 
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/TokenRenewer.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/Hdfs.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenIdentifier.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestResolveHdfsSymlink.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationToken.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/OfflineEditsViewerHelper.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestDelegationTokenFetcher.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF/services
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Cluster.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/DelegationTokenRenewal.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/JobTokenIdentifier.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/delegation/DelegationTokenIdentifier.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ApplicationTokenIdentifier.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ContainerTokenIdentifier.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/mai

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13128021#comment-13128021
 ] 

Hudson commented on MAPREDUCE-2764:
---

Integrated in Hadoop-Hdfs-0.23-Commit #4 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/4/])
Merged r1183187 from trunk for MAPREDUCE-2764.

jitendra : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183576
Files : 
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java
* 
/hadoop/common/branches/branch-0.23/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/TokenRenewer.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/Hdfs.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenIdentifier.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestResolveHdfsSymlink.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationToken.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/OfflineEditsViewerHelper.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestDelegationTokenFetcher.java
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF/services
* 
/hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Cluster.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/DelegationTokenRenewal.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/JobTokenIdentifier.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/delegation/DelegationTokenIdentifier.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ApplicationTokenIdentifier.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ContainerTokenIdentifier.java
* 
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/re

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Jitendra Nath Pandey (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13128018#comment-13128018
 ] 

Jitendra Nath Pandey commented on MAPREDUCE-2764:
-

Merged to 23 as well.

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0, 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764-trunk.patch, 
> MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764.patch, 
> delegation.patch, token-renew-trunk.patch, token-renew.patch, 
> token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Siddharth Seth (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13128011#comment-13128011
 ] 

Siddharth Seth commented on MAPREDUCE-2764:
---

Does this need to go into the 23 branch as well ?

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0, 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764-trunk.patch, 
> MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764.patch, 
> delegation.patch, token-renew-trunk.patch, token-renew.patch, 
> token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Jitendra Nath Pandey (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127798#comment-13127798
 ] 

Jitendra Nath Pandey commented on MAPREDUCE-2764:
-

Created jira MAPREDUCE-3192 to fix the javadoc warning.

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0, 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764-trunk.patch, 
> MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764.patch, 
> delegation.patch, token-renew-trunk.patch, token-renew.patch, 
> token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Mahadev konar (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127638#comment-13127638
 ] 

Mahadev konar commented on MAPREDUCE-2764:
--

Jitendra,
 Looks like this patch added some javadoc warnings. Can you please check on it?

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0, 0.23.0, 0.24.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764-trunk.patch, 
> MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764.patch, 
> delegation.patch, token-renew-trunk.patch, token-renew.patch, 
> token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127524#comment-13127524
 ] 

Hudson commented on MAPREDUCE-2764:
---

Integrated in Hadoop-Mapreduce-trunk #860 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/860/])
MAPREDUCE-2764. Fix renewal of dfs delegation tokens. Contributed by Owen.

jitendra : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183187
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/TokenRenewer.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/Hdfs.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenIdentifier.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestResolveHdfsSymlink.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationToken.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/OfflineEditsViewerHelper.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestDelegationTokenFetcher.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF/services
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Cluster.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/DelegationTokenRenewal.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/JobTokenIdentifier.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/delegation/DelegationTokenIdentifier.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ApplicationTokenIdentifier.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ContainerTokenIdentifier.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/META-INF
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/META-INF/services
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/test/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/token/TestDeleg

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127505#comment-13127505
 ] 

Hudson commented on MAPREDUCE-2764:
---

Integrated in Hadoop-Hdfs-trunk #830 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/830/])
MAPREDUCE-2764. Fix renewal of dfs delegation tokens. Contributed by Owen.

jitendra : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183187
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/TokenRenewer.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/Hdfs.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenIdentifier.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestResolveHdfsSymlink.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationToken.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/OfflineEditsViewerHelper.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestDelegationTokenFetcher.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF/services
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Cluster.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/DelegationTokenRenewal.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/JobTokenIdentifier.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/delegation/DelegationTokenIdentifier.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ApplicationTokenIdentifier.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ContainerTokenIdentifier.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/META-INF
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/META-INF/services
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/test/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/token/TestDelegationToken

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127237#comment-13127237
 ] 

Hudson commented on MAPREDUCE-2764:
---

Integrated in Hadoop-Mapreduce-trunk-Commit #1099 (See 
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1099/])
MAPREDUCE-2764. Fix renewal of dfs delegation tokens. Contributed by Owen.

jitendra : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183187
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/TokenRenewer.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/Hdfs.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenIdentifier.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestResolveHdfsSymlink.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationToken.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/OfflineEditsViewerHelper.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestDelegationTokenFetcher.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF/services
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Cluster.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/DelegationTokenRenewal.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/JobTokenIdentifier.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/delegation/DelegationTokenIdentifier.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ApplicationTokenIdentifier.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ContainerTokenIdentifier.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/META-INF
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/META-INF/services
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/test/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127219#comment-13127219
 ] 

Hudson commented on MAPREDUCE-2764:
---

Integrated in Hadoop-Hdfs-trunk-Commit #1158 (See 
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1158/])
MAPREDUCE-2764. Fix renewal of dfs delegation tokens. Contributed by Owen.

jitendra : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183187
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/TokenRenewer.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/Hdfs.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenIdentifier.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestResolveHdfsSymlink.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationToken.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/OfflineEditsViewerHelper.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestDelegationTokenFetcher.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF/services
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Cluster.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/DelegationTokenRenewal.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/JobTokenIdentifier.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/delegation/DelegationTokenIdentifier.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ApplicationTokenIdentifier.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ContainerTokenIdentifier.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/META-INF
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/META-INF/services
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/test/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/token/Tes

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Hudson (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127217#comment-13127217
 ] 

Hudson commented on MAPREDUCE-2764:
---

Integrated in Hadoop-Common-trunk-Commit #1080 (See 
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1080/])
MAPREDUCE-2764. Fix renewal of dfs delegation tokens. Contributed by Owen.

jitendra : 
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1183187
Files : 
* /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/SecurityUtil.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/UserGroupInformation.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/Token.java
* 
/hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/security/token/TokenRenewer.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/Hdfs.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/block/BlockTokenIdentifier.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/fs/TestResolveHdfsSymlink.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/security/TestDelegationToken.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/OfflineEditsViewerHelper.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/tools/TestDelegationTokenFetcher.java
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF/services
* 
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/JobClient.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/Cluster.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/DelegationTokenRenewal.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/JobTokenIdentifier.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/security/token/delegation/DelegationTokenIdentifier.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ApplicationTokenIdentifier.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/security/ContainerTokenIdentifier.java
* 
/hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/META-INF
* /hadoop/common/trunk/hadoop-mapreduce-project/src/test/META-INF/services
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/test/META-INF/services/org.apache.hadoop.security.token.TokenRenewer
* 
/hadoop/common/trunk/hadoop-mapreduce-project/src/test/mapred/org/apache/hadoop/mapreduce/security/token

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127203#comment-13127203
 ] 

Hadoop QA commented on MAPREDUCE-2764:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12498967/MAPREDUCE-2764-trunk.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 19 new or modified tests.

-1 patch.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1015//console

This message is automatically generated.

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0, 0.23.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764-trunk.patch, 
> MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764.patch, 
> delegation.patch, token-renew-trunk.patch, token-renew.patch, 
> token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Suresh Srinivas (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127170#comment-13127170
 ] 

Suresh Srinivas commented on MAPREDUCE-2764:


Comments:
# UserGroupInformation: Can we create a jira instead of adding "TODO"
# Interface classification missing for TrivialRenewer, TokenRenewer
# There are some tabs in DFSClient.java, HftpFileSystem.java, 
BlockTokenIdentifier.java, DelegationTokenFetcher.java
# Indentation is not correct at several places, due to tab. One without tab 
below:
{noformat}
+  SecurityUtil.buildDTServiceName(name, 
+  DFSConfigKeys.DFS_HTTPS_PORT_DEFAULT);
{noformat}

+1 with those changes.

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0, 0.23.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764-trunk.patch, 
> MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764.patch, delegation.patch, 
> token-renew-trunk.patch, token-renew.patch, token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Jitendra Nath Pandey (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127121#comment-13127121
 ] 

Jitendra Nath Pandey commented on MAPREDUCE-2764:
-

I manually ran tests for common, hdfs, yarn. Following tests failed
  TestFailoverProxy,
  TestJMXJsonServlet
  TestDistributedFileSystem
  TestRestartDFS
  TestDfsOverAvroRpc
  TestDeleteBlockPool
  TestNMWebServer 

  They fail on trunk without this patch as well. 
  (TestDistributedFileSystem fails with java.io.IOException: Unknown protocol: 
org.apache.hadoop.hdfs.protocol.ClientProtocol)
 

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0, 0.23.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764-trunk.patch, 
> MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764.patch, delegation.patch, 
> token-renew-trunk.patch, token-renew.patch, token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127095#comment-13127095
 ] 

Hadoop QA commented on MAPREDUCE-2764:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12498952/MAPREDUCE-2764-trunk.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 19 new or modified tests.

-1 patch.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1012//console

This message is automatically generated.

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0, 0.23.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764-trunk.patch, 
> MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764.patch, delegation.patch, 
> token-renew-trunk.patch, token-renew.patch, token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Jitendra Nath Pandey (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126421#comment-13126421
 ] 

Jitendra Nath Pandey commented on MAPREDUCE-2764:
-

test patch results:

-1 overall.  

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 19 new or modified tests.

+1 javadoc.  The javadoc tool did not generate any warning messages.

+1 javac.  The applied patch does not increase the total number of javac 
compiler warnings.

+1 findbugs.  The patch does not introduce any new Findbugs (version ) 
warnings.

-1 release audit.  The applied patch generated 1 release audit warnings 
(more than the trunk's current 0 warnings).

The release audit warning is related to Apache License header for 
hadoop-assemblies/src/main/resources/assemblies/hadoop-mapreduce-dist.xml. It 
is unrelated to this patch.

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0, 0.23.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764.patch, 
> delegation.patch, token-renew-trunk.patch, token-renew.patch, 
> token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126419#comment-13126419
 ] 

Hadoop QA commented on MAPREDUCE-2764:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12498858/MAPREDUCE-2764-trunk.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 19 new or modified tests.

-1 patch.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1002//console

This message is automatically generated.

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0, 0.23.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764-trunk.patch, MAPREDUCE-2764.patch, 
> delegation.patch, token-renew-trunk.patch, token-renew.patch, 
> token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Jitendra Nath Pandey (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126263#comment-13126263
 ] 

Jitendra Nath Pandey commented on MAPREDUCE-2764:
-

I agree with Daryn regarding the use_ip change. Apart from that, the trunk 
patch looks good to me. +1

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0, 0.23.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764.patch, delegation.patch, 
> token-renew-trunk.patch, token-renew.patch, token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Todd Lipcon (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13126186#comment-13126186
 ] 

Todd Lipcon commented on MAPREDUCE-2764:


Are you still working on getting this into trunk?

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0, 0.23.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764.patch, delegation.patch, 
> token-renew-trunk.patch, token-renew.patch, token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Daryn Sharp (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121489#comment-13121489
 ] 

Daryn Sharp commented on MAPREDUCE-2764:


Arg, too many checkedout trees.  The earlier comment regarding 
{{SecurityUtil.buildTokenService()}} is actually an issue.  If you revert to 
how it worked when the change went into 205, I'll handle bringing the 
{{use_ip}} changes in via HADOOP-7510.

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0, 0.23.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764.patch, delegation.patch, 
> token-renew-trunk.patch, token-renew.patch, token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Daryn Sharp (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13121330#comment-13121330
 ] 

Daryn Sharp commented on MAPREDUCE-2764:


I misapplied the patches, so earlier comment is void.  A question: should the 
{{ContainerToken}} be renewing its token instead of using a 
{{TrivialTokenRenewer}}?

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0, 0.23.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764.patch, delegation.patch, 
> token-renew-trunk.patch, token-renew.patch, token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Hadoop QA (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117732#comment-13117732
 ] 

Hadoop QA commented on MAPREDUCE-2764:
--

-1 overall.  Here are the results of testing the latest attachment 
  
http://issues.apache.org/jira/secure/attachment/12497056/token-renew-trunk.patch
  against trunk revision .

+1 @author.  The patch does not contain any @author tags.

+1 tests included.  The patch appears to include 10 new or modified tests.

-1 patch.  The patch command could not apply the patch.

Console output: 
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/901//console

This message is automatically generated.

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0, 0.23.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764.patch, delegation.patch, 
> token-renew-trunk.patch, token-renew.patch, token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Daryn Sharp (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13117650#comment-13117650
 ] 

Daryn Sharp commented on MAPREDUCE-2764:


I don't think the trunk patch will work because {{buildTokenService}} is 
unconditionally creating token services with hostnames.  However, using 
hostnames relies on HADOOP-7510.

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0, 0.23.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764.patch, delegation.patch, 
> token-renew-trunk.patch, token-renew.patch, token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Daryn Sharp (Commented) (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114946#comment-13114946
 ] 

Daryn Sharp commented on MAPREDUCE-2764:


Is there an estimate for when this jira's trunk patch will be available?  
HADOOP-7510's trunk patch will depend on this jira's trunk patch.  Thanks! 

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764.patch, delegation.patch, 
> token-renew.patch, token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: 
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Owen O'Malley (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101601#comment-13101601
 ] 

Owen O'Malley commented on MAPREDUCE-2764:
--

I've committed this to branch-0.20-security. I'm still working on the forward 
port to trunk, which is taking longer than I expected due to the difference 
between the two branches.

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764.patch, delegation.patch, 
> token-renew.patch, token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Jitendra Nath Pandey (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101580#comment-13101580
 ] 

Jitendra Nath Pandey commented on MAPREDUCE-2764:
-

A minor comment: Can we use TokenSelector(s) to get the token from ugi in 
HftpFileSystem#initialize? 

The patch looks good to me. +1

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764.patch, delegation.patch, 
> token-renew.patch, token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Owen O'Malley (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101453#comment-13101453
 ] 

Owen O'Malley commented on MAPREDUCE-2764:
--

Daryn,
 It is awkward to call setTokenService, since it requires a InetSocketAddress 
as a parameter, which I would need to create for precisely that call. My patch 
uses the underlying methods in SecurityUtil, all of which were already public.

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764.patch, delegation.patch, 
> token-renew.patch, token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Owen O'Malley (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101365#comment-13101365
 ] 

Owen O'Malley commented on MAPREDUCE-2764:
--

{quote}
Ah, I see, however getDTFromRemote isn't the right place to set the service. 
Forcing all tokens to be converted to hftp tokens in this outside method is 
artificially limiting.
{quote}

Arguably, the code needs to be refactored. However, I need the hadoop fetchdt 
command to work the same way. getDTfromRemote is the primitive that is used to 
fetch hftp tokens for both the hftp filesystem and fetchdt. I was trying to 
minimize refactoring to the bare minimum, since this was urgent for 205. 

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764.patch, delegation.patch, 
> token-renew.patch, token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Daryn Sharp (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101356#comment-13101356
 ] 

Daryn Sharp commented on MAPREDUCE-2764:


bq.  Daryn, the service is set by the DelegationTokenFetcher, since it has to 
happen over there anyways.

Ah, I see, however {{getDTFromRemote}} isn't the right place to set the 
service.  Forcing all tokens to be converted to hftp tokens in this outside 
method is artificially limiting -- there's no reason why other tokens couldn't 
be retrieved via the http protocol.  It would equivalent to the rpc client 
forcing all tokens to be a particular type.

{{HftpFileSystem}} is the correct place to stamp a token as an hftp token.

bq.  The service isn't a generic field that is required to be :, it 
is for the use of each kind of token. The current tokens use that format, but 
it isn't a requirement. Obviously a shared library of methods to help are a 
good thing and should be used.

Fully agreed.  Tokens like job tokens are free to use and interpret the field 
as they see fit.

However, connection based tokens must use {{setTokenService}} to abstract the 
format of the service field.  The method was added to eliminate the rampant 
copy-n-paste code that made it very difficult to enable something as simple as 
ip->host change.  The abstraction also eliminated the extremely tight coupling 
between the token selectors and a connection-based token's service.

Given the abstraction to encode the field, an equivalent function is needed to 
abstract the decoding the field.  Otherwise, n-many places will need to be 
changed if the encoding or desired behavior (ex. ip->host!) needs to be 
universally changed.

bq.  That said, the current SecurityUtil class needs cleanup to remove all of 
the redundant methods and provide a much cleaner and better documented 
abstraction.

Also fully agreed!!  But it's what we have for the sustaining release to avoid 
incompatible API changes.


> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764.patch, delegation.patch, 
> token-renew.patch, token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Owen O'Malley (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101306#comment-13101306
 ] 

Owen O'Malley commented on MAPREDUCE-2764:
--

Daryn, the service is set by the DelegationTokenFetcher, since it has to happen 
over there anyways.

The service isn't a generic field that is required to be :, it is 
for the use of each kind of token. The current tokens use that format, but it 
isn't a requirement. Obviously a shared library of methods to help are a good 
thing and should be used.

That said, the current SecurityUtil class needs cleanup to remove all of the 
redundant methods and provide a much cleaner and better documented abstraction.

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764.patch, delegation.patch, 
> token-renew.patch, token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Daryn Sharp (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101298#comment-13101298
 ] 

Daryn Sharp commented on MAPREDUCE-2764:


I have other comments/concerns, but the most important is that setting of the 
token service in {{HftpFileSystem}} was removed.  It must be added back or it 
will break the ip->host change for hftp tokens which is where it is most 
needed.  Please use {{SecurityUtils.setTokenService}}.

We'll also have to add a {{SecurityUtil.getTokenServiceAddress}} that decodes 
the token's addresses and performs another lookup via 
{{NetUtil.createSocketAddr}}.  All the places directly accessing the service 
should be updated.

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764.patch, delegation.patch, 
> token-renew.patch, token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Owen O'Malley (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101272#comment-13101272
 ] 

Owen O'Malley commented on MAPREDUCE-2764:
--

The structure of the change means that since hftp tokens are renewed via http, 
that it should only be hampered by breakage of the http protocol. I haven't 
tested it and I'm in the middle of the forward port, which is taxing because of 
the large number of changes between 2xx and trunk.

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764.patch, delegation.patch, 
> token-renew.patch, token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Nathan Roberts (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13101201#comment-13101201
 ] 

Nathan Roberts commented on MAPREDUCE-2764:
---

Owen, Have you had a chance to test interop with 23? If my understanding is 
correct, the original issue is that 23 and 20x didn't get along when distcp was 
needing to renew delegation tokens. 

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Owen O'Malley
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764-4.patch, MAPREDUCE-2764.patch, delegation.patch, 
> token-renew.patch, token-renew.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Daryn Sharp (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13096188#comment-13096188
 ] 

Daryn Sharp commented on MAPREDUCE-2764:


Once my laptop is functional, will post patch that abstracts away the direct 
token management from the {{obtainTokensForNamenodes}}.  It will just tell the 
{{Credentials}} to get the tokens.  Renewal will use {{TokenManager}}s instead 
of {{Token}}s.

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764-3.patch, 
> MAPREDUCE-2764.patch, delegation.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Owen O'Malley (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13095463#comment-13095463
 ] 

Owen O'Malley commented on MAPREDUCE-2764:
--

Yeah, I had to track through the code to make sure that I had it right, but 
StreamFile calls down into JspHelper.getUGI. That clobbers the service to point 
to the NameNode's rpc address. So as long as the HFTP client sets the kind back 
to HDFS, we will stay backwards compatible and still work.

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764.patch, 
> delegation.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Nathan Roberts (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13095387#comment-13095387
 ] 

Nathan Roberts commented on MAPREDUCE-2764:
---

After jitendra latest comment on HADOOP-7510, I think I know the answer to #2 
(please tell me if this if incorrect). When a new (205) client gets redirected 
to an old (204) datanode to stream the file contents, the service information 
within the token is ignored. The kind has to be right, but the service can be 
pretty much anything because the servlet is going to reconstruct the service 
field anyway.

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764.patch, 
> delegation.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Nathan Roberts (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094983#comment-13094983
 ] 

Nathan Roberts commented on MAPREDUCE-2764:
---

Thanks Owen. Getting closer.

.bq Given that 203 and 204 are in production, we can't make incompatible 
changes, therefore the HFTP service needs to continue to send and receive HDFS 
delegation tokens. 

Definitely agree on the compatibility requirement.

.bq Therefore, having the HFTP client manipulate the tokens into the form that 
should have been sent seems like the best alternative.

Where am I going wrong in this use case:
* I launch an MR job from a client box
* As part of this launch, the client will authenticate with all the required 
NNs and gather up all the necessary delegation tokens. Prior to serializing the 
credentials, the HFTP client would have already manipulated any tokens it has 
received into something like (kind=hftp, service=host/hftp_port)
* There will be 2 uses of the token when read later from the credentials.
*# The JT will want to renew it (so it needs HFTP host/port information, so far 
so good). 
*# The map task might use it to fetch data from the remote cluster (it can talk 
to the remote namenode fine but doesn't it need the HDFS host/port information 
to pass to the servlet running on the datanode that will serve up the file's 
contents? Since this information is no longer associated with this token, how 
will it be able to do this?)


> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764.patch, 
> delegation.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Owen O'Malley (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094802#comment-13094802
 ] 

Owen O'Malley commented on MAPREDUCE-2764:
--

The service field was a per-kind specific information that contained 
information about which instance of the service generated the token. It is used 
in two cases:
  1. picking which token to use when opening an rpc connection to the service 
(eg. NameNode, JobTracker, Oozie, HBase, ...)
  2. connecting to the service to renew or cancel the token

For HDFS delegation tokens the service consists of the host and port of the 
NameNode. For HFTP delegation tokens, it would have been ideal if it had always 
contained the host and port of the HFTP server. Then all of the mapping would 
have been done in the HFTP servlet code as it converted an HDFS token into a 
HFTP one. Since the access mechanism is *totally* different for the two types 
of tokens, they should be pulled apart into separate kinds so that they can 
have different renewal/canceling mechanisms.

Given that 203 and 204 are in production, we can't make incompatible changes, 
therefore the HFTP service needs to continue to send and receive HDFS 
delegation tokens. Therefore, having the HFTP client manipulate the tokens into 
the form that *should* have been sent seems like the best alternative.

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764.patch, 
> delegation.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Nathan Roberts (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094759#comment-13094759
 ] 

Nathan Roberts commented on MAPREDUCE-2764:
---

bq. It seems to make much more sense to separate the HFTP tokens out and use 
the service field to refer to the HFTP address, in the way it was intended to 
be used.

Can you say a little about what the service field was intended to do? What I'm 
stuck on is that the client is going to authenticate with the remote NN and get 
back an HFTP token with an HFTP address in the service field, but then that 
same token is going to eventually get to a remote DN that needs to use it to 
talk to the NN via RPC. What am I not understanding?

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764.patch, 
> delegation.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Daryn Sharp (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094738#comment-13094738
 ] 

Daryn Sharp commented on MAPREDUCE-2764:


Yes, the patch was hurried to be a demonstration of the latest approach I am 
pursuing.As posted, tests also don't pass.

bq. When delegation tokens were added, there was agreement that adding those 
methods to FileSystem wasn't appropriate. We can re-open that discussion, but 
it needs to be discussed more widely.

I removed {{FileSystem.getDelegationToken}} after you expressed displeasure 
with it ever being there.  I can re-add it for now if you feel a larger debate 
is required?

bq. I also assume that the change to fixFontPaths is accidental.

Yes, I didn't mean to include it in the patch.  It allows forrest to run on 
java 6.

bq. I also like the isRenewable method, although it also controls whether it is 
cancelable so probably needs a better name like isManaged.

Sure.

bq. Please don't remove InterruptedException from APIs. At some point we'll fix 
RPC to properly throw InterruptedException when it is waiting.

Ok.

bq. Changing the semantics of JobClient.addAll isn't ok.

Granted, I forget to use the boolean replace field, but it's not changing the 
semantics of the existing method.  The new {{addAll(Credentials, boolean)}} 
method prevents overwriting existing credentials.  This fixes an edge case in 
the JT where the client has already acquired valid renewable tokens for a 
service, but they are replaced by bad credentials on disk.

bq. We've been burned badly before by the idiom of using static blocks to 
register types. The ServiceLookup is much better.

I was concerned about that too, so I made the operation be a simple map 
insertion.  I found a number of other places that have similar static blocks 
for registering things so I thought it might be ok.  I'd like to get the rest 
of the implementation nailed down, then switch it to a SL.  Is that ok?

bq. Adding a transient field that is lost when going over RPC is problematic 
from a maintenance point of view.

The issuer is intentionally transient so: a) no binary compat issues b) it's 
only relevant to the client that requested the token.  The NN, DN, etc that 
uses the token doesn't care how the client acquired the token.

bq. It also doesn't feel justified, just so that we can continue to intermingle 
HFTP delegation tokens with HFTP delegation tokens.

The issuer is much larger than that.  Http(s) is merely the transport used to 
acquire/renew/cancel the token.  However, the token acquired is not authorizing 
future http(s) connections.  The token authorizes hdfs rpc connections.  +The 
protocol to manage the token is completely orthogonal to what the token is 
authorizing+.  Having the client stomp on the kind and service fields 
completely changes the token's semantics and renders it useless unless the 
original kind and service fields are swapped back.

The options are then:
* (your proposal) The client does hack-ish swapping of kind and service within 
the token so it knows the transport used to manage the token.  The client is no 
longer allowed to use the token for what it is actually authorizing.  Ex. The 
token now looks like it's for http (which it's not!), so the client can't use 
it for rpc.  The original values will need to be marshaled into existing fields 
during serialization, or the token becomes useless.

* (my early approach) The client creates an hftp token that is a wrapper for 
the real token that is acquired.  This prevents the swapping of kind and 
service within the token, but I found it unexpectedly difficult to implement.  
Note that the client still cannot directly use the token for what it 
authorizes, which was an earlier concern by Jitendra.

* (my current approach) The client tracks how it acquired the token via an 
issuer field.  The issuer field is only persistent within the client's 
credentials.  The client uses the issuer to determine the transport to manage 
the token.  Unlike the other approaches, the token is completely binary 
compatible and, the client can use the token for what it authorizes, and 
there's no guesswork in how to renew it!

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764.patch, 
> delegation.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Owen O'Malley (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094633#comment-13094633
 ] 

Owen O'Malley commented on MAPREDUCE-2764:
--

Daryn,

Your patch doesn't compile because you are using @Override on the cancel/renew 
token methods in DistributedFileSystem. I assume you forgot to include 
FileSystem in your patch. When delegation tokens were added, there was 
agreement that adding those methods to FileSystem wasn't appropriate. We can 
re-open that discussion, but it needs to be discussed more widely.

I also assume that the change to fixFontPaths is accidental.

I'm glad to see the move to use methods on Token, I think that is appropriate. 
I also like the isRenewable method, although it also controls whether it is 
cancelable so probably needs a better name like isManaged.

Please don't remove InterruptedException from APIs. At some point we'll fix RPC 
to properly throw InterruptedException when it is waiting.

We've been burned badly before by the idiom of using static blocks to register 
types. The ServiceLookup is much better.

Adding a transient field that is lost when going over RPC is problematic from a 
maintenance point of view. It also doesn't feel justified, just so that we can 
continue to intermingle HFTP delegation tokens with HFTP delegation tokens. It 
seems to make much more sense to separate the HFTP tokens out and use the 
service field to refer to the HFTP address, in the way it was intended to be 
used.

I'm still looking through the patch, but wanted to provide the high level 
feedback.

Changing the semantics of JobClient.addAll isn't ok.

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764-2.patch, MAPREDUCE-2764.patch, 
> delegation.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Daryn Sharp (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13085754#comment-13085754
 ] 

Daryn Sharp commented on MAPREDUCE-2764:


Note the current patch is not intended for review.  I'm attempting to resolve 
the issues Jitendra cited by breaking the patch into multiple parts for easier 
review/discussion.

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Owen O'Malley (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084404#comment-13084404
 ] 

Owen O'Malley commented on MAPREDUCE-2764:
--

I think it is far cleaner to fix the hftp tokens to properly reflect that they 
are for hftp instead of hdfs. It removes all of the current guess work that is 
currently done.

hdfs tokens -> always use nn:rpc-port, only used for rpc
hftp tokens -> always use nn:http-port, only used for hftp

by having a renewer for each kind of token, they are simpler.

The only hiccup is needing to have the hftp client translate the token to hdfs 
to provide wire compatibility.

Of course the long term solution has to be to move to a serialization library 
that allows us to add/remove fields. Writable doesn't. :(



> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Daryn Sharp (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080320#comment-13080320
 ] 

Daryn Sharp commented on MAPREDUCE-2764:


I don't think the goal should be to minimize change in the hopes of reducing 
risk, but rather to improve the code base to reduce risk. First, a brief(ish) 
and polite rebuttal:
# Changes to namenode and rpc client were trivial.  Every token producer should 
not know how to properly encode the socket into a service string.  
{code}
-  String s = NameNode.getAddress(conf).getAddress().getHostAddress()
- + ":" + NameNode.getAddress(conf).getPort();
-  token.setService(new Text(s));
+  token.setService(NameNode.getAddress(conf));
{code}
# Changes to rpc client are trivial.  Just like the namenode changes, the 
encoding of the service is abstracted.
{code}
-  InetSocketAddress addr = remoteId.getAddress();
-  token = tokenSelector.selectToken(new Text(addr.getAddress()
-  .getHostAddress() + ":" + addr.getPort()), 
-  ticket.getTokens());
+  token = tokenSelector.selectToken(server, ticket.getTokens());
{code}
# The refactor of the token selectors is essentially creating a base class to 
eliminate all the copy-n-paste in the individual selectors.  There's not much 
risk there.  Adding URI support was very simple.

The additional cons to the counter-proposal:
* The token renewer will be hardcoded such that mapreduce will need to be 
recompiled when another filesystem is added.
* Mapred will require a mapping of token type to schemes.
* Can't simply subclass an existing filesystem with a new scheme w/o 
recompilation.
* The tokens and selectors are generics-based.  The type is used to do 
unchecked-casting, so adding the ability to change the type is playing with 
fire.
* HftpFilesystem should not be guessing the rpc port when the rpc port is in 
the original token.
* Is not a sustainable design pattern.

IMHO, the token renewer should be "dumb" and not require knowledge of every 
filesystem.  Ergo, all filesystem tokens should have the same type.  All 
filesystem tokens should be routed to their filesystem object.  The filesystem 
object handles renewal.  To solve the hftp/remote-hdfs token issue, I'd prefer 
for the hftp token to simply wrap/contain the remote dfs token instead of 
twiddling its fields.

Pros:
* Overall simple and clean(er) design
* No editing of a token type to scheme mapping for new filesystems
* No recompilation of mapreduce to add a filesystem 
* Eliminates brittle & risky copy-n-paste in the token producers
* Prepares the code to be more flexible and extensible with future service types
* Does not guess the remote rpc port

Cons:
* Slices across multiple components
* It's a bit more work, mainly due to cleanup of existing code

The "nice thing" about this type of low-level change is it will break 
immediately if implemented incorrectly...

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Jitendra Nath Pandey (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080263#comment-13080263
 ] 

Jitendra Nath Pandey commented on MAPREDUCE-2764:
-

I had a long discussion with Owen. Here is the new proposal:

- Add setKind method in the token
- Add a new kind for tokens i.e. HFTP
- HftpFileSystem client fetches the token and sets the kind to HFTP and the 
service to ip:http-port.
- Renewer determines the filesystem to use (hftp or DFS) using the kind of the 
token. The hftp port is also obtained from the token.
- Hftp client looks for HFTP tokens in the UGI when making an hftp call to nn.
- Hftp client sends the token in the url to the namenode. Before serializing 
the token in the url, Hftp client changes the service to ip:rpc-port and kind 
of the token back to HDFS.

Pros:
  No change to Namenode.
  No change to rpc.
  No change to token selectors.
  Most of the changes are confined to HftpFileSystem class.
  HftpFilesystem already has a mechanism to map the http port to rpc port, 
which it can use to set the service before encoding the token in the url.
  Changing the kind will make sure that this token is not accidentally used for 
rpc connection.

Cons:
  Token obtained over hftp, cannot be used over rpc by clients.

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Jitendra Nath Pandey (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080128#comment-13080128
 ] 

Jitendra Nath Pandey commented on MAPREDUCE-2764:
-

> I'll probably need to add a FileSystem.getDelegationToken-s- which will 
> return a collection of tokens.
 I didn't get that, how will multiple tokens solve this problem? Currently, 
only one token can be passed in the URL for hftp calls, and it is that token 
which needs to be renewed by JT and the same token is eventually used for rpc.

> Iff we are going back to copy-n-paste...
 There may be some code repeated in current code base. All I am recommending is 
to do that refactoring in a separate jira. My point was that, if we don't do 
that refactoring in this patch only DelegationTokenSelector will need to 
change. 

> The getServiceAuthority() is extracting the authority from the service if 
> it's a uri, else it uses the 
> service itself if it's not a uri.
>The the service is currently "authority", and I changed it to be either 
>"authority" or >"scheme://authority". I'd like to eventually move everything 
>to "scheme://authority" hence the >compatibility.

 The problem is that we can't use same authority for both rpc and hftp, because 
ports are different.

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Daryn Sharp (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080034#comment-13080034
 ] 

Daryn Sharp commented on MAPREDUCE-2764:


I just re-read your comments.  There is indeed a problem with hftp, but it's 
not insurmountable, and it's not incompatible with this change.

Other DFS filesystems, and h-s-ftp need only one token.  Hftp needs two tokens: 
http & https.  I think this explains the non-standard behavior of hftp setting 
up its own token renewal thread.

Unfortunately the existing {{FileSystem.getDelegationToken()}} only returns a 
single token.  It's not unreasonable to think that filesystems, like hftp, will 
require multiple tokens.  I'll probably need to add a 
{{FileSystem.getDelegationToken-s-}} which will return a collection of tokens.

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Daryn Sharp (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080014#comment-13080014
 ] 

Daryn Sharp commented on MAPREDUCE-2764:


bq. The token is used for rpc connection and when token selector does a match 
it needs the service to have rpc port. But if we store hftp uri in the token, 
the port will be http port. How is that being handled?

The {{getServiceAuthority()}} is extracting the authority from the service if 
it's a uri, else it uses the service itself if it's not a uri.

bq. One approach could be to store both issuer and service in the current 
service field.

The the service is currently "authority", and I changed it to be either 
"authority" or "scheme://authority".  I'd like to eventually move everything to 
"scheme://authority" hence the compatibility.

bq. I think the AbstractDelegationtionTokenSelector change and introduction of 
TokenMatcher is unnecessary.  BlockTokenSelector or JobTokenSelector shouldn't 
need to change at all.

That means the token selectors go back to copy-n-paste...  The base 
{{selectTokens()}} method filters the tokens to find the expected token kind.  
This eliminates the copy-n-paste code which limits the ability to alter the 
standard behavior.

The {{TokenMatcher}} is used to allow tokens, if those so chose, to provide 
custom selection criteria above and beyond the standard service match.  This is 
also used to match a RPC socket to its token, else it just adds even more 
copy-n-paste.

{{JobTokenSelector}} only removed methods now in the base class.  
{{BlockTokenSelector}} oddly doesn't care about service at all, so it's 
{{selectToken()}} is overridden to provide a {{TokenMatcher}} that ignores 
service.

bq. A change in hdfs's DelegationTokenSelector should suffice for this jira.

Iff we are going back to copy-n-paste...

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Jitendra Nath Pandey (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13079804#comment-13079804
 ] 

Jitendra Nath Pandey commented on MAPREDUCE-2764:
-

   The token is used for rpc connection and when token selector does a match it 
needs the service to have rpc port. But if we store hftp uri in the token, the 
port will be http port. How is that being handled?
   In my opinion the basic problem is that we are using the same service field 
for two different purposes. One purpose is where token should be used and other 
is to store how token was issued. That is why I suggested to introduce 
additional issuer field. But backward compatibility issue is a good point. One 
approach could be to store both issuer and service in the current service field.

  I think the AbstractDelegationtionTokenSelector change and introduction of 
TokenMatcher is unnecessary. BlockTokenSelector or JobTokenSelector shouldn't 
need to change at all. A change in hdfs's DelegationTokenSelector should 
suffice for this jira.
  
  
 
  

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Daryn Sharp (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13079681#comment-13079681
 ] 

Daryn Sharp commented on MAPREDUCE-2764:


The changes aren't entirely independent so I'm not sure if I can easily tease 
them apart, but I'll start trying to break it up into more jiras...  The token 
selectors were all just copy-n-paste, so I would have been further perpetrating 
unnecessary duplication if I didn't commonize them.  The {{HftpFileSystem}} 
changes were the fulcrum of this bug.

To help clarify the approach I took in case there is misunderstanding: I let 
the "service" be one of two formats: "scheme://authority" or "authority" (for 
compatibility).  I added {{getServiceAuthority}} in order to easily get the 
authority irregardless of the format.  The benefits to supporting both formats 
in the same field is to avoid unnecessary duplication of the same data, and 
more importantly to avoid introducing binary incompatibility to the tokens.  
Unless I'm mistaken, your recommendation for a new issuer field would have 
those downsides.  Are there other benefits that would outweigh those issues?


> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Jitendra Nath Pandey (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13079654#comment-13079654
 ] 

Jitendra Nath Pandey commented on MAPREDUCE-2764:
-

1. The patch introduces some refactoring around token selector, and also 
probably code cleanup in HftpFileSystem.java, but it would be good if that is 
done in a separate jira. This will keep this patch smaller and focused. 
2. I like the idea to store uri in the token which is issuer of the token. So 
going forward we have two things in the token:
 - service: where the token should be used. This should contain rpc port. This 
remains same as we have today, so no changes should be needed to set/getService 
APIs.
 - issuer: The uri of the file system that issued this token and add 
set/getIssuer APIs to the Token.
  Can we rename serviceUri to something like issuer or something else, just to 
differentiate it from service.
  So, the DelegationTokenRenewal looks at the issuer, while token selectors 
continue to use service, as they do today.

3. I will recommend the issuer should be set by every filesystem that 
implements getDelegationToken.
4. So we add only two APIs to Token: set/getIssuer. We don't need to add others 
like getServiceAuthority.
5. The patch didn't compile for me. Probably a typo in 
TestDelegationTokenRenewal.

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Daryn Sharp (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13079143#comment-13079143
 ] 

Daryn Sharp commented on MAPREDUCE-2764:


Oops, standard method to set service based InetSocketAddr will greatly ease the 
change for HADOOP-7510.

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

[jira] [Commented] (MAPREDUCE-2764) Fix renewal of dfs delegation tokens

[Daryn Sharp (JIRA)]

[ 
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13075991#comment-13075991
 ] 

Daryn Sharp commented on MAPREDUCE-2764:


The renewal problem can be solved far more easily, w/o false coupling, and w/o 
an assumption that hftp uses https.  A DFS issues and renews tokens.  The 
problem is the lack of traceability from a token to its origin DFS.  Setting a 
DFS DT's service field to be the DFS uri, instead of ip:port, will allow a 
trivial {{FileSystem.get}} to obtain the DFS.  All guessing is removed, and the 
hftp fs encapsulates that it's using https.

The RPC layer and the token selectors will require minor modification to use 
the authority of the uri in the service field.  The semantics of other tokens 
should not be affected.

> Fix renewal of dfs delegation tokens
> 
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
>  Issue Type: Bug
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Fix For: 0.20.205.0
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.  
> The problem is the JT's delegation token renewal code is built on brittle 
> assumptions.  The token's service field contains only the "ip:port" pair.  
> The renewal process assumes that the scheme must be hdfs.  If that fails due 
> to a {{VersionMismatchException}}, it tries https based on another assumption 
> that it must be hftp if it's not hdfs.  A number of other exceptions, most 
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since 
> it won't fallback to https.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira