[
https://issues.apache.org/jira/browse/MAPREDUCE-2764?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13080034#comment-13080034
]
Daryn Sharp commented on MAPREDUCE-2764:
----------------------------------------
I just re-read your comments. There is indeed a problem with hftp, but it's
not insurmountable, and it's not incompatible with this change.
Other DFS filesystems, and h-s-ftp need only one token. Hftp needs two tokens:
http & https. I think this explains the non-standard behavior of hftp setting
up its own token renewal thread.
Unfortunately the existing {{FileSystem.getDelegationToken()}} only returns a
single token. It's not unreasonable to think that filesystems, like hftp, will
require multiple tokens. I'll probably need to add a
{{FileSystem.getDelegationToken-s-}} which will return a collection of tokens.
> Fix renewal of dfs delegation tokens
> ------------------------------------
>
> Key: MAPREDUCE-2764
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2764
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Reporter: Daryn Sharp
> Assignee: Daryn Sharp
> Fix For: 0.20.205.0
>
> Attachments: MAPREDUCE-2764.patch
>
>
> The JT may have issues renewing hftp tokens which disrupt long distcp jobs.
> The problem is the JT's delegation token renewal code is built on brittle
> assumptions. The token's service field contains only the "ip:port" pair.
> The renewal process assumes that the scheme must be hdfs. If that fails due
> to a {{VersionMismatchException}}, it tries https based on another assumption
> that it must be hftp if it's not hdfs. A number of other exceptions, most
> commonly {{IOExceptions}}, can be generated which fouls up the renewal since
> it won't fallback to https.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
