[
https://issues.apache.org/jira/browse/MAPREDUCE-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13430564#comment-13430564
]
Jason Lowe commented on MAPREDUCE-3782:
---------------------------------------
Took a closer look at this. The log message about getting a delegation token
for HDFS is not in error, as that token is needed to setup the staging
directory for the job. So that is working as-expected.
Teragen failing to get a delegation token for webhdfs *is* a bug in
TeraOutputFormat, as it is responsible for grabbing any necessary delegation
tokens in the {{checkOutputSpecs}} method.
Terasort grabs two tokens, one from webhdfs (for the input files and would also
for the output directory once TeraOutputFormat is fixed) and one from HDFS for
the job staging area. We won't see two delegation token messages for the same
filesystem since it doesn't grab a new token if it already has one.
The range error is a known problem that has been already fixed, see HDFS-3101.
So to sum up, I think there's only one remaining bug here: TeraOutputFormat is
not grabbing a delegation token for the output directory when
{{checkOutputSpecs}} is called.
> teragen terasort jobs fail when using webhdfs://
> -------------------------------------------------
>
> Key: MAPREDUCE-3782
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3782
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Arpit Gupta
> Priority: Critical
>
> When running a teragen job with a webhdfs:// url the delegation token that is
> retrieved is an hdfs delegation token.
> And the subsequent terasort job on the output fails with java io exception
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
