[
https://issues.apache.org/jira/browse/MAPREDUCE-6876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15969499#comment-15969499
]
Jason Lowe commented on MAPREDUCE-6876:
---------------------------------------
When you say {{listStatus}} do you really mean {{getSplits}}? We could move
the delegation token stuff out of {{listStatus}} and into a separate method
that {{getSplits}} calls internally to get the tokens, but I'm assuming that is
not going to solve your issue.
Here's what I think is being proposed, please correct me if I missed something:
# Add an API method to InputFormat, e.g.: {{getSplitsTokens}}, that the job
submitter code would call to obtain tokens needed for the input. By default,
the method does nothing.
# Input formats would need to move the token-fetching code out of their
{{getSplits}} code path and into the new API method to avoid breaking Spark
> FileInputFormat.listStatus should not fetch delegation tokens
> -------------------------------------------------------------
>
> Key: MAPREDUCE-6876
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6876
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Reporter: Michael Gummelt
>
> {{FileInputFormat.listStatus}} fetches delegation tokens:
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java#L213
> AFAICT, this is unnecessary. {{listStatus}} doesn't delegate those tokens to
> another process. This is causing issues described in the attached Spark
> Kerberos ticket, because {{TokenCache.obtainTokensForNameNodes}}, which is
> used to fetch the delegation tokens, assumes that certain MapReduce
> configuration variables are set, which isn't true in the Spark calling code.
> This is a separate problem, but nonetheless it wouldn't have arisen if
> {{listStatus}} weren't fetching delegation tokens.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
