Michael Gummelt created MAPREDUCE-6876:
------------------------------------------
Summary: FileInputFormat.listStatus should not fetch delegation
tokens
Key: MAPREDUCE-6876
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6876
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Michael Gummelt
{{FileInputFormat.listStatus}} fetches delegation tokens:
https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java#L213
AFAICT, this is unnecessary. {{listStatus}} doesn't delegate those tokens to
another process. This is causing issues described in the attached Spark
Kerberos ticket, because {{TokenCache.obtainTokensForNameNodes}}, which is used
to fetch the delegation tokens, assumes that certain MapReduce configuration
variables are set, which isn't true in the Spark calling code. This is a
separate problem, but nonetheless it wouldn't have arisen if {{listStatus}}
weren't fetching delegation tokens.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
