[
https://issues.apache.org/jira/browse/YARN-10311?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17138189#comment-17138189
]
Prabhu Joseph commented on YARN-10311:
--------------------------------------
Hi [~eyang], [~BilwaST]
Have faced same issue. This issue (Log Aggregation fails for job) happens for
any YARN Job which uses a FileSystem for input / output path (fs.defaultFS)
other than the one configured for YARN logs
(yarn.nodemanager.remote-app-log-dir).
For Example:
MapReduce job input path is set to HDFS where as
yarn.nodemanager.remote-app-log-dir set to ABFS will succeed but Log
Aggregation will fail as NM won't have ABFS Token from end user credentials to
aggregate logs.
MR Jobs & Spark Jobs can set the other filesystems for which delegation token
will be obtained by client side using mapreduce.job.hdfs-servers and
spark.yarn.access.hadoopFileSystems respectively.
But this option is not there in Tez, Yarn Native Service and custom YARN
Application Type. I think it will be useful to have a generic fix which can
obtain token for the filesystem specified at
yarn.nodemanager.remote-app-log-dir for any YARN Job.
> Yarn Service should support obtaining tokens from multiple name services
> ------------------------------------------------------------------------
>
> Key: YARN-10311
> URL: https://issues.apache.org/jira/browse/YARN-10311
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Bilwa S T
> Assignee: Bilwa S T
> Priority: Major
> Attachments: YARN-10311.001.patch, YARN-10311.002.patch
>
>
> Currently yarn services support single name service tokens. We can add a new
> conf called
> "yarn.service.hdfs-servers" for supporting this
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
