[
https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16416325#comment-16416325
]
Wangda Tan commented on YARN-8079:
----------------------------------
Thanks [~billie.rinaldi] / [~eyang] for suggestions.
I just reverted changes for the original ConfigFile and added a new code block
to handle two additional FileType: ARCHIVE/STATIC. Also added test cases.
[~eyang],
For the NFS suggestion, it makes sense for some scenarios, but not everybody
configs NFS in their cluster. Localization of static files is still an
important requirement. For example, user could export pre-trained model from
cluster A to cluster B.
Just uploaded ver.2 patch, please feel free to share your thoughts.
> YARN native service should respect source file of ConfigFile inside
> Service/Component spec
> ------------------------------------------------------------------------------------------
>
> Key: YARN-8079
> URL: https://issues.apache.org/jira/browse/YARN-8079
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Wangda Tan
> Assignee: Wangda Tan
> Priority: Blocker
> Attachments: YARN-8079.001.patch, YARN-8079.002.patch
>
>
> Currently, {{srcFile}} is not respected. {{ProviderUtils}} doesn't properly
> read srcFile, instead it always construct {{remoteFile}} by using
> componentDir and fileName of {{destFile}}:
> {code}
> Path remoteFile = new Path(compInstanceDir, fileName);
> {code}
> To me it is a common use case which services have some files existed in HDFS
> and need to be localized when components get launched. (For example, if we
> want to serve a Tensorflow model, we need to localize Tensorflow model
> (typically not huge, less than GB) to local disk. Otherwise launched docker
> container has to access HDFS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
