[
https://issues.apache.org/jira/browse/YARN-8079?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16482076#comment-16482076
]
Eric Yang edited comment on YARN-8079 at 5/21/18 12:20 AM:
-----------------------------------------------------------
[~leftnoteasy] When the files are placed in resources directory, patch 10
implementation prevents mistake to overwrite system level generated files, such
as .token file, and launch_container.sh. However, this design can created
inconvenience for some users because existing Hadoop workload may already be
using the top level localized directory instead of resource directory. We may
not need to worry about launch_container.sh getting overwritten because
container-executor generates the file after static files are localized. Apps
will try to avoid .token files because they can not contact HDFS from
containers, if they overwrites the token files.
With resources directory, it maybe easier for end user to specify a single
relative directory to bind-mount instead of specifying individual files to
bind-mount in yarnfile. By removing the resources directory, user will need to
think a bit more on how to manage the bind-mount directories to reduce wordy
syntax.
With both approaches considered, it all comes down to usability of which
approach is easiest to use, while not creating too much clutters. In summary,
it might be safe to remove the requirement of "resources" directory from my
point of view.
was (Author: eyang):
[~leftnoteasy] When the files are placed in resources directory, patch 10
implementation prevents mistake to overwrite system level generated files, such
as .token file, and launch_container.sh. However, this design can created
inconvenience for some users because existing Hadoop workload may already be
using the top level localized directory instead of resource directory. We may
not need to worry about launch_container.sh getting overwritten because
container-executor generates the file after static files are localized. Apps
will try to avoid .token files because they can not contact HDFS from
containers, if they overwrites the token files. In summary, it is likely safe
to remove the requirement of "resources" directory from my point of view.
> Support static and archive unmodified local resources in service AM
> -------------------------------------------------------------------
>
> Key: YARN-8079
> URL: https://issues.apache.org/jira/browse/YARN-8079
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Wangda Tan
> Assignee: Suma Shivaprasad
> Priority: Critical
> Fix For: 3.2.0, 3.1.1
>
> Attachments: YARN-8079.001.patch, YARN-8079.002.patch,
> YARN-8079.003.patch, YARN-8079.004.patch, YARN-8079.005.patch,
> YARN-8079.006.patch, YARN-8079.007.patch, YARN-8079.008.patch,
> YARN-8079.009.patch, YARN-8079.010.patch
>
>
> Currently, {{srcFile}} is not respected. {{ProviderUtils}} doesn't properly
> read srcFile, instead it always construct {{remoteFile}} by using
> componentDir and fileName of {{destFile}}:
> {code}
> Path remoteFile = new Path(compInstanceDir, fileName);
> {code}
> To me it is a common use case which services have some files existed in HDFS
> and need to be localized when components get launched. (For example, if we
> want to serve a Tensorflow model, we need to localize Tensorflow model
> (typically not huge, less than GB) to local disk. Otherwise launched docker
> container has to access HDFS.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
