[ https://issues.apache.org/jira/browse/YARN-8714?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16692696#comment-16692696 ]
Zhankun Tang edited comment on YARN-8714 at 11/20/18 5:55 AM: -------------------------------------------------------------- [~liuxun323], [~leftnoteasy], [~yuan_zac]. Updated a new version. More tests are in progress. But please help to review in caseĀ our direction is not right. For all ps and worker component, it provides --localization to support both HDFS and local directory (may download, then zip it in temp dir and then upload to HDFS) as the first part of the parameter. And it also supports mount into the container with permission(default is rw) Goals are: {code:java} --localization hdfs:///user/yarn/mydir:.:ro # download, zip and upload the zip file to HDFS. then mount into container word dir's "mydir" folder. {PWD}/mydir --localization /user/yarn/mydir2:/opt/dir2 # zip local dir and upload the zip file to HDFS. set mount into container's /opt/dir2 --localization /user/yarn/script1.py:.:ro # upload a script to HDFS and set mount into container {PWD}/script1.py --localization /user/yarn/script1.py:/opt/script2.py:rw # upload and mount into container /opt/script2.py {code} was (Author: tangzhankun): [~liuxun323], [~leftnoteasy], [~yuan_zac]. Updated a new version. More tests are needed. But please help to review incase the changes are too big. For all ps and worker component, it provides --localization to support both HDFS and local directory (may download, then zip it in temp dir and then upload to HDFS) as the first part of the parameter. And it also supports mount into the container with permission(default is rw) Goals are: {code:java} --localization hdfs:///user/yarn/mydir:.:ro # download, zip and upload the zip file to HDFS. then mount into container word dir's "mydir" folder. {PWD}/mydir --localization /user/yarn/mydir2:/opt/dir2 # zip local dir and upload the zip file to HDFS. set mount into container's /opt/dir2 --localization /user/yarn/script1.py:.:ro # upload a script to HDFS and set mount into container {PWD}/script1.py --localization /user/yarn/script1.py:/opt/script2.py:rw # upload and mount into container /opt/script2.py {code} > [Submarine] Support files/tarballs to be localized for a training job. > ---------------------------------------------------------------------- > > Key: YARN-8714 > URL: https://issues.apache.org/jira/browse/YARN-8714 > Project: Hadoop YARN > Issue Type: Sub-task > Reporter: Wangda Tan > Assignee: Zhankun Tang > Priority: Major > Attachments: YARN-8714-WIP1-trunk-001.patch, > YARN-8714-WIP1-trunk-002.patch > > > See > https://docs.google.com/document/d/199J4pB3blqgV9SCNvBbTqkEoQdjoyGMjESV4MktCo0k/edit#heading=h.vkxp9edl11m7, > {{job run --localizations ...}} -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org