[jira] [Commented] (YARN-7799) YARN Service dependency follow up work
[ https://issues.apache.org/jira/browse/YARN-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460205#comment-16460205 ] Hudson commented on YARN-7799: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #14101 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/14101/]) YARN-7799. Improved YARN service jar file handling.(eyang: rev 24eeea8b18749e02ea05b03eb18e3cf0455077c5) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/client/ServiceClient.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/utils/CoreFileSystem.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/client/TestServiceCLI.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/pom.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/yarn-service/Configurations.md > YARN Service dependency follow up work > -- > > Key: YARN-7799 > URL: https://issues.apache.org/jira/browse/YARN-7799 > Project: Hadoop YARN > Issue Type: Bug > Components: client, resourcemanager >Reporter: Gour Saha >Assignee: Billie Rinaldi >Priority: Critical > Attachments: YARN-7799.1.patch, YARN-7799.2.patch, YARN-7799.3.patch, > YARN-7799.4.patch, YARN-7799.5.patch > > > As per [~jianhe] these are some followup items that make sense to do after > YARN-7766. Quoting Jian's comment below - > Currently, if user doesn't supply location when run yarn app > -enableFastLaunch, the jars will be put under this location > {code} > hdfs:///yarn-services//service-dep.tar.gz > {code} > Since API server is embedded in RM, should RM look for this location too if > "yarn.service.framework.path" is not specified ? > And if "yarn.service.framework.path" is not specified and still the file > doesn't exist at above default location, I think RM can try to upload the > jars to above default location instead, currently RM is uploading the jars to > the location defined by below code. This folder is per app and also > inconsistent with CLI location. > {code} > protected Path addJarResource(String serviceName, > MaplocalResources) > throws IOException, SliderException { > Path libPath = fs.buildClusterDirPath(serviceName); > {code} > By doing this, the next time a submission request comes, RM doesn't need to > upload the jars again. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7799) YARN Service dependency follow up work
[ https://issues.apache.org/jira/browse/YARN-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460180#comment-16460180 ] Billie Rinaldi commented on YARN-7799: -- Thanks [~eyang] and [~gsaha]! Gour, that's right, the user will only be allowed to upload the tarball if they appear on the yarn.admin.acl or dfs.cluster.administrators lists. So for a typical install it could be the yarn user or the hdfs user. The yarn.admin.acl property does default to *, though, so that value would allow any user to perform the upload. (On the other hand, the default upload location under /yarn-services would not be writable by all users.) > YARN Service dependency follow up work > -- > > Key: YARN-7799 > URL: https://issues.apache.org/jira/browse/YARN-7799 > Project: Hadoop YARN > Issue Type: Bug > Components: client, resourcemanager >Reporter: Gour Saha >Assignee: Billie Rinaldi >Priority: Critical > Attachments: YARN-7799.1.patch, YARN-7799.2.patch, YARN-7799.3.patch, > YARN-7799.4.patch, YARN-7799.5.patch > > > As per [~jianhe] these are some followup items that make sense to do after > YARN-7766. Quoting Jian's comment below - > Currently, if user doesn't supply location when run yarn app > -enableFastLaunch, the jars will be put under this location > {code} > hdfs:///yarn-services//service-dep.tar.gz > {code} > Since API server is embedded in RM, should RM look for this location too if > "yarn.service.framework.path" is not specified ? > And if "yarn.service.framework.path" is not specified and still the file > doesn't exist at above default location, I think RM can try to upload the > jars to above default location instead, currently RM is uploading the jars to > the location defined by below code. This folder is per app and also > inconsistent with CLI location. > {code} > protected Path addJarResource(String serviceName, > MaplocalResources) > throws IOException, SliderException { > Path libPath = fs.buildClusterDirPath(serviceName); > {code} > By doing this, the next time a submission request comes, RM doesn't need to > upload the jars again. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7799) YARN Service dependency follow up work
[ https://issues.apache.org/jira/browse/YARN-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16460167#comment-16460167 ] Gour Saha commented on YARN-7799: - Ok, I think it makes sense. We can do the system service later. So just to make sure I understand the latest patch, only if a user with HDFS admin ACLs (like hdfs user) submits a service launch is when the dependency tarball will get uploaded, right? > YARN Service dependency follow up work > -- > > Key: YARN-7799 > URL: https://issues.apache.org/jira/browse/YARN-7799 > Project: Hadoop YARN > Issue Type: Bug > Components: client, resourcemanager >Reporter: Gour Saha >Assignee: Billie Rinaldi >Priority: Critical > Attachments: YARN-7799.1.patch, YARN-7799.2.patch, YARN-7799.3.patch, > YARN-7799.4.patch, YARN-7799.5.patch > > > As per [~jianhe] these are some followup items that make sense to do after > YARN-7766. Quoting Jian's comment below - > Currently, if user doesn't supply location when run yarn app > -enableFastLaunch, the jars will be put under this location > {code} > hdfs:///yarn-services//service-dep.tar.gz > {code} > Since API server is embedded in RM, should RM look for this location too if > "yarn.service.framework.path" is not specified ? > And if "yarn.service.framework.path" is not specified and still the file > doesn't exist at above default location, I think RM can try to upload the > jars to above default location instead, currently RM is uploading the jars to > the location defined by below code. This folder is per app and also > inconsistent with CLI location. > {code} > protected Path addJarResource(String serviceName, > MaplocalResources) > throws IOException, SliderException { > Path libPath = fs.buildClusterDirPath(serviceName); > {code} > By doing this, the next time a submission request comes, RM doesn't need to > upload the jars again. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7799) YARN Service dependency follow up work
[ https://issues.apache.org/jira/browse/YARN-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458688#comment-16458688 ] genericqa commented on YARN-7799: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 49s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 12s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 4s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 58s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 34s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 9m 12s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 45s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 33s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 33s{color} | {color:green} hadoop-yarn-services-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 13s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 24s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 77m 27s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd | | JIRA Issue | YARN-7799 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12921255/YARN-7799.5.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit
[jira] [Commented] (YARN-7799) YARN Service dependency follow up work
[ https://issues.apache.org/jira/browse/YARN-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458225#comment-16458225 ] genericqa commented on YARN-7799: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 32s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 33s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 31s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 22s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 4s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 13m 5s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 45s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 46s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 40s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 18s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 20s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 57 unchanged - 0 fixed = 58 total (was 57) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 8s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 54s{color} | {color:red} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core generated 1 new + 0 unchanged - 0 fixed = 1 total (was 0) {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 44s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 10m 31s{color} | {color:green} hadoop-yarn-services-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 20s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 35s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 85m 50s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | FindBugs |
[jira] [Commented] (YARN-7799) YARN Service dependency follow up work
[ https://issues.apache.org/jira/browse/YARN-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16458207#comment-16458207 ] Billie Rinaldi commented on YARN-7799: -- I don't think we can avoid checking on every launch. If the file gets removed for some reason, the RM should still upload the jars and launch the service. It's only a file existence check, so that isn't too much work per service (especially as compared to uploading jars for each service, which was one of the possible behaviors before this patch). We could still consider adding a system service to perform enableFastLaunch in a future ticket, which would make the first app launched by the RM faster. I think we would need YARN-8080 to make this work. > YARN Service dependency follow up work > -- > > Key: YARN-7799 > URL: https://issues.apache.org/jira/browse/YARN-7799 > Project: Hadoop YARN > Issue Type: Bug > Components: client, resourcemanager >Reporter: Gour Saha >Assignee: Billie Rinaldi >Priority: Critical > Attachments: YARN-7799.1.patch, YARN-7799.2.patch, YARN-7799.3.patch, > YARN-7799.4.patch > > > As per [~jianhe] these are some followup items that make sense to do after > YARN-7766. Quoting Jian's comment below - > Currently, if user doesn't supply location when run yarn app > -enableFastLaunch, the jars will be put under this location > {code} > hdfs:///yarn-services//service-dep.tar.gz > {code} > Since API server is embedded in RM, should RM look for this location too if > "yarn.service.framework.path" is not specified ? > And if "yarn.service.framework.path" is not specified and still the file > doesn't exist at above default location, I think RM can try to upload the > jars to above default location instead, currently RM is uploading the jars to > the location defined by below code. This folder is per app and also > inconsistent with CLI location. > {code} > protected Path addJarResource(String serviceName, > MaplocalResources) > throws IOException, SliderException { > Path libPath = fs.buildClusterDirPath(serviceName); > {code} > By doing this, the next time a submission request comes, RM doesn't need to > upload the jars again. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7799) YARN Service dependency follow up work
[ https://issues.apache.org/jira/browse/YARN-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457762#comment-16457762 ] Gour Saha commented on YARN-7799: - {quote}Oh dear, I think patch 2 has messed up file permissions in the precommit jenkins workspace. {quote} This is funĀ :) > YARN Service dependency follow up work > -- > > Key: YARN-7799 > URL: https://issues.apache.org/jira/browse/YARN-7799 > Project: Hadoop YARN > Issue Type: Bug > Components: client, resourcemanager >Reporter: Gour Saha >Assignee: Billie Rinaldi >Priority: Critical > Attachments: YARN-7799.1.patch, YARN-7799.2.patch, YARN-7799.3.patch > > > As per [~jianhe] these are some followup items that make sense to do after > YARN-7766. Quoting Jian's comment below - > Currently, if user doesn't supply location when run yarn app > -enableFastLaunch, the jars will be put under this location > {code} > hdfs:///yarn-services//service-dep.tar.gz > {code} > Since API server is embedded in RM, should RM look for this location too if > "yarn.service.framework.path" is not specified ? > And if "yarn.service.framework.path" is not specified and still the file > doesn't exist at above default location, I think RM can try to upload the > jars to above default location instead, currently RM is uploading the jars to > the location defined by below code. This folder is per app and also > inconsistent with CLI location. > {code} > protected Path addJarResource(String serviceName, > MaplocalResources) > throws IOException, SliderException { > Path libPath = fs.buildClusterDirPath(serviceName); > {code} > By doing this, the next time a submission request comes, RM doesn't need to > upload the jars again. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7799) YARN Service dependency follow up work
[ https://issues.apache.org/jira/browse/YARN-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457761#comment-16457761 ] Gour Saha commented on YARN-7799: - Just thinking aloud - to make this simple and to avoid checking on every service launch request of every user, do you think we can seed a simple system service as the HDFS user (even the sleeper service will do and async mode should be fine too)? During RM start, the dependency tarball will get created and uploaded as HDFS user and be available for all services going forward. We can even give some special name to this system service spec (say dependency-loader.yarnfile or fast-launch-enabler.yarnfile or something better) and treat it as a special service, such that on every RM restart (presuming jars might have been updated) we can restart this service which will fire enableFastLaunch with overwrite set to true. Thoughts? /cc [~rohithsharma] (ref: YARN-8048) > YARN Service dependency follow up work > -- > > Key: YARN-7799 > URL: https://issues.apache.org/jira/browse/YARN-7799 > Project: Hadoop YARN > Issue Type: Bug > Components: client, resourcemanager >Reporter: Gour Saha >Assignee: Billie Rinaldi >Priority: Critical > Attachments: YARN-7799.1.patch, YARN-7799.2.patch, YARN-7799.3.patch > > > As per [~jianhe] these are some followup items that make sense to do after > YARN-7766. Quoting Jian's comment below - > Currently, if user doesn't supply location when run yarn app > -enableFastLaunch, the jars will be put under this location > {code} > hdfs:///yarn-services//service-dep.tar.gz > {code} > Since API server is embedded in RM, should RM look for this location too if > "yarn.service.framework.path" is not specified ? > And if "yarn.service.framework.path" is not specified and still the file > doesn't exist at above default location, I think RM can try to upload the > jars to above default location instead, currently RM is uploading the jars to > the location defined by below code. This folder is per app and also > inconsistent with CLI location. > {code} > protected Path addJarResource(String serviceName, > MaplocalResources) > throws IOException, SliderException { > Path libPath = fs.buildClusterDirPath(serviceName); > {code} > By doing this, the next time a submission request comes, RM doesn't need to > upload the jars again. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7799) YARN Service dependency follow up work
[ https://issues.apache.org/jira/browse/YARN-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457706#comment-16457706 ] Billie Rinaldi commented on YARN-7799: -- Oh dear, I think patch 2 has messed up file permissions in the precommit jenkins workspace. I'm reaching out to infra to try to get this fixed. > YARN Service dependency follow up work > -- > > Key: YARN-7799 > URL: https://issues.apache.org/jira/browse/YARN-7799 > Project: Hadoop YARN > Issue Type: Bug > Components: client, resourcemanager >Reporter: Gour Saha >Assignee: Billie Rinaldi >Priority: Critical > Attachments: YARN-7799.1.patch, YARN-7799.2.patch, YARN-7799.3.patch > > > As per [~jianhe] these are some followup items that make sense to do after > YARN-7766. Quoting Jian's comment below - > Currently, if user doesn't supply location when run yarn app > -enableFastLaunch, the jars will be put under this location > {code} > hdfs:///yarn-services//service-dep.tar.gz > {code} > Since API server is embedded in RM, should RM look for this location too if > "yarn.service.framework.path" is not specified ? > And if "yarn.service.framework.path" is not specified and still the file > doesn't exist at above default location, I think RM can try to upload the > jars to above default location instead, currently RM is uploading the jars to > the location defined by below code. This folder is per app and also > inconsistent with CLI location. > {code} > protected Path addJarResource(String serviceName, > MaplocalResources) > throws IOException, SliderException { > Path libPath = fs.buildClusterDirPath(serviceName); > {code} > By doing this, the next time a submission request comes, RM doesn't need to > upload the jars again. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7799) YARN Service dependency follow up work
[ https://issues.apache.org/jira/browse/YARN-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457170#comment-16457170 ] genericqa commented on YARN-7799: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 25s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 26m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 47s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 13s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 6s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 33s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 47s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 15s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 48s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 44s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 44s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 16s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 1 new + 56 unchanged - 0 fixed = 57 total (was 56) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 26s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 52s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 39s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 11m 12s{color} | {color:red} hadoop-yarn-services-core in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 22s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 35s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 86m 12s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.service.client.TestServiceCLI | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:abb62dd | | JIRA Issue | YARN-7799 | | JIRA
[jira] [Commented] (YARN-7799) YARN Service dependency follow up work
[ https://issues.apache.org/jira/browse/YARN-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16457065#comment-16457065 ] Billie Rinaldi commented on YARN-7799: -- Attached patch 2 based on the discussion. In the case when the calling user is in yarn.admin.acl, we don't need to check whether yarn.service.framework.path has been preconfigured. If the hdfs admin user has given the yarn user permission to write to /yarn-services, the file can be uploaded. If not, the upload will fail. Oh, I just noticed I forgot to check for executable permission on the parent directories. I'll upload patch 3 for that. > YARN Service dependency follow up work > -- > > Key: YARN-7799 > URL: https://issues.apache.org/jira/browse/YARN-7799 > Project: Hadoop YARN > Issue Type: Bug > Components: client, resourcemanager >Reporter: Gour Saha >Assignee: Billie Rinaldi >Priority: Critical > Attachments: YARN-7799.1.patch, YARN-7799.2.patch > > > As per [~jianhe] these are some followup items that make sense to do after > YARN-7766. Quoting Jian's comment below - > Currently, if user doesn't supply location when run yarn app > -enableFastLaunch, the jars will be put under this location > {code} > hdfs:///yarn-services//service-dep.tar.gz > {code} > Since API server is embedded in RM, should RM look for this location too if > "yarn.service.framework.path" is not specified ? > And if "yarn.service.framework.path" is not specified and still the file > doesn't exist at above default location, I think RM can try to upload the > jars to above default location instead, currently RM is uploading the jars to > the location defined by below code. This folder is per app and also > inconsistent with CLI location. > {code} > protected Path addJarResource(String serviceName, > MaplocalResources) > throws IOException, SliderException { > Path libPath = fs.buildClusterDirPath(serviceName); > {code} > By doing this, the next time a submission request comes, RM doesn't need to > upload the jars again. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7799) YARN Service dependency follow up work
[ https://issues.apache.org/jira/browse/YARN-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456808#comment-16456808 ] Eric Yang commented on YARN-7799: - [~billie.rinaldi] The summary of our discussion: 1. We can check prefix directory of yarn.service.framework.path to ensure all sub-directories are world readable and executable to ensure other user can access this path. 2. If the calling user to -enableFastLaunch is one of yarn.admin.acl, and yarn.service.framework.path is pre-conigured. User is allowed to upload service-dep.tar.gz. 3. If the calling user is dfs.cluster.administrators, user is allowed to upload service-dep.tar.gz. 4. Auto-upload follows the same logic. > YARN Service dependency follow up work > -- > > Key: YARN-7799 > URL: https://issues.apache.org/jira/browse/YARN-7799 > Project: Hadoop YARN > Issue Type: Bug > Components: client, resourcemanager >Reporter: Gour Saha >Assignee: Billie Rinaldi >Priority: Critical > Attachments: YARN-7799.1.patch > > > As per [~jianhe] these are some followup items that make sense to do after > YARN-7766. Quoting Jian's comment below - > Currently, if user doesn't supply location when run yarn app > -enableFastLaunch, the jars will be put under this location > {code} > hdfs:///yarn-services//service-dep.tar.gz > {code} > Since API server is embedded in RM, should RM look for this location too if > "yarn.service.framework.path" is not specified ? > And if "yarn.service.framework.path" is not specified and still the file > doesn't exist at above default location, I think RM can try to upload the > jars to above default location instead, currently RM is uploading the jars to > the location defined by below code. This folder is per app and also > inconsistent with CLI location. > {code} > protected Path addJarResource(String serviceName, > MaplocalResources) > throws IOException, SliderException { > Path libPath = fs.buildClusterDirPath(serviceName); > {code} > By doing this, the next time a submission request comes, RM doesn't need to > upload the jars again. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7799) YARN Service dependency follow up work
[ https://issues.apache.org/jira/browse/YARN-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16456647#comment-16456647 ] Billie Rinaldi commented on YARN-7799: -- {quote}Can this new feature create a problem with prefix directory file permission that prevent other people from accessing the dependency jar file? {quote} This is a good point. Typically the hdfs user would upload the dependency tarball with the enableFastLaunch command and other users would only be allowed to read the file. If we make the change to allow the RM to automatically upload the file, at a minimum we'd be changing how the directory permissions would need to be set up, since the yarn user would need to be given permission to write the file. I was reluctant to make the auto upload change anyway, so I am thinking about removing this change from the patch and only fixing the default behavior of enableFastLaunch. (Removing auto upload would only be a one line change in the patch, and it would also address Gour's suggestions.) What do you think, [~gsaha] [~eyang]? > YARN Service dependency follow up work > -- > > Key: YARN-7799 > URL: https://issues.apache.org/jira/browse/YARN-7799 > Project: Hadoop YARN > Issue Type: Bug > Components: client, resourcemanager >Reporter: Gour Saha >Assignee: Billie Rinaldi >Priority: Critical > Attachments: YARN-7799.1.patch > > > As per [~jianhe] these are some followup items that make sense to do after > YARN-7766. Quoting Jian's comment below - > Currently, if user doesn't supply location when run yarn app > -enableFastLaunch, the jars will be put under this location > {code} > hdfs:///yarn-services//service-dep.tar.gz > {code} > Since API server is embedded in RM, should RM look for this location too if > "yarn.service.framework.path" is not specified ? > And if "yarn.service.framework.path" is not specified and still the file > doesn't exist at above default location, I think RM can try to upload the > jars to above default location instead, currently RM is uploading the jars to > the location defined by below code. This folder is per app and also > inconsistent with CLI location. > {code} > protected Path addJarResource(String serviceName, > MaplocalResources) > throws IOException, SliderException { > Path libPath = fs.buildClusterDirPath(serviceName); > {code} > By doing this, the next time a submission request comes, RM doesn't need to > upload the jars again. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7799) YARN Service dependency follow up work
[ https://issues.apache.org/jira/browse/YARN-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16455986#comment-16455986 ] Gour Saha commented on YARN-7799: - [~billie.rinaldi] Few points - h5. ServiceClient.java {code} System.out.println(String.format( "Dependency libs are already uploaded to %s.", dependencyLibTarGzip .toUri())); {code} Seems like this is going to be printed for every call to launch/start an app. Is there some way we can suppress this and print only when enableFastLaunch is called? h5. Configurations.md {code} |yarn.service.framework.path | HDFS URI of the service AM dependency tarball. When no file exists at this location, AM dependencies will be uploaded each time a service is started or launched. If unspecified, value will be assumed to be /yarn-services/$\{hadoop.version\}/service-dep.tar.gz.| {code} With this patch, I think this statement won't be valid anymore - "AM dependencies will be uploaded each time a service is started or launched". It will be uploaded only for the first service launch, right? > YARN Service dependency follow up work > -- > > Key: YARN-7799 > URL: https://issues.apache.org/jira/browse/YARN-7799 > Project: Hadoop YARN > Issue Type: Bug > Components: client, resourcemanager >Reporter: Gour Saha >Assignee: Billie Rinaldi >Priority: Critical > Attachments: YARN-7799.1.patch > > > As per [~jianhe] these are some followup items that make sense to do after > YARN-7766. Quoting Jian's comment below - > Currently, if user doesn't supply location when run yarn app > -enableFastLaunch, the jars will be put under this location > {code} > hdfs:///yarn-services//service-dep.tar.gz > {code} > Since API server is embedded in RM, should RM look for this location too if > "yarn.service.framework.path" is not specified ? > And if "yarn.service.framework.path" is not specified and still the file > doesn't exist at above default location, I think RM can try to upload the > jars to above default location instead, currently RM is uploading the jars to > the location defined by below code. This folder is per app and also > inconsistent with CLI location. > {code} > protected Path addJarResource(String serviceName, > MaplocalResources) > throws IOException, SliderException { > Path libPath = fs.buildClusterDirPath(serviceName); > {code} > By doing this, the next time a submission request comes, RM doesn't need to > upload the jars again. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7799) YARN Service dependency follow up work
[ https://issues.apache.org/jira/browse/YARN-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454824#comment-16454824 ] Eric Yang commented on YARN-7799: - [~billie.rinaldi] It looks like there was a conflict of code changes with YARN-7939 and patch 7799.1. Could you rebase the patch to current trunk? Thanks > YARN Service dependency follow up work > -- > > Key: YARN-7799 > URL: https://issues.apache.org/jira/browse/YARN-7799 > Project: Hadoop YARN > Issue Type: Bug > Components: client, resourcemanager >Reporter: Gour Saha >Assignee: Billie Rinaldi >Priority: Critical > Attachments: YARN-7799.1.patch > > > As per [~jianhe] these are some followup items that make sense to do after > YARN-7766. Quoting Jian's comment below - > Currently, if user doesn't supply location when run yarn app > -enableFastLaunch, the jars will be put under this location > {code} > hdfs:///yarn-services//service-dep.tar.gz > {code} > Since API server is embedded in RM, should RM look for this location too if > "yarn.service.framework.path" is not specified ? > And if "yarn.service.framework.path" is not specified and still the file > doesn't exist at above default location, I think RM can try to upload the > jars to above default location instead, currently RM is uploading the jars to > the location defined by below code. This folder is per app and also > inconsistent with CLI location. > {code} > protected Path addJarResource(String serviceName, > MaplocalResources) > throws IOException, SliderException { > Path libPath = fs.buildClusterDirPath(serviceName); > {code} > By doing this, the next time a submission request comes, RM doesn't need to > upload the jars again. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7799) YARN Service dependency follow up work
[ https://issues.apache.org/jira/browse/YARN-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16454552#comment-16454552 ] genericqa commented on YARN-7799: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 28s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 21s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 43s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 8s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 50s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 39s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 36s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 34s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 30s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 0s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 25s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s{color} | {color:blue} Skipped patched modules with no Java source: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 51s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 45s{color} | {color:green} hadoop-yarn-services-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 14s{color} | {color:green} hadoop-yarn-site in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 29s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 76m 6s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:b78c94f | | JIRA Issue | YARN-7799 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12920823/YARN-7799.1.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 2713494048e8 4.4.0-64-generic #85-Ubuntu SMP Mon Feb 20 11:50:30 UTC 2017 x86_64 x86_64 x86_64
[jira] [Commented] (YARN-7799) YARN Service dependency follow up work
[ https://issues.apache.org/jira/browse/YARN-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437353#comment-16437353 ] Billie Rinaldi commented on YARN-7799: -- I think this is a critical issue that we should try to get done for 3.1.1. > YARN Service dependency follow up work > -- > > Key: YARN-7799 > URL: https://issues.apache.org/jira/browse/YARN-7799 > Project: Hadoop YARN > Issue Type: Bug > Components: client, resourcemanager >Reporter: Gour Saha >Assignee: Gour Saha >Priority: Critical > > As per [~jianhe] these are some followup items that make sense to do after > YARN-7766. Quoting Jian's comment below - > Currently, if user doesn't supply location when run yarn app > -enableFastLaunch, the jars will be put under this location > {code} > hdfs:///yarn-services//service-dep.tar.gz > {code} > Since API server is embedded in RM, should RM look for this location too if > "yarn.service.framework.path" is not specified ? > And if "yarn.service.framework.path" is not specified and still the file > doesn't exist at above default location, I think RM can try to upload the > jars to above default location instead, currently RM is uploading the jars to > the location defined by below code. This folder is per app and also > inconsistent with CLI location. > {code} > protected Path addJarResource(String serviceName, > MaplocalResources) > throws IOException, SliderException { > Path libPath = fs.buildClusterDirPath(serviceName); > {code} > By doing this, the next time a submission request comes, RM doesn't need to > upload the jars again. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7799) YARN Service dependency follow up work
[ https://issues.apache.org/jira/browse/YARN-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368431#comment-16368431 ] Wangda Tan commented on YARN-7799: -- [~gsaha], we originally plan to start vote by end of this weekend. So moved it to 3.2.0, thanks. > YARN Service dependency follow up work > -- > > Key: YARN-7799 > URL: https://issues.apache.org/jira/browse/YARN-7799 > Project: Hadoop YARN > Issue Type: Bug > Components: client, resourcemanager >Reporter: Gour Saha >Assignee: Gour Saha >Priority: Critical > > As per [~jianhe] these are some followup items that make sense to do after > YARN-7766. Quoting Jian's comment below - > Currently, if user doesn't supply location when run yarn app > -enableFastLaunch, the jars will be put under this location > {code} > hdfs:///yarn-services//service-dep.tar.gz > {code} > Since API server is embedded in RM, should RM look for this location too if > "yarn.service.framework.path" is not specified ? > And if "yarn.service.framework.path" is not specified and still the file > doesn't exist at above default location, I think RM can try to upload the > jars to above default location instead, currently RM is uploading the jars to > the location defined by below code. This folder is per app and also > inconsistent with CLI location. > {code} > protected Path addJarResource(String serviceName, > MaplocalResources) > throws IOException, SliderException { > Path libPath = fs.buildClusterDirPath(serviceName); > {code} > By doing this, the next time a submission request comes, RM doesn't need to > upload the jars again. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7799) YARN Service dependency follow up work
[ https://issues.apache.org/jira/browse/YARN-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16367436#comment-16367436 ] Gour Saha commented on YARN-7799: - [~leftnoteasy] what is timeline for 3.1.0? It's not really a critical issue so I think 3.2.0 should be fine. > YARN Service dependency follow up work > -- > > Key: YARN-7799 > URL: https://issues.apache.org/jira/browse/YARN-7799 > Project: Hadoop YARN > Issue Type: Bug > Components: client, resourcemanager >Reporter: Gour Saha >Assignee: Gour Saha >Priority: Critical > > As per [~jianhe] these are some followup items that make sense to do after > YARN-7766. Quoting Jian's comment below - > Currently, if user doesn't supply location when run yarn app > -enableFastLaunch, the jars will be put under this location > {code} > hdfs:///yarn-services//service-dep.tar.gz > {code} > Since API server is embedded in RM, should RM look for this location too if > "yarn.service.framework.path" is not specified ? > And if "yarn.service.framework.path" is not specified and still the file > doesn't exist at above default location, I think RM can try to upload the > jars to above default location instead, currently RM is uploading the jars to > the location defined by below code. This folder is per app and also > inconsistent with CLI location. > {code} > protected Path addJarResource(String serviceName, > MaplocalResources) > throws IOException, SliderException { > Path libPath = fs.buildClusterDirPath(serviceName); > {code} > By doing this, the next time a submission request comes, RM doesn't need to > upload the jars again. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7799) YARN Service dependency follow up work
[ https://issues.apache.org/jira/browse/YARN-7799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16366582#comment-16366582 ] Wangda Tan commented on YARN-7799: -- [~gsaha], do you plan to finish this patch soon or it will be fine to be moved to 3.2.0? > YARN Service dependency follow up work > -- > > Key: YARN-7799 > URL: https://issues.apache.org/jira/browse/YARN-7799 > Project: Hadoop YARN > Issue Type: Bug > Components: client, resourcemanager >Reporter: Gour Saha >Assignee: Gour Saha >Priority: Critical > > As per [~jianhe] these are some followup items that make sense to do after > YARN-7766. Quoting Jian's comment below - > Currently, if user doesn't supply location when run yarn app > -enableFastLaunch, the jars will be put under this location > {code} > hdfs:///yarn-services//service-dep.tar.gz > {code} > Since API server is embedded in RM, should RM look for this location too if > "yarn.service.framework.path" is not specified ? > And if "yarn.service.framework.path" is not specified and still the file > doesn't exist at above default location, I think RM can try to upload the > jars to above default location instead, currently RM is uploading the jars to > the location defined by below code. This folder is per app and also > inconsistent with CLI location. > {code} > protected Path addJarResource(String serviceName, > MaplocalResources) > throws IOException, SliderException { > Path libPath = fs.buildClusterDirPath(serviceName); > {code} > By doing this, the next time a submission request comes, RM doesn't need to > upload the jars again. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org