[jira] [Commented] (YARN-7540) Convert yarn app cli to call yarn api services
[ https://issues.apache.org/jira/browse/YARN-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16336769#comment-16336769 ] Hudson commented on YARN-7540: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13541 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13541/]) YARN-7540 and YARN-7605. Convert yarn app cli to call yarn api services (billie: rev e307edcb472207a39d1cbe4be6f7fcddc7b4fd6d) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/Resource.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AppAdminClient.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/pom.xml * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/ServiceState.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/ApiServer.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/TestYarnNativeServices.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/client/ServiceClient.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/client/TestServiceCLI.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/test/resources/example-app.json * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/test/java/org/apache/hadoop/yarn/service/client/TestApiServiceClient.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/test/resources/log4j.properties * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/webapp/WebApps.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/test/java/org/apache/hadoop/yarn/service/TestApiServer.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/impl/zk/RegistrySecurity.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/utils/JsonSerDeser.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/cli/ApplicationCLI.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-registry/src/main/java/org/apache/hadoop/registry/client/api/RegistryConstants.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/client/TestBuildExternalComponents.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/ReadinessCheck.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/client/ApiServiceClient.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/package-info.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/http/HttpServer2.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceManager.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/client/package-info.java > Convert yarn app cli to call yarn api services > -- > > Key: YARN-7540 > URL: https://issues.apache.org/jira/browse/YARN-7540 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang >Priority: Major > Fix For: 3.1.0 > > Attachments: YARN-7540.001.patch, YARN-7540.002.patch, > YARN-7540.003.patch, YARN-7540.004.patch, YARN-7540.005.patch, > YARN-7540.006.patch > > > For YARN docker application to launch through CLI, it works differently from > launching through REST API. All application launched through REST API is > currently stored in yarn u
[jira] [Commented] (YARN-7540) Convert yarn app cli to call yarn api services
[ https://issues.apache.org/jira/browse/YARN-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314423#comment-16314423 ] Eric Yang commented on YARN-7540: - Hadoop officially supports two distinct security modes, SIMPLE and Kerberos. Simple mode was designed to run everything in the same user space for single user mode. Kerberos supports multi-user mode using in combination with Linux task controller to provide security. However, Linux task controller with SIMPLE security creates a third combination which should not be support because this combination has a privilege escalation security hole that it allows any user to impersonate as any other user without any verification of end user credential. The implementation of YARN-7540 and YARN-7605 blocked the third mode from working because REST API without authentication fallback to {{hadoop.http.staticuser.user}} setting to look for deployment artifacts. This is the reason that Gour is seeing dr.who when YARN-7605 is applied. If down stream project depends on the third mode, then I recommend to evaluate the usage of down stream project to prevent opening up more security holes. Security problem is not going to be solved by reverting this patch, quite the opposite that you allow security hole to remain in the system, and potentially assisted to open up more security holes in downstream projects. This is the reason that I take no part in reverting this patch. Feel free to commit again once you verified YARN-7605 matches your expectation. > Convert yarn app cli to call yarn api services > -- > > Key: YARN-7540 > URL: https://issues.apache.org/jira/browse/YARN-7540 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang > Fix For: yarn-native-services > > Attachments: YARN-7540.001.patch, YARN-7540.002.patch, > YARN-7540.003.patch, YARN-7540.004.patch, YARN-7540.005.patch, > YARN-7540.006.patch > > > For YARN docker application to launch through CLI, it works differently from > launching through REST API. All application launched through REST API is > currently stored in yarn user HDFS home directory. Application managed > through CLI are stored into individual user's HDFS home directory. For > consistency, we want to have yarn app cli to interact with API service to > manage applications. For performance reason, it is easier to implement list > all applications from one user's home directory instead of crawling all > user's home directories. For security reason, it is safer to access only one > user home directory instead of all users. Given the reasons above, the > proposal is to change how {{yarn app -launch}}, {{yarn app -list}} and {{yarn > app -destroy}} work. Instead of calling HDFS API and RM API to launch > containers, CLI will be converted to call API service REST API resides in RM. > RM perform the persist and operations to launch the actual application. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7540) Convert yarn app cli to call yarn api services
[ https://issues.apache.org/jira/browse/YARN-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314320#comment-16314320 ] Hudson commented on YARN-7540: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13457 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13457/]) Revert "YARN-7540. Route YARN service CLI function through YARN Service (jianhe: rev 836e3c45e8232fc4c8c795c0f93d2f3d7353f392) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/ApiServer.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/Resource.java * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/client/ApiServiceClient.java * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/test/resources/log4j.properties * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/test/java/org/apache/hadoop/yarn/service/client/TestApiServiceClient.java * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/package-info.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/ReadinessCheck.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/ServiceState.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/utils/JsonSerDeser.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/client/TestServiceCLI.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AppAdminClient.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/client/TestBuildExternalComponents.java * (delete) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/test/resources/example-app.json * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/pom.xml > Convert yarn app cli to call yarn api services > -- > > Key: YARN-7540 > URL: https://issues.apache.org/jira/browse/YARN-7540 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang > Fix For: yarn-native-services > > Attachments: YARN-7540.001.patch, YARN-7540.002.patch, > YARN-7540.003.patch, YARN-7540.004.patch, YARN-7540.005.patch, > YARN-7540.006.patch > > > For YARN docker application to launch through CLI, it works differently from > launching through REST API. All application launched through REST API is > currently stored in yarn user HDFS home directory. Application managed > through CLI are stored into individual user's HDFS home directory. For > consistency, we want to have yarn app cli to interact with API service to > manage applications. For performance reason, it is easier to implement list > all applications from one user's home directory instead of crawling all > user's home directories. For security reason, it is safer to access only one > user home directory instead of all users. Given the reasons above, the > proposal is to change how {{yarn app -launch}}, {{yarn app -list}} and {{yarn > app -destroy}} work. Instead of calling HDFS API and RM API to launch > containers, CLI will be converted to call API service REST API resides in RM. > RM perform the persist and operations to launch the actual application. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7540) Convert yarn app cli to call yarn api services
[ https://issues.apache.org/jira/browse/YARN-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16314311#comment-16314311 ] Jian He commented on YARN-7540: --- Since this is partially completed and blocking downstream project, I'm reverting this patch for now. Will commit this and YARN-7605 together after this is ready. > Convert yarn app cli to call yarn api services > -- > > Key: YARN-7540 > URL: https://issues.apache.org/jira/browse/YARN-7540 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang > Fix For: yarn-native-services > > Attachments: YARN-7540.001.patch, YARN-7540.002.patch, > YARN-7540.003.patch, YARN-7540.004.patch, YARN-7540.005.patch, > YARN-7540.006.patch > > > For YARN docker application to launch through CLI, it works differently from > launching through REST API. All application launched through REST API is > currently stored in yarn user HDFS home directory. Application managed > through CLI are stored into individual user's HDFS home directory. For > consistency, we want to have yarn app cli to interact with API service to > manage applications. For performance reason, it is easier to implement list > all applications from one user's home directory instead of crawling all > user's home directories. For security reason, it is safer to access only one > user home directory instead of all users. Given the reasons above, the > proposal is to change how {{yarn app -launch}}, {{yarn app -list}} and {{yarn > app -destroy}} work. Instead of calling HDFS API and RM API to launch > containers, CLI will be converted to call API service REST API resides in RM. > RM perform the persist and operations to launch the actual application. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7540) Convert yarn app cli to call yarn api services
[ https://issues.apache.org/jira/browse/YARN-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16300951#comment-16300951 ] Gour Saha commented on YARN-7540: - This patch breaks the most basic functionality that existed in Yarn Services untill it was committed (and in Slider forever) - In an unsecure cluster, a non-yarn user (say hive) uploads a tarball to HDFS and submits a service create request (say LLAP). The service create request fails with the below exception because it tries to create it as user yarn and looks for the tarball under yarn's HDFS home directory as well. {code} 17/12/21 10:04:25 ERROR client.ApiServiceClient: Artifact tarball does not exist /user/yarn/.yarn/package/LLAP/llap-21Dec2017.tar.gz {code} It also breaks the functionality where different users can create services of the same name because the namespace was separated by user dir. With this patch, you cannot anymore, since you are always in the yarn namespace. The patch in YARN-7605 tries to solve the problem by implementing doAs which is the right approach. However, it is not implemented correctly and breaks the functionality even further. With that patch the service gets submitted as user dr.who. This patch is a blocker for all basic Yarn Services functionality and needs to be reverted until we can make this patch and YARN-7605 work together. > Convert yarn app cli to call yarn api services > -- > > Key: YARN-7540 > URL: https://issues.apache.org/jira/browse/YARN-7540 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang > Fix For: yarn-native-services > > Attachments: YARN-7540.001.patch, YARN-7540.002.patch, > YARN-7540.003.patch, YARN-7540.004.patch, YARN-7540.005.patch, > YARN-7540.006.patch > > > For YARN docker application to launch through CLI, it works differently from > launching through REST API. All application launched through REST API is > currently stored in yarn user HDFS home directory. Application managed > through CLI are stored into individual user's HDFS home directory. For > consistency, we want to have yarn app cli to interact with API service to > manage applications. For performance reason, it is easier to implement list > all applications from one user's home directory instead of crawling all > user's home directories. For security reason, it is safer to access only one > user home directory instead of all users. Given the reasons above, the > proposal is to change how {{yarn app -launch}}, {{yarn app -list}} and {{yarn > app -destroy}} work. Instead of calling HDFS API and RM API to launch > containers, CLI will be converted to call API service REST API resides in RM. > RM perform the persist and operations to launch the actual application. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7540) Convert yarn app cli to call yarn api services
[ https://issues.apache.org/jira/browse/YARN-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16290167#comment-16290167 ] Hudson commented on YARN-7540: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #13371 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/13371/]) YARN-7540. Route YARN service CLI function through YARN Service API. (eyang: rev 438c1d333ebc0a3071bb556532ed959a4bd1e6d6) * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/client/TestServiceCLI.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/ReadinessCheck.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/test/java/org/apache/hadoop/yarn/service/client/TestApiServiceClient.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/package-info.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/webapp/ApiServer.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/Resource.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/utils/JsonSerDeser.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/test/java/org/apache/hadoop/yarn/service/client/TestBuildExternalComponents.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/test/resources/log4j.properties * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/component/Component.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/AppAdminClient.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/test/resources/example-app.json * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services/hadoop-yarn-services-core/src/main/java/org/apache/hadoop/yarn/service/api/records/ServiceState.java * (add) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/src/main/java/org/apache/hadoop/yarn/service/client/ApiServiceClient.java * (edit) hadoop-yarn-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-services-api/pom.xml > Convert yarn app cli to call yarn api services > -- > > Key: YARN-7540 > URL: https://issues.apache.org/jira/browse/YARN-7540 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang > Fix For: yarn-native-services > > Attachments: YARN-7540.001.patch, YARN-7540.002.patch, > YARN-7540.003.patch, YARN-7540.004.patch, YARN-7540.005.patch, > YARN-7540.006.patch > > > For YARN docker application to launch through CLI, it works differently from > launching through REST API. All application launched through REST API is > currently stored in yarn user HDFS home directory. Application managed > through CLI are stored into individual user's HDFS home directory. For > consistency, we want to have yarn app cli to interact with API service to > manage applications. For performance reason, it is easier to implement list > all applications from one user's home directory instead of crawling all > user's home directories. For security reason, it is safer to access only one > user home directory instead of all users. Given the reasons above, the > proposal is to change how {{yarn app -launch}}, {{yarn app -list}} and {{yarn > app -destroy}} work. Instead of calling HDFS API and RM API to launch > containers, CLI will be converted to call API service REST API resides in RM. > RM perform the persist and operations to launch the actual application. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7540) Convert yarn app cli to call yarn api services
[ https://issues.apache.org/jira/browse/YARN-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16289536#comment-16289536 ] Billie Rinaldi commented on YARN-7540: -- +1 for patch 006. This resolves issues with flex and save that I was seeing in earlier patches. I noticed a bug in the ApiServer where it fails to destroy a service that is stopped, but we can address this elsewhere since it is not introduced in this patch. > Convert yarn app cli to call yarn api services > -- > > Key: YARN-7540 > URL: https://issues.apache.org/jira/browse/YARN-7540 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang > Fix For: yarn-native-services > > Attachments: YARN-7540.001.patch, YARN-7540.002.patch, > YARN-7540.003.patch, YARN-7540.004.patch, YARN-7540.005.patch, > YARN-7540.006.patch > > > For YARN docker application to launch through CLI, it works differently from > launching through REST API. All application launched through REST API is > currently stored in yarn user HDFS home directory. Application managed > through CLI are stored into individual user's HDFS home directory. For > consistency, we want to have yarn app cli to interact with API service to > manage applications. For performance reason, it is easier to implement list > all applications from one user's home directory instead of crawling all > user's home directories. For security reason, it is safer to access only one > user home directory instead of all users. Given the reasons above, the > proposal is to change how {{yarn app -launch}}, {{yarn app -list}} and {{yarn > app -destroy}} work. Instead of calling HDFS API and RM API to launch > containers, CLI will be converted to call API service REST API resides in RM. > RM perform the persist and operations to launch the actual application. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7540) Convert yarn app cli to call yarn api services
[ https://issues.apache.org/jira/browse/YARN-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16288362#comment-16288362 ] genericqa commented on YARN-7540: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 5 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 25s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 32s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 47s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 52s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 4s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 4s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 58s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 3 new + 59 unchanged - 1 fixed = 62 total (was 60) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 36s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 56s{color} | {color:red} hadoop-yarn-client in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 44s{color} | {color:green} hadoop-yarn-services-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 32s{color} | {color:green} hadoop-yarn-services-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 32s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 94m 42s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.client.api.impl.TestAMRMClientOnRMRestart | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | YARN-7540 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12901759/YARN-7540.006.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle | | u
[jira] [Commented] (YARN-7540) Convert yarn app cli to call yarn api services
[ https://issues.apache.org/jira/browse/YARN-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16288271#comment-16288271 ] genericqa commented on YARN-7540: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 5 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 6s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 59s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 32s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 40s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 52s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 6s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 59s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 59s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 58s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 59 unchanged - 0 fixed = 61 total (was 59) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 26s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 59s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 51s{color} | {color:red} hadoop-yarn-client in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 55s{color} | {color:green} hadoop-yarn-services-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 37s{color} | {color:green} hadoop-yarn-services-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 33s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 93m 50s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.client.api.impl.TestAMRMClientOnRMRestart | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | YARN-7540 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12901752/YARN-7540.005.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle | | u
[jira] [Commented] (YARN-7540) Convert yarn app cli to call yarn api services
[ https://issues.apache.org/jira/browse/YARN-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16284456#comment-16284456 ] Eric Yang commented on YARN-7540: - The failed unit test is not related to this JIRA. > Convert yarn app cli to call yarn api services > -- > > Key: YARN-7540 > URL: https://issues.apache.org/jira/browse/YARN-7540 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang > Fix For: yarn-native-services > > Attachments: YARN-7540.001.patch, YARN-7540.002.patch, > YARN-7540.003.patch, YARN-7540.004.patch > > > For YARN docker application to launch through CLI, it works differently from > launching through REST API. All application launched through REST API is > currently stored in yarn user HDFS home directory. Application managed > through CLI are stored into individual user's HDFS home directory. For > consistency, we want to have yarn app cli to interact with API service to > manage applications. For performance reason, it is easier to implement list > all applications from one user's home directory instead of crawling all > user's home directories. For security reason, it is safer to access only one > user home directory instead of all users. Given the reasons above, the > proposal is to change how {{yarn app -launch}}, {{yarn app -list}} and {{yarn > app -destroy}} work. Instead of calling HDFS API and RM API to launch > containers, CLI will be converted to call API service REST API resides in RM. > RM perform the persist and operations to launch the actual application. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7540) Convert yarn app cli to call yarn api services
[ https://issues.apache.org/jira/browse/YARN-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16284443#comment-16284443 ] genericqa commented on YARN-7540: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 5 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 42s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 1s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 59s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 39s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 50s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 49s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 49s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 1s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 55 unchanged - 0 fixed = 57 total (was 55) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 29s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 11m 4s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 19s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 56s{color} | {color:red} hadoop-yarn-client in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 55s{color} | {color:green} hadoop-yarn-services-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 36s{color} | {color:green} hadoop-yarn-services-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 30s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 95m 8s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.client.api.impl.TestAMRMClientOnRMRestart | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | YARN-7540 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12901314/YARN-7540.004.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle | | u
[jira] [Commented] (YARN-7540) Convert yarn app cli to call yarn api services
[ https://issues.apache.org/jira/browse/YARN-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16284309#comment-16284309 ] Jian He commented on YARN-7540: --- - config name for https is "webapp.https.address" {code} String rmAddress = conf.get("yarn.resourcemanager.webapp.address"); if(conf.getBoolean("hadoop.ssl.enabled", false)) { scheme = "https://";; } {code} > Convert yarn app cli to call yarn api services > -- > > Key: YARN-7540 > URL: https://issues.apache.org/jira/browse/YARN-7540 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang > Fix For: yarn-native-services > > Attachments: YARN-7540.001.patch, YARN-7540.002.patch, > YARN-7540.003.patch > > > For YARN docker application to launch through CLI, it works differently from > launching through REST API. All application launched through REST API is > currently stored in yarn user HDFS home directory. Application managed > through CLI are stored into individual user's HDFS home directory. For > consistency, we want to have yarn app cli to interact with API service to > manage applications. For performance reason, it is easier to implement list > all applications from one user's home directory instead of crawling all > user's home directories. For security reason, it is safer to access only one > user home directory instead of all users. Given the reasons above, the > proposal is to change how {{yarn app -launch}}, {{yarn app -list}} and {{yarn > app -destroy}} work. Instead of calling HDFS API and RM API to launch > containers, CLI will be converted to call API service REST API resides in RM. > RM perform the persist and operations to launch the actual application. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7540) Convert yarn app cli to call yarn api services
[ https://issues.apache.org/jira/browse/YARN-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16284128#comment-16284128 ] genericqa commented on YARN-7540: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 4m 10s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 5 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 48s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 40s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 0s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 53s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 11s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 9s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 53s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 53s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 56s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 55 unchanged - 0 fixed = 57 total (was 55) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 27s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 50s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:red}-1{color} | {color:red} unit {color} | {color:red} 20m 50s{color} | {color:red} hadoop-yarn-client in the patch failed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 43s{color} | {color:green} hadoop-yarn-services-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 30s{color} | {color:green} hadoop-yarn-services-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 32s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 98m 49s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | Failed junit tests | hadoop.yarn.client.api.impl.TestAMRMClientOnRMRestart | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | YARN-7540 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12901284/YARN-7540.003.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle | | u
[jira] [Commented] (YARN-7540) Convert yarn app cli to call yarn api services
[ https://issues.apache.org/jira/browse/YARN-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16280786#comment-16280786 ] Billie Rinaldi commented on YARN-7540: -- Regarding enableFastLaunch, I think we need to keep the command because we shouldn't require the RM to be restarted to update the dependency tarball in HDFS. It would be okay to make the RM do the upload automatically on start, but the command is still needed. If you add the following after instantiating the ServiceClient in ApiServiceClient, that should fix the issue. {noformat} sc.init(getConfig()); sc.start(); {noformat} > Convert yarn app cli to call yarn api services > -- > > Key: YARN-7540 > URL: https://issues.apache.org/jira/browse/YARN-7540 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang > Fix For: yarn-native-services > > Attachments: YARN-7540.001.patch, YARN-7540.002.patch > > > For YARN docker application to launch through CLI, it works differently from > launching through REST API. All application launched through REST API is > currently stored in yarn user HDFS home directory. Application managed > through CLI are stored into individual user's HDFS home directory. For > consistency, we want to have yarn app cli to interact with API service to > manage applications. For performance reason, it is easier to implement list > all applications from one user's home directory instead of crawling all > user's home directories. For security reason, it is safer to access only one > user home directory instead of all users. Given the reasons above, the > proposal is to change how {{yarn app -launch}}, {{yarn app -list}} and {{yarn > app -destroy}} work. Instead of calling HDFS API and RM API to launch > containers, CLI will be converted to call API service REST API resides in RM. > RM perform the persist and operations to launch the actual application. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7540) Convert yarn app cli to call yarn api services
[ https://issues.apache.org/jira/browse/YARN-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16279389#comment-16279389 ] Eric Yang commented on YARN-7540: - [~billie.rinaldi] # I will reduce config object. # I will reuse RMHAUtils.getRMHAWebappAddress. # EnableFastLaunch seems to be a problem because Rest API server doesn't have this functionality. I am more inclined to fix this issue by start up of Resource Manager to check if the required libraries are uploaded to hdfs, and remove the enableFastLaunch option. Thoughts? # I will fix the conflict introduced via YARN-6669. # I agree that NATIVE_TYPE is confusing, and we need this special type for unit tests. Therefore, I am inclined to rename NATIVE_TYPE to "unit-test" type for correctness. > Convert yarn app cli to call yarn api services > -- > > Key: YARN-7540 > URL: https://issues.apache.org/jira/browse/YARN-7540 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang > Fix For: yarn-native-services > > Attachments: YARN-7540.001.patch, YARN-7540.002.patch > > > For YARN docker application to launch through CLI, it works differently from > launching through REST API. All application launched through REST API is > currently stored in yarn user HDFS home directory. Application managed > through CLI are stored into individual user's HDFS home directory. For > consistency, we want to have yarn app cli to interact with API service to > manage applications. For performance reason, it is easier to implement list > all applications from one user's home directory instead of crawling all > user's home directories. For security reason, it is safer to access only one > user home directory instead of all users. Given the reasons above, the > proposal is to change how {{yarn app -launch}}, {{yarn app -list}} and {{yarn > app -destroy}} work. Instead of calling HDFS API and RM API to launch > containers, CLI will be converted to call API service REST API resides in RM. > RM perform the persist and operations to launch the actual application. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7540) Convert yarn app cli to call yarn api services
[ https://issues.apache.org/jira/browse/YARN-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16278862#comment-16278862 ] Billie Rinaldi commented on YARN-7540: -- I haven't tested the patch manually yet. Hopefully I will have time for this later this week. These are the comments I have so far: * ApiServiceClient doesn't need to store a copy of the configuration; it can call getConfig() to access the configuration that will be initialized in AbstractService.serviceInit * Instead of this implementation of ApiServiceClient.getRMAddress, I suspect we should be doing something similar to RMWebApp.buildRedirectPath or reusing existing utility methods like RMHAUtils.getRMHAWebappAddresses * The enableFastLaunch method probably isn't working because the ServiceClient needs to be inited * The patch doesn't apply because the log statement added in ApiServer has already been added * There are a couple of problems with the NATIVE_TYPE. For one, the name is confusing; I don't think we should use "native" for this. Also, there is no way to create an application that has NATIVE_TYPE, because YARN service is coded to use "yarn-service" through the YarnServiceConstants.APP_TYPE variable. I would think we should just make the new ApiServiceClient the only provided AppAdminClient, but I guess that would leave us unable to unit test some features that it's good to have unit tests for. What do you think? > Convert yarn app cli to call yarn api services > -- > > Key: YARN-7540 > URL: https://issues.apache.org/jira/browse/YARN-7540 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Eric Yang >Assignee: Eric Yang > Fix For: yarn-native-services > > Attachments: YARN-7540.001.patch, YARN-7540.002.patch > > > For YARN docker application to launch through CLI, it works differently from > launching through REST API. All application launched through REST API is > currently stored in yarn user HDFS home directory. Application managed > through CLI are stored into individual user's HDFS home directory. For > consistency, we want to have yarn app cli to interact with API service to > manage applications. For performance reason, it is easier to implement list > all applications from one user's home directory instead of crawling all > user's home directories. For security reason, it is safer to access only one > user home directory instead of all users. Given the reasons above, the > proposal is to change how {{yarn app -launch}}, {{yarn app -list}} and {{yarn > app -destroy}} work. Instead of calling HDFS API and RM API to launch > containers, CLI will be converted to call API service REST API resides in RM. > RM perform the persist and operations to launch the actual application. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-7540) Convert yarn app cli to call yarn api services
[ https://issues.apache.org/jira/browse/YARN-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16272098#comment-16272098 ] genericqa commented on YARN-7540: - | (/) *{color:green}+1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 5 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 13s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m 24s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 18s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m 2s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 56s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 56s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 7m 17s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 1m 2s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 2 new + 59 unchanged - 0 fixed = 61 total (was 59) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 32s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s{color} | {color:green} The patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 44s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 11s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m 17s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 44s{color} | {color:green} hadoop-yarn-services-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 30s{color} | {color:green} hadoop-yarn-services-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 32s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 95m 11s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | YARN-7540 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12899927/YARN-7540.002.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle | | uname | Linux 01d24b26a546 3.13.0-129-generic #178-Ubuntu SMP Fri Aug 11 12:48:20 UTC 2017 x86_64
[jira] [Commented] (YARN-7540) Convert yarn app cli to call yarn api services
[ https://issues.apache.org/jira/browse/YARN-7540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16271901#comment-16271901 ] genericqa commented on YARN-7540: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s{color} | {color:blue} Docker mode activated. {color} | || || || || {color:brown} Prechecks {color} || | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 5 new or modified test files. {color} | || || || || {color:brown} trunk Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 12s{color} | {color:blue} Maven dependency ordering for branch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m 38s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 11s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 59s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 33s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 12m 44s{color} | {color:green} branch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m 53s{color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 3s{color} | {color:green} trunk passed {color} | || || || || {color:brown} Patch Compile Tests {color} || | {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s{color} | {color:blue} Maven dependency ordering for patch {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m 8s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 6m 55s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 6m 55s{color} | {color:green} the patch passed {color} | | {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange} 0m 57s{color} | {color:orange} hadoop-yarn-project/hadoop-yarn: The patch generated 6 new + 59 unchanged - 0 fixed = 65 total (was 59) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 25s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 5 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s{color} | {color:green} The patch has no ill-formed XML file. {color} | | {color:green}+1{color} | {color:green} shadedclient {color} | {color:green} 10m 50s{color} | {color:green} patch has no errors when building and testing our client artifacts. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 31s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 2s{color} | {color:green} the patch passed {color} | || || || || {color:brown} Other Tests {color} || | {color:green}+1{color} | {color:green} unit {color} | {color:green} 20m 57s{color} | {color:green} hadoop-yarn-client in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 3m 44s{color} | {color:green} hadoop-yarn-services-core in the patch passed. {color} | | {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 29s{color} | {color:green} hadoop-yarn-services-api in the patch passed. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 34s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 93m 38s{color} | {color:black} {color} | \\ \\ || Subsystem || Report/Notes || | Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:5b98639 | | JIRA Issue | YARN-7540 | | JIRA Patch URL | https://issues.apache.org/jira/secure/attachment/12899896/YARN-7540.001.patch | | Optional Tests | asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml findbugs checkstyle | | uname | Linux 7a43