[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression
[ https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16035730#comment-16035730 ] Jonathan Hung commented on YARN-4925: - Thanks [~shv]! > ContainerRequest in AMRMClient, application should be able to specify > nodes/racks together with nodeLabelExpression > --- > > Key: YARN-4925 > URL: https://issues.apache.org/jira/browse/YARN-4925 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Fix For: 2.8.0, 2.7.4, 3.0.0-alpha1 > > Attachments: 0001-YARN-4925.patch, 0002-YARN-4925.patch, > YARN-4925-branch-2.7.001.patch > > > Currently with nodelabel AMRMClient will not be able to specify nodelabels > with Node/Rack requests.For application like spark NODE_LOCAL requests cannot > be asked with label expression. > As per the check in {{AMRMClientImpl#checkNodeLabelExpression}} > {noformat} > // Don't allow specify node label against ANY request > if ((containerRequest.getRacks() != null && > (!containerRequest.getRacks().isEmpty())) > || > (containerRequest.getNodes() != null && > (!containerRequest.getNodes().isEmpty( { > throw new InvalidContainerRequestException( > "Cannot specify node label with rack and node"); > } > {noformat} > {{AppSchedulingInfo#updateResourceRequests}} we do reset of labels to that of > OFF-SWITCH. > The above check is not required for ContainerRequest ask /cc [~wangda] thank > you for confirming -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression
[ https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16035651#comment-16035651 ] Konstantin Shvachko commented on YARN-4925: --- Turned out the *Client tests were failing due to unusual local URI configuration, which should be addressed in YARN-6684. I was able to successfully run the tests locally with the work around. > ContainerRequest in AMRMClient, application should be able to specify > nodes/racks together with nodeLabelExpression > --- > > Key: YARN-4925 > URL: https://issues.apache.org/jira/browse/YARN-4925 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Labels: release-blocker > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: 0001-YARN-4925.patch, 0002-YARN-4925.patch, > YARN-4925-branch-2.7.001.patch > > > Currently with nodelabel AMRMClient will not be able to specify nodelabels > with Node/Rack requests.For application like spark NODE_LOCAL requests cannot > be asked with label expression. > As per the check in {{AMRMClientImpl#checkNodeLabelExpression}} > {noformat} > // Don't allow specify node label against ANY request > if ((containerRequest.getRacks() != null && > (!containerRequest.getRacks().isEmpty())) > || > (containerRequest.getNodes() != null && > (!containerRequest.getNodes().isEmpty( { > throw new InvalidContainerRequestException( > "Cannot specify node label with rack and node"); > } > {noformat} > {{AppSchedulingInfo#updateResourceRequests}} we do reset of labels to that of > OFF-SWITCH. > The above check is not required for ContainerRequest ask /cc [~wangda] thank > you for confirming -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression
[ https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16034043#comment-16034043 ] Jonathan Hung commented on YARN-4925: - Created YARN-6684 for this issue. Seems the unit test failures are unrelated to this patch. > ContainerRequest in AMRMClient, application should be able to specify > nodes/racks together with nodeLabelExpression > --- > > Key: YARN-4925 > URL: https://issues.apache.org/jira/browse/YARN-4925 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Labels: release-blocker > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: 0001-YARN-4925.patch, 0002-YARN-4925.patch, > YARN-4925-branch-2.7.001.patch > > > Currently with nodelabel AMRMClient will not be able to specify nodelabels > with Node/Rack requests.For application like spark NODE_LOCAL requests cannot > be asked with label expression. > As per the check in {{AMRMClientImpl#checkNodeLabelExpression}} > {noformat} > // Don't allow specify node label against ANY request > if ((containerRequest.getRacks() != null && > (!containerRequest.getRacks().isEmpty())) > || > (containerRequest.getNodes() != null && > (!containerRequest.getNodes().isEmpty( { > throw new InvalidContainerRequestException( > "Cannot specify node label with rack and node"); > } > {noformat} > {{AppSchedulingInfo#updateResourceRequests}} we do reset of labels to that of > OFF-SWITCH. > The above check is not required for ContainerRequest ask /cc [~wangda] thank > you for confirming -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression
[ https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16031876#comment-16031876 ] Konstantin Shvachko commented on YARN-4925: --- I've done quite a few runs for the client tests. I see {{TestAMRMClient}} and other client tests failing consistently on my Linux box. On Mac they succeed most of the time, but also fail once in a while. I don't think it is related to this particular patch, but I don't know for sure. So I think we should get some understanding on why the tests are failing, which will give us an idea if we can commit this jira and focus on fixing the tests for branch-2.7. > ContainerRequest in AMRMClient, application should be able to specify > nodes/racks together with nodeLabelExpression > --- > > Key: YARN-4925 > URL: https://issues.apache.org/jira/browse/YARN-4925 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Labels: release-blocker > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: 0001-YARN-4925.patch, 0002-YARN-4925.patch, > YARN-4925-branch-2.7.001.patch > > > Currently with nodelabel AMRMClient will not be able to specify nodelabels > with Node/Rack requests.For application like spark NODE_LOCAL requests cannot > be asked with label expression. > As per the check in {{AMRMClientImpl#checkNodeLabelExpression}} > {noformat} > // Don't allow specify node label against ANY request > if ((containerRequest.getRacks() != null && > (!containerRequest.getRacks().isEmpty())) > || > (containerRequest.getNodes() != null && > (!containerRequest.getNodes().isEmpty( { > throw new InvalidContainerRequestException( > "Cannot specify node label with rack and node"); > } > {noformat} > {{AppSchedulingInfo#updateResourceRequests}} we do reset of labels to that of > OFF-SWITCH. > The above check is not required for ContainerRequest ask /cc [~wangda] thank > you for confirming -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression
[ https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16030530#comment-16030530 ] Jonathan Hung commented on YARN-4925: - [~shv] the branch-2.7 patch passes locally for me: {noformat}--- T E S T S --- --- T E S T S --- Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=768m; support was removed in 8.0 Running org.apache.hadoop.yarn.client.api.impl.TestAMRMClient Tests run: 10, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 50.551 sec - in org.apache.hadoop.yarn.client.api.impl.TestAMRMClient Results : Tests run: 10, Failures: 0, Errors: 0, Skipped: 0{noformat} > ContainerRequest in AMRMClient, application should be able to specify > nodes/racks together with nodeLabelExpression > --- > > Key: YARN-4925 > URL: https://issues.apache.org/jira/browse/YARN-4925 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Labels: release-blocker > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: 0001-YARN-4925.patch, 0002-YARN-4925.patch, > YARN-4925-branch-2.7.001.patch > > > Currently with nodelabel AMRMClient will not be able to specify nodelabels > with Node/Rack requests.For application like spark NODE_LOCAL requests cannot > be asked with label expression. > As per the check in {{AMRMClientImpl#checkNodeLabelExpression}} > {noformat} > // Don't allow specify node label against ANY request > if ((containerRequest.getRacks() != null && > (!containerRequest.getRacks().isEmpty())) > || > (containerRequest.getNodes() != null && > (!containerRequest.getNodes().isEmpty( { > throw new InvalidContainerRequestException( > "Cannot specify node label with rack and node"); > } > {noformat} > {{AppSchedulingInfo#updateResourceRequests}} we do reset of labels to that of > OFF-SWITCH. > The above check is not required for ContainerRequest ask /cc [~wangda] thank > you for confirming -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression
[ https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16027623#comment-16027623 ] Jonathan Hung commented on YARN-4925: - Thanks [~bibinchundatt] I am planning on working on this as well. Seems the test failures are possibly related to YARN-5208, which depends on HADOOP-12954. Created YARN-6662 and HADOOP-14463 for these issues, respectively. > ContainerRequest in AMRMClient, application should be able to specify > nodes/racks together with nodeLabelExpression > --- > > Key: YARN-4925 > URL: https://issues.apache.org/jira/browse/YARN-4925 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Labels: release-blocker > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: 0001-YARN-4925.patch, 0002-YARN-4925.patch, > YARN-4925-branch-2.7.001.patch > > > Currently with nodelabel AMRMClient will not be able to specify nodelabels > with Node/Rack requests.For application like spark NODE_LOCAL requests cannot > be asked with label expression. > As per the check in {{AMRMClientImpl#checkNodeLabelExpression}} > {noformat} > // Don't allow specify node label against ANY request > if ((containerRequest.getRacks() != null && > (!containerRequest.getRacks().isEmpty())) > || > (containerRequest.getNodes() != null && > (!containerRequest.getNodes().isEmpty( { > throw new InvalidContainerRequestException( > "Cannot specify node label with rack and node"); > } > {noformat} > {{AppSchedulingInfo#updateResourceRequests}} we do reset of labels to that of > OFF-SWITCH. > The above check is not required for ContainerRequest ask /cc [~wangda] thank > you for confirming -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression
[ https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16024866#comment-16024866 ] Bibin A Chundatt commented on YARN-4925: We should also port dependent YARN-4140 > ContainerRequest in AMRMClient, application should be able to specify > nodes/racks together with nodeLabelExpression > --- > > Key: YARN-4925 > URL: https://issues.apache.org/jira/browse/YARN-4925 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Labels: release-blocker > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: 0001-YARN-4925.patch, 0002-YARN-4925.patch, > YARN-4925-branch-2.7.001.patch > > > Currently with nodelabel AMRMClient will not be able to specify nodelabels > with Node/Rack requests.For application like spark NODE_LOCAL requests cannot > be asked with label expression. > As per the check in {{AMRMClientImpl#checkNodeLabelExpression}} > {noformat} > // Don't allow specify node label against ANY request > if ((containerRequest.getRacks() != null && > (!containerRequest.getRacks().isEmpty())) > || > (containerRequest.getNodes() != null && > (!containerRequest.getNodes().isEmpty( { > throw new InvalidContainerRequestException( > "Cannot specify node label with rack and node"); > } > {noformat} > {{AppSchedulingInfo#updateResourceRequests}} we do reset of labels to that of > OFF-SWITCH. > The above check is not required for ContainerRequest ask /cc [~wangda] thank > you for confirming -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression
[ https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020578#comment-16020578 ] Jonathan Hung commented on YARN-4925: - I'm not able to reproduce some of the test failures: {noformat}--- T E S T S --- Running org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA Tests run: 17, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 79.298 sec - in org.apache.hadoop.yarn.client.TestApplicationClientProtocolOnHA Running org.apache.hadoop.yarn.client.TestApplicationMasterServiceProtocolOnHA Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.953 sec - in org.apache.hadoop.yarn.client.TestApplicationMasterServiceProtocolOnHA Running org.apache.hadoop.yarn.client.TestGetGroups Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.963 sec - in org.apache.hadoop.yarn.client.TestGetGroups Running org.apache.hadoop.yarn.client.TestResourceTrackerOnHA Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 7.216 sec - in org.apache.hadoop.yarn.client.TestResourceTrackerOnHA Running org.apache.hadoop.yarn.client.TestRMFailover Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.319 sec - in org.apache.hadoop.yarn.client.TestRMFailover Results : Tests run: 32, Failures: 0, Errors: 0, Skipped: 0 {noformat} Seems to be related to the jenkins environment. Will look into TestAMRMClient, TestYarnClient, TestNMClient > ContainerRequest in AMRMClient, application should be able to specify > nodes/racks together with nodeLabelExpression > --- > > Key: YARN-4925 > URL: https://issues.apache.org/jira/browse/YARN-4925 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Labels: release-blocker > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: 0001-YARN-4925.patch, 0002-YARN-4925.patch, > YARN-4925-branch-2.7.001.patch > > > Currently with nodelabel AMRMClient will not be able to specify nodelabels > with Node/Rack requests.For application like spark NODE_LOCAL requests cannot > be asked with label expression. > As per the check in {{AMRMClientImpl#checkNodeLabelExpression}} > {noformat} > // Don't allow specify node label against ANY request > if ((containerRequest.getRacks() != null && > (!containerRequest.getRacks().isEmpty())) > || > (containerRequest.getNodes() != null && > (!containerRequest.getNodes().isEmpty( { > throw new InvalidContainerRequestException( > "Cannot specify node label with rack and node"); > } > {noformat} > {{AppSchedulingInfo#updateResourceRequests}} we do reset of labels to that of > OFF-SWITCH. > The above check is not required for ContainerRequest ask /cc [~wangda] thank > you for confirming -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression
[ https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16018974#comment-16018974 ] Konstantin Shvachko commented on YARN-4925: --- Hey [~jhung] looks like there are problems with a few tests, including TestAMRClient on branch-2.7. Could you please check. > ContainerRequest in AMRMClient, application should be able to specify > nodes/racks together with nodeLabelExpression > --- > > Key: YARN-4925 > URL: https://issues.apache.org/jira/browse/YARN-4925 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Labels: release-blocker > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: 0001-YARN-4925.patch, 0002-YARN-4925.patch, > YARN-4925-branch-2.7.001.patch > > > Currently with nodelabel AMRMClient will not be able to specify nodelabels > with Node/Rack requests.For application like spark NODE_LOCAL requests cannot > be asked with label expression. > As per the check in {{AMRMClientImpl#checkNodeLabelExpression}} > {noformat} > // Don't allow specify node label against ANY request > if ((containerRequest.getRacks() != null && > (!containerRequest.getRacks().isEmpty())) > || > (containerRequest.getNodes() != null && > (!containerRequest.getNodes().isEmpty( { > throw new InvalidContainerRequestException( > "Cannot specify node label with rack and node"); > } > {noformat} > {{AppSchedulingInfo#updateResourceRequests}} we do reset of labels to that of > OFF-SWITCH. > The above check is not required for ContainerRequest ask /cc [~wangda] thank > you for confirming -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression
[ https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16018220#comment-16018220 ] Hadoop QA commented on YARN-4925: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 16m 3s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 18s{color} | {color:green} branch-2.7 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_131 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 21s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_131 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 17s{color} | {color:green} branch-2.7 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 26s{color} | {color:green} branch-2.7 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 14s{color} | {color:green} branch-2.7 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 34s{color} | {color:green} branch-2.7 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_131 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_131 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 17s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s{color} | {color:green} the patch passed with JDK v1.8.0_131 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 13s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 13s{color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The patch generated 0 new + 215 unchanged - 6 fixed = 215 total (was 221) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 20s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 11s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 2346 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 55s{color} | {color:red} The patch 75 line(s) with tabs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 43s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s{color} | {color:green} the patch passed with JDK v1.8.0_131 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 46m 41s{color} | {color:red} hadoop-yarn-client in the patch failed with JDK v1.7.0_131. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 24s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}142m 36s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | JDK v1.8.0_131 Failed junit tests | hadoop.yarn.client.TestApplicationClientProtocolOnHA | | | hadoop.yarn.client.TestApplicationMasterServiceProtocolOnHA | | | hadoop.yarn.client.TestGetGroups | | | hadoop.yarn.client.TestResourceTrackerOnHA | | JDK v1.8.0_131 Timed out junit tests | org.apache.hadoop.yarn.client.TestRMFailover | | | org.apache.hadoop.yarn.client.api.impl.TestYarnClient | | |
[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression
[ https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16018128#comment-16018128 ] Hadoop QA commented on YARN-4925: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s{color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s{color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s{color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m 42s{color} | {color:green} branch-2.7 passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_131 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_131 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s{color} | {color:green} branch-2.7 passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 20s{color} | {color:green} branch-2.7 passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 10s{color} | {color:green} branch-2.7 passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 29s{color} | {color:green} branch-2.7 passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s{color} | {color:green} branch-2.7 passed with JDK v1.8.0_131 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s{color} | {color:green} branch-2.7 passed with JDK v1.7.0_131 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 10s{color} | {color:green} the patch passed with JDK v1.8.0_131 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 10s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 15s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 10s{color} | {color:green} hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client: The patch generated 0 new + 215 unchanged - 6 fixed = 215 total (was 221) {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 18s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 9s{color} | {color:green} the patch passed {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s{color} | {color:red} The patch has 1999 line(s) that end in whitespace. Use git apply --whitespace=fix <>. Refer https://git-scm.com/docs/git-apply {color} | | {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 29s{color} | {color:red} The patch 75 line(s) with tabs. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 37s{color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 8s{color} | {color:green} the patch passed with JDK v1.8.0_131 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s{color} | {color:green} the patch passed with JDK v1.7.0_131 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 46m 39s{color} | {color:red} hadoop-yarn-client in the patch failed with JDK v1.7.0_131. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 14s{color} | {color:green} The patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black}120m 6s{color} | {color:black} {color} | \\ \\ || Reason || Tests || | JDK v1.8.0_131 Failed junit tests | hadoop.yarn.client.TestGetGroups | | | hadoop.yarn.client.TestResourceTrackerOnHA | | | hadoop.yarn.client.TestApplicationClientProtocolOnHA | | | hadoop.yarn.client.TestApplicationMasterServiceProtocolOnHA | | JDK v1.8.0_131 Timed out junit tests | org.apache.hadoop.yarn.client.TestRMFailover | | | org.apache.hadoop.yarn.client.api.impl.TestAMRMClient | | | o
[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression
[ https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16017897#comment-16017897 ] Jonathan Hung commented on YARN-4925: - Attached a branch-2.7 patch. > ContainerRequest in AMRMClient, application should be able to specify > nodes/racks together with nodeLabelExpression > --- > > Key: YARN-4925 > URL: https://issues.apache.org/jira/browse/YARN-4925 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Labels: release-blocker > Fix For: 2.8.0, 3.0.0-alpha1 > > Attachments: 0001-YARN-4925.patch, 0002-YARN-4925.patch, > YARN-4925-branch-2.7.001.patch > > > Currently with nodelabel AMRMClient will not be able to specify nodelabels > with Node/Rack requests.For application like spark NODE_LOCAL requests cannot > be asked with label expression. > As per the check in {{AMRMClientImpl#checkNodeLabelExpression}} > {noformat} > // Don't allow specify node label against ANY request > if ((containerRequest.getRacks() != null && > (!containerRequest.getRacks().isEmpty())) > || > (containerRequest.getNodes() != null && > (!containerRequest.getNodes().isEmpty( { > throw new InvalidContainerRequestException( > "Cannot specify node label with rack and node"); > } > {noformat} > {{AppSchedulingInfo#updateResourceRequests}} we do reset of labels to that of > OFF-SWITCH. > The above check is not required for ContainerRequest ask /cc [~wangda] thank > you for confirming -- This message was sent by Atlassian JIRA (v6.3.15#6346) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression
[ https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289440#comment-15289440 ] Hudson commented on YARN-4925: -- SUCCESS: Integrated in Hadoop-trunk-Commit #9816 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/9816/]) YARN-4925. ContainerRequest in AMRMClient, application should be able to (naganarasimha_gr: rev f04c81c9ce93512bc714531a7731debbe6b794ce) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClient.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/main/java/org/apache/hadoop/yarn/client/api/impl/AMRMClientImpl.java > ContainerRequest in AMRMClient, application should be able to specify > nodes/racks together with nodeLabelExpression > --- > > Key: YARN-4925 > URL: https://issues.apache.org/jira/browse/YARN-4925 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Fix For: 2.8.0 > > Attachments: 0001-YARN-4925.patch, 0002-YARN-4925.patch > > > Currently with nodelabel AMRMClient will not be able to specify nodelabels > with Node/Rack requests.For application like spark NODE_LOCAL requests cannot > be asked with label expression. > As per the check in {{AMRMClientImpl#checkNodeLabelExpression}} > {noformat} > // Don't allow specify node label against ANY request > if ((containerRequest.getRacks() != null && > (!containerRequest.getRacks().isEmpty())) > || > (containerRequest.getNodes() != null && > (!containerRequest.getNodes().isEmpty( { > throw new InvalidContainerRequestException( > "Cannot specify node label with rack and node"); > } > {noformat} > {{AppSchedulingInfo#updateResourceRequests}} we do reset of labels to that of > OFF-SWITCH. > The above check is not required for ContainerRequest ask /cc [~wangda] thank > you for confirming -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression
[ https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15289412#comment-15289412 ] Naganarasimha G R commented on YARN-4925: - Committed it to branch 2.8, branch 2 & Thanks for working on this patch > ContainerRequest in AMRMClient, application should be able to specify > nodes/racks together with nodeLabelExpression > --- > > Key: YARN-4925 > URL: https://issues.apache.org/jira/browse/YARN-4925 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-4925.patch, 0002-YARN-4925.patch > > > Currently with nodelabel AMRMClient will not be able to specify nodelabels > with Node/Rack requests.For application like spark NODE_LOCAL requests cannot > be asked with label expression. > As per the check in {{AMRMClientImpl#checkNodeLabelExpression}} > {noformat} > // Don't allow specify node label against ANY request > if ((containerRequest.getRacks() != null && > (!containerRequest.getRacks().isEmpty())) > || > (containerRequest.getNodes() != null && > (!containerRequest.getNodes().isEmpty( { > throw new InvalidContainerRequestException( > "Cannot specify node label with rack and node"); > } > {noformat} > {{AppSchedulingInfo#updateResourceRequests}} we do reset of labels to that of > OFF-SWITCH. > The above check is not required for ContainerRequest ask /cc [~wangda] thank > you for confirming -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression
[ https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275663#comment-15275663 ] Naganarasimha G R commented on YARN-4925: - Will commit it, if no one has any objections... > ContainerRequest in AMRMClient, application should be able to specify > nodes/racks together with nodeLabelExpression > --- > > Key: YARN-4925 > URL: https://issues.apache.org/jira/browse/YARN-4925 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-4925.patch, 0002-YARN-4925.patch > > > Currently with nodelabel AMRMClient will not be able to specify nodelabels > with Node/Rack requests.For application like spark NODE_LOCAL requests cannot > be asked with label expression. > As per the check in {{AMRMClientImpl#checkNodeLabelExpression}} > {noformat} > // Don't allow specify node label against ANY request > if ((containerRequest.getRacks() != null && > (!containerRequest.getRacks().isEmpty())) > || > (containerRequest.getNodes() != null && > (!containerRequest.getNodes().isEmpty( { > throw new InvalidContainerRequestException( > "Cannot specify node label with rack and node"); > } > {noformat} > {{AppSchedulingInfo#updateResourceRequests}} we do reset of labels to that of > OFF-SWITCH. > The above check is not required for ContainerRequest ask /cc [~wangda] thank > you for confirming -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression
[ https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15275662#comment-15275662 ] Naganarasimha G R commented on YARN-4925: - +1 Patch LG > ContainerRequest in AMRMClient, application should be able to specify > nodes/racks together with nodeLabelExpression > --- > > Key: YARN-4925 > URL: https://issues.apache.org/jira/browse/YARN-4925 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-4925.patch, 0002-YARN-4925.patch > > > Currently with nodelabel AMRMClient will not be able to specify nodelabels > with Node/Rack requests.For application like spark NODE_LOCAL requests cannot > be asked with label expression. > As per the check in {{AMRMClientImpl#checkNodeLabelExpression}} > {noformat} > // Don't allow specify node label against ANY request > if ((containerRequest.getRacks() != null && > (!containerRequest.getRacks().isEmpty())) > || > (containerRequest.getNodes() != null && > (!containerRequest.getNodes().isEmpty( { > throw new InvalidContainerRequestException( > "Cannot specify node label with rack and node"); > } > {noformat} > {{AppSchedulingInfo#updateResourceRequests}} we do reset of labels to that of > OFF-SWITCH. > The above check is not required for ContainerRequest ask /cc [~wangda] thank > you for confirming -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression
[ https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15269411#comment-15269411 ] Bibin A Chundatt commented on YARN-4925: [~Naganarasimha]/[~leftnoteasy] Could you please review patch attached. > ContainerRequest in AMRMClient, application should be able to specify > nodes/racks together with nodeLabelExpression > --- > > Key: YARN-4925 > URL: https://issues.apache.org/jira/browse/YARN-4925 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-4925.patch, 0002-YARN-4925.patch > > > Currently with nodelabel AMRMClient will not be able to specify nodelabels > with Node/Rack requests.For application like spark NODE_LOCAL requests cannot > be asked with label expression. > As per the check in {{AMRMClientImpl#checkNodeLabelExpression}} > {noformat} > // Don't allow specify node label against ANY request > if ((containerRequest.getRacks() != null && > (!containerRequest.getRacks().isEmpty())) > || > (containerRequest.getNodes() != null && > (!containerRequest.getNodes().isEmpty( { > throw new InvalidContainerRequestException( > "Cannot specify node label with rack and node"); > } > {noformat} > {{AppSchedulingInfo#updateResourceRequests}} we do reset of labels to that of > OFF-SWITCH. > The above check is not required for ContainerRequest ask /cc [~wangda] thank > you for confirming -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression
[ https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15265278#comment-15265278 ] Hadoop QA commented on YARN-4925: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 39s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s {color} | {color:green} trunk passed with JDK v1.8.0_92 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 19s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 14s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 22s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 13s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 34s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s {color} | {color:green} trunk passed with JDK v1.8.0_92 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 18s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 12s {color} | {color:green} the patch passed with JDK v1.8.0_92 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 12s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 20s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 10s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 42s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s {color} | {color:green} the patch passed with JDK v1.8.0_92 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 1s {color} | {color:red} hadoop-yarn-client in the patch failed with JDK v1.8.0_92. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 15s {color} | {color:red} hadoop-yarn-client in the patch failed with JDK v1.7.0_95. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 19s {color} | {color:green} Patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 145m 34s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | JDK v1.8.0_92 Failed junit tests | hadoop.yarn.client.TestGetGroups | | | hadoop.yarn.client.api.impl.TestAMRMProxy | | JDK v1.8.0_92 Timed out junit tests | org.apache.hadoop.yarn.client.cli.TestYarnCLI | | | org.apache.hadoop.yarn.client.api.impl.TestAMRMClient | | | org.apache.hadoop.yarn.client.api.impl.TestYarnClient | | | org.apache.hadoop.yarn.client.api.impl.TestNMClient | | JDK v1.7.0_95 Failed junit tests | hadoop.yarn.client.TestGetGroups | | | hadoop.yarn.client.api.impl.TestAMRMProxy | | JDK v1.7.0_95 Timed out junit tests | org.apache.hadoop.yarn.client.cli.TestYarnCLI | | | org.apache.hadoop.yarn.client.api.i
[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression
[ https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255834#comment-15255834 ] Naganarasimha G R commented on YARN-4925: - Thanks for the patch [~bibinchundatt], seems like test cases are mostly not related to the patch please cross check once and hope a test case can also be added. > ContainerRequest in AMRMClient, application should be able to specify > nodes/racks together with nodeLabelExpression > --- > > Key: YARN-4925 > URL: https://issues.apache.org/jira/browse/YARN-4925 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > Attachments: 0001-YARN-4925.patch > > > Currently with nodelabel AMRMClient will not be able to specify nodelabels > with Node/Rack requests.For application like spark NODE_LOCAL requests cannot > be asked with label expression. > As per the check in {{AMRMClientImpl#checkNodeLabelExpression}} > {noformat} > // Don't allow specify node label against ANY request > if ((containerRequest.getRacks() != null && > (!containerRequest.getRacks().isEmpty())) > || > (containerRequest.getNodes() != null && > (!containerRequest.getNodes().isEmpty( { > throw new InvalidContainerRequestException( > "Cannot specify node label with rack and node"); > } > {noformat} > {{AppSchedulingInfo#updateResourceRequests}} we do reset of labels to that of > OFF-SWITCH. > The above check is not required for ContainerRequest ask /cc [~wangda] thank > you for confirming -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression
[ https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15255647#comment-15255647 ] Hadoop QA commented on YARN-4925: - | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 15s {color} | {color:blue} Docker mode activated. {color} | | {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s {color} | {color:green} The patch does not contain any @author tags. {color} | | {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m 0s {color} | {color:green} The patch appears to include 1 new or modified test files. {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m 46s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 17s {color} | {color:green} trunk passed with JDK v1.8.0_77 {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 18s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 15s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 21s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 12s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 34s {color} | {color:green} trunk passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s {color} | {color:green} trunk passed with JDK v1.8.0_77 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 18s {color} | {color:green} trunk passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m 18s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 13s {color} | {color:green} the patch passed with JDK v1.8.0_77 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 13s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 16s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 16s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m 12s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 19s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m 11s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m 0s {color} | {color:green} Patch has no whitespace issues. {color} | | {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 42s {color} | {color:green} the patch passed {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s {color} | {color:green} the patch passed with JDK v1.8.0_77 {color} | | {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s {color} | {color:green} the patch passed with JDK v1.7.0_95 {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 11s {color} | {color:red} hadoop-yarn-client in the patch failed with JDK v1.8.0_77. {color} | | {color:red}-1{color} | {color:red} unit {color} | {color:red} 66m 23s {color} | {color:red} hadoop-yarn-client in the patch failed with JDK v1.7.0_95. {color} | | {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m 18s {color} | {color:green} Patch does not generate ASF License warnings. {color} | | {color:black}{color} | {color:black} {color} | {color:black} 146m 0s {color} | {color:black} {color} | \\ \\ || Reason || Tests || | JDK v1.8.0_77 Failed junit tests | hadoop.yarn.client.api.impl.TestAMRMProxy | | | hadoop.yarn.client.TestGetGroups | | JDK v1.8.0_77 Timed out junit tests | org.apache.hadoop.yarn.client.cli.TestYarnCLI | | | org.apache.hadoop.yarn.client.api.impl.TestYarnClient | | | org.apache.hadoop.yarn.client.api.impl.TestAMRMClient | | | org.apache.hadoop.yarn.client.api.impl.TestNMClient | | JDK v1.7.0_95 Failed junit tests | hadoop.yarn.client.api.impl.TestAMRMProxy | | | hadoop.yarn.client.TestGetGroups | | JDK v1.7.0_95 Timed out junit tests | org.apache.hadoop.yarn.client.cli.TestYarnCLI | | | org.apache.hadoop.yarn.client.api.i
[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression
[ https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229472#comment-15229472 ] Sunil G commented on YARN-4925: --- Thanks [~bibinchundatt]. Yes, NODE_LOCAL and OFF-SWITCH should not be set differently. So a change in ANY can reset other 2 types which is fine. I was just ensuring this point. No issue :) > ContainerRequest in AMRMClient, application should be able to specify > nodes/racks together with nodeLabelExpression > --- > > Key: YARN-4925 > URL: https://issues.apache.org/jira/browse/YARN-4925 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > > Currently with nodelabel AMRMClient will not be able to specify nodelabels > with Node/Rack requests.For application like spark NODE_LOCAL requests cannot > be asked with label expression. > As per the check in {{AMRMClientImpl#checkNodeLabelExpression}} > {noformat} > // Don't allow specify node label against ANY request > if ((containerRequest.getRacks() != null && > (!containerRequest.getRacks().isEmpty())) > || > (containerRequest.getNodes() != null && > (!containerRequest.getNodes().isEmpty( { > throw new InvalidContainerRequestException( > "Cannot specify node label with rack and node"); > } > {noformat} > {{AppSchedulingInfo#updateResourceRequests}} we do reset of labels to that of > OFF-SWITCH. > The above check is not required for ContainerRequest ask /cc [~wangda] thank > you for confirming -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression
[ https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15229469#comment-15229469 ] Bibin A Chundatt commented on YARN-4925: [~sunilg] and [~Naganarasimha] Thank you for looking into the issue. The check is for {{ContainerRequest}} and IIUC after the discussion with [~wangda] the check was done to *properly account how much resources an app requests for each partitions* considering that label for NODE_LOCAL and OFF-SWITCH shouldnt be set differently, which is not possible if we are setting {{ContainerRequest}} . {noformat} public static class ContainerRequest { final Resource capability; final List nodes; final List racks; final Priority priority; final boolean relaxLocality; final String nodeLabelsExpression; } {noformat} Consider in one NODELABEL there are 100 Nodes and need container to be started on NODE where data is available this is not allowed.For issue we face was for spark container needed be node local and was not allowed. cc/ [~wangda] > ContainerRequest in AMRMClient, application should be able to specify > nodes/racks together with nodeLabelExpression > --- > > Key: YARN-4925 > URL: https://issues.apache.org/jira/browse/YARN-4925 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > > Currently with nodelabel AMRMClient will not be able to specify nodelabels > with Node/Rack requests.For application like spark NODE_LOCAL requests cannot > be asked with label expression. > As per the check in {{AMRMClientImpl#checkNodeLabelExpression}} > {noformat} > // Don't allow specify node label against ANY request > if ((containerRequest.getRacks() != null && > (!containerRequest.getRacks().isEmpty())) > || > (containerRequest.getNodes() != null && > (!containerRequest.getNodes().isEmpty( { > throw new InvalidContainerRequestException( > "Cannot specify node label with rack and node"); > } > {noformat} > {{AppSchedulingInfo#updateResourceRequests}} we do reset of labels to that of > OFF-SWITCH. > The above check is not required for ContainerRequest ask /cc [~wangda] thank > you for confirming -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression
[ https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15228836#comment-15228836 ] Sunil G commented on YARN-4925: --- Thanks [~Naganarasimha Garla] for the clarification. One final doubt, in this case if we change the labelExpression for ANY containerRequest , it will replace the expression set by node/rack local earlier. Correct. > ContainerRequest in AMRMClient, application should be able to specify > nodes/racks together with nodeLabelExpression > --- > > Key: YARN-4925 > URL: https://issues.apache.org/jira/browse/YARN-4925 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > > Currently with nodelabel AMRMClient will not be able to specify nodelabels > with Node/Rack requests.For application like spark NODE_LOCAL requests cannot > be asked with label expression. > As per the check in {{AMRMClientImpl#checkNodeLabelExpression}} > {noformat} > // Don't allow specify node label against ANY request > if ((containerRequest.getRacks() != null && > (!containerRequest.getRacks().isEmpty())) > || > (containerRequest.getNodes() != null && > (!containerRequest.getNodes().isEmpty( { > throw new InvalidContainerRequestException( > "Cannot specify node label with rack and node"); > } > {noformat} > {{AppSchedulingInfo#updateResourceRequests}} we do reset of labels to that of > OFF-SWITCH. > The above check is not required for ContainerRequest ask /cc [~wangda] thank > you for confirming -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression
[ https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15228824#comment-15228824 ] Naganarasimha G R commented on YARN-4925: - IIUC, [~bibinchundatt]'s solution was to keep the code same i.e. set the nodelabel expression present containerRequest for only *any* request thus user will be able to specify node/rack locality with label expression. > ContainerRequest in AMRMClient, application should be able to specify > nodes/racks together with nodeLabelExpression > --- > > Key: YARN-4925 > URL: https://issues.apache.org/jira/browse/YARN-4925 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > > Currently with nodelabel AMRMClient will not be able to specify nodelabels > with Node/Rack requests.For application like spark NODE_LOCAL requests cannot > be asked with label expression. > As per the check in {{AMRMClientImpl#checkNodeLabelExpression}} > {noformat} > // Don't allow specify node label against ANY request > if ((containerRequest.getRacks() != null && > (!containerRequest.getRacks().isEmpty())) > || > (containerRequest.getNodes() != null && > (!containerRequest.getNodes().isEmpty( { > throw new InvalidContainerRequestException( > "Cannot specify node label with rack and node"); > } > {noformat} > {{AppSchedulingInfo#updateResourceRequests}} we do reset of labels to that of > OFF-SWITCH. > The above check is not required for ContainerRequest ask /cc [~wangda] thank > you for confirming -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression
[ https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15228803#comment-15228803 ] Sunil G commented on YARN-4925: --- Hi [~bibinchundatt] I have one doubt here. After removing this check, now ResourceRequest can have label exp for NodeLocal or RackLocal or ANY. With YARN-4140, label specified in ANY will reset all other ResourceRequest's label expression (for same priority). Is this intended OR will it solve the case mentioned in Spark scenario? > ContainerRequest in AMRMClient, application should be able to specify > nodes/racks together with nodeLabelExpression > --- > > Key: YARN-4925 > URL: https://issues.apache.org/jira/browse/YARN-4925 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > > Currently with nodelabel AMRMClient will not be able to specify nodelabels > with Node/Rack requests.For application like spark NODE_LOCAL requests cannot > be asked with label expression. > As per the check in {{AMRMClientImpl#checkNodeLabelExpression}} > {noformat} > // Don't allow specify node label against ANY request > if ((containerRequest.getRacks() != null && > (!containerRequest.getRacks().isEmpty())) > || > (containerRequest.getNodes() != null && > (!containerRequest.getNodes().isEmpty( { > throw new InvalidContainerRequestException( > "Cannot specify node label with rack and node"); > } > {noformat} > {{AppSchedulingInfo#updateResourceRequests}} we do reset of labels to that of > OFF-SWITCH. > The above check is not required for ContainerRequest ask /cc [~wangda] thank > you for confirming -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-4925) ContainerRequest in AMRMClient, application should be able to specify nodes/racks together with nodeLabelExpression
[ https://issues.apache.org/jira/browse/YARN-4925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15227695#comment-15227695 ] Naganarasimha G R commented on YARN-4925: - [~bibinchundatt], At first glance seems to be a valid issue, IIUC we can directly apply the nodelabel expression to the any request and no need to have this check. > ContainerRequest in AMRMClient, application should be able to specify > nodes/racks together with nodeLabelExpression > --- > > Key: YARN-4925 > URL: https://issues.apache.org/jira/browse/YARN-4925 > Project: Hadoop YARN > Issue Type: Bug >Reporter: Bibin A Chundatt >Assignee: Bibin A Chundatt > > Currently with nodelabel AMRMClient will not be able to specify nodelabels > with Node/Rack requests.For application like spark NODE_LOCAL requests cannot > be asked with label expression. > As per the check in {{AMRMClientImpl#checkNodeLabelExpression}} > {noformat} > // Don't allow specify node label against ANY request > if ((containerRequest.getRacks() != null && > (!containerRequest.getRacks().isEmpty())) > || > (containerRequest.getNodes() != null && > (!containerRequest.getNodes().isEmpty( { > throw new InvalidContainerRequestException( > "Cannot specify node label with rack and node"); > } > {noformat} > {{AppSchedulingInfo#updateResourceRequests}} we do reset of labels to that of > OFF-SWITCH. > The above check is not required for ContainerRequest ask /cc [~wangda] thank > you for confirming -- This message was sent by Atlassian JIRA (v6.3.4#6332)