[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14723493#comment-14723493 ] MENG DING commented on YARN-1651: - [~leftnoteasy], I just realized that there is one issue not discussed in the protocol design regarding the {{ContainerResourceChangeRequestProto}} that may affect the scheduler: {code} message ContainerResourceChangeRequestProto { optional ContainerIdProto container_id = 1; optional ResourceProto capability = 2; } {code} Shall we add a priority field to {{ContainerResourceChangeRequestProto}}? Without a priority field, how does the scheduler decide the priority between increase/decrease request and new allocation request within the same application? Does it simply assume that the increase/decrease request has the highest priority within the same application? If so, that may not be the correct thing to do ... What do you think? > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14723572#comment-14723572 ] MENG DING commented on YARN-1651: - Correction, the concern is only for resource increase request. The decrease request should be irrelevant in this context. > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14723608#comment-14723608 ] Wangda Tan commented on YARN-1651: -- Hi [~mding], I think for now, we can assume increase has higher priority. We can add priority if people think it's important. Attaching WIP patch, I think most of functionalities are completed, there're few pending items need to add some tests, and some part of code should be polished. The patch assumes increase request has higher priority than regular request. > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-WIP.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14724606#comment-14724606 ] Hadoop QA commented on YARN-1651: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:red}-1{color} | pre-patch | 21m 15s | Findbugs (version ) appears to be broken on YARN-1197. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 20 new or modified test files. | | {color:red}-1{color} | javac | 9m 5s | The applied patch generated 1 additional warning messages. | | {color:red}-1{color} | javadoc | 12m 3s | The applied patch generated 2 additional warning messages. | | {color:red}-1{color} | release audit | 0m 22s | The applied patch generated 1 release audit warnings. | | {color:green}+1{color} | checkstyle | 2m 3s | There were no new checkstyle issues. | | {color:red}-1{color} | whitespace | 26m 37s | The patch has 151 line(s) that end in whitespace. Use git apply --whitespace=fix. | | {color:green}+1{color} | install | 2m 3s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 40s | The patch built with eclipse:eclipse. | | {color:red}-1{color} | findbugs | 6m 47s | The patch appears to introduce 8 new Findbugs (version 3.0.0) warnings. | | {color:green}+1{color} | mapreduce tests | 10m 17s | Tests passed in hadoop-mapreduce-client-app. | | {color:green}+1{color} | tools/hadoop tests | 0m 57s | Tests passed in hadoop-sls. | | {color:red}-1{color} | yarn tests | 7m 26s | Tests failed in hadoop-yarn-client. | | {color:green}+1{color} | yarn tests | 0m 26s | Tests passed in hadoop-yarn-server-common. | | {color:red}-1{color} | yarn tests | 49m 36s | Tests failed in hadoop-yarn-server-resourcemanager. | | | | 149m 45s | | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-yarn-server-common | | FindBugs | module:hadoop-yarn-server-resourcemanager | | Failed unit tests | hadoop.yarn.client.TestApplicationClientProtocolOnHA | | | hadoop.yarn.client.api.impl.TestYarnClient | | | hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCNodeUpdates | | | hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl | | | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation | | | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesAppsModification | | | hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesApps | | | hadoop.yarn.server.resourcemanager.TestRMRestart | | | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing | | | hadoop.yarn.server.resourcemanager.TestResourceManager | | | hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart | | | hadoop.yarn.server.resourcemanager.webapp.TestRMWebAppFairScheduler | | | hadoop.yarn.server.resourcemanager.rmapp.attempt.TestRMAppAttemptTransitions | | | hadoop.yarn.server.resourcemanager.webapp.dao.TestFairSchedulerQueueInfo | | | hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRMRPCResponseId | | | hadoop.yarn.server.resourcemanager.TestWorkPreservingRMRestart | | Timed out tests | org.apache.hadoop.yarn.server.resourcemanager.security.TestAMRMTokens | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12753408/YARN-1651-1.YARN-1197.patch | | Optional Tests | javac unit findbugs checkstyle javadoc | | git revision | YARN-1197 / f35a945 | | javac | https://builds.apache.org/job/PreCommit-YARN-Build/8955/artifact/patchprocess/diffJavacWarnings.txt | | javadoc | https://builds.apache.org/job/PreCommit-YARN-Build/8955/artifact/patchprocess/diffJavadocWarnings.txt | | Release Audit | https://builds.apache.org/job/PreCommit-YARN-Build/8955/artifact/patchprocess/patchReleaseAuditProblems.txt | | whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/8955/artifact/patchprocess/whitespace.txt | | Findbugs warnings | https://builds.apache.org/job/PreCommit-YARN-Build/8955/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html | | Findbugs warnings | https://builds.apache.org/job/PreCommit-YARN-Build/8955/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html | | hadoop-mapreduce-client-app test log | https://builds.apache.org/job/PreCommit-YARN-Build/8955/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt | | hadoop-sls test log | https://builds.apache.org/job/PreCommit-YARN-Build/8955/artifact/patchprocess/testrun_hadoop-sls.txt | | hadoop-yarn-client test log | https://builds.apache.org/job/PreCommit-YARN-Build/8955/artifact/patchprocess/testrun_hadoop-yarn-client.txt | | hadoop-yarn-server-common test log | https://builds.apache.org/job/
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14725533#comment-14725533 ] MENG DING commented on YARN-1651: - Hi, [~leftnoteasy], thanks so much for posting the patch. I do have one question regarding the patch. Recall during the design discussion, we agreed that as long as an increase has not yet completed for a container, we should not process any other increase/decrease requests for the same container. It seems that this patch will still process decrease/increase requests even an increase action is ongoing? If the following sequence of events happen: Example 1: 1. AM sends container increase request to RM 2. RM allocates the resource and gives out increase token to AM 3. AM sends decrease request to RM for the same container 4. AM uses the increase token to increase resource on NM 5. NM reports container status back to RM IIUC, at step 3, this patch will decrease the container size, and remove the container from allocation expirer. At step 5, this patch will see that the RM container size is smaller than the reported NM container size, and will tell NM to decrease the container resource. The concern I have with this approach is that in step 4, the user will think that the increase is successfully done in NM, but in fact it won't. Also, what will happen in the following sequence of events? Example 2: 1. AM sends container increase request to RM 2. RM allocates the resource and gives out increase token (token1) to AM 3. AM sends a new container increase request for the same container to RM with more resource 4. RM allocates the resource and gives out increase token (token2) to AM 5. AM uses token1 (the one with smaller size) to increase resource on NM, but not token2 IIUC, when RM receives the increase report from NM, it will find out that the RM container size is larger than the reported NM container size, and do nothing about it, later on when token2 expires, the entire container will be killed according to the current implementation. I think this behavior could be quite confusing to the user. IMHO, at least for the case in example 2, we should delay processing of the second increase request until the first increase action is completed. > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch, > YARN-1651-WIP.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14725644#comment-14725644 ] MENG DING commented on YARN-1651: - Some additional questions: 1. For the following, I was wondering why not handling duplicate containers in ApplicationMasterService? This way, the list passed to CapacityScheduler will be unique, and we can avoid unnecessary insert/remove? {code:title=CapacityScheduler#allocate} + // Decrease containers, put it after processing increase container request + // because it's possible AM ask increase/decrease same container in a same + // heartbeat. + decreaseContainers(decreaseRequests, application); {code} 2. In RMContainerImpl.java, I think reusing the ACQUIRED, and EXPIRE events for container increase could be a little confusing? Can we make it very clear by using a separate event, like INCREASE_ACQUIRED, INCREASE_EXPIRE? Also, for NM_DONE_CHANGE_RESOURCE, how about using RESOURCE_CHANGED_IN_NM, or RESOURCE_INCREASED_IN_NM? 3. For the following, will it be completed in the 2.8 time frame? I think the penalty is a little too harsh, especially for long running services to be killed because of this. {code} +// TODO, we can do better for this: roll back container resource to the +// resource before increase, and notify scheduler about this decrease as +// well. Will do that in a separated JIRA. +container.containerIncreasedAndAcquired = false; {code} 4. For the following, I think the writeLock is not needed, as the transition is already guarded with writeLock in handle(RMNodeEvent event) {code:title=RMNodeImpl} + public static class DecreaseContainersTransition + implements SingleArcTransition { + +@Override +public void transition(RMNodeImpl rmNode, RMNodeEvent event) { + try { +rmNode.writeLock.lock(); +RMNodeDecreaseContainerEvent de = (RMNodeDecreaseContainerEvent) event; + +for (Container c : de.getToBeDecreasedContainers()) { + rmNode.toBeDecreasedContainers.put(c.getId(), c); +} + } finally { +rmNode.writeLock.unlock(); + } +} + } {code} 5. How about renaming {{CombinedContainerAllocator}} to {{GeneralContainerAllocator}}, and renaming {{RegularContainerAllocator}} to {{NewContainerAllocator}}? > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch, > YARN-1651-WIP.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14725859#comment-14725859 ] Wangda Tan commented on YARN-1651: -- Hi [~mding], Thanks for reviewing patch, it's very helpful! *1)* Regarding to: bq. Recall during the design discussion, we agreed that as long as an increase has not yet completed for a container, we should not process any other increase/decrease requests for the same container. It seems that this patch will still process decrease/increase requests even an increase action is ongoing? For the example 1, I think it's not very confusing. IIRC, we have discussed this before. We should track latest container increase/decrease status in AMRMClient. If AM send increase token to NM after it asks decrease the same container, it's AM's fault. For example 2, actually this is handled by existing code: {{AbstractYarnScheduler#containerIncreasedOnNode}} {code} else if (Resources.fitsIn(nmContainerResource, rmContainerResource)) { // when rmContainerResource >= nmContainerResource, we won't do anything, // it is possible a container increased is issued by RM, but AM hasn't // told NM. } {code} I think it's fine to me. Killing the container after increase expires is just a temp solution. I did this only because existing patch is too big, I don't want to add more confusing to it :). We should fix it as soon as we can, and it should be in the same release of YARN-1197. Do you think does it still confuse you if we fix the killing container after increase expires issue? *2)* bq. For the following, I was wondering why not handling duplicate containers in ApplicationMasterService? This way, the list passed to CapacityScheduler will be unique, and we can avoid unnecessary insert/remove? Good suggestion, will fix. *3)* bq. In RMContainerImpl.java, I think reusing the ACQUIRED, and EXPIRE events for container increase could be a little confusing? Can we make it very clear by using a separate event, like INCREASE_ACQUIRED Agree, will fix. *4)* bq. For the following, I think the writeLock is not needed, as the transition is already guarded with writeLock in handle(RMNodeEvent event) You're correct, I'm under wrong impression that handle uses synchronized lock, will fix. *5)* bq. How about renaming CombinedContainerAllocator to GeneralContainerAllocator, and renaming RegularContainerAllocator to NewContainerAllocator? I disucssed with [~jianhe] about this long time ago. NewContainerAllocator was used in early patches of YARN-3983. Actually NewContainerAllocator is same confusing, it looks like some "OldContainerAllocator" exists. GeneralContainerAllocator cannot reflect it's the parent container allocator for "regular-" and "increase-". Frankly I don't like existing naming as well, but I don't get a better name. Please let me know if your have any other thoughts. > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14725861#comment-14725861 ] Wangda Tan commented on YARN-1651: -- Removed WIP patch since it is very close to .1 patch and may cause people look at wrong version. > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14726075#comment-14726075 ] MENG DING commented on YARN-1651: - Thanks [~leftnoteasy] for the explanation. bq. For the example 1, I think it's not very confusing. IIRC, we have discussed this before. We should track latest container increase/decrease status in AMRMClient. If AM send increase token to NM after it asks decrease the same container, it's AM's fault This makes sense to me now. bq. For example 2, actually this is handled by existing code: AbstractYarnScheduler#containerIncreasedOnNode. Killing the container after increase expires is just a temp solution. I did this only because existing patch is too big, I don't want to add more confusing to it . bq. We should fix it as soon as we can, and it should be in the same release of YARN-1197. Do you think does it still confuse you if we fix the killing container after increase expires issue? This should be fine as long as we handle resource rollback correctly for multiple increase requests expiration. For example: 1. Initially a container uses 4GB 2. The first increase request increases it to 6GB (token1) 3. The second increase request increases it to 8GB (token2) If only token2 is used, expiration of token1 will be ignored If token1 is used, but token2 is expired, scheduler should be able to roll back to 6GB. If both token1 and token2 expire, scheduler must be able to roll back to 4GB. In other words, the increase expiration logic must remember all ongoing increase requests for a container, not just the latest one. bq. Frankly I don't like existing naming as well, but I don't get a better name. I think it is the name of {{CombinedContainerAllocator}} that is most confusing to me :-). How about just naming {{CombinedContainerAllocator}} as {{ContainerAllocator}}, and then renaming the current {{ContainerAllocator}} as {{AbstractContainerAllocator}}? Anyway, maybe other people will have better suggestions. I also have another suggestion regarding {{newlyDecreasedContainers}}. Right now, as soon as the scheduler decrease the container resource, it will add the container to this list, which will be pulled by AM in the next heartbeat. I think this doesn't have too much value. I was wondering, if it can be enhanced such that the list will only be updated when the decreased containers list has been sent to NM? Implementation wise, it can be similar to the idea of RMAppAttemptImpl.justFinishedContainers, and RMAppAttemptImpl.finishedContainersSentToAM. Basically RMNodeImpl keeps the containers from the *toBeDecreasedContainers* list in the memory for one more heartbeat cycle, and once RM receives the next heartbeat request from NM, it implies that the toBeDecreasedContainers from last heartbeat response has been successfully received by NM. The benefit of this approach is that once AM receives the decreased list, it knows that the decrease has been received by NM. Maybe we can take a step further to make the decrease action synchronous in NodeStatusUpdaterImpl (involves one line code change, and we already do blocking increase on NM), then once AM receives the decreased list, it knows that decrease has been completed in NM, eliminating the needs for status polling. Thoughts? > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14726303#comment-14726303 ] Wangda Tan commented on YARN-1651: -- bq. In other words, the increase expiration logic must remember all ongoing increase requests for a container, not just the latest one. I think we only need to remember *latest confirmed capacity (including decreased, confirmed from NM)*. In your example, we only need to remember 4G, token2 will overwrite token1 and expiration timout will be reset when second increase request approved. bq. renaming the current ContainerAllocator as AbstractContainerAllocator? Anyway, maybe other people will have better suggestions. This seems better than existing, thoughts? [~jianhe] bq. I also have another suggestion regarding newlyDecreasedContainers... This suggestion changes semantics of decreasedContainers in AllocateResponse. Currently, both of increasedContainers and decreasedContainers are all added to AllocateResponse when approved by RM. I think it may confuse people if AM waits few cycle to get the decreasedContainers. In addition, I cannot see a big value here: AM decreases containers just to release resources, it shouldn't care when the resource becomes available to YARN, it should decrease resource usage before sending the decrease container request to RM. Do you have any use case in your mind that AM should know about when/if container is decreased by NM? > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14727355#comment-14727355 ] MENG DING commented on YARN-1651: - bq. Do you have any use case in your mind that AM should know about when/if container is decreased by NM? You are right that there doesn't seem to be a compelling use case where AM needs to know if container resource has decreased or not in NM. We should keep it as is. bq. I think we only need to remember latest confirmed capacity (including decreased, confirmed from NM). In your example, we only need to remember 4G, token2 will overwrite token1 and expiration timout will be reset when second increase request approved. So, IIUC, if we only remember last confirmed capacity, using my example, the sequence of events will be: 1. Initially a container uses 4GB 2. The first increase request increases it to 6GB (token1) 3. The second increase request increases it to 8GB (token2). The rollback resource is still 4GB, and expiration timeout will be reset in RM. 4. AM uses the first token to increase container resource to 6GB in NM 5. NM reports 6GB to RM during heartbeat, and RM will do nothing because of the following: {code} else if (Resources.fitsIn(nmContainerResource, rmContainerResource)) { // when rmContainerResource >= nmContainerResource, we won't do anything, // it is possible a container increased is issued by RM, but AM hasn't // told NM. } {code} 6. token2 is never used by AM, and is expired. 7. Expirer now roll back container resource to 4GB in RM. Now RM and NM has inconsistent view of resource allocation ... Did I miss anything? > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14727613#comment-14727613 ] Wangda Tan commented on YARN-1651: -- [~mding], I think for your last example, if RM decides to rollback resources, it should send a container decrease event to NM as well, NM can ignore the decrease event if target size of container is equals to existing container size. Sounds like a plan? > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14727683#comment-14727683 ] MENG DING commented on YARN-1651: - I think that should work. We probably need to make sure to properly log warning messages so that user will find out what is going on and why under these circumstances. I know that both of my examples are edge cases (mostly through incorrect logic in AM), but it could be frustrating to the user when they think their container has been granted certain resource in NM, only to find out at a later time that it is not. > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14727686#comment-14727686 ] Wangda Tan commented on YARN-1651: -- [~mding], that's make sense, maybe we need to put these roll back containers to next AM-RM heartbeat. > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14728495#comment-14728495 ] Hadoop QA commented on YARN-1651: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:red}-1{color} | pre-patch | 17m 32s | Findbugs (version ) appears to be broken on YARN-1197. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 20 new or modified test files. | | {color:red}-1{color} | javac | 8m 1s | The applied patch generated 1 additional warning messages. | | {color:green}+1{color} | javadoc | 10m 8s | There were no new javadoc warning messages. | | {color:green}+1{color} | release audit | 0m 22s | The applied patch does not increase the total number of release audit warnings. | | {color:green}+1{color} | checkstyle | 1m 47s | There were no new checkstyle issues. | | {color:red}-1{color} | whitespace | 29m 39s | The patch has 159 line(s) that end in whitespace. Use git apply --whitespace=fix. | | {color:green}+1{color} | install | 1m 34s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. | | {color:red}-1{color} | findbugs | 5m 30s | The patch appears to introduce 7 new Findbugs (version 3.0.0) warnings. | | {color:green}+1{color} | mapreduce tests | 9m 19s | Tests passed in hadoop-mapreduce-client-app. | | {color:red}-1{color} | tools/hadoop tests | 0m 27s | Tests failed in hadoop-sls. | | {color:green}+1{color} | yarn tests | 6m 58s | Tests passed in hadoop-yarn-client. | | {color:green}+1{color} | yarn tests | 0m 25s | Tests passed in hadoop-yarn-server-common. | | {color:red}-1{color} | yarn tests | 57m 14s | Tests failed in hadoop-yarn-server-resourcemanager. | | | | 149m 40s | | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-yarn-server-common | | Failed unit tests | hadoop.yarn.sls.TestSLSRunner | | | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation | | | hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart | | | hadoop.yarn.server.resourcemanager.recovery.TestFSRMStateStore | | | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12753880/YARN-1651-2.YARN-1197.patch | | Optional Tests | javac unit findbugs checkstyle javadoc | | git revision | YARN-1197 / f35a945 | | javac | https://builds.apache.org/job/PreCommit-YARN-Build/8991/artifact/patchprocess/diffJavacWarnings.txt | | whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/8991/artifact/patchprocess/whitespace.txt | | Findbugs warnings | https://builds.apache.org/job/PreCommit-YARN-Build/8991/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html | | hadoop-mapreduce-client-app test log | https://builds.apache.org/job/PreCommit-YARN-Build/8991/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt | | hadoop-sls test log | https://builds.apache.org/job/PreCommit-YARN-Build/8991/artifact/patchprocess/testrun_hadoop-sls.txt | | hadoop-yarn-client test log | https://builds.apache.org/job/PreCommit-YARN-Build/8991/artifact/patchprocess/testrun_hadoop-yarn-client.txt | | hadoop-yarn-server-common test log | https://builds.apache.org/job/PreCommit-YARN-Build/8991/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt | | hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/8991/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/8991/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/8991/console | This message was automatically generated. > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch, YARN-1651-2.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14729706#comment-14729706 ] MENG DING commented on YARN-1651: - Hi, [~leftnoteasy] I think it is fine to reuse Expire for now for container increase expiration. We probably need to address this properly in the JIRA that tracks container resource increase roll back. (I think Container resource increase expiration should be tracked as a Scheduler Event, e.g., SchedulerEventType.CONTAINER_INCREASE_EXPIRE) I have a few more comments or questions regarding the patch: * Regarding sanity checks: ** The following functions can be removed? {{ApplicationMaster.checkDuplicatedIncreaseDecreaseRequest()}} ** About {{RMServerUtils.checkDuplicatedIncreaseDecreaseRequest()}}. It seems that this function throws exception whenever there is a duplicated id. Shall we handle the case where if there are both increase and decrease requests for the same id, we can ignore the increase but keep the decrease request? ** Will it be better to combine all sanity checks into one function, e.g., {{validateIncreaseDecreaseRequest(List incRequests, List decRequests)}}, such that it will check both duplicated IDs, and the resource validity for increase and decrease requests? ** For {{validateIncreaseDecreaseRequest}}, we don't check minimum allocation now, is it intended? I see that later on you normalize the request so that it will be at least minimum allocation. Just want to confirm. * For {{SchedulerApplicationAttempt.pullNewlyUpdatedContainers}}. ** This function is used by both pullNewlyIncreasedContainers(), and pullNewlyDecreasedContainers(). Why do we need to call {{updateContainerAndNMToken}} for decreased containers? It also unnecessarily send a ACQUIRE_UPDATED_CONTAINER event for every decreased container? ** We should probably check null before adding updatedContainer? {code:title=pullNewlyUpdatedContainers} Container updatedContainer = updateContainerAndNMToken(rmContainer, false); returnContainerList.add(updatedContainer); {code} * It seems {{RMNodeImpl.pullNewlyIncreasedContainers()}} is empty? * The following function doesn't seem to be used? {code:title=AppSchedulingInfo} public synchronized void notifyContainerStopped(RMContainer rmContainer) { // remove from pending increase request map if it exists removeIncreaseRequest(rmContainer.getAllocatedNode(), rmContainer.getAllocatedPriority(), rmContainer.getContainerId()); } {code} * In {{IncreaseContainerAllocator.assignContainers}}: ** I think the following is a typo, should be {{if (cannotAllocateAnything)}}, right? {code} if (shouldUnreserve) { LOG.debug("We cannot allocate anything because of low headroom, " + "headroom=" + resourceLimits.getHeadroom()); } {code} ** Not sure if I understand the logic. Why only break when node.getReservedContainer() == null? Shouldn't we break out of the loop here no matter what? {code} while (iter.hasNext()) { ... ... // Try to allocate the increase request assigned = allocateIncreaseRequest(node, increaseRequest); if (node.getReservedContainer() == null) { // if it's not a reserved increase request, we will record // priority/containerId so that we can remove the request later increasedContainerPriority = priority; increasedContainerId = rmContainer.getContainerId(); break; } } {code} ** Is the following needed? {code} if (increasedContainerId != null) { // If we increased (not reserved) a new increase request, we should // remove it from request map. application.removeIncreaseRequest(nodeId, increasedContainerPriority, increasedContainerId); } {code} I think earlier in the {{allocateIncreaseRequest()}} function, if a new increase is successfully allocated, {{application.increaseContainer(increaseRequest)}} will have removed the increase request already? * In {{RMContainerImpl.java}} IIUC, {{containerIncreased}} indicates that a increase is done in scheduler, and {{containerIncreasedAndAcquired}} indicates that a increase has been acquired by AM. If so, then in {{NMReportedContainerChangeIsDoneTransition}} {code} public void transition(RMContainerImpl container, RMContainerEvent event) { if (container.containerIncreased) { // If container is increased but not acquired by AM, we will start // containerAllocationExpirer for this container in this transition. container.containerAllocationExpirer.unregister(event.getContainerId()); container.containerIncreasedAndAcquired = false; } } {code} Shouldn't it be changed to: {code} public void transition(RMContainerImpl container, RMContainerEvent event) { if (cont
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731505#comment-14731505 ] Wangda Tan commented on YARN-1651: -- Hi Meng, Thanks for comments: bq. We probably need to address this properly in the JIRA that tracks container resource increase roll back. (I think Container resource increase expiration should be tracked as a Scheduler Event, e.g., SchedulerEventType.CONTAINER_INCREASE_EXPIRE) I think we can do that either via adding CONTAINER_INCREASE_EXPIRE or directly call decrease of scheduler from RMContainer, I'm not sure which one is better, let's figure it out when doing it. bq. It seems that this function throws exception whenever there is a duplicated id. Shall we handle the case where if there are both increase and decrease requests for the same id, we can ignore the increase but keep the decrease request? I have thought about this before, I think it's hard to decide when two request with same containerId but different target resource exists, which one will be chosen. And it's not like an expected allocate request as well. So I prefer to reject both. bq. Will it be better to combine all sanity checks into one function Done bq. For validateIncreaseDecreaseRequest, we don't check minimum allocation now, is it intended? Yes it's intended, because we will normalize it later, so no need to throw exception. bq. This function is used by both pullNewlyIncreasedContainers(), and pullNewlyDecreasedContainers(). Why do we need to call updateContainerAndNMToken for decreased containers? It also unnecessarily send a ACQUIRE_UPDATED_CONTAINER event for every decreased container? This is majorly makes logic correct and consistent, we don't use the container token for now, but I think we should make it updated before return to app. Unless we have any performance issue of doing this, I prefer to keep existing behavior. bq. We should probably check null before adding updatedContainer? It seems no need to do the null check here. When it becomes null? I prefer to keep it as-is and it will throw NPE if any fatal issue happens. bq. RMNodeImpl.pullNewlyIncreasedContainers() Implemented. bq. AppSchedulingInfo#notifyContainerStopped not being used. Removed, we handled this in LeafQueue#completedContainer. bq. I think the following is a typo, should be if (cannotAllocateAnything), right? Correct, fixed. bq. Not sure if I understand the logic. Why only break when node.getReservedContainer() == null? Shouldn't we break out of the loop here no matter what? Nice catch! I fixed this, we should break when we allocated or reserved anything. bq. I think earlier in the allocateIncreaseRequest() function, if a new increase is successfully allocated, application.increaseContainer(increaseRequest) will have removed the increase request already? Another nice catch! Yes we should already handled it in application.increaseContainer bq. RMContainerImpl...Shouldn't it be changed to... Yes, it should do as you said, updated. bq. Also, is container.containerIncreased really needed? It's needed when we don't know if a acquired event is for an increasedContainer or decreasedContainer, added isIncreaseContainer to acquire event (now it's RMContainerUpdatesAcquiredEvent). And removed RMContainerImpl.containerIncreased. Typos: fixed. > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch, YARN-1651-2.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14731696#comment-14731696 ] Hadoop QA commented on YARN-1651: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:red}-1{color} | pre-patch | 17m 31s | Findbugs (version ) appears to be broken on YARN-1197. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 20 new or modified test files. | | {color:red}-1{color} | javac | 7m 57s | The applied patch generated 1 additional warning messages. | | {color:green}+1{color} | javadoc | 10m 5s | There were no new javadoc warning messages. | | {color:green}+1{color} | release audit | 0m 24s | The applied patch does not increase the total number of release audit warnings. | | {color:green}+1{color} | checkstyle | 1m 52s | There were no new checkstyle issues. | | {color:red}-1{color} | whitespace | 30m 2s | The patch has 162 line(s) that end in whitespace. Use git apply --whitespace=fix. | | {color:green}+1{color} | install | 1m 31s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with eclipse:eclipse. | | {color:red}-1{color} | findbugs | 5m 25s | The patch appears to introduce 7 new Findbugs (version 3.0.0) warnings. | | {color:green}+1{color} | mapreduce tests | 9m 16s | Tests passed in hadoop-mapreduce-client-app. | | {color:green}+1{color} | tools/hadoop tests | 0m 51s | Tests passed in hadoop-sls. | | {color:green}+1{color} | yarn tests | 6m 59s | Tests passed in hadoop-yarn-client. | | {color:green}+1{color} | yarn tests | 0m 24s | Tests passed in hadoop-yarn-server-common. | | {color:red}-1{color} | yarn tests | 58m 15s | Tests failed in hadoop-yarn-server-resourcemanager. | | | | 151m 13s | | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-yarn-server-common | | Failed unit tests | hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart | | | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation | | | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12754278/YARN-1651-3.YARN-1197.patch | | Optional Tests | javac unit findbugs checkstyle javadoc | | git revision | YARN-1197 / f86eae1 | | javac | https://builds.apache.org/job/PreCommit-YARN-Build/9013/artifact/patchprocess/diffJavacWarnings.txt | | whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/9013/artifact/patchprocess/whitespace.txt | | Findbugs warnings | https://builds.apache.org/job/PreCommit-YARN-Build/9013/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html | | hadoop-mapreduce-client-app test log | https://builds.apache.org/job/PreCommit-YARN-Build/9013/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt | | hadoop-sls test log | https://builds.apache.org/job/PreCommit-YARN-Build/9013/artifact/patchprocess/testrun_hadoop-sls.txt | | hadoop-yarn-client test log | https://builds.apache.org/job/PreCommit-YARN-Build/9013/artifact/patchprocess/testrun_hadoop-yarn-client.txt | | hadoop-yarn-server-common test log | https://builds.apache.org/job/PreCommit-YARN-Build/9013/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt | | hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/9013/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/9013/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/9013/console | This message was automatically generated. > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch, > YARN-1651-2.YARN-1197.patch, YARN-1651-3.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14735070#comment-14735070 ] MENG DING commented on YARN-1651: - Hi, [~leftnoteasy] I am ok with most of the reply comments. Thanks. bq. It seems no need to do the null check here. When it becomes null? I prefer to keep it as-is and it will throw NPE if any fatal issue happens. The {{updateContainerAndNMToken}} may return null: {code} Container updatedContainer = updateContainerAndNMToken(rmContainer, false, increase); returnContainerList.add(updatedContainer); {code} I only mention this because {{pullNewlyAllocatedContainers()}} has a check for null for the same logic, so I think we may want to make it consistent? Some remaining comments: * As you mentioned in the code, currently reserved resource increase request does not participate in the continuous reservation looking logic. So, based on my understanding, if an application has reserved some resource for a container resource increase request on a node, that amount of resource should never be unreserved in order for the application to allocate a regular container on some other node. But that doesn't seem to be the case right now? Can you confirm? If so, I am thinking a simple solution would be to *exclude* resources reserved for increased containers when trying to find an unreserved container for regular container allocation. {code:title=RegularContainerAllocator.assignContainer} ... ... unreservedContainer = application.findNodeToUnreserve(clusterResource, node, priority, <= Don't consider resources reserved for container increase request resourceNeedToUnReserve); ... {code} * I think it will be desirable to implement a {{pendingDecrease}} set in {{SchedulerApplicationAttempt}}, and corresponding logic, just like {{SchedulerApplicationAttempt.pendingRelease}}. This is to guard against the situation *when decrease requests are received while RM is in the middle of recovery, and has not received all container statuses from NM yet*. * Some nits ** Comments in {{NMReportedContainerChangeIsDoneTransition}} doesn't seem right. ** IncreaseContainerAllocator: {{LOG.debug(" Headroom is satisifed, skip..");}} --> {{LOG.debug(" Headroom is not satisfied, skip..");}} > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch, > YARN-1651-2.YARN-1197.patch, YARN-1651-3.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14735676#comment-14735676 ] MENG DING commented on YARN-1651: - Hi, [~leftnoteasy] bq. I agree the general idea, and we should do the similar thing. However, I'm not sure caching in RM is a good idea, potentially a malicious AM can send millions of unknown-to-be-decreased-containers to RM when RM started. Maybe it's better to cache in AMRMClient side. I think we can do this in a separated JIRA? Could you file a new JIRA for this if you agree? Your proposal makes sense. I will file a JIRA for this. Thanks for addressing my comments. I don't have more comments for now. As per our discussion, I will come up with an end-to-end test based on distributedshell, and post onto this JIRA for review. > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch, > YARN-1651-2.YARN-1197.patch, YARN-1651-3.YARN-1197.patch, > YARN-1651-4.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14735859#comment-14735859 ] Hadoop QA commented on YARN-1651: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:red}-1{color} | pre-patch | 18m 2s | Findbugs (version ) appears to be broken on YARN-1197. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 20 new or modified test files. | | {color:red}-1{color} | javac | 8m 10s | The applied patch generated 1 additional warning messages. | | {color:green}+1{color} | javadoc | 10m 17s | There were no new javadoc warning messages. | | {color:green}+1{color} | release audit | 0m 24s | The applied patch does not increase the total number of release audit warnings. | | {color:green}+1{color} | checkstyle | 1m 55s | There were no new checkstyle issues. | | {color:red}-1{color} | whitespace | 31m 2s | The patch has 163 line(s) that end in whitespace. Use git apply --whitespace=fix. | | {color:green}+1{color} | install | 1m 35s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with eclipse:eclipse. | | {color:red}-1{color} | findbugs | 5m 29s | The patch appears to introduce 7 new Findbugs (version 3.0.0) warnings. | | {color:green}+1{color} | mapreduce tests | 9m 26s | Tests passed in hadoop-mapreduce-client-app. | | {color:green}+1{color} | tools/hadoop tests | 0m 53s | Tests passed in hadoop-sls. | | {color:green}+1{color} | yarn tests | 6m 58s | Tests passed in hadoop-yarn-client. | | {color:green}+1{color} | yarn tests | 0m 26s | Tests passed in hadoop-yarn-server-common. | | {color:red}-1{color} | yarn tests | 59m 24s | Tests failed in hadoop-yarn-server-resourcemanager. | | | | 154m 43s | | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-yarn-server-common | | Failed unit tests | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestReservations | | | hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart | | | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer | | | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12754736/YARN-1651-4.YARN-1197.patch | | Optional Tests | javac unit findbugs checkstyle javadoc | | git revision | YARN-1197 / f86eae1 | | javac | https://builds.apache.org/job/PreCommit-YARN-Build/9045/artifact/patchprocess/diffJavacWarnings.txt | | whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/9045/artifact/patchprocess/whitespace.txt | | Findbugs warnings | https://builds.apache.org/job/PreCommit-YARN-Build/9045/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html | | hadoop-mapreduce-client-app test log | https://builds.apache.org/job/PreCommit-YARN-Build/9045/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt | | hadoop-sls test log | https://builds.apache.org/job/PreCommit-YARN-Build/9045/artifact/patchprocess/testrun_hadoop-sls.txt | | hadoop-yarn-client test log | https://builds.apache.org/job/PreCommit-YARN-Build/9045/artifact/patchprocess/testrun_hadoop-yarn-client.txt | | hadoop-yarn-server-common test log | https://builds.apache.org/job/PreCommit-YARN-Build/9045/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt | | hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/9045/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/9045/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/9045/console | This message was automatically generated. > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch, > YARN-1651-2.YARN-1197.patch, YARN-1651-3.YARN-1197.patch, > YARN-1651-4.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14736779#comment-14736779 ] Jian He commented on YARN-1651: --- Thanks Wangda, some comments on my side: - Remove ApplicationMasterService#checkDuplicatedIncreaseDecreaseRequest - revert ClientRMService change - AbstractYarnScheduler#checkAndNormalizeContainerChangeRequests -- attempt parameter is not used -- this will never be null ? {code} if (null == sr) {
 continue;
} {code} -- pass exception as a parameter, instead of string concatenation ? {code} LOG.warn("Error happens when checking increase request:" + e
+ ". Ignoring this request"); {code} -- the invalid resize requests are simply ignored. How will the AM know that their resize requests are ignored and so lost ? Similar thing for the toBeRemovedRequests inside IncreaseContainerAllocator. - updateIncreaseRequests: originalResource-> prevChangeRequest; similarly for the logging {{original capacity = }} - use decreaseRequest#getSchedulerNode instead of csContext.getNode(decreaseRequest.getNodeId()) {code} unreserveContainerIncreaseRequest(clusterResource, app,
 csContext.getNode(decreaseRequest.getNodeId()), rmContainer); csContext.getNode(decreaseRequest.getNodeId())
 .decreaseContainer(decreaseRequest.getContainerId(), absDelta); {code} - remove SchedulerApplicationAttempt#hasIncreaseRequestForContainer - newlyIncreasedContainers will not be null {code} List newlyIncreasedContainers =
 nm.pullNewlyIncreasedContainers(); 
if (null != newlyIncreasedContainers) { {code} - why is the decision made to give increase request highest priority ? I think we need to augment the ContainerResourceChangeRequest API to have the priority too so that it can be compared against regular resource requests? {code} /*
 * Try to allocate increase container first, and if we failed to allocate
 * anything, we will try to allocate regular container
 */ {code} - SchedulerContainerResourceChangeRequest, this name is so long that it reduces code readability a lot as it’s used in so many places, how about SchedContainerChangeRequest ? or any other concise name you can think of ? - It is true that node resource would change. But changing node resource is such a rare event that the probability of hitting this much lower than hitting it in the allocate call. I suggest we should also fail the allocate call in this case. {code} // The reason of doing check here instead of adding increase request
// to scheduler because node's resource could be updated after
// request added. {code} - why equal ? {code} if (null == request
|| reservedContainer.getContainerState() != ContainerState.RUNNING
|| Resources.equals(reservedContainer.getReservedResource(),
 request.getDeltaCapacity())) { {code} - allocateIncreaseRequestFromReservedContainer- remove the unused parameter - I think killing a container in this case is too harsh. Nothing is wrong with the container itself. The container can run fine. I think we should revert the increased resource and have some way to signal back that the increaseRequest failed, maybe in the allocate response ? {code} // When the container expired, and it has a pending increased request, we
// will kill the container. 
new KillTransition().transition(container, event); {code} - I think the container.containerIncreasedAndAcquired boolean flag is not needed? - unreserveContainerIncreaseRequest-> unreserveIncreasedContainer - internalReleaseResource: remove unused application parameter - fix the format {code} RMAppAttempt appAttempt = rmContext .getRMApps() .get( container.getId().getApplicationAttemptId() .getApplicationId()).getCurrentAppAttempt(); {code} > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch, > YARN-1651-2.YARN-1197.patch, YARN-1651-3.YARN-1197.patch, > YARN-1651-4.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14737174#comment-14737174 ] MENG DING commented on YARN-1651: - bq. I think killing a container in this case is too harsh. Nothing is wrong with the container itself. The container can run fine. I think we should revert the increased resource and have some way to signal back that the increaseRequest failed, maybe in the allocate response ? Hi, [~jianhe], a sub-task (YARN-4138) has been logged for this issue. Based on our current discussion, when an increase token expires, RM will revert the increased resource back and also send a decrease event to NM (NM can ignore the decrease event if target size of container is equal to existing container size). As a bonus, the decrease can be pulled by AM in AllocateResponse.decreased_containers which acts as a signal for the expired increase action. > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch, > YARN-1651-2.YARN-1197.patch, YARN-1651-3.YARN-1197.patch, > YARN-1651-4.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14737947#comment-14737947 ] Hadoop QA commented on YARN-1651: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:red}-1{color} | pre-patch | 18m 30s | Findbugs (version ) appears to be broken on YARN-1197. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 22 new or modified test files. | | {color:red}-1{color} | javac | 8m 1s | The applied patch generated 1 additional warning messages. | | {color:green}+1{color} | javadoc | 10m 18s | There were no new javadoc warning messages. | | {color:green}+1{color} | release audit | 0m 29s | The applied patch does not increase the total number of release audit warnings. | | {color:green}+1{color} | checkstyle | 2m 40s | There were no new checkstyle issues. | | {color:red}-1{color} | whitespace | 34m 45s | The patch has 163 line(s) that end in whitespace. Use git apply --whitespace=fix. | | {color:green}+1{color} | install | 1m 38s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with eclipse:eclipse. | | {color:red}-1{color} | findbugs | 8m 14s | The patch appears to introduce 7 new Findbugs (version 3.0.0) warnings. | | {color:green}+1{color} | mapreduce tests | 9m 18s | Tests passed in hadoop-mapreduce-client-app. | | {color:green}+1{color} | tools/hadoop tests | 0m 53s | Tests passed in hadoop-sls. | | {color:green}+1{color} | yarn tests | 6m 53s | Tests passed in hadoop-yarn-client. | | {color:green}+1{color} | yarn tests | 2m 2s | Tests passed in hadoop-yarn-common. | | {color:green}+1{color} | yarn tests | 0m 25s | Tests passed in hadoop-yarn-server-common. | | {color:green}+1{color} | yarn tests | 8m 21s | Tests passed in hadoop-yarn-server-nodemanager. | | {color:red}-1{color} | yarn tests | 63m 9s | Tests failed in hadoop-yarn-server-resourcemanager. | | | | 176m 17s | | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-yarn-server-common | | Failed unit tests | hadoop.yarn.server.resourcemanager.rmcontainer.TestRMContainerImpl | | | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerAllocation | | | hadoop.yarn.server.resourcemanager.scheduler.capacity.TestContainerResizing | | | hadoop.yarn.server.resourcemanager.applicationsmanager.TestAMRestart | | | hadoop.yarn.server.resourcemanager.security.TestDelegationTokenRenewer | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12754992/YARN-1651-5.YARN-1197.patch | | Optional Tests | javac unit findbugs checkstyle javadoc | | git revision | YARN-1197 / f86eae1 | | javac | https://builds.apache.org/job/PreCommit-YARN-Build/9069/artifact/patchprocess/diffJavacWarnings.txt | | whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/9069/artifact/patchprocess/whitespace.txt | | Findbugs warnings | https://builds.apache.org/job/PreCommit-YARN-Build/9069/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html | | hadoop-mapreduce-client-app test log | https://builds.apache.org/job/PreCommit-YARN-Build/9069/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt | | hadoop-sls test log | https://builds.apache.org/job/PreCommit-YARN-Build/9069/artifact/patchprocess/testrun_hadoop-sls.txt | | hadoop-yarn-client test log | https://builds.apache.org/job/PreCommit-YARN-Build/9069/artifact/patchprocess/testrun_hadoop-yarn-client.txt | | hadoop-yarn-common test log | https://builds.apache.org/job/PreCommit-YARN-Build/9069/artifact/patchprocess/testrun_hadoop-yarn-common.txt | | hadoop-yarn-server-common test log | https://builds.apache.org/job/PreCommit-YARN-Build/9069/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt | | hadoop-yarn-server-nodemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/9069/artifact/patchprocess/testrun_hadoop-yarn-server-nodemanager.txt | | hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/9069/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/9069/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/9069/console | This message was automatically generated. > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Ke
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738568#comment-14738568 ] Jian He commented on YARN-1651: --- bq. I think we may need add such information to AMRMProtocol to make sure AM will be notified. For now, we can keep them as-is. Users can still get such information from RM logs. I think for now we can fail the allocate call explicitly on those very clear situations in checkAndNormalizeContainerChangeRequest ?, e.g. the situation that rmContainer doesn't exist That's more explicit to users. Digging through logs is not an easy thing for application writer. thanks for updating, Wangda ! some more comments focusing on decreasing code path. - this may be not correct, because reserve event can happen on RESERVE state too, i.e. reReservation {code} if (container.getState() != RMContainerState.NEW) { container.hasIncreaseReservation = true; } {code} - RMNodeImpl#toBeDecreasedContainers - no need to be a map, it can be a list ? and therefore NodeHeartBeatResponse and Impl change is not needed; similarly nmReportedIncreasedContainers can be a list. - When decreasing a container, should it send RMNodeDecreaseContainerEvent too ? - revert ContainerManagerImpl change - Remove SchedulerApplicationAttempt#getIncreaseRequests - In AbstractYarnScheduler#deceraseContainers() move checkAndNormalizeContainerChangeRequests(decreaseRequests, false) to the same place as checkAndNormalizeContainerChangeRequests(increaseRequests, false) for consistency. - this if condition is not needed. {code} public boolean unreserve(Priority priority, FiCaSchedulerNode node, RMContainer rmContainer) { if (rmContainer.hasIncreaseReservation()) { rmContainer.cancelIncreaseReservation(); } {code} - looks like when decreasing reservedIncreasedContainer, it will unreserve the *whole* extra reserved resource, should it only unreserve the extra resources being decresed ? - In general, I think we should be able to decrease/increase a regular reserved container or a increasedReservedContainer ? - In ParentQueue, this null check is not needed. {code} @Override public void decreaseContainer(Resource clusterResource, SchedContainerChangeRequest decreaseRequest, FiCaSchedulerApp app) { if (app != null) { {code} - allocate call is specifically marked as noLock, but now every allocate call holds the global scheduler lock which is too expensive. we can move decreaseContainer to application itself. {code} protected synchronized void decreaseContainer( {code} It is also now holding queue Lock on allocate, which is also expensive, because that means a bunch of malicious AMs can effectively block queue's execuation. > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch, > YARN-1651-2.YARN-1197.patch, YARN-1651-3.YARN-1197.patch, > YARN-1651-4.YARN-1197.patch, YARN-1651-5.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14738830#comment-14738830 ] MENG DING commented on YARN-1651: - Hi, [~leftnoteasy] One comment I forgot to post is that we may want to synchronize the RMContainerImpl.getAllocatedResource() call? Because the container resource may be updated at any time, e.g: {code:title=RMContainerImpl.java} @Override public Resource getAllocatedResource() { -return container.getResource(); +try { + readLock.lock(); + return Resources.clone(container.getResource()); +} finally { + readLock.unlock(); +} } {code} > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch, > YARN-1651-2.YARN-1197.patch, YARN-1651-3.YARN-1197.patch, > YARN-1651-4.YARN-1197.patch, YARN-1651-5.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14739742#comment-14739742 ] Wangda Tan commented on YARN-1651: -- Thanks review! [~jianhe]. bq. I think for now we can fail the allocate call explicitly on those very clear situations in checkAndNormalizeContainerChangeRequest ?, e.g. the situation that rmContainer doesn't exist That's more explicit to users. Digging through logs is not an easy thing for application writer. Done, now we will check it in both AMS/Scheduler, exception will be thrown in AMS. Doing both check because AMS doesn't acquire scheduler lock, so it is still possbile that RMContainer state changed when adding to scheduler. bq. RMNodeImpl#toBeDecreasedContainers - no need to be a map, it can be a list ? and therefore NodeHeartBeatResponse and Impl change is not needed; similarly nmReportedIncreasedContainers can be a list. This is to avoid AM decrease same container multiple times between same NM heartbeats, this is a rare edge case. Similar for NM reports increasedContainers, if we decouple NM heartbeat and scheduler allocation, we could have container increased multiple times between scheduler looks at NM. bq. When decreasing a container, should it send RMNodeDecreaseContainerEvent too ? Done, added test to confirm this as well. bq. looks like when decreasing reservedIncreasedContainer, it will unreserve the whole extra reserved resource, should it only unreserve the extra resources being decresed ? Decrease container is decrease resource of a container to lower than confirmed resource. If a container is 2G, AM asks to increase to 4G, it can only decrease it to less than 2G before increase issued. So I think we need to unreserve the whole extra reserved resource. bq. In general, I think we should be able to decrease/increase a regular reserved container or a increasedReservedContainer ? Container reservation is an internal state of scheduler, AM doesn't know about the reserved container at know, so far I think we don't need to expose that to user. bq. allocate call is specifically marked as noLock, but now every allocate call holds the global scheduler lock which is too expensive. we can move decreaseContainer to application itself. DecreaseContainer is as same as completedContainer, both acquire scheduler lock and queue lock. I think we can optimize it in the future, which we can add them to something like "pendingReleased" list, and will be traversed periodically. I added comments to CS#allocate to explain about this, the "NoLock" is not 100% acurate. And addressed all other comments. [~mding] Comment addressed. > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch, > YARN-1651-2.YARN-1197.patch, YARN-1651-3.YARN-1197.patch, > YARN-1651-4.YARN-1197.patch, YARN-1651-5.YARN-1197.patch, > YARN-1651-6.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14740114#comment-14740114 ] Hadoop QA commented on YARN-1651: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:red}-1{color} | pre-patch | 18m 19s | Findbugs (version ) appears to be broken on YARN-1197. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 23 new or modified test files. | | {color:red}-1{color} | javac | 8m 18s | The applied patch generated 1 additional warning messages. | | {color:green}+1{color} | javadoc | 11m 39s | There were no new javadoc warning messages. | | {color:green}+1{color} | release audit | 0m 26s | The applied patch does not increase the total number of release audit warnings. | | {color:green}+1{color} | checkstyle | 2m 30s | There were no new checkstyle issues. | | {color:red}-1{color} | whitespace | 43m 12s | The patch has 177 line(s) that end in whitespace. Use git apply --whitespace=fix. | | {color:green}+1{color} | install | 1m 48s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 46s | The patch built with eclipse:eclipse. | | {color:red}-1{color} | findbugs | 9m 13s | The patch appears to introduce 7 new Findbugs (version 3.0.0) warnings. | | {color:green}+1{color} | mapreduce tests | 9m 52s | Tests passed in hadoop-mapreduce-client-app. | | {color:green}+1{color} | tools/hadoop tests | 1m 0s | Tests passed in hadoop-sls. | | {color:red}-1{color} | yarn tests | 6m 43s | Tests failed in hadoop-yarn-client. | | {color:green}+1{color} | yarn tests | 1m 59s | Tests passed in hadoop-yarn-common. | | {color:green}+1{color} | yarn tests | 0m 26s | Tests passed in hadoop-yarn-server-common. | | {color:green}+1{color} | yarn tests | 55m 30s | Tests passed in hadoop-yarn-server-resourcemanager. | | | | 171m 51s | | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-yarn-server-common | | Failed unit tests | hadoop.yarn.client.api.impl.TestYarnClient | | Timed out tests | org.apache.hadoop.yarn.client.api.impl.TestNMClient | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12755257/YARN-1651-6.YARN-1197.patch | | Optional Tests | javac unit findbugs checkstyle javadoc | | git revision | YARN-1197 / f86eae1 | | javac | https://builds.apache.org/job/PreCommit-YARN-Build/9079/artifact/patchprocess/diffJavacWarnings.txt | | whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/9079/artifact/patchprocess/whitespace.txt | | Findbugs warnings | https://builds.apache.org/job/PreCommit-YARN-Build/9079/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html | | hadoop-mapreduce-client-app test log | https://builds.apache.org/job/PreCommit-YARN-Build/9079/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt | | hadoop-sls test log | https://builds.apache.org/job/PreCommit-YARN-Build/9079/artifact/patchprocess/testrun_hadoop-sls.txt | | hadoop-yarn-client test log | https://builds.apache.org/job/PreCommit-YARN-Build/9079/artifact/patchprocess/testrun_hadoop-yarn-client.txt | | hadoop-yarn-common test log | https://builds.apache.org/job/PreCommit-YARN-Build/9079/artifact/patchprocess/testrun_hadoop-yarn-common.txt | | hadoop-yarn-server-common test log | https://builds.apache.org/job/PreCommit-YARN-Build/9079/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt | | hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/9079/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/9079/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/9079/console | This message was automatically generated. > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch, > YARN-1651-2.YARN-1197.patch, YARN-1651-3.YARN-1197.patch, > YARN-1651-4.YARN-1197.patch, YARN-1651-5.YARN-1197.patch, > YARN-1651-6.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14740717#comment-14740717 ] Jian He commented on YARN-1651: --- few more comments: - schedulerNode#increaseContainer is not invoked when increasing regular container ? add a test? - remove ContainersAndNMTokensAllocation in SchedulerApplicationAttempt - FiCaSchedulerApp#unreserve -> unreserveIncreasedContainer to avoid name confliction. {code} public boolean unreserve(Priority priority, FiCaSchedulerNode node, RMContainer rmContainer) { {code} - revert RMWebServices, AssignmentInformation changes - MockAM#allocateChangeContainerRequests -> resizeContainers > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch, > YARN-1651-2.YARN-1197.patch, YARN-1651-3.YARN-1197.patch, > YARN-1651-4.YARN-1197.patch, YARN-1651-5.YARN-1197.patch, > YARN-1651-6.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14741371#comment-14741371 ] Wangda Tan commented on YARN-1651: -- Also rebased YARN-1197 branch to latest trunk. > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch, > YARN-1651-2.YARN-1197.patch, YARN-1651-3.YARN-1197.patch, > YARN-1651-4.YARN-1197.patch, YARN-1651-5.YARN-1197.patch, > YARN-1651-6.YARN-1197.patch, YARN-1651-7.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14742963#comment-14742963 ] Hadoop QA commented on YARN-1651: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:red}-1{color} | pre-patch | 3m 10s | YARN-1197 compilation may be broken. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 23 new or modified test files. | | {color:green}+1{color} | javac | 8m 15s | There were no new javac warning messages. | | {color:red}-1{color} | javadoc | 10m 17s | The applied patch generated 65 additional warning messages. | | {color:green}+1{color} | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. | | {color:green}+1{color} | checkstyle | 2m 21s | There were no new checkstyle issues. | | {color:red}-1{color} | whitespace | 37m 18s | The patch has 177 line(s) that end in whitespace. Use git apply --whitespace=fix. | | {color:green}+1{color} | install | 1m 37s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with eclipse:eclipse. | | {color:red}-1{color} | findbugs | 7m 9s | The patch appears to introduce 8 new Findbugs (version 3.0.0) warnings. | | {color:green}+1{color} | mapreduce tests | 9m 25s | Tests passed in hadoop-mapreduce-client-app. | | {color:green}+1{color} | tools/hadoop tests | 0m 53s | Tests passed in hadoop-sls. | | {color:green}+1{color} | yarn tests | 6m 58s | Tests passed in hadoop-yarn-client. | | {color:green}+1{color} | yarn tests | 2m 3s | Tests passed in hadoop-yarn-common. | | {color:green}+1{color} | yarn tests | 0m 26s | Tests passed in hadoop-yarn-server-common. | | {color:green}+1{color} | yarn tests | 55m 46s | Tests passed in hadoop-yarn-server-resourcemanager. | | | | 146m 45s | | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-yarn-server-common | | FindBugs | module:hadoop-yarn-server-resourcemanager | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12755643/YARN-1651-8.YARN-1197.patch | | Optional Tests | javac unit findbugs checkstyle javadoc | | git revision | YARN-1197 / 78ad04d | | javadoc | https://builds.apache.org/job/PreCommit-YARN-Build/9110/artifact/patchprocess/diffJavadocWarnings.txt | | whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/9110/artifact/patchprocess/whitespace.txt | | Findbugs warnings | https://builds.apache.org/job/PreCommit-YARN-Build/9110/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html | | Findbugs warnings | https://builds.apache.org/job/PreCommit-YARN-Build/9110/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html | | hadoop-mapreduce-client-app test log | https://builds.apache.org/job/PreCommit-YARN-Build/9110/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt | | hadoop-sls test log | https://builds.apache.org/job/PreCommit-YARN-Build/9110/artifact/patchprocess/testrun_hadoop-sls.txt | | hadoop-yarn-client test log | https://builds.apache.org/job/PreCommit-YARN-Build/9110/artifact/patchprocess/testrun_hadoop-yarn-client.txt | | hadoop-yarn-common test log | https://builds.apache.org/job/PreCommit-YARN-Build/9110/artifact/patchprocess/testrun_hadoop-yarn-common.txt | | hadoop-yarn-server-common test log | https://builds.apache.org/job/PreCommit-YARN-Build/9110/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt | | hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/9110/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/9110/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/9110/console | This message was automatically generated. > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch, > YARN-1651-2.YARN-1197.patch, YARN-1651-3.YARN-1197.patch, > YARN-1651-4.YARN-1197.patch, YARN-1651-5.YARN-1197.patch, > YARN-1651-6.YARN-1197.patch, YARN
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14742978#comment-14742978 ] Hadoop QA commented on YARN-1651: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:red}-1{color} | pre-patch | 2m 59s | YARN-1197 compilation may be broken. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 23 new or modified test files. | | {color:green}+1{color} | javac | 7m 38s | There were no new javac warning messages. | | {color:red}-1{color} | javadoc | 10m 1s | The applied patch generated 65 additional warning messages. | | {color:green}+1{color} | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. | | {color:green}+1{color} | checkstyle | 2m 16s | There were no new checkstyle issues. | | {color:red}-1{color} | whitespace | 51m 21s | The patch has 177 line(s) that end in whitespace. Use git apply --whitespace=fix. | | {color:green}+1{color} | install | 1m 30s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. | | {color:red}-1{color} | findbugs | 7m 0s | The patch appears to introduce 8 new Findbugs (version 3.0.0) warnings. | | {color:red}-1{color} | mapreduce tests | 9m 9s | Tests failed in hadoop-mapreduce-client-app. | | {color:green}+1{color} | tools/hadoop tests | 0m 53s | Tests passed in hadoop-sls. | | {color:green}+1{color} | yarn tests | 6m 51s | Tests passed in hadoop-yarn-client. | | {color:green}+1{color} | yarn tests | 2m 0s | Tests passed in hadoop-yarn-common. | | {color:green}+1{color} | yarn tests | 0m 25s | Tests passed in hadoop-yarn-server-common. | | {color:green}+1{color} | yarn tests | 55m 28s | Tests passed in hadoop-yarn-server-resourcemanager. | | | | 158m 38s | | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-yarn-server-common | | FindBugs | module:hadoop-yarn-server-resourcemanager | | Failed unit tests | hadoop.mapreduce.v2.app.webapp.TestAMWebServicesJobs | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12755643/YARN-1651-8.YARN-1197.patch | | Optional Tests | javac unit findbugs checkstyle javadoc | | git revision | YARN-1197 / 78ad04d | | javadoc | https://builds.apache.org/job/PreCommit-YARN-Build/9111/artifact/patchprocess/diffJavadocWarnings.txt | | whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/9111/artifact/patchprocess/whitespace.txt | | Findbugs warnings | https://builds.apache.org/job/PreCommit-YARN-Build/9111/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html | | Findbugs warnings | https://builds.apache.org/job/PreCommit-YARN-Build/9111/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html | | hadoop-mapreduce-client-app test log | https://builds.apache.org/job/PreCommit-YARN-Build/9111/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt | | hadoop-sls test log | https://builds.apache.org/job/PreCommit-YARN-Build/9111/artifact/patchprocess/testrun_hadoop-sls.txt | | hadoop-yarn-client test log | https://builds.apache.org/job/PreCommit-YARN-Build/9111/artifact/patchprocess/testrun_hadoop-yarn-client.txt | | hadoop-yarn-common test log | https://builds.apache.org/job/PreCommit-YARN-Build/9111/artifact/patchprocess/testrun_hadoop-yarn-common.txt | | hadoop-yarn-server-common test log | https://builds.apache.org/job/PreCommit-YARN-Build/9111/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt | | hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/9111/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/9111/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/9111/console | This message was automatically generated. > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch, > YARN-1651-2.YARN-1197.patch, YARN-1651-3.YARN-1197.patch, > YARN-1651-4.YARN-11
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743002#comment-14743002 ] Hadoop QA commented on YARN-1651: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:red}-1{color} | pre-patch | 3m 10s | YARN-1197 compilation may be broken. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 23 new or modified test files. | | {color:green}+1{color} | javac | 8m 10s | There were no new javac warning messages. | | {color:red}-1{color} | javadoc | 10m 16s | The applied patch generated 65 additional warning messages. | | {color:green}+1{color} | release audit | 0m 26s | The applied patch does not increase the total number of release audit warnings. | | {color:green}+1{color} | checkstyle | 2m 16s | There were no new checkstyle issues. | | {color:red}-1{color} | whitespace | 37m 6s | The patch has 177 line(s) that end in whitespace. Use git apply --whitespace=fix. | | {color:green}+1{color} | install | 1m 34s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with eclipse:eclipse. | | {color:red}-1{color} | findbugs | 7m 11s | The patch appears to introduce 8 new Findbugs (version 3.0.0) warnings. | | {color:green}+1{color} | mapreduce tests | 9m 26s | Tests passed in hadoop-mapreduce-client-app. | | {color:green}+1{color} | tools/hadoop tests | 0m 54s | Tests passed in hadoop-sls. | | {color:red}-1{color} | yarn tests | 6m 53s | Tests failed in hadoop-yarn-client. | | {color:green}+1{color} | yarn tests | 2m 3s | Tests passed in hadoop-yarn-common. | | {color:green}+1{color} | yarn tests | 0m 26s | Tests passed in hadoop-yarn-server-common. | | {color:green}+1{color} | yarn tests | 59m 52s | Tests passed in hadoop-yarn-server-resourcemanager. | | | | 150m 26s | | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-yarn-server-common | | FindBugs | module:hadoop-yarn-server-resourcemanager | | Failed unit tests | hadoop.yarn.client.api.impl.TestYarnClient | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12755652/YARN-1651-9.YARN-1197.patch | | Optional Tests | javac unit findbugs checkstyle javadoc | | git revision | YARN-1197 / 78ad04d | | javadoc | https://builds.apache.org/job/PreCommit-YARN-Build/9114/artifact/patchprocess/diffJavadocWarnings.txt | | whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/9114/artifact/patchprocess/whitespace.txt | | Findbugs warnings | https://builds.apache.org/job/PreCommit-YARN-Build/9114/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html | | Findbugs warnings | https://builds.apache.org/job/PreCommit-YARN-Build/9114/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html | | hadoop-mapreduce-client-app test log | https://builds.apache.org/job/PreCommit-YARN-Build/9114/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt | | hadoop-sls test log | https://builds.apache.org/job/PreCommit-YARN-Build/9114/artifact/patchprocess/testrun_hadoop-sls.txt | | hadoop-yarn-client test log | https://builds.apache.org/job/PreCommit-YARN-Build/9114/artifact/patchprocess/testrun_hadoop-yarn-client.txt | | hadoop-yarn-common test log | https://builds.apache.org/job/PreCommit-YARN-Build/9114/artifact/patchprocess/testrun_hadoop-yarn-common.txt | | hadoop-yarn-server-common test log | https://builds.apache.org/job/PreCommit-YARN-Build/9114/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt | | hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/9114/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/9114/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/9114/console | This message was automatically generated. > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch, > YARN-1651-10.YARN-1197.patch, YARN-1651-2.YARN-1197.patch, > YARN-1651-3.YARN-1197.patch,
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743191#comment-14743191 ] Hadoop QA commented on YARN-1651: - \\ \\ | (x) *{color:red}-1 overall{color}* | \\ \\ || Vote || Subsystem || Runtime || Comment || | {color:red}-1{color} | pre-patch | 3m 13s | YARN-1197 compilation may be broken. | | {color:green}+1{color} | @author | 0m 0s | The patch does not contain any @author tags. | | {color:green}+1{color} | tests included | 0m 0s | The patch appears to include 23 new or modified test files. | | {color:green}+1{color} | javac | 7m 52s | There were no new javac warning messages. | | {color:red}-1{color} | javadoc | 9m 54s | The applied patch generated 65 additional warning messages. | | {color:green}+1{color} | release audit | 0m 23s | The applied patch does not increase the total number of release audit warnings. | | {color:green}+1{color} | checkstyle | 2m 17s | There were no new checkstyle issues. | | {color:red}-1{color} | whitespace | 35m 36s | The patch has 177 line(s) that end in whitespace. Use git apply --whitespace=fix. | | {color:green}+1{color} | install | 1m 31s | mvn install still works. | | {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with eclipse:eclipse. | | {color:red}-1{color} | findbugs | 7m 3s | The patch appears to introduce 8 new Findbugs (version 3.0.0) warnings. | | {color:red}-1{color} | mapreduce tests | 9m 19s | Tests failed in hadoop-mapreduce-client-app. | | {color:green}+1{color} | tools/hadoop tests | 0m 52s | Tests passed in hadoop-sls. | | {color:green}+1{color} | yarn tests | 6m 56s | Tests passed in hadoop-yarn-client. | | {color:green}+1{color} | yarn tests | 2m 0s | Tests passed in hadoop-yarn-common. | | {color:green}+1{color} | yarn tests | 0m 25s | Tests passed in hadoop-yarn-server-common. | | {color:green}+1{color} | yarn tests | 55m 37s | Tests passed in hadoop-yarn-server-resourcemanager. | | | | 143m 38s | | \\ \\ || Reason || Tests || | FindBugs | module:hadoop-yarn-server-common | | FindBugs | module:hadoop-yarn-server-resourcemanager | | Failed unit tests | hadoop.mapreduce.v2.app.rm.TestRMContainerAllocator | \\ \\ || Subsystem || Report/Notes || | Patch URL | http://issues.apache.org/jira/secure/attachment/12755671/YARN-1651-10.YARN-1197.patch | | Optional Tests | javac unit findbugs checkstyle javadoc | | git revision | YARN-1197 / 78ad04d | | javadoc | https://builds.apache.org/job/PreCommit-YARN-Build/9117/artifact/patchprocess/diffJavadocWarnings.txt | | whitespace | https://builds.apache.org/job/PreCommit-YARN-Build/9117/artifact/patchprocess/whitespace.txt | | Findbugs warnings | https://builds.apache.org/job/PreCommit-YARN-Build/9117/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html | | Findbugs warnings | https://builds.apache.org/job/PreCommit-YARN-Build/9117/artifact/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html | | hadoop-mapreduce-client-app test log | https://builds.apache.org/job/PreCommit-YARN-Build/9117/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt | | hadoop-sls test log | https://builds.apache.org/job/PreCommit-YARN-Build/9117/artifact/patchprocess/testrun_hadoop-sls.txt | | hadoop-yarn-client test log | https://builds.apache.org/job/PreCommit-YARN-Build/9117/artifact/patchprocess/testrun_hadoop-yarn-client.txt | | hadoop-yarn-common test log | https://builds.apache.org/job/PreCommit-YARN-Build/9117/artifact/patchprocess/testrun_hadoop-yarn-common.txt | | hadoop-yarn-server-common test log | https://builds.apache.org/job/PreCommit-YARN-Build/9117/artifact/patchprocess/testrun_hadoop-yarn-server-common.txt | | hadoop-yarn-server-resourcemanager test log | https://builds.apache.org/job/PreCommit-YARN-Build/9117/artifact/patchprocess/testrun_hadoop-yarn-server-resourcemanager.txt | | Test Results | https://builds.apache.org/job/PreCommit-YARN-Build/9117/testReport/ | | Java | 1.7.0_55 | | uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux | | Console output | https://builds.apache.org/job/PreCommit-YARN-Build/9117/console | This message was automatically generated. > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch, > YARN-1651-10.YARN-1197.patch, YARN-1651-2.YARN-1197.patch, > YARN-1651-3.YARN-1
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14743646#comment-14743646 ] Wangda Tan commented on YARN-1651: -- javadocs warnings are not related to this patch, mr test failure tracked by: https://issues.apache.org/jira/browse/MAPREDUCE-6475. Findbugs warnings are not related to this patch: https://issues.apache.org/jira/browse/YARN-1644?focusedCommentId=14739800&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14739800. > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Attachments: YARN-1651-1.YARN-1197.patch, > YARN-1651-10.YARN-1197.patch, YARN-1651-2.YARN-1197.patch, > YARN-1651-3.YARN-1197.patch, YARN-1651-4.YARN-1197.patch, > YARN-1651-5.YARN-1197.patch, YARN-1651-6.YARN-1197.patch, > YARN-1651-7.YARN-1197.patch, YARN-1651-8.YARN-1197.patch, > YARN-1651-9.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14744806#comment-14744806 ] Wangda Tan commented on YARN-1651: -- Thanks [~jianhe] and [~mding] for very thorough review! > CapacityScheduler side changes to support increase/decrease container > resource. > --- > > Key: YARN-1651 > URL: https://issues.apache.org/jira/browse/YARN-1651 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager, scheduler >Reporter: Wangda Tan >Assignee: Wangda Tan > Fix For: YARN-1197 > > Attachments: YARN-1651-1.YARN-1197.patch, > YARN-1651-10.YARN-1197.patch, YARN-1651-2.YARN-1197.patch, > YARN-1651-3.YARN-1197.patch, YARN-1651-4.YARN-1197.patch, > YARN-1651-5.YARN-1197.patch, YARN-1651-6.YARN-1197.patch, > YARN-1651-7.YARN-1197.patch, YARN-1651-8.YARN-1197.patch, > YARN-1651-9.YARN-1197.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14905314#comment-14905314 ] Hudson commented on YARN-1651: -- FAILURE: Integrated in Hadoop-trunk-Commit #8505 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8505/]) YARN-1651. CapacityScheduler side changes to support container resize. Contributed by Wangda Tan (wangda: rev 89cab1ba5f0671f8ef30dbe7432079c18362b434) * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/RMNodeWrapper.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerNode.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/Application.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeDecreaseContainerEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/AbstractContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java * hadoop-yarn-project/CHANGES.txt * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerUpdatesAcquiredEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerChangeResourceEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerEventType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14905348#comment-14905348 ] Hudson commented on YARN-1651: -- FAILURE: Integrated in Hadoop-Yarn-trunk-Java8 #436 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk-Java8/436/]) YARN-1651. CapacityScheduler side changes to support container resize. Contributed by Wangda Tan (wangda: rev 89cab1ba5f0671f8ef30dbe7432079c18362b434) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeEventType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/TestRMContainerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/AbstractContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/IncreaseContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/TestFifoScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAuditLogger.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockAM.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeDecreaseContainerEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/Allocation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestContinuousScheduling.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/nodemanager/NodeInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/ResourceCalculator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/ma
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14905421#comment-14905421 ] Hudson commented on YARN-1651: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk-Java8 #429 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Java8/429/]) YARN-1651. CapacityScheduler side changes to support container resize. Contributed by Wangda Tan (wangda: rev 89cab1ba5f0671f8ef30dbe7432079c18362b434) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/DominantResourceCalculator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/nodemanager/NodeInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerChangeResourceEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedContainerChangeRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/logaggregationstatus/TestRMAppLogAggregationStatus.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/RMNodeWrapper.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/IncreaseContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockRM.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplication.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourceman
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14905528#comment-14905528 ] Hudson commented on YARN-1651: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #1169 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/1169/]) YARN-1651. CapacityScheduler side changes to support container resize. Contributed by Wangda Tan (wangda: rev 89cab1ba5f0671f8ef30dbe7432079c18362b434) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/Application.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainer.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/AbstractContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/Allocation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/logaggregationstatus/TestRMAppLogAggregationStatus.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerUpdatesAcquiredEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplication.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/DefaultResourceCalculator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java * hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/rm/TestRMContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerChangeResourceEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestContinuousScheduling.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/o
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14905633#comment-14905633 ] Hudson commented on YARN-1651: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #409 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/409/]) YARN-1651. CapacityScheduler side changes to support container resize. Contributed by Wangda Tan (wangda: rev 89cab1ba5f0671f8ef30dbe7432079c18362b434) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/ResourceCalculator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AbstractYarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/DominantResourceCalculator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/Allocation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairScheduler.java * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/RMNodeWrapper.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerChangeResourceEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNode.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedContainerChangeRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ApplicationMasterService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplication.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/FairSchedulerTestBase.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerUpdatesAcquiredEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-client/src/test/java/org/apache/hadoop/yarn/client/api/impl/TestAMRMClientOnRMRestart.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeEventType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/Application.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/AbstractContainerAllocator.java * hadoop-y
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14905702#comment-14905702 ] Hudson commented on YARN-1651: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #2375 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/2375/]) YARN-1651. CapacityScheduler side changes to support container resize. Contributed by Wangda Tan (wangda: rev 89cab1ba5f0671f8ef30dbe7432079c18362b434) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerAllocation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/impl/pb/NodeHeartbeatResponsePBImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/Application.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestApplicationMasterService.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/AbstractCSQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/QueueMetrics.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedContainerChangeRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestChildQueueOrder.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/test/java/org/apache/hadoop/yarn/util/resource/TestResourceCalculator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestContinuousScheduling.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/SLSCapacityScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeStatusEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/ContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeEventType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplication.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSAssignment.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerChangeResourceEvent.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/Resources.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/Allocation.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-common/src/main/java/org/apache/hadoop/yarn/server/api/protocolrecords/NodeHeartbeatResponse.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-serve
[jira] [Commented] (YARN-1651) CapacityScheduler side changes to support increase/decrease container resource.
[ https://issues.apache.org/jira/browse/YARN-1651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14905751#comment-14905751 ] Hudson commented on YARN-1651: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #2348 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/2348/]) YARN-1651. CapacityScheduler side changes to support container resize. Contributed by Wangda Tan (wangda: rev 89cab1ba5f0671f8ef30dbe7432079c18362b434) * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/Application.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestContinuousScheduling.java * hadoop-tools/hadoop-sls/src/main/java/org/apache/hadoop/yarn/sls/scheduler/RMNodeWrapper.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeEventType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/allocator/RegularContainerAllocator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/RMAppAttemptImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApplicationAttempt.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/YarnScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockAM.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmcontainer/RMContainerEventType.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestContainerResizing.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedContainerChangeRequest.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/util/resource/DominantResourceCalculator.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMAuditLogger.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fair/TestFairScheduler.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/common/fica/FiCaSchedulerApp.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/rmapp/attempt/TestRMAppAttemptTransitions.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/logaggregationstatus/TestRMAppLogAggregationStatus.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/applicationsmanager/TestAMRestart.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/RMServerUtils.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/AppSchedulingInfo.java * hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestReservations.java * hadoop-ya