[jira] [Updated] (YARN-2022) Preempting an Application Master container can be kept as least priority when multiple applications are marked for preemption by ProportionalCapacityPreemptionPolicy
[ https://issues.apache.org/jira/browse/YARN-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-2022: -- Description: Cluster Size = 16GB [2NM's] Queue A Capacity = 50% Queue B Capacity = 50% Consider there are 3 applications running in Queue A which has taken the full cluster capacity. J1 = 2GB AM + 1GB * 4 Maps J2 = 2GB AM + 1GB * 4 Maps J3 = 2GB AM + 1GB * 2 Maps Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ]. Currently in this scenario, Jobs J3 will get killed including its AM. It is better if AM can be given least priority among multiple applications. In this same scenario, map tasks from J3 and J4 can be preempted. Later when cluster is free, maps can be allocated to these Jobs. was: Cluster Size = 16GB [2NM's] Queue A Capacity = 50% Queue B Capacity = 50% Consider there are 3 applications running in Queue A which has taken the full cluster capacity. J1 = 2GB AM + 1GB * 4 Maps J2 = 2GB AM + 1GB * 4 Maps J3 = 2GB AM + 1GB * 2 Maps Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ]. Currently in this scenario, Jobs J3 will get killed including its AM. It is better if AM can be given least priroity among multiple applications. In this same scenario, map tasks from J3 and J4 can be preempted. Later when cluster is free, maps can be allocated to these Jobs. > Preempting an Application Master container can be kept as least priority when > multiple applications are marked for preemption by > ProportionalCapacityPreemptionPolicy > - > > Key: YARN-2022 > URL: https://issues.apache.org/jira/browse/YARN-2022 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Sunil G >Assignee: Sunil G > > Cluster Size = 16GB [2NM's] > Queue A Capacity = 50% > Queue B Capacity = 50% > Consider there are 3 applications running in Queue A which has taken the full > cluster capacity. > J1 = 2GB AM + 1GB * 4 Maps > J2 = 2GB AM + 1GB * 4 Maps > J3 = 2GB AM + 1GB * 2 Maps > Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ]. > Currently in this scenario, Jobs J3 will get killed including its AM. > It is better if AM can be given least priority among multiple applications. > In this same scenario, map tasks from J3 and J4 can be preempted. > Later when cluster is free, maps can be allocated to these Jobs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2022) Preempting an Application Master container can be kept as least priority when multiple applications are marked for preemption by ProportionalCapacityPreemptionPolicy
[ https://issues.apache.org/jira/browse/YARN-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-2022: -- Issue Type: Sub-task (was: Improvement) Parent: YARN-45 > Preempting an Application Master container can be kept as least priority when > multiple applications are marked for preemption by > ProportionalCapacityPreemptionPolicy > - > > Key: YARN-2022 > URL: https://issues.apache.org/jira/browse/YARN-2022 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Sunil G >Assignee: Sunil G > Attachments: Yarn-2022.1.patch > > > Cluster Size = 16GB [2NM's] > Queue A Capacity = 50% > Queue B Capacity = 50% > Consider there are 3 applications running in Queue A which has taken the full > cluster capacity. > J1 = 2GB AM + 1GB * 4 Maps > J2 = 2GB AM + 1GB * 4 Maps > J3 = 2GB AM + 1GB * 2 Maps > Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ]. > Currently in this scenario, Jobs J3 will get killed including its AM. > It is better if AM can be given least priority among multiple applications. > In this same scenario, map tasks from J3 and J2 can be preempted. > Later when cluster is free, maps can be allocated to these Jobs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2022) Preempting an Application Master container can be kept as least priority when multiple applications are marked for preemption by ProportionalCapacityPreemptionPolicy
[ https://issues.apache.org/jira/browse/YARN-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-2022: -- Attachment: YARN-2022-DesignDraft.docx Hi [~curino] I have attached a Design Draft document and I tried to capture the corner cases. This draft also includes the approach to handle the same. Please review the same and share your thoughts. > Preempting an Application Master container can be kept as least priority when > multiple applications are marked for preemption by > ProportionalCapacityPreemptionPolicy > - > > Key: YARN-2022 > URL: https://issues.apache.org/jira/browse/YARN-2022 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-2022-DesignDraft.docx, Yarn-2022.1.patch > > > Cluster Size = 16GB [2NM's] > Queue A Capacity = 50% > Queue B Capacity = 50% > Consider there are 3 applications running in Queue A which has taken the full > cluster capacity. > J1 = 2GB AM + 1GB * 4 Maps > J2 = 2GB AM + 1GB * 4 Maps > J3 = 2GB AM + 1GB * 2 Maps > Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ]. > Currently in this scenario, Jobs J3 will get killed including its AM. > It is better if AM can be given least priority among multiple applications. > In this same scenario, map tasks from J3 and J2 can be preempted. > Later when cluster is free, maps can be allocated to these Jobs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2022) Preempting an Application Master container can be kept as least priority when multiple applications are marked for preemption by ProportionalCapacityPreemptionPolicy
[ https://issues.apache.org/jira/browse/YARN-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-2022: -- Attachment: YARN-2022.2.patch Thank you Mayank and Carlo. I have attached a patch as per discussion. Kindly review the same. > Preempting an Application Master container can be kept as least priority when > multiple applications are marked for preemption by > ProportionalCapacityPreemptionPolicy > - > > Key: YARN-2022 > URL: https://issues.apache.org/jira/browse/YARN-2022 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-2022-DesignDraft.docx, YARN-2022.2.patch, > Yarn-2022.1.patch > > > Cluster Size = 16GB [2NM's] > Queue A Capacity = 50% > Queue B Capacity = 50% > Consider there are 3 applications running in Queue A which has taken the full > cluster capacity. > J1 = 2GB AM + 1GB * 4 Maps > J2 = 2GB AM + 1GB * 4 Maps > J3 = 2GB AM + 1GB * 2 Maps > Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ]. > Currently in this scenario, Jobs J3 will get killed including its AM. > It is better if AM can be given least priority among multiple applications. > In this same scenario, map tasks from J3 and J2 can be preempted. > Later when cluster is free, maps can be allocated to these Jobs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2022) Preempting an Application Master container can be kept as least priority when multiple applications are marked for preemption by ProportionalCapacityPreemptionPolicy
[ https://issues.apache.org/jira/browse/YARN-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-2022: -- Attachment: YARN-2022.3.patch Updating patch to correct test failures. > Preempting an Application Master container can be kept as least priority when > multiple applications are marked for preemption by > ProportionalCapacityPreemptionPolicy > - > > Key: YARN-2022 > URL: https://issues.apache.org/jira/browse/YARN-2022 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-2022-DesignDraft.docx, YARN-2022.2.patch, > YARN-2022.3.patch, Yarn-2022.1.patch > > > Cluster Size = 16GB [2NM's] > Queue A Capacity = 50% > Queue B Capacity = 50% > Consider there are 3 applications running in Queue A which has taken the full > cluster capacity. > J1 = 2GB AM + 1GB * 4 Maps > J2 = 2GB AM + 1GB * 4 Maps > J3 = 2GB AM + 1GB * 2 Maps > Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ]. > Currently in this scenario, Jobs J3 will get killed including its AM. > It is better if AM can be given least priority among multiple applications. > In this same scenario, map tasks from J3 and J2 can be preempted. > Later when cluster is free, maps can be allocated to these Jobs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2022) Preempting an Application Master container can be kept as least priority when multiple applications are marked for preemption by ProportionalCapacityPreemptionPolicy
[ https://issues.apache.org/jira/browse/YARN-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-2022: -- Attachment: YARN-2022.4.patch Thank you [~mayank_bansal] for the review. I updated a patch as per the review comments. yarn.scheduler.capacity..maximum-am-resource-percent will always default to yarn.scheduler.capacity.maximum-am-resource-percent, if not set. Hence I was using *LeafQueue#getMaxAMResourcePerQueuePercent* only. getMaxAMResourcePercent was used only in default case of yarn.scheduler.capacity..maximum-am-resource-percent. Kindly check the same and please let me know your comments > Preempting an Application Master container can be kept as least priority when > multiple applications are marked for preemption by > ProportionalCapacityPreemptionPolicy > - > > Key: YARN-2022 > URL: https://issues.apache.org/jira/browse/YARN-2022 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-2022-DesignDraft.docx, YARN-2022.2.patch, > YARN-2022.3.patch, YARN-2022.4.patch, Yarn-2022.1.patch > > > Cluster Size = 16GB [2NM's] > Queue A Capacity = 50% > Queue B Capacity = 50% > Consider there are 3 applications running in Queue A which has taken the full > cluster capacity. > J1 = 2GB AM + 1GB * 4 Maps > J2 = 2GB AM + 1GB * 4 Maps > J3 = 2GB AM + 1GB * 2 Maps > Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ]. > Currently in this scenario, Jobs J3 will get killed including its AM. > It is better if AM can be given least priority among multiple applications. > In this same scenario, map tasks from J3 and J2 can be preempted. > Later when cluster is free, maps can be allocated to these Jobs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2022) Preempting an Application Master container can be kept as least priority when multiple applications are marked for preemption by ProportionalCapacityPreemptionPolicy
[ https://issues.apache.org/jira/browse/YARN-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-2022: -- Attachment: YARN-2022.5.patch Thank you Mayank. I have updated the patch as per the comments. Also I did test on real cluster, and found that AM Container are spared by the proportional policy. Basic scenarios are tested as part of this. > Preempting an Application Master container can be kept as least priority when > multiple applications are marked for preemption by > ProportionalCapacityPreemptionPolicy > - > > Key: YARN-2022 > URL: https://issues.apache.org/jira/browse/YARN-2022 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-2022-DesignDraft.docx, YARN-2022.2.patch, > YARN-2022.3.patch, YARN-2022.4.patch, YARN-2022.5.patch, Yarn-2022.1.patch > > > Cluster Size = 16GB [2NM's] > Queue A Capacity = 50% > Queue B Capacity = 50% > Consider there are 3 applications running in Queue A which has taken the full > cluster capacity. > J1 = 2GB AM + 1GB * 4 Maps > J2 = 2GB AM + 1GB * 4 Maps > J3 = 2GB AM + 1GB * 2 Maps > Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ]. > Currently in this scenario, Jobs J3 will get killed including its AM. > It is better if AM can be given least priority among multiple applications. > In this same scenario, map tasks from J3 and J2 can be preempted. > Later when cluster is free, maps can be allocated to these Jobs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2022) Preempting an Application Master container can be kept as least priority when multiple applications are marked for preemption by ProportionalCapacityPreemptionPolicy
[ https://issues.apache.org/jira/browse/YARN-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-2022: -- Attachment: YARN-2022.6.patch Thank you [~mayank_bansal]. I have updated the patch with new test case named testAMResourcePercentForSkippedAMContainers for AMResourcePercent test. Kindly review. > Preempting an Application Master container can be kept as least priority when > multiple applications are marked for preemption by > ProportionalCapacityPreemptionPolicy > - > > Key: YARN-2022 > URL: https://issues.apache.org/jira/browse/YARN-2022 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-2022-DesignDraft.docx, YARN-2022.2.patch, > YARN-2022.3.patch, YARN-2022.4.patch, YARN-2022.5.patch, YARN-2022.6.patch, > Yarn-2022.1.patch > > > Cluster Size = 16GB [2NM's] > Queue A Capacity = 50% > Queue B Capacity = 50% > Consider there are 3 applications running in Queue A which has taken the full > cluster capacity. > J1 = 2GB AM + 1GB * 4 Maps > J2 = 2GB AM + 1GB * 4 Maps > J3 = 2GB AM + 1GB * 2 Maps > Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ]. > Currently in this scenario, Jobs J3 will get killed including its AM. > It is better if AM can be given least priority among multiple applications. > In this same scenario, map tasks from J3 and J2 can be preempted. > Later when cluster is free, maps can be allocated to these Jobs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2022) Preempting an Application Master container can be kept as least priority when multiple applications are marked for preemption by ProportionalCapacityPreemptionPolicy
[ https://issues.apache.org/jira/browse/YARN-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-2022: -- Attachment: YARN-2022.7.patch I have updated the patch w.r.t YARN-1368. Also added test case to verify whether RMContainer is marked as AM Container even after RM restart/failover. Thank you [~leftnoteasy] for pointing this. Please review. > Preempting an Application Master container can be kept as least priority when > multiple applications are marked for preemption by > ProportionalCapacityPreemptionPolicy > - > > Key: YARN-2022 > URL: https://issues.apache.org/jira/browse/YARN-2022 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-2022-DesignDraft.docx, YARN-2022.2.patch, > YARN-2022.3.patch, YARN-2022.4.patch, YARN-2022.5.patch, YARN-2022.6.patch, > YARN-2022.7.patch, Yarn-2022.1.patch > > > Cluster Size = 16GB [2NM's] > Queue A Capacity = 50% > Queue B Capacity = 50% > Consider there are 3 applications running in Queue A which has taken the full > cluster capacity. > J1 = 2GB AM + 1GB * 4 Maps > J2 = 2GB AM + 1GB * 4 Maps > J3 = 2GB AM + 1GB * 2 Maps > Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ]. > Currently in this scenario, Jobs J3 will get killed including its AM. > It is better if AM can be given least priority among multiple applications. > In this same scenario, map tasks from J3 and J2 can be preempted. > Later when cluster is free, maps can be allocated to these Jobs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2022) Preempting an Application Master container can be kept as least priority when multiple applications are marked for preemption by ProportionalCapacityPreemptionPolicy
[ https://issues.apache.org/jira/browse/YARN-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-2022: -- Attachment: YARN-2022.8.patch Thank you [~mayank_bansal] for the review. I have updated the patch by changing to getMaxCapacity. Please review. > Preempting an Application Master container can be kept as least priority when > multiple applications are marked for preemption by > ProportionalCapacityPreemptionPolicy > - > > Key: YARN-2022 > URL: https://issues.apache.org/jira/browse/YARN-2022 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-2022-DesignDraft.docx, YARN-2022.2.patch, > YARN-2022.3.patch, YARN-2022.4.patch, YARN-2022.5.patch, YARN-2022.6.patch, > YARN-2022.7.patch, YARN-2022.8.patch, Yarn-2022.1.patch > > > Cluster Size = 16GB [2NM's] > Queue A Capacity = 50% > Queue B Capacity = 50% > Consider there are 3 applications running in Queue A which has taken the full > cluster capacity. > J1 = 2GB AM + 1GB * 4 Maps > J2 = 2GB AM + 1GB * 4 Maps > J3 = 2GB AM + 1GB * 2 Maps > Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ]. > Currently in this scenario, Jobs J3 will get killed including its AM. > It is better if AM can be given least priority among multiple applications. > In this same scenario, map tasks from J3 and J2 can be preempted. > Later when cluster is free, maps can be allocated to these Jobs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2022) Preempting an Application Master container can be kept as least priority when multiple applications are marked for preemption by ProportionalCapacityPreemptionPolicy
[ https://issues.apache.org/jira/browse/YARN-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-2022: -- Attachment: YARN-2022.9.patch Thank you [~vinodkv] for the comments. bq.The other option that we may do is to skip the config completely and hard-code the skipping. Yes. I also feel we can hardcode for now. bq.RMContainer usually is a read only interface. setMasterContainer() doesn't belong here, please move it to RMContainerImpl. I moved setMasterContainer to RMContainerImpl. In this case, if I need to invoke setMasterContainer then I may need to raise an Event to RMContainerImpl (downcast may not be good). Please suggest your opinion. I also fixed other comments. Kindly review the updated patch. > Preempting an Application Master container can be kept as least priority when > multiple applications are marked for preemption by > ProportionalCapacityPreemptionPolicy > - > > Key: YARN-2022 > URL: https://issues.apache.org/jira/browse/YARN-2022 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-2022-DesignDraft.docx, YARN-2022.2.patch, > YARN-2022.3.patch, YARN-2022.4.patch, YARN-2022.5.patch, YARN-2022.6.patch, > YARN-2022.7.patch, YARN-2022.8.patch, YARN-2022.9.patch, Yarn-2022.1.patch > > > Cluster Size = 16GB [2NM's] > Queue A Capacity = 50% > Queue B Capacity = 50% > Consider there are 3 applications running in Queue A which has taken the full > cluster capacity. > J1 = 2GB AM + 1GB * 4 Maps > J2 = 2GB AM + 1GB * 4 Maps > J3 = 2GB AM + 1GB * 2 Maps > Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ]. > Currently in this scenario, Jobs J3 will get killed including its AM. > It is better if AM can be given least priority among multiple applications. > In this same scenario, map tasks from J3 and J2 can be preempted. > Later when cluster is free, maps can be allocated to these Jobs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2022) Preempting an Application Master container can be kept as least priority when multiple applications are marked for preemption by ProportionalCapacityPreemptionPolicy
[ https://issues.apache.org/jira/browse/YARN-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-2022: -- Attachment: (was: YARN-2022.9.patch) > Preempting an Application Master container can be kept as least priority when > multiple applications are marked for preemption by > ProportionalCapacityPreemptionPolicy > - > > Key: YARN-2022 > URL: https://issues.apache.org/jira/browse/YARN-2022 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-2022-DesignDraft.docx, YARN-2022.2.patch, > YARN-2022.3.patch, YARN-2022.4.patch, YARN-2022.5.patch, YARN-2022.6.patch, > YARN-2022.7.patch, YARN-2022.8.patch, Yarn-2022.1.patch > > > Cluster Size = 16GB [2NM's] > Queue A Capacity = 50% > Queue B Capacity = 50% > Consider there are 3 applications running in Queue A which has taken the full > cluster capacity. > J1 = 2GB AM + 1GB * 4 Maps > J2 = 2GB AM + 1GB * 4 Maps > J3 = 2GB AM + 1GB * 2 Maps > Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ]. > Currently in this scenario, Jobs J3 will get killed including its AM. > It is better if AM can be given least priority among multiple applications. > In this same scenario, map tasks from J3 and J2 can be preempted. > Later when cluster is free, maps can be allocated to these Jobs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2022) Preempting an Application Master container can be kept as least priority when multiple applications are marked for preemption by ProportionalCapacityPreemptionPolicy
[ https://issues.apache.org/jira/browse/YARN-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-2022: -- Attachment: YARN-2022.9.patch Thank you [~vinodkv]. I updated the patch as per the comment. Please review. > Preempting an Application Master container can be kept as least priority when > multiple applications are marked for preemption by > ProportionalCapacityPreemptionPolicy > - > > Key: YARN-2022 > URL: https://issues.apache.org/jira/browse/YARN-2022 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-2022-DesignDraft.docx, YARN-2022.2.patch, > YARN-2022.3.patch, YARN-2022.4.patch, YARN-2022.5.patch, YARN-2022.6.patch, > YARN-2022.7.patch, YARN-2022.8.patch, YARN-2022.9.patch, Yarn-2022.1.patch > > > Cluster Size = 16GB [2NM's] > Queue A Capacity = 50% > Queue B Capacity = 50% > Consider there are 3 applications running in Queue A which has taken the full > cluster capacity. > J1 = 2GB AM + 1GB * 4 Maps > J2 = 2GB AM + 1GB * 4 Maps > J3 = 2GB AM + 1GB * 2 Maps > Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ]. > Currently in this scenario, Jobs J3 will get killed including its AM. > It is better if AM can be given least priority among multiple applications. > In this same scenario, map tasks from J3 and J2 can be preempted. > Later when cluster is free, maps can be allocated to these Jobs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2022) Preempting an Application Master container can be kept as least priority when multiple applications are marked for preemption by ProportionalCapacityPreemptionPolicy
[ https://issues.apache.org/jira/browse/YARN-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-2022: -- Attachment: YARN-2022.10.patch Thank you [~mayank_bansal] , [~vinodkv] and [~leftnoteasy] for the comments. I have updated the patch. Please review. > Preempting an Application Master container can be kept as least priority when > multiple applications are marked for preemption by > ProportionalCapacityPreemptionPolicy > - > > Key: YARN-2022 > URL: https://issues.apache.org/jira/browse/YARN-2022 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-2022-DesignDraft.docx, YARN-2022.10.patch, > YARN-2022.2.patch, YARN-2022.3.patch, YARN-2022.4.patch, YARN-2022.5.patch, > YARN-2022.6.patch, YARN-2022.7.patch, YARN-2022.8.patch, YARN-2022.9.patch, > Yarn-2022.1.patch > > > Cluster Size = 16GB [2NM's] > Queue A Capacity = 50% > Queue B Capacity = 50% > Consider there are 3 applications running in Queue A which has taken the full > cluster capacity. > J1 = 2GB AM + 1GB * 4 Maps > J2 = 2GB AM + 1GB * 4 Maps > J3 = 2GB AM + 1GB * 2 Maps > Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ]. > Currently in this scenario, Jobs J3 will get killed including its AM. > It is better if AM can be given least priority among multiple applications. > In this same scenario, map tasks from J3 and J2 can be preempted. > Later when cluster is free, maps can be allocated to these Jobs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2022) Preempting an Application Master container can be kept as least priority when multiple applications are marked for preemption by ProportionalCapacityPreemptionPolicy
[ https://issues.apache.org/jira/browse/YARN-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-2022: -- Attachment: (was: YARN-2022.10.patch) > Preempting an Application Master container can be kept as least priority when > multiple applications are marked for preemption by > ProportionalCapacityPreemptionPolicy > - > > Key: YARN-2022 > URL: https://issues.apache.org/jira/browse/YARN-2022 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-2022-DesignDraft.docx, YARN-2022.10.patch, > YARN-2022.2.patch, YARN-2022.3.patch, YARN-2022.4.patch, YARN-2022.5.patch, > YARN-2022.6.patch, YARN-2022.7.patch, YARN-2022.8.patch, YARN-2022.9.patch, > Yarn-2022.1.patch > > > Cluster Size = 16GB [2NM's] > Queue A Capacity = 50% > Queue B Capacity = 50% > Consider there are 3 applications running in Queue A which has taken the full > cluster capacity. > J1 = 2GB AM + 1GB * 4 Maps > J2 = 2GB AM + 1GB * 4 Maps > J3 = 2GB AM + 1GB * 2 Maps > Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ]. > Currently in this scenario, Jobs J3 will get killed including its AM. > It is better if AM can be given least priority among multiple applications. > In this same scenario, map tasks from J3 and J2 can be preempted. > Later when cluster is free, maps can be allocated to these Jobs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2022) Preempting an Application Master container can be kept as least priority when multiple applications are marked for preemption by ProportionalCapacityPreemptionPolicy
[ https://issues.apache.org/jira/browse/YARN-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated YARN-2022: -- Attachment: YARN-2022.10.patch > Preempting an Application Master container can be kept as least priority when > multiple applications are marked for preemption by > ProportionalCapacityPreemptionPolicy > - > > Key: YARN-2022 > URL: https://issues.apache.org/jira/browse/YARN-2022 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Sunil G >Assignee: Sunil G > Attachments: YARN-2022-DesignDraft.docx, YARN-2022.10.patch, > YARN-2022.2.patch, YARN-2022.3.patch, YARN-2022.4.patch, YARN-2022.5.patch, > YARN-2022.6.patch, YARN-2022.7.patch, YARN-2022.8.patch, YARN-2022.9.patch, > Yarn-2022.1.patch > > > Cluster Size = 16GB [2NM's] > Queue A Capacity = 50% > Queue B Capacity = 50% > Consider there are 3 applications running in Queue A which has taken the full > cluster capacity. > J1 = 2GB AM + 1GB * 4 Maps > J2 = 2GB AM + 1GB * 4 Maps > J3 = 2GB AM + 1GB * 2 Maps > Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ]. > Currently in this scenario, Jobs J3 will get killed including its AM. > It is better if AM can be given least priority among multiple applications. > In this same scenario, map tasks from J3 and J2 can be preempted. > Later when cluster is free, maps can be allocated to these Jobs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (YARN-2022) Preempting an Application Master container can be kept as least priority when multiple applications are marked for preemption by ProportionalCapacityPreemptionPolicy
[ https://issues.apache.org/jira/browse/YARN-2022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-2022: -- Fix Version/s: 2.5.0 > Preempting an Application Master container can be kept as least priority when > multiple applications are marked for preemption by > ProportionalCapacityPreemptionPolicy > - > > Key: YARN-2022 > URL: https://issues.apache.org/jira/browse/YARN-2022 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Affects Versions: 2.4.0 >Reporter: Sunil G >Assignee: Sunil G > Fix For: 2.5.0 > > Attachments: YARN-2022-DesignDraft.docx, YARN-2022.10.patch, > YARN-2022.2.patch, YARN-2022.3.patch, YARN-2022.4.patch, YARN-2022.5.patch, > YARN-2022.6.patch, YARN-2022.7.patch, YARN-2022.8.patch, YARN-2022.9.patch, > Yarn-2022.1.patch > > > Cluster Size = 16GB [2NM's] > Queue A Capacity = 50% > Queue B Capacity = 50% > Consider there are 3 applications running in Queue A which has taken the full > cluster capacity. > J1 = 2GB AM + 1GB * 4 Maps > J2 = 2GB AM + 1GB * 4 Maps > J3 = 2GB AM + 1GB * 2 Maps > Another Job J4 is submitted in Queue B [J4 needs a 2GB AM + 1GB * 2 Maps ]. > Currently in this scenario, Jobs J3 will get killed including its AM. > It is better if AM can be given least priority among multiple applications. > In this same scenario, map tasks from J3 and J2 can be preempted. > Later when cluster is free, maps can be allocated to these Jobs. -- This message was sent by Atlassian JIRA (v6.2#6252)
