[jira] [Updated] (YARN-567) RM changes to support preemption for FairScheduler and CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-567: - Issue Type: Bug (was: Sub-task) Parent: (was: YARN-397) > RM changes to support preemption for FairScheduler and CapacityScheduler > > > Key: YARN-567 > URL: https://issues.apache.org/jira/browse/YARN-567 > Project: Hadoop YARN > Issue Type: Bug > Components: resourcemanager >Reporter: Carlo Curino >Assignee: Carlo Curino > Fix For: 2.1.0-beta > > Attachments: YARN-567.patch, YARN-567.patch > > > A common tradeoff in scheduling jobs is between keeping the cluster busy and > enforcing capacity/fairness properties. FairScheduler and CapacityScheduler > takes opposite stance on how to achieve this. > The FairScheduler, leverages task-killing to quickly reclaim resources from > currently running jobs and redistributing them among new jobs, thus keeping > the cluster busy but waste useful work. The CapacityScheduler is typically > tuned > to limit the portion of the cluster used by each queue so that the likelihood > of violating capacity is low, thus never wasting work, but risking to keep > the cluster underutilized or have jobs waiting to obtain their rightful > capacity. > By introducing the notion of a work-preserving preemption we can remove this > tradeoff. This requires a protocol for preemption (YARN-45), and > ApplicationMasters that can answer to preemption efficiently (e.g., by > saving their intermediate state, this will be posted for MapReduce in a > separate JIRA soon), together with a scheduler that can issues preemption > requests (discussed in separate JIRAs YARN-568 and YARN-569). > The changes we track with this JIRA are common to FairScheduler and > CapacityScheduler, and are mostly propagation of preemption decisions through > the ApplicationMastersService. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-567) RM changes to support preemption for FairScheduler and CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinod Kumar Vavilapalli updated YARN-567: - Issue Type: Sub-task (was: Bug) Parent: YARN-45 > RM changes to support preemption for FairScheduler and CapacityScheduler > > > Key: YARN-567 > URL: https://issues.apache.org/jira/browse/YARN-567 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Carlo Curino >Assignee: Carlo Curino > Fix For: 2.1.0-beta > > Attachments: YARN-567.patch, YARN-567.patch > > > A common tradeoff in scheduling jobs is between keeping the cluster busy and > enforcing capacity/fairness properties. FairScheduler and CapacityScheduler > takes opposite stance on how to achieve this. > The FairScheduler, leverages task-killing to quickly reclaim resources from > currently running jobs and redistributing them among new jobs, thus keeping > the cluster busy but waste useful work. The CapacityScheduler is typically > tuned > to limit the portion of the cluster used by each queue so that the likelihood > of violating capacity is low, thus never wasting work, but risking to keep > the cluster underutilized or have jobs waiting to obtain their rightful > capacity. > By introducing the notion of a work-preserving preemption we can remove this > tradeoff. This requires a protocol for preemption (YARN-45), and > ApplicationMasters that can answer to preemption efficiently (e.g., by > saving their intermediate state, this will be posted for MapReduce in a > separate JIRA soon), together with a scheduler that can issues preemption > requests (discussed in separate JIRAs YARN-568 and YARN-569). > The changes we track with this JIRA are common to FairScheduler and > CapacityScheduler, and are mostly propagation of preemption decisions through > the ApplicationMastersService. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (YARN-567) RM changes to support preemption for FairScheduler and CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-567: -- Description: A common tradeoff in scheduling jobs is between keeping the cluster busy and enforcing capacity/fairness properties. FairScheduler and CapacityScheduler takes opposite stance on how to achieve this. The FairScheduler, leverages task-killing to quickly reclaim resources from currently running jobs and redistributing them among new jobs, thus keeping the cluster busy but waste useful work. The CapacityScheduler is typically tuned to limit the portion of the cluster used by each queue so that the likelihood of violating capacity is low, thus never wasting work, but risking to keep the cluster underutilized or have jobs waiting to obtain their rightful capacity. By introducing the notion of a work-preserving preemption we can remove this tradeoff. This requires a protocol for preemption (YARN-45), and ApplicationMasters that can answer to preemption efficiently (e.g., by saving their intermediate state, this will be posted for MapReduce in a separate JIRA soon), together with a scheduler that can issues preemption requests (discussed in separate JIRAs YARN-568 and YARN-569). The changes we track with this JIRA are common to FairScheduler and CapacityScheduler, and are mostly propagation of preemption decisions through the ApplicationMastersService. was: A common tradeoff in scheduling jobs is between keeping the cluster busy and enforcing capacity/fairness properties. FairScheduler and CapacityScheduler takes opposite stance on how to achieve this. The FairScheduler, leverages task-killing to quickly reclaim resources from currently running jobs and redistributing them among new jobs, thus keeping the cluster busy but waste useful work. The CapacityScheduler is typically tuned to limit the portion of the cluster used by each queue so that the likelihood of violating capacity is low, thus never wasting work, but risking to keep the cluster underutilized or have jobs waiting to obtain their rightful capacity. By introducing the notion of a work-preserving preemption we can remove this tradeoff. This requires a protocol for preemption (YARN-45), and ApplicationMasters that can answer to preemption efficiently (e.g., by saving their intermediate state, this will be posted for MapReduce in a separate JIRA soon), together with a scheduler that can issues preemption requests (discussed in separate JIRAs). The changes we track with this JIRA are common to FairScheduler and CapacityScheduler, and are mostly propagation of preemption decisions through the ApplicationMastersService. > RM changes to support preemption for FairScheduler and CapacityScheduler > > > Key: YARN-567 > URL: https://issues.apache.org/jira/browse/YARN-567 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Carlo Curino >Assignee: Carlo Curino > > A common tradeoff in scheduling jobs is between keeping the cluster busy and > enforcing capacity/fairness properties. FairScheduler and CapacityScheduler > takes opposite stance on how to achieve this. > The FairScheduler, leverages task-killing to quickly reclaim resources from > currently running jobs and redistributing them among new jobs, thus keeping > the cluster busy but waste useful work. The CapacityScheduler is typically > tuned > to limit the portion of the cluster used by each queue so that the likelihood > of violating capacity is low, thus never wasting work, but risking to keep > the cluster underutilized or have jobs waiting to obtain their rightful > capacity. > By introducing the notion of a work-preserving preemption we can remove this > tradeoff. This requires a protocol for preemption (YARN-45), and > ApplicationMasters that can answer to preemption efficiently (e.g., by > saving their intermediate state, this will be posted for MapReduce in a > separate JIRA soon), together with a scheduler that can issues preemption > requests (discussed in separate JIRAs YARN-568 and YARN-569). > The changes we track with this JIRA are common to FairScheduler and > CapacityScheduler, and are mostly propagation of preemption decisions through > the ApplicationMastersService. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-567) RM changes to support preemption for FairScheduler and CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-567: -- Attachment: common.patch > RM changes to support preemption for FairScheduler and CapacityScheduler > > > Key: YARN-567 > URL: https://issues.apache.org/jira/browse/YARN-567 > Project: Hadoop YARN > Issue Type: Improvement > Components: resourcemanager >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: common.patch > > > A common tradeoff in scheduling jobs is between keeping the cluster busy and > enforcing capacity/fairness properties. FairScheduler and CapacityScheduler > takes opposite stance on how to achieve this. > The FairScheduler, leverages task-killing to quickly reclaim resources from > currently running jobs and redistributing them among new jobs, thus keeping > the cluster busy but waste useful work. The CapacityScheduler is typically > tuned > to limit the portion of the cluster used by each queue so that the likelihood > of violating capacity is low, thus never wasting work, but risking to keep > the cluster underutilized or have jobs waiting to obtain their rightful > capacity. > By introducing the notion of a work-preserving preemption we can remove this > tradeoff. This requires a protocol for preemption (YARN-45), and > ApplicationMasters that can answer to preemption efficiently (e.g., by > saving their intermediate state, this will be posted for MapReduce in a > separate JIRA soon), together with a scheduler that can issues preemption > requests (discussed in separate JIRAs YARN-568 and YARN-569). > The changes we track with this JIRA are common to FairScheduler and > CapacityScheduler, and are mostly propagation of preemption decisions through > the ApplicationMastersService. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-567) RM changes to support preemption for FairScheduler and CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-567: -- Issue Type: Sub-task (was: Improvement) Parent: YARN-397 > RM changes to support preemption for FairScheduler and CapacityScheduler > > > Key: YARN-567 > URL: https://issues.apache.org/jira/browse/YARN-567 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: common.patch > > > A common tradeoff in scheduling jobs is between keeping the cluster busy and > enforcing capacity/fairness properties. FairScheduler and CapacityScheduler > takes opposite stance on how to achieve this. > The FairScheduler, leverages task-killing to quickly reclaim resources from > currently running jobs and redistributing them among new jobs, thus keeping > the cluster busy but waste useful work. The CapacityScheduler is typically > tuned > to limit the portion of the cluster used by each queue so that the likelihood > of violating capacity is low, thus never wasting work, but risking to keep > the cluster underutilized or have jobs waiting to obtain their rightful > capacity. > By introducing the notion of a work-preserving preemption we can remove this > tradeoff. This requires a protocol for preemption (YARN-45), and > ApplicationMasters that can answer to preemption efficiently (e.g., by > saving their intermediate state, this will be posted for MapReduce in a > separate JIRA soon), together with a scheduler that can issues preemption > requests (discussed in separate JIRAs YARN-568 and YARN-569). > The changes we track with this JIRA are common to FairScheduler and > CapacityScheduler, and are mostly propagation of preemption decisions through > the ApplicationMastersService. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-567) RM changes to support preemption for FairScheduler and CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-567: -- Attachment: YARN-567.patch > RM changes to support preemption for FairScheduler and CapacityScheduler > > > Key: YARN-567 > URL: https://issues.apache.org/jira/browse/YARN-567 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-567.patch > > > A common tradeoff in scheduling jobs is between keeping the cluster busy and > enforcing capacity/fairness properties. FairScheduler and CapacityScheduler > takes opposite stance on how to achieve this. > The FairScheduler, leverages task-killing to quickly reclaim resources from > currently running jobs and redistributing them among new jobs, thus keeping > the cluster busy but waste useful work. The CapacityScheduler is typically > tuned > to limit the portion of the cluster used by each queue so that the likelihood > of violating capacity is low, thus never wasting work, but risking to keep > the cluster underutilized or have jobs waiting to obtain their rightful > capacity. > By introducing the notion of a work-preserving preemption we can remove this > tradeoff. This requires a protocol for preemption (YARN-45), and > ApplicationMasters that can answer to preemption efficiently (e.g., by > saving their intermediate state, this will be posted for MapReduce in a > separate JIRA soon), together with a scheduler that can issues preemption > requests (discussed in separate JIRAs YARN-568 and YARN-569). > The changes we track with this JIRA are common to FairScheduler and > CapacityScheduler, and are mostly propagation of preemption decisions through > the ApplicationMastersService. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-567) RM changes to support preemption for FairScheduler and CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-567: -- Attachment: (was: common.patch) > RM changes to support preemption for FairScheduler and CapacityScheduler > > > Key: YARN-567 > URL: https://issues.apache.org/jira/browse/YARN-567 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-567.patch > > > A common tradeoff in scheduling jobs is between keeping the cluster busy and > enforcing capacity/fairness properties. FairScheduler and CapacityScheduler > takes opposite stance on how to achieve this. > The FairScheduler, leverages task-killing to quickly reclaim resources from > currently running jobs and redistributing them among new jobs, thus keeping > the cluster busy but waste useful work. The CapacityScheduler is typically > tuned > to limit the portion of the cluster used by each queue so that the likelihood > of violating capacity is low, thus never wasting work, but risking to keep > the cluster underutilized or have jobs waiting to obtain their rightful > capacity. > By introducing the notion of a work-preserving preemption we can remove this > tradeoff. This requires a protocol for preemption (YARN-45), and > ApplicationMasters that can answer to preemption efficiently (e.g., by > saving their intermediate state, this will be posted for MapReduce in a > separate JIRA soon), together with a scheduler that can issues preemption > requests (discussed in separate JIRAs YARN-568 and YARN-569). > The changes we track with this JIRA are common to FairScheduler and > CapacityScheduler, and are mostly propagation of preemption decisions through > the ApplicationMastersService. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-567) RM changes to support preemption for FairScheduler and CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Carlo Curino updated YARN-567: -- Attachment: YARN-567.patch updating patch, accordingly to evolution in YARN-45 > RM changes to support preemption for FairScheduler and CapacityScheduler > > > Key: YARN-567 > URL: https://issues.apache.org/jira/browse/YARN-567 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-567.patch, YARN-567.patch > > > A common tradeoff in scheduling jobs is between keeping the cluster busy and > enforcing capacity/fairness properties. FairScheduler and CapacityScheduler > takes opposite stance on how to achieve this. > The FairScheduler, leverages task-killing to quickly reclaim resources from > currently running jobs and redistributing them among new jobs, thus keeping > the cluster busy but waste useful work. The CapacityScheduler is typically > tuned > to limit the portion of the cluster used by each queue so that the likelihood > of violating capacity is low, thus never wasting work, but risking to keep > the cluster underutilized or have jobs waiting to obtain their rightful > capacity. > By introducing the notion of a work-preserving preemption we can remove this > tradeoff. This requires a protocol for preemption (YARN-45), and > ApplicationMasters that can answer to preemption efficiently (e.g., by > saving their intermediate state, this will be posted for MapReduce in a > separate JIRA soon), together with a scheduler that can issues preemption > requests (discussed in separate JIRAs YARN-568 and YARN-569). > The changes we track with this JIRA are common to FairScheduler and > CapacityScheduler, and are mostly propagation of preemption decisions through > the ApplicationMastersService. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-567) RM changes to support preemption for FairScheduler and CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated YARN-567: --- Attachment: (was: YARN-567-1.patch) > RM changes to support preemption for FairScheduler and CapacityScheduler > > > Key: YARN-567 > URL: https://issues.apache.org/jira/browse/YARN-567 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-567.patch, YARN-567.patch > > > A common tradeoff in scheduling jobs is between keeping the cluster busy and > enforcing capacity/fairness properties. FairScheduler and CapacityScheduler > takes opposite stance on how to achieve this. > The FairScheduler, leverages task-killing to quickly reclaim resources from > currently running jobs and redistributing them among new jobs, thus keeping > the cluster busy but waste useful work. The CapacityScheduler is typically > tuned > to limit the portion of the cluster used by each queue so that the likelihood > of violating capacity is low, thus never wasting work, but risking to keep > the cluster underutilized or have jobs waiting to obtain their rightful > capacity. > By introducing the notion of a work-preserving preemption we can remove this > tradeoff. This requires a protocol for preemption (YARN-45), and > ApplicationMasters that can answer to preemption efficiently (e.g., by > saving their intermediate state, this will be posted for MapReduce in a > separate JIRA soon), together with a scheduler that can issues preemption > requests (discussed in separate JIRAs YARN-568 and YARN-569). > The changes we track with this JIRA are common to FairScheduler and > CapacityScheduler, and are mostly propagation of preemption decisions through > the ApplicationMastersService. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-567) RM changes to support preemption for FairScheduler and CapacityScheduler
[ https://issues.apache.org/jira/browse/YARN-567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Douglas updated YARN-567: --- Attachment: YARN-567-1.patch > RM changes to support preemption for FairScheduler and CapacityScheduler > > > Key: YARN-567 > URL: https://issues.apache.org/jira/browse/YARN-567 > Project: Hadoop YARN > Issue Type: Sub-task > Components: resourcemanager >Reporter: Carlo Curino >Assignee: Carlo Curino > Attachments: YARN-567.patch, YARN-567.patch > > > A common tradeoff in scheduling jobs is between keeping the cluster busy and > enforcing capacity/fairness properties. FairScheduler and CapacityScheduler > takes opposite stance on how to achieve this. > The FairScheduler, leverages task-killing to quickly reclaim resources from > currently running jobs and redistributing them among new jobs, thus keeping > the cluster busy but waste useful work. The CapacityScheduler is typically > tuned > to limit the portion of the cluster used by each queue so that the likelihood > of violating capacity is low, thus never wasting work, but risking to keep > the cluster underutilized or have jobs waiting to obtain their rightful > capacity. > By introducing the notion of a work-preserving preemption we can remove this > tradeoff. This requires a protocol for preemption (YARN-45), and > ApplicationMasters that can answer to preemption efficiently (e.g., by > saving their intermediate state, this will be posted for MapReduce in a > separate JIRA soon), together with a scheduler that can issues preemption > requests (discussed in separate JIRAs YARN-568 and YARN-569). > The changes we track with this JIRA are common to FairScheduler and > CapacityScheduler, and are mostly propagation of preemption decisions through > the ApplicationMastersService. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
