[jira] Commented: (MAPREDUCE-698) Per-pool task limits for the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12792018#action_12792018 ] Kevin Peterson commented on MAPREDUCE-698: -- That was for checking if the allocations were consistent (min < max), I moved this into the loop where they read but missed this bit it looks like. > Per-pool task limits for the fair scheduler > --- > > Key: MAPREDUCE-698 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-698 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Reporter: Matei Zaharia >Assignee: Kevin Peterson > Fix For: 0.21.0 > > Attachments: MAPREDUCE-698-prelim.patch, mapreduce-698-trunk-3.patch, > mapreduce-698-trunk-4.patch, mapreduce-698-trunk.patch, > mapreduce-698-trunk.patch > > > The fair scheduler could use a way to cap the share of a given pool similar > to MAPREDUCE-532. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-698) Per-pool task limits for the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Peterson updated MAPREDUCE-698: - Status: Open (was: Patch Available) cancel and resubmit to pick up newer patch. > Per-pool task limits for the fair scheduler > --- > > Key: MAPREDUCE-698 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-698 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Reporter: Matei Zaharia >Assignee: Kevin Peterson > Fix For: 0.21.0 > > Attachments: MAPREDUCE-698-prelim.patch, mapreduce-698-trunk-3.patch, > mapreduce-698-trunk-4.patch, mapreduce-698-trunk.patch, > mapreduce-698-trunk.patch > > > The fair scheduler could use a way to cap the share of a given pool similar > to MAPREDUCE-532. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-698) Per-pool task limits for the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Peterson updated MAPREDUCE-698: - Status: Patch Available (was: Open) Should be ready to go. > Per-pool task limits for the fair scheduler > --- > > Key: MAPREDUCE-698 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-698 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Reporter: Matei Zaharia >Assignee: Kevin Peterson > Fix For: 0.21.0 > > Attachments: MAPREDUCE-698-prelim.patch, mapreduce-698-trunk-3.patch, > mapreduce-698-trunk-4.patch, mapreduce-698-trunk.patch, > mapreduce-698-trunk.patch > > > The fair scheduler could use a way to cap the share of a given pool similar > to MAPREDUCE-532. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-698) Per-pool task limits for the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Peterson updated MAPREDUCE-698: - Attachment: mapreduce-698-trunk-4.patch Added some warnings if it's configured with max < min, fixed layout in servlet to match current version. Also, I believe the tests have been fixed in trunk, so this should pass release audit now. > Per-pool task limits for the fair scheduler > --- > > Key: MAPREDUCE-698 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-698 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Reporter: Matei Zaharia >Assignee: Kevin Peterson > Fix For: 0.21.0 > > Attachments: MAPREDUCE-698-prelim.patch, mapreduce-698-trunk-3.patch, > mapreduce-698-trunk-4.patch, mapreduce-698-trunk.patch, > mapreduce-698-trunk.patch > > > The fair scheduler could use a way to cap the share of a given pool similar > to MAPREDUCE-532. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-698) Per-pool task limits for the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Peterson updated MAPREDUCE-698: - Assignee: Kevin Peterson Release Note: Per-pool map and reduce caps for Fair Scheduler. Status: Patch Available (was: Open) > Per-pool task limits for the fair scheduler > --- > > Key: MAPREDUCE-698 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-698 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Reporter: Matei Zaharia >Assignee: Kevin Peterson > Fix For: 0.21.0 > > Attachments: MAPREDUCE-698-prelim.patch, mapreduce-698-trunk-3.patch, > mapreduce-698-trunk.patch, mapreduce-698-trunk.patch > > > The fair scheduler could use a way to cap the share of a given pool similar > to MAPREDUCE-532. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-698) Per-pool task limits for the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Peterson updated MAPREDUCE-698: - Attachment: mapreduce-698-trunk-3.patch changes from previous patch: - extra newline in fairscheduler.java removed - removed "single test" changes from build-contrib.xml (they didn't accomplish what I wanted -- to run just a single test method) - Regarding checkAssignment, I made the change you suggested, but I'm not sure I'm testing things in the best way. The only thing I'm concerned with is that it ends up scheduling the right number from each pool, the only way I was able to get it to actually assign the jobs was to use checkAssignment. - in the UI, labels are "Max Share" - Removed Pool.numRunningTasks since it was only used from within PoolSchedulable, where this data is already available. - Moved cap from getDemand() to updateDemand(). - Documentation updated - Removed tabs. > Per-pool task limits for the fair scheduler > --- > > Key: MAPREDUCE-698 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-698 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Reporter: Matei Zaharia > Fix For: 0.21.0 > > Attachments: MAPREDUCE-698-prelim.patch, mapreduce-698-trunk-3.patch, > mapreduce-698-trunk.patch, mapreduce-698-trunk.patch > > > The fair scheduler could use a way to cap the share of a given pool similar > to MAPREDUCE-532. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-698) Per-pool task limits for the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Peterson updated MAPREDUCE-698: - Attachment: mapreduce-698-trunk.patch Update patch to display caps in the scheduler UI. > Per-pool task limits for the fair scheduler > --- > > Key: MAPREDUCE-698 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-698 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Reporter: Matei Zaharia > Fix For: 0.21.0 > > Attachments: MAPREDUCE-698-prelim.patch, mapreduce-698-trunk.patch, > mapreduce-698-trunk.patch > > > The fair scheduler could use a way to cap the share of a given pool similar > to MAPREDUCE-532. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-698) Per-pool task limits for the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Peterson updated MAPREDUCE-698: - Attachment: mapreduce-698-trunk.patch Patch relative to head. Overrides demand() and assign(), unit test included. I'm able to run the test I added locally, but I'm having trouble with some other test when I run the suite. I haven't tested this on a real cluster, just in unit tests. Still need to add the display in scheduler servlet. It's sitting on my machine at work. > Per-pool task limits for the fair scheduler > --- > > Key: MAPREDUCE-698 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-698 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Reporter: Matei Zaharia > Fix For: 0.21.0 > > Attachments: MAPREDUCE-698-prelim.patch, mapreduce-698-trunk.patch > > > The fair scheduler could use a way to cap the share of a given pool similar > to MAPREDUCE-532. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Updated: (MAPREDUCE-698) Per-pool task limits for the fair scheduler
[ https://issues.apache.org/jira/browse/MAPREDUCE-698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Peterson updated MAPREDUCE-698: - Attachment: MAPREDUCE-698-prelim.patch Attached patch works for me, but I think I need to make some changes for interaction with the multiple-assign before it's ready to commit. Patch is relative to 0.20 branch. It allows two new fields in the pools definition, "maxMaps" and "maxReduces". My need for this is interacting with outside services (in this case, an HBase cluster that's significantly smaller than my Hadoop cluster). > Per-pool task limits for the fair scheduler > --- > > Key: MAPREDUCE-698 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-698 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Reporter: Matei Zaharia > Fix For: 0.21.0 > > Attachments: MAPREDUCE-698-prelim.patch > > > The fair scheduler could use a way to cap the share of a given pool similar > to MAPREDUCE-532. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
