[jira] [Commented] (MAPREDUCE-2198) Allow FairScheduler to control the number of slots on each TaskTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105119#comment-13105119 ] Scott Chen commented on MAPREDUCE-2198: --- Arun: Thanks for the comments. You are right. I guess this is not an issue since we have MRv2. Closing this now. > Allow FairScheduler to control the number of slots on each TaskTracker > -- > > Key: MAPREDUCE-2198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2198 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Affects Versions: 0.23.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2198-v2.txt, MAPREDUCE-2198.txt > > > We can set the number of slots on the TaskTracker to be high and let > FairScheduler handles the slots. > This approach allows us to change the number of slots on each node > dynamically. > The administrator can change the number of slots with a CLI tool. > One use case of this is for upgrading the MapReduce. > Instead of restarting the cluster, we can run the new MapReduce on the same > cluster. > And use the CLI tool to gradually migrate the slots. > This way we don't lost the progress fo the jobs that's already executed. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2198) Allow FairScheduler to control the number of slots on each TaskTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999876#comment-12999876 ] Hadoop QA commented on MAPREDUCE-2198: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12464776/MAPREDUCE-2198-v2.txt against trunk revision 1074251. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed core unit tests. -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/51//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/51//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-MAPREDUCE-Build/51//console This message is automatically generated. > Allow FairScheduler to control the number of slots on each TaskTracker > -- > > Key: MAPREDUCE-2198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2198 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Affects Versions: 0.23.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2198-v2.txt, MAPREDUCE-2198.txt > > > We can set the number of slots on the TaskTracker to be high and let > FairScheduler handles the slots. > This approach allows us to change the number of slots on each node > dynamically. > The administrator can change the number of slots with a CLI tool. > One use case of this is for upgrading the MapReduce. > Instead of restarting the cluster, we can run the new MapReduce on the same > cluster. > And use the CLI tool to gradually migrate the slots. > This way we don't lost the progress fo the jobs that's already executed. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (MAPREDUCE-2198) Allow FairScheduler to control the number of slots on each TaskTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12968862#action_12968862 ] Joydeep Sen Sarma commented on MAPREDUCE-2198: -- +1 on Matei's previous comments about needing to wait until slots are actually released. We should have another api to request the actual number of [map/reduce] slots in use on any given tracker and only claim slots when they are actually confirmed to be released. > Allow FairScheduler to control the number of slots on each TaskTracker > -- > > Key: MAPREDUCE-2198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2198 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Affects Versions: 0.23.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2198-v2.txt, MAPREDUCE-2198.txt > > > We can set the number of slots on the TaskTracker to be high and let > FairScheduler handles the slots. > This approach allows us to change the number of slots on each node > dynamically. > The administrator can change the number of slots with a CLI tool. > One use case of this is for upgrading the MapReduce. > Instead of restarting the cluster, we can run the new MapReduce on the same > cluster. > And use the CLI tool to gradually migrate the slots. > This way we don't lost the progress fo the jobs that's already executed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2198) Allow FairScheduler to control the number of slots on each TaskTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12965478#action_12965478 ] Scott Chen commented on MAPREDUCE-2198: --- bq. However, I think this can already be done through the MRAdmin shell command / API. I also prefer using the existing interface and keep this one simpler. > Allow FairScheduler to control the number of slots on each TaskTracker > -- > > Key: MAPREDUCE-2198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2198 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Affects Versions: 0.23.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2198-v2.txt, MAPREDUCE-2198.txt > > > We can set the number of slots on the TaskTracker to be high and let > FairScheduler handles the slots. > This approach allows us to change the number of slots on each node > dynamically. > The administrator can change the number of slots with a CLI tool. > One use case of this is for upgrading the MapReduce. > Instead of restarting the cluster, we can run the new MapReduce on the same > cluster. > And use the CLI tool to gradually migrate the slots. > This way we don't lost the progress fo the jobs that's already executed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2198) Allow FairScheduler to control the number of slots on each TaskTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12964503#action_12964503 ] dhruba borthakur commented on MAPREDUCE-2198: - > Either have the process that's scaling down the cluster watch the running > tasks before giving the slots to someone else I like this idea (instead of having callbacks, keeps the design simple) > Allow FairScheduler to control the number of slots on each TaskTracker > -- > > Key: MAPREDUCE-2198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2198 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Affects Versions: 0.23.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2198-v2.txt, MAPREDUCE-2198.txt > > > We can set the number of slots on the TaskTracker to be high and let > FairScheduler handles the slots. > This approach allows us to change the number of slots on each node > dynamically. > The administrator can change the number of slots with a CLI tool. > One use case of this is for upgrading the MapReduce. > Instead of restarting the cluster, we can run the new MapReduce on the same > cluster. > And use the CLI tool to gradually migrate the slots. > This way we don't lost the progress fo the jobs that's already executed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2198) Allow FairScheduler to control the number of slots on each TaskTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12964477#action_12964477 ] Matei Zaharia commented on MAPREDUCE-2198: -- The changes look good, but I thought about one other issue: What should we do when we are asked to lower the slots on a node to below the number of running tasks on it? In the current version, the scheduler won't launch tasks on that node until its running task count falls below its slot count. However, if we wanted to use this for rollover, we'd probably want to wait until enough of those tasks are done before giving a slot to the new JobTracker. There are two ways we can do this: Either have the process that's scaling down the cluster watch the running tasks before giving the slots to someone else, or include an API that somehow makes a callback when the number of running tasks has decreased below the target slot count. What are your thoughts on this? One other thing we may want to support is killing tasks after a timeout if the cluster hasn't scaled down. However, I think this can already be done through the MRAdmin shell command / API. In either case, we probably need some API to see what's running on the cluster. Some of the commands in MRAdmin might be enough, but we may want to add something there. However, this can be a different JIRA. > Allow FairScheduler to control the number of slots on each TaskTracker > -- > > Key: MAPREDUCE-2198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2198 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Affects Versions: 0.23.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2198-v2.txt, MAPREDUCE-2198.txt > > > We can set the number of slots on the TaskTracker to be high and let > FairScheduler handles the slots. > This approach allows us to change the number of slots on each node > dynamically. > The administrator can change the number of slots with a CLI tool. > One use case of this is for upgrading the MapReduce. > Instead of restarting the cluster, we can run the new MapReduce on the same > cluster. > And use the CLI tool to gradually migrate the slots. > This way we don't lost the progress fo the jobs that's already executed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2198) Allow FairScheduler to control the number of slots on each TaskTracker
[
https://issues.apache.org/jira/browse/MAPREDUCE-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12936147#action_12936147
]
Scott Chen commented on MAPREDUCE-2198:
---
Hey Matei, Thanks for the review.
{quote}
How can you run the FairSchedulerShell from the command line? It doesn't seem
to have a main method (so just using bin/hadoop
org.apache.hadoop.mapred.FairSchedulerShell doesn't work), and I don't see it
registered as a tool anywhere.
{quote}
It's my bad. I forgot to put a main method in FairSchedulerShell. I will update
it.
{quote}
Will the slot counts set by the scheduler be visible in the JobTracker web UI?
It looks like jobtracker.jsp looks at ClusterMetrics and machines.jsp looks at
TaskTrackerStatus objects.
{quote}
For the ClusterMetircs, it's OK because we use TaskScheduler.getMaxSlots() to
calculate the total slots in JobTracker. But you are right about machines.jsp.
I should change it so that it also pulls the information from scheduler.
> Allow FairScheduler to control the number of slots on each TaskTracker
> --
>
> Key: MAPREDUCE-2198
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2198
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: contrib/fair-share
>Affects Versions: 0.23.0
>Reporter: Scott Chen
>Assignee: Scott Chen
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2198.txt
>
>
> We can set the number of slots on the TaskTracker to be high and let
> FairScheduler handles the slots.
> This approach allows us to change the number of slots on each node
> dynamically.
> The administrator can change the number of slots with a CLI tool.
> One use case of this is for upgrading the MapReduce.
> Instead of restarting the cluster, we can run the new MapReduce on the same
> cluster.
> And use the CLI tool to gradually migrate the slots.
> This way we don't lost the progress fo the jobs that's already executed.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2198) Allow FairScheduler to control the number of slots on each TaskTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12936124#action_12936124 ] Matei Zaharia commented on MAPREDUCE-2198: -- The approach in the patch looks good, but I have two questions: - How can you run the FairSchedulerShell from the command line? It doesn't seem to have a main method (so just using bin/hadoop org.apache.hadoop.mapred.FairSchedulerShell doesn't work), and I don't see it registered as a tool anywhere. - Will the slot counts set by the scheduler be visible in the JobTracker web UI? It looks like jobtracker.jsp looks at ClusterMetrics and machines.jsp looks at TaskTrackerStatus objects. > Allow FairScheduler to control the number of slots on each TaskTracker > -- > > Key: MAPREDUCE-2198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2198 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Affects Versions: 0.23.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2198.txt > > > We can set the number of slots on the TaskTracker to be high and let > FairScheduler handles the slots. > This approach allows us to change the number of slots on each node > dynamically. > The administrator can change the number of slots with a CLI tool. > One use case of this is for upgrading the MapReduce. > Instead of restarting the cluster, we can run the new MapReduce on the same > cluster. > And use the CLI tool to gradually migrate the slots. > This way we don't lost the progress fo the jobs that's already executed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2198) Allow FairScheduler to control the number of slots on each TaskTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935935#action_12935935 ] Scott Chen commented on MAPREDUCE-2198: --- I tried using review board. But I always get 500 error. > Allow FairScheduler to control the number of slots on each TaskTracker > -- > > Key: MAPREDUCE-2198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2198 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Affects Versions: 0.23.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.23.0 > > Attachments: MAPREDUCE-2198.txt > > > We can set the number of slots on the TaskTracker to be high and let > FairScheduler handles the slots. > This approach allows us to change the number of slots on each node > dynamically. > The administrator can change the number of slots with a CLI tool. > One use case of this is for upgrading the MapReduce. > Instead of restarting the cluster, we can run the new MapReduce on the same > cluster. > And use the CLI tool to gradually migrate the slots. > This way we don't lost the progress fo the jobs that's already executed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2198) Allow FairScheduler to control the number of slots on each TaskTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935577#action_12935577 ] Matei Zaharia commented on MAPREDUCE-2198: -- One other minor comment: if the fair scheduler is over-scheduling task trackers, maybe we should consider that a bug. I don't think it was intended to do that, although in trunk at least, it looks like it may do it if mapAssignCap and reduceAssignCap are set to something less than infinity. (Otherwise, it looks at the number of slots free on the TT and does not assign more than that.) To deal with any sort of race condition that occurs if you lower a slot count while a heartbeat is in progress, I'd suggest making the TT report over-scheduled tasks as killed and drop them. > Allow FairScheduler to control the number of slots on each TaskTracker > -- > > Key: MAPREDUCE-2198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2198 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > > We can set the number of slots on the TaskTracker to be high and let > FairScheduler handles the slots. > This approach allows us to change the number of slots on each node > dynamically. > The administrator can change the number of slots with a CLI tool. > One use case of this is for upgrading the MapReduce. > Instead of restarting the cluster, we can run the new MapReduce on the same > cluster. > And use the CLI tool to gradually migrate the slots. > This way we don't lost the progress fo the jobs that's already executed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2198) Allow FairScheduler to control the number of slots on each TaskTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935568#action_12935568 ] Matei Zaharia commented on MAPREDUCE-2198: -- Do setup and cleanup tasks do a significant amount of work? I can see two simple ways of dealing with them other than moving control to the scheduler: Either always allow them to run (even if the TT is full on map and reduce slots), or allow them to run but limit the number of such tasks per node to 1 (i.e. have a "setup slot"). If they do need to do CPU-intensive work, then it makes sense to give control of them to the scheduler. > Allow FairScheduler to control the number of slots on each TaskTracker > -- > > Key: MAPREDUCE-2198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2198 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > > We can set the number of slots on the TaskTracker to be high and let > FairScheduler handles the slots. > This approach allows us to change the number of slots on each node > dynamically. > The administrator can change the number of slots with a CLI tool. > One use case of this is for upgrading the MapReduce. > Instead of restarting the cluster, we can run the new MapReduce on the same > cluster. > And use the CLI tool to gradually migrate the slots. > This way we don't lost the progress fo the jobs that's already executed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2198) Allow FairScheduler to control the number of slots on each TaskTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935119#action_12935119 ] Arun C Murthy commented on MAPREDUCE-2198: -- Right. Also, you need to worry about task-cleanup-tasks i.e. responses to move tasks from COMMIT_PENDING to SUCCESS/KILLED. > Allow FairScheduler to control the number of slots on each TaskTracker > -- > > Key: MAPREDUCE-2198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2198 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > > We can set the number of slots on the TaskTracker to be high and let > FairScheduler handles the slots. > This approach allows us to change the number of slots on each node > dynamically. > The administrator can change the number of slots with a CLI tool. > One use case of this is for upgrading the MapReduce. > Instead of restarting the cluster, we can run the new MapReduce on the same > cluster. > And use the CLI tool to gradually migrate the slots. > This way we don't lost the progress fo the jobs that's already executed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2198) Allow FairScheduler to control the number of slots on each TaskTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935052#action_12935052 ] Scott Chen commented on MAPREDUCE-2198: --- Hey Arun, bq. You also have to worry about piggy-backing of task-cleanup tasks done by the JT... I read some codes. Now I see your point. JobTracker.getSetupAndCleanupTasks() is not controlled by the Scheduler. So we need to put some logic there to make it aware of this task limit. Or we might be over-scheduled. > Allow FairScheduler to control the number of slots on each TaskTracker > -- > > Key: MAPREDUCE-2198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2198 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > > We can set the number of slots on the TaskTracker to be high and let > FairScheduler handles the slots. > This approach allows us to change the number of slots on each node > dynamically. > The administrator can change the number of slots with a CLI tool. > One use case of this is for upgrading the MapReduce. > Instead of restarting the cluster, we can run the new MapReduce on the same > cluster. > And use the CLI tool to gradually migrate the slots. > This way we don't lost the progress fo the jobs that's already executed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2198) Allow FairScheduler to control the number of slots on each TaskTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12935001#action_12935001 ] Scott Chen commented on MAPREDUCE-2198: --- Hey Arun and Joydeep, bq. How do you handle heterogeneous clusters? Or will your CLI command be per TT? Yes, the CLI command will be per TT. bq. in addition - one bug/feature that we need to fix as part of this where the JT overschedules TTs (when configured to schedule multiple tasks per hbt). this is benign today (TT puts those tasks in unassigned state) - but in this world will not be so benign. Yes, we should never let JT submit more tasks than TT's limit. Scheduler should incorporate this logic. bq. You also have to worry about piggy-backing of task-cleanup tasks done by the JT... I am not very clear about this part. Will read more codes. > Allow FairScheduler to control the number of slots on each TaskTracker > -- > > Key: MAPREDUCE-2198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2198 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > > We can set the number of slots on the TaskTracker to be high and let > FairScheduler handles the slots. > This approach allows us to change the number of slots on each node > dynamically. > The administrator can change the number of slots with a CLI tool. > One use case of this is for upgrading the MapReduce. > Instead of restarting the cluster, we can run the new MapReduce on the same > cluster. > And use the CLI tool to gradually migrate the slots. > This way we don't lost the progress fo the jobs that's already executed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2198) Allow FairScheduler to control the number of slots on each TaskTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934939#action_12934939 ] Arun C Murthy commented on MAPREDUCE-2198: -- bq. Also, as Joydeep pointed out there are several issues with unassigned slots in TTs... You also have to worry about piggy-backing of task-cleanup tasks done by the JT... > Allow FairScheduler to control the number of slots on each TaskTracker > -- > > Key: MAPREDUCE-2198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2198 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > > We can set the number of slots on the TaskTracker to be high and let > FairScheduler handles the slots. > This approach allows us to change the number of slots on each node > dynamically. > The administrator can change the number of slots with a CLI tool. > One use case of this is for upgrading the MapReduce. > Instead of restarting the cluster, we can run the new MapReduce on the same > cluster. > And use the CLI tool to gradually migrate the slots. > This way we don't lost the progress fo the jobs that's already executed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2198) Allow FairScheduler to control the number of slots on each TaskTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934931#action_12934931 ] Arun C Murthy commented on MAPREDUCE-2198: -- bq. Right now if you want to change the number of slots, you have to change the conf on every TT and restart. How do you handle heterogeneous clusters? Or will your CLI command be per TT? Also, as Joydeep pointed out there are several issues with unassigned slots in TTs... > Allow FairScheduler to control the number of slots on each TaskTracker > -- > > Key: MAPREDUCE-2198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2198 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > > We can set the number of slots on the TaskTracker to be high and let > FairScheduler handles the slots. > This approach allows us to change the number of slots on each node > dynamically. > The administrator can change the number of slots with a CLI tool. > One use case of this is for upgrading the MapReduce. > Instead of restarting the cluster, we can run the new MapReduce on the same > cluster. > And use the CLI tool to gradually migrate the slots. > This way we don't lost the progress fo the jobs that's already executed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2198) Allow FairScheduler to control the number of slots on each TaskTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934753#action_12934753 ] Joydeep Sen Sarma commented on MAPREDUCE-2198: -- what is fairscheduler specific about this? could we not make this a change in the JT directly to scale the slots advertised by TT? in addition - one bug/feature that we need to fix as part of this where the JT overschedules TTs (when configured to schedule multiple tasks per hbt). this is benign today (TT puts those tasks in unassigned state) - but in this world will not be so benign. > Allow FairScheduler to control the number of slots on each TaskTracker > -- > > Key: MAPREDUCE-2198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2198 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > > We can set the number of slots on the TaskTracker to be high and let > FairScheduler handles the slots. > This approach allows us to change the number of slots on each node > dynamically. > The administrator can change the number of slots with a CLI tool. > One use case of this is for upgrading the MapReduce. > Instead of restarting the cluster, we can run the new MapReduce on the same > cluster. > And use the CLI tool to gradually migrate the slots. > This way we don't lost the progress fo the jobs that's already executed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2198) Allow FairScheduler to control the number of slots on each TaskTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934739#action_12934739 ] Scott Chen commented on MAPREDUCE-2198: --- Hey M.C. Yes, we can set a higher slot limit on TT and let scheduler manage the slots. bq. Are we running two TT's on each node, that talk to different JT's, and "migrating" slots from one to the other? Yes. The motivation here is that when deploying new JT and TT. We need to restart the cluster and we lose all the running jobs. This can be solved by the way you described. Other use case is that people can experiment with the best slot settings by using the CLI without restarting the cluster. Right now if you want to change the number of slots, you have to change the conf on every TT and restart. Scott > Allow FairScheduler to control the number of slots on each TaskTracker > -- > > Key: MAPREDUCE-2198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2198 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > > We can set the number of slots on the TaskTracker to be high and let > FairScheduler handles the slots. > This approach allows us to change the number of slots on each node > dynamically. > The administrator can change the number of slots with a CLI tool. > One use case of this is for upgrading the MapReduce. > Instead of restarting the cluster, we can run the new MapReduce on the same > cluster. > And use the CLI tool to gradually migrate the slots. > This way we don't lost the progress fo the jobs that's already executed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2198) Allow FairScheduler to control the number of slots on each TaskTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934738#action_12934738 ] dhruba borthakur commented on MAPREDUCE-2198: - > Are we running two TT's on each node, that talk to different JT's, and > "migrating" slots from one to the other? Precisely. We have a solution that is used to deploy new software to the JT-TT. Earlier, the JT/TT have to be shutdown, new code deployed, and then the cluster is restarted. This means that the cluster was unavailable for a while and currently running jobs all fails when the cluster is shutdown. The modified approach is to direct all new jobs to the newly created JT instance, and then slowly (and proportionally) migrate slots from the old JT instance to the new JT instance. This allows JT software upgrades without incurring any cluster downtime. > Allow FairScheduler to control the number of slots on each TaskTracker > -- > > Key: MAPREDUCE-2198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2198 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > > We can set the number of slots on the TaskTracker to be high and let > FairScheduler handles the slots. > This approach allows us to change the number of slots on each node > dynamically. > The administrator can change the number of slots with a CLI tool. > One use case of this is for upgrading the MapReduce. > Instead of restarting the cluster, we can run the new MapReduce on the same > cluster. > And use the CLI tool to gradually migrate the slots. > This way we don't lost the progress fo the jobs that's already executed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
[jira] Commented: (MAPREDUCE-2198) Allow FairScheduler to control the number of slots on each TaskTracker
[ https://issues.apache.org/jira/browse/MAPREDUCE-2198?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934732#action_12934732 ] M. C. Srivas commented on MAPREDUCE-2198: - The scheduler can easily choose to ignore the slot information while scheduling, correct? Then why is this needed? Are we running two TT's on each node, that talk to different JT's, and "migrating" slots from one to the other? > Allow FairScheduler to control the number of slots on each TaskTracker > -- > > Key: MAPREDUCE-2198 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-2198 > Project: Hadoop Map/Reduce > Issue Type: New Feature > Components: contrib/fair-share >Affects Versions: 0.22.0 >Reporter: Scott Chen >Assignee: Scott Chen > Fix For: 0.22.0 > > > We can set the number of slots on the TaskTracker to be high and let > FairScheduler handles the slots. > This approach allows us to change the number of slots on each node > dynamically. > The administrator can change the number of slots with a CLI tool. > One use case of this is for upgrading the MapReduce. > Instead of restarting the cluster, we can run the new MapReduce on the same > cluster. > And use the CLI tool to gradually migrate the slots. > This way we don't lost the progress fo the jobs that's already executed. -- This message is automatically generated by JIRA. - You can reply to this email to add a comment to the issue online.
