[
https://issues.apache.org/jira/browse/HADOOP-2573?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12557770#action_12557770
]
[EMAIL PROTECTED] edited comment on HADOOP-2573 at 1/10/08 12:28 PM:
---------------------------------------------------------------
(oops... yes, doug anticipated this in his comment and I didn't read very well)
Presumably the limit could be made dynamic. The limit could be
max(static_limit, number of cores in cluster / # active jobs)
On further reflection, I should note that my big jobs are all limited in pretty
much the way that Doug suggests because they are processing a few large files
that are unsplittable. This limits the number of slots these big jobs can eat
up.
The result is pretty OK. My little jobs with lots of maps can slide through
the cracks most of the time and everything runs pretty well.
was (Author: [EMAIL PROTECTED]):
Presumably the limit could be made dynamic. The limit could be
max(static_limit, number of cores in cluster / # active jobs)
> limit running tasks per job
> ---------------------------
>
> Key: HADOOP-2573
> URL: https://issues.apache.org/jira/browse/HADOOP-2573
> Project: Hadoop
> Issue Type: New Feature
> Components: mapred
> Reporter: Doug Cutting
> Fix For: 0.17.0
>
>
> It should be possible to specify a limit to the number of tasks per job
> permitted to run simultaneously. If, for example, you have a cluster of 50
> nodes, with 100 map task slots and 100 reduce task slots, and the configured
> limit is 25 simultaneous tasks/job, then four or more jobs will be able to
> run at a time. This will permit short jobs to pass longer-running jobs.
> This also avoids some problems we've seen with HOD, where nodes are
> underutilized in their tail, and it should permit improved input locality.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
