[
https://issues.apache.org/jira/browse/MAPREDUCE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14339096#comment-14339096
]
Jason Lowe commented on MAPREDUCE-5583:
---------------------------------------
bq. If so, I think we should keep the logic consistent here. Many places seems
to treat limit=0 the same way as <0
The limit I referred to before is a _temporary_ limit applied to the current
ask. This limit is the one the user specified, which is effectively constant
for the lifetime of the job. It makes no sense to configure a job with a
maximum concurrent task limit of zero -- the job would just hang. Therefore
the code treats a user-specified limit of 0 as no limit. However during a
limit-enabled job's lifetime it may need to temporarily set the ask to zero
because it's already running at the configured limit. That's why one of them
uses limit=0 to really mean no tasks will be scheduled and the other does not.
bq. Assume the same request for 3 containers on node0, rack0 and ANY, it could
adjust to 2 containers for ANY in client side, but the real allocation could
still be 3 containers. Am I missing something here?
Yes, it looks like the FIFO scheduler handles this differently and IMHO has a
bug. Both the CapacityScheduler and, from my brief glance at it, the
FairScheduler will only assign as many containers as specified by ANY. Each
container allocated will always decrement the current ANY ask, and the
scheduler checks at the top of the assignment loop if there are still ANY asks
before proceeding. See LeafQueue#assignContainers and
SchedulerApplicationAttempt#getTotalRequiredResources. It's true that it will
consider node-level asks and rack-level asks, but it only assigns one
container. As I stated before, the problem with the AM tuning down the rack-
and node- level requests is knowing *which* racks and nodes are the correct
ones to limit. The AM cannot know this, but the RM can. That's why the full
locality information should be sent to the RM and let it decide where to
allocate the limited number of containers.
bq. do we add back filtered asks later when limit is not hit then?
Yes, RMContainerAllocator performs the current limit calculation every
heartbeat and will send the current requested ask (i.e.: how many containers we
are short of hitting the limit). See applyConcurrentTaskLimits. So if the
limit is 10 and we get 7 containers, it will turn around and ask for 3 the next
heartbeat. If two tasks then complete, it will update the ask from 3 to 5
(assuming no more containers were granted in the interim).
> Ability to limit running map and reduce tasks
> ---------------------------------------------
>
> Key: MAPREDUCE-5583
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5583
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am, mrv2
> Affects Versions: 0.23.9, 2.1.1-beta
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Attachments: MAPREDUCE-5583-branch2.4.1.patch,
> MAPREDUCE-5583v1.patch, MAPREDUCE-5583v2.patch, MAPREDUCE-5583v3.patch,
> MAPREDUCE-5583v4.patch
>
>
> It would be nice if users could specify a limit to the number of map or
> reduce tasks that are running simultaneously. Occasionally users are
> performing operations in tasks that can lead to DDoS scenarios if too many
> tasks run simultaneously (e.g.: accessing a database, web service, etc.).
> Having the ability to throttle the number of tasks simultaneously running
> would provide users a way to mitigate issues with too many tasks on a large
> cluster attempting to access a serivce at any one time.
> This is similar to the functionality requested by MAPREDUCE-224 and
> implemented by HADOOP-3412 but was dropped in mrv2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
