[
https://issues.apache.org/jira/browse/MAPREDUCE-5583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14338866#comment-14338866
]
Junping Du commented on MAPREDUCE-5583:
---------------------------------------
Thanks [~jlowe] for updating the patch!
bq. In the common case, the list of containers to assign is empty and that
will preclude the need to call canAssignMaps. Since the "early out" at the
beginning of the method doesn't save much I didn't make this change.
Agree. "early out" doesn't seem to have much value here.
bq. However I left it as "if (limit < 0)" because there are cases where we
want the limit to really be 0 (i.e.: to not request any more containers of that
priority).
If so, I think we should keep the logic consistent here. Many places seems to
treat limit=0 the same way as <0, e.g.
{code}
+ private boolean canAssignMaps() {
+ return (maxRunningMaps <= 0
+ || assignedRequests.maps.size() < maxRunningMaps);
+ }
+
+ private boolean canAssignReduces() {
+ return (maxRunningReduces <= 0
+ || assignedRequests.reduces.size() < maxRunningReduces);
+ }
{code}
bq. ANY limits the total number of containers the application is requesting.
Any new container being requested, whether that's node-local or rack-local,
requires the ANY level to be incremented by one so the RM knows it needs to
allocate one more container.
Agree.
bq. The RM will only grant as many containers as the app requested at the ANY
level regardless of how many it asked at the rack or node level.
I am not sure if big impact for any inconsistent among rack-local, node-local
and ANY in request here. From what I quick check on FIFOScheduler (which is
simpler than CS or FS), looks like it will first check maxContainer from ANY
request like you said. However, when maxContainer >0 (request for ANY is
exist), it will go ahead to assign container according to requests in sequence
of Node-local, Rack-local or ANY. Assume the same request for 3 containers on
node0, rack0 and ANY, it could adjust to 2 containers for ANY in client side,
but the real allocation could still be 3 containers. Am I missing something
here?
bq. It's important we only limit at the ANY level because the AM has no idea
what locality is available on the cluster. Essentially what we're doing with
this patch is giving the RM the full information of what we want wrt. locality
but then also telling it to not give us all of those containers right now. That
allows the RM to make the decision of which containers to grant, and it will
try to do its best with locality.
I can see we are making request for node or local can have more rich info for
locality reason. However, I have a quick question here: do we add back filtered
asks later when limit is not hit then? I didn't find this important logic there.
> Ability to limit running map and reduce tasks
> ---------------------------------------------
>
> Key: MAPREDUCE-5583
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5583
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mr-am, mrv2
> Affects Versions: 0.23.9, 2.1.1-beta
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Attachments: MAPREDUCE-5583-branch2.4.1.patch,
> MAPREDUCE-5583v1.patch, MAPREDUCE-5583v2.patch, MAPREDUCE-5583v3.patch,
> MAPREDUCE-5583v4.patch
>
>
> It would be nice if users could specify a limit to the number of map or
> reduce tasks that are running simultaneously. Occasionally users are
> performing operations in tasks that can lead to DDoS scenarios if too many
> tasks run simultaneously (e.g.: accessing a database, web service, etc.).
> Having the ability to throttle the number of tasks simultaneously running
> would provide users a way to mitigate issues with too many tasks on a large
> cluster attempting to access a serivce at any one time.
> This is similar to the functionality requested by MAPREDUCE-224 and
> implemented by HADOOP-3412 but was dropped in mrv2.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
