[
https://issues.apache.org/jira/browse/MAPREDUCE-7243?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16938773#comment-16938773
]
David Mollitor commented on MAPREDUCE-7243:
-------------------------------------------
I was thinking that the list gets generated once and then provided to each
task. So the list is random, but all the tasks load the files still in the
same order. However, it looks like the list is generated for each Task Attempt.
https://github.com/apache/hadoop/blob/499c70eda5f6315cf7e13a4b3bbefc7e187bc457/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/job/impl/TaskAttemptImpl.java#L913
> Randomize List of Resources to Localize
> ---------------------------------------
>
> Key: MAPREDUCE-7243
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7243
> Project: Hadoop Map/Reduce
> Issue Type: New Feature
> Components: nodemanager, performance
> Reporter: David Mollitor
> Assignee: David Mollitor
> Priority: Minor
>
> https://github.com/apache/hadoop/blob/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-common/src/main/java/org/apache/hadoop/mapreduce/v2/util/LocalResourceBuilder.java
> Add a new parameter to {{LocalResourceBuilder}} that allows the list of
> resources to be shuffled randomly. This will allow the Localizer to spread
> the load of requests so that not all of the NodeManagers are requesting to
> localize the same files, in the same order, from the same DataNodes,
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
