[ https://issues.apache.org/jira/browse/MAPREDUCE-3710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192560#comment-13192560 ]
Hudson commented on MAPREDUCE-3710: ----------------------------------- Integrated in Hadoop-Common-0.23-Commit #413 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/413/]) MAPREDUCE-3710. Improved FileInputFormat to return better locality for the last split. Contributed by Siddarth Seth. svn merge --ignore-ancestry -c 1235510 ../../trunk/ vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235511 Files : * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapred/FileInputFormat.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/FileInputFormat.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapred/TestFileInputFormat.java * /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestFileInputFormat.java > last split generated by FileInputFormat.getSplits may not have the best > locality > -------------------------------------------------------------------------------- > > Key: MAPREDUCE-3710 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-3710 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, mrv2 > Affects Versions: 0.23.0, 1.0.0 > Reporter: Siddharth Seth > Assignee: Siddharth Seth > Fix For: 0.23.1 > > Attachments: MR-3710_v1.txt > > > The last split generated by FileInputFormat.getSplits considers > {{blkLocations.length-1}} to be the hosts for the split. > The last split may be larger than the rest (SPLIT_SLOP=1.1 by default) - in > which case locality is picked up from a smaller block. > e.g. 1027MB file with a 128MB split size. The last split ends up being 131MB. > The hosts for locality end up being the nodes containing the 3MB block > instead of the 128MB block. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira