[ https://issues.apache.org/jira/browse/MAPREDUCE-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494163#comment-13494163 ]
Hudson commented on MAPREDUCE-4782: ----------------------------------- Integrated in Hadoop-Mapreduce-trunk #1252 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1252/]) MAPREDUCE-4782. NLineInputFormat skips first line of last InputSplit (Mark Fuhs via bobby) (Revision 1407505) Result = FAILURE bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407505 Files : * /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/NLineInputFormat.java * /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestNLineInputFormat.java > NLineInputFormat skips first line of last InputSplit > ---------------------------------------------------- > > Key: MAPREDUCE-4782 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4782 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client > Affects Versions: 0.22.0, 0.23.0, 1.0.0, 2.0.0-alpha, trunk > Environment: job.setMapperClass(Mapper.class); // just pass text > lines through to output > job.setInputFormatClass(NLineInputFormat.class); > NLineInputFormat.setNumLinesPerSplit(job, 100); > NLineInputFormat.setInputPaths(job, "/path/to/a_file_with_many_lines.txt"); > Reporter: Mark Fuhs > Assignee: Mark Fuhs > Priority: Blocker > Fix For: 1.1.1, 1.2.0, 3.0.0, 2.0.3-alpha, 0.23.5 > > Attachments: MAPREDUCE-4782.patch, MR-4782-branch-1.txt, MR-4782.txt > > > NLineInputFormat creates FileSplits that are then used by LineRecordReader to > generate Text values. To deal with an idiosyncrasy of LineRecordReader, the > begin and length fields of the FileSplit are constructed differently for the > first FileSplit vs. the rest. > After looping through all lines of a file, the final FileSplit is created, > but the creation does not respect the difference of how the first vs. the > rest of the FileSplits are created. > This results in the first line of the final InputSplit being skipped. I've > created a patch to NLineInputFormat, and this fixes the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira