[ https://issues.apache.org/jira/browse/MAPREDUCE-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4782: ------------------------------------------- Attachment: MR-4782.txt I was able to reproduce the issue, and I have updated the test case to reproduce it as well. The original test case did not check the last split, I don't know why. I also found out that this exists in branch-1 as well. > NLineInputFormat skips first line of last InputSplit > ---------------------------------------------------- > > Key: MAPREDUCE-4782 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4782 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client > Affects Versions: 0.22.0, 0.23.0, 1.0.0, 2.0.0-alpha, trunk > Environment: job.setMapperClass(Mapper.class); // just pass text > lines through to output > job.setInputFormatClass(NLineInputFormat.class); > NLineInputFormat.setNumLinesPerSplit(job, 100); > NLineInputFormat.setInputPaths(job, "/path/to/a_file_with_many_lines.txt"); > Reporter: Mark Fuhs > Priority: Critical > Attachments: MAPREDUCE-4782.patch, MR-4782.txt > > > NLineInputFormat creates FileSplits that are then used by LineRecordReader to > generate Text values. To deal with an idiosyncrasy of LineRecordReader, the > begin and length fields of the FileSplit are constructed differently for the > first FileSplit vs. the rest. > After looping through all lines of a file, the final FileSplit is created, > but the creation does not respect the difference of how the first vs. the > rest of the FileSplits are created. > This results in the first line of the final InputSplit being skipped. I've > created a patch to NLineInputFormat, and this fixes the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira