[ https://issues.apache.org/jira/browse/MAPREDUCE-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4782: ------------------------------------------- Resolution: Fixed Fix Version/s: 0.23.5 2.0.3-alpha 3.0.0 1.2.0 1.1.1 Status: Resolved (was: Patch Available) Thanks Mark, This is a great catch, I just wish we had found it sooner. I put this into trunk, branch-2, branch-0.23, branch-1, and branch-1.1. If I missed any branches that people want it in please let me know and I will see what I can do. > NLineInputFormat skips first line of last InputSplit > ---------------------------------------------------- > > Key: MAPREDUCE-4782 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4782 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: client > Affects Versions: 0.22.0, 0.23.0, 1.0.0, 2.0.0-alpha, trunk > Environment: job.setMapperClass(Mapper.class); // just pass text > lines through to output > job.setInputFormatClass(NLineInputFormat.class); > NLineInputFormat.setNumLinesPerSplit(job, 100); > NLineInputFormat.setInputPaths(job, "/path/to/a_file_with_many_lines.txt"); > Reporter: Mark Fuhs > Assignee: Mark Fuhs > Priority: Blocker > Fix For: 1.1.1, 1.2.0, 3.0.0, 2.0.3-alpha, 0.23.5 > > Attachments: MAPREDUCE-4782.patch, MR-4782-branch-1.txt, MR-4782.txt > > > NLineInputFormat creates FileSplits that are then used by LineRecordReader to > generate Text values. To deal with an idiosyncrasy of LineRecordReader, the > begin and length fields of the FileSplit are constructed differently for the > first FileSplit vs. the rest. > After looping through all lines of a file, the final FileSplit is created, > but the creation does not respect the difference of how the first vs. the > rest of the FileSplits are created. > This results in the first line of the final InputSplit being skipped. I've > created a patch to NLineInputFormat, and this fixes the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira