[
https://issues.apache.org/jira/browse/MAPREDUCE-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Matt Foley updated MAPREDUCE-4782:
----------------------------------
Target Version/s: 1.1.1, 0.23.5 (was: 0.23.5)
> NLineInputFormat skips first line of last InputSplit
> ----------------------------------------------------
>
> Key: MAPREDUCE-4782
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4782
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 0.22.0, 0.23.0, 1.0.0, 2.0.0-alpha, trunk
> Environment: job.setMapperClass(Mapper.class); // just pass text
> lines through to output
> job.setInputFormatClass(NLineInputFormat.class);
> NLineInputFormat.setNumLinesPerSplit(job, 100);
> NLineInputFormat.setInputPaths(job, "/path/to/a_file_with_many_lines.txt");
> Reporter: Mark Fuhs
> Priority: Critical
> Attachments: MAPREDUCE-4782.patch, MR-4782.txt
>
>
> NLineInputFormat creates FileSplits that are then used by LineRecordReader to
> generate Text values. To deal with an idiosyncrasy of LineRecordReader, the
> begin and length fields of the FileSplit are constructed differently for the
> first FileSplit vs. the rest.
> After looping through all lines of a file, the final FileSplit is created,
> but the creation does not respect the difference of how the first vs. the
> rest of the FileSplits are created.
> This results in the first line of the final InputSplit being skipped. I've
> created a patch to NLineInputFormat, and this fixes the problem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
