[
https://issues.apache.org/jira/browse/MAPREDUCE-4782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13494650#comment-13494650
]
Hudson commented on MAPREDUCE-4782:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #431 (See
[https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/431/])
svn merge -c 1407505 FIXES: MAPREDUCE-4782. NLineInputFormat skips first
line of last InputSplit (Mark Fuhs via bobby) (Revision 1407507)
Result = UNSTABLE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1407507
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/lib/input/NLineInputFormat.java
*
/hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-jobclient/src/test/java/org/apache/hadoop/mapreduce/lib/input/TestNLineInputFormat.java
> NLineInputFormat skips first line of last InputSplit
> ----------------------------------------------------
>
> Key: MAPREDUCE-4782
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4782
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
> Affects Versions: 0.22.0, 0.23.0, 1.0.0, 2.0.0-alpha, trunk
> Environment: job.setMapperClass(Mapper.class); // just pass text
> lines through to output
> job.setInputFormatClass(NLineInputFormat.class);
> NLineInputFormat.setNumLinesPerSplit(job, 100);
> NLineInputFormat.setInputPaths(job, "/path/to/a_file_with_many_lines.txt");
> Reporter: Mark Fuhs
> Assignee: Mark Fuhs
> Priority: Blocker
> Fix For: 1.1.1, 1.2.0, 3.0.0, 2.0.3-alpha, 0.23.5
>
> Attachments: MAPREDUCE-4782.patch, MR-4782-branch-1.txt, MR-4782.txt
>
>
> NLineInputFormat creates FileSplits that are then used by LineRecordReader to
> generate Text values. To deal with an idiosyncrasy of LineRecordReader, the
> begin and length fields of the FileSplit are constructed differently for the
> first FileSplit vs. the rest.
> After looping through all lines of a file, the final FileSplit is created,
> but the creation does not respect the difference of how the first vs. the
> rest of the FileSplits are created.
> This results in the first line of the final InputSplit being skipped. I've
> created a patch to NLineInputFormat, and this fixes the problem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
