[ https://issues.apache.org/jira/browse/HADOOP-9867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14046138#comment-14046138 ]
Jason Lowe commented on HADOOP-9867: ------------------------------------ Actually I agree with Rushabh that there are at least two somewhat different problems here. The original problem reported in the JIRA has to do with records being dropped with uncompressed inputs. We should fix that issue so we don't drop data when using an uncompressed input. I'm assuming Rushabh's patch solves that issue, but I haven't looked at it in detail just yet. There's another issue related to mistaken record delimiter recognition where the subsequent split reader can accidentally think it found a delimiter when in fact the real record delimiter is somewhere else. If the subsequent split reader sees 'xxxxyzxxx' at the beginning of its split then it will toss out the first record (i.e.: the first 'xxx') then read 'xyz' as the next record. However that may or may not be the correct behavior, because with that kind of delimiter and data the correct behavior depends upon the _previous_ split's data. If the previous split ended with 'abc' then the behavior was correct and there are two records in the stream: 'abc' and 'xyz'. If the previous split ended with 'abcx' then that's the incorrect behavior. The records should be 'abc' and 'xxyz' but the second split reader will report an 'xyz' record that shouldn't exist. To solve that problem either a split reader would have to examine the prior-split's data to distinguish this case, or the split reader would have to realize it's an ambiguous situation and leave the record processing to the previous split reader to handle. The former can be very expensive if the prior split is compressed, as it has to potentially unpack the entire split. Also this can get very tricky and a reader may need to read more than one other split to resolve it. For example, if the data stream is 'axxxxxxxxxxxxx......xxxxxxbxxxxxx......xxxxxcxxxxxx' then a reader may have to scan far down into subsequent splits since only it knows where the true record boundaries are. Simply tacking on an extra character at the beginning of that input changes where the record boundaries are and the record contents even the last split in the input. Solving this requires a different high-level algorithm to split processing than what we have today (i.e.: throw away the first record and go), so I believe that's something better left to a followup JIRA. It'd be nice to solve the dropped-record problem for scenarios where we don't have to worry about mistaken record delimiter recognition in the data, as that's an incremental improvement from where we are today. I'll try to get some time to review the latest patch and provide comments soon. > org.apache.hadoop.mapred.LineRecordReader does not handle multibyte record > delimiters well > ------------------------------------------------------------------------------------------ > > Key: HADOOP-9867 > URL: https://issues.apache.org/jira/browse/HADOOP-9867 > Project: Hadoop Common > Issue Type: Bug > Components: io > Affects Versions: 0.20.2, 0.23.9, 2.2.0 > Environment: CDH3U2 Redhat linux 5.7 > Reporter: Kris Geusebroek > Assignee: Vinayakumar B > Priority: Critical > Attachments: HADOOP-9867.patch, HADOOP-9867.patch, HADOOP-9867.patch, > HADOOP-9867.patch > > > Having defined a recorddelimiter of multiple bytes in a new InputFileFormat > sometimes has the effect of skipping records from the input. > This happens when the input splits are split off just after a > recordseparator. Starting point for the next split would be non zero and > skipFirstLine would be true. A seek into the file is done to start - 1 and > the text until the first recorddelimiter is ignored (due to the presumption > that this record is already handled by the previous maptask). Since the re > ord delimiter is multibyte the seek only got the last byte of the delimiter > into scope and its not recognized as a full delimiter. So the text is skipped > until the next delimiter (ignoring a full record!!) -- This message was sent by Atlassian JIRA (v6.2#6252)