[ https://issues.apache.org/jira/browse/MAPREDUCE-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15006839#comment-15006839 ]
Dustin Cote commented on MAPREDUCE-6549: ---------------------------------------- [~wilfreds] Are you sure your fix is the right one? The reason I changed the tests was because they validate an incomplete record for some reason at the end (which is the part of my fix that is breaking tests in the mapred package since I forgot to change them). The reason I'm saying this is because I would expect the following: Input: abcdefghij++kl++mno Output records: 1) abcdefghij 2) kl It looks like your tests do the same thing. mno doesn't have a delimiter at the end, so isn't that garbage data as an incomplete record? I would expect that to be the behavior if I were using the API, but I don't see any real documentation for this on multibyte delimiters. If we're going to commit to the last part of the data without a delimiter at the end as being a record, then that should be documented as well. Otherwise, I'd rather merge our patches together and verify the functionality so that the above scenario is what happens instead of pulling in undelimited data at the end of the file. > multibyte delimiters with LineRecordReader cause duplicate records > ------------------------------------------------------------------ > > Key: MAPREDUCE-6549 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6549 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1, mrv2 > Affects Versions: 2.7.2 > Reporter: Dustin Cote > Assignee: Wilfred Spiegelenburg > Attachments: MAPREDUCE-6549-1.patch, MAPREDUCE-6549-2.patch > > > LineRecorderReader currently produces duplicate records under certain > scenarios such as: > 1) input string: "abc+++def++ghi++" > delimiter string: "+++" > test passes with all sizes of the split > 2) input string: "abc++def+++ghi++" > delimiter string: "+++" > test fails with a split size of 4 > 2) input string: "abc+++def++ghi++" > delimiter string: "++" > test fails with a split size of 5 > 3) input string "abc+++defg++hij++" > delimiter string: "++" > test fails with a split size of 4 > 4) input string "abc++def+++ghi++" > delimiter string: "++" > test fails with a split size of 9 -- This message was sent by Atlassian JIRA (v6.3.4#6332)