[ https://issues.apache.org/jira/browse/MAPREDUCE-5767?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15641821#comment-15641821 ]
Adam Ringel commented on MAPREDUCE-5767: ---------------------------------------- Is this possibly still a problem? I'm getting the error after a long running map job on Hadoop 2.7.3. I can't increase mapreduce.task.io.sort.mb > 2047 because of [https://github.com/apache/hadoop-mapreduce/blob/HDFS-641/src/java/org/apache/hadoop/mapred/MapTask.java?source=cc#L746] Interestingly enough some other searches from Google suggest _decreasing_ the parameter, e.g. [http://comments.gmane.org/gmane.comp.jakarta.lucene.hadoop.user/44152] However, since it takes ~5hrs to reach the error point doing trial and error testing is problematic. {noformat} Task: attempt_1478217421435_0024_m_000000_0 - exited : java.lang.ArrayIndexOutOfBoundsException at org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1453) at java.io.DataOutputStream.writeLong(DataOutputStream.java:224) at org.apache.hadoop.io.LongWritable.write(LongWritable.java:52) ... at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:98) at org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:82) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1149) at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:715) at org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89) at org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112) at org.apache.hadoop.mapreduce.Mapper.map(Mapper.java:125) at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:146) at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:787) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:164) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:422) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1698) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158) {noformat} > Data corruption when single value exceeds map buffer size (io.sort.mb) > ---------------------------------------------------------------------- > > Key: MAPREDUCE-5767 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5767 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv1 > Affects Versions: 0.20.1 > Reporter: Ben Roling > > There is an issue in org.apache.hadoop.mapred.MapTask in 0.20 that can cause > data corruption when the size of a single value produced by the mapper > exceeds the size of the map output buffer (roughly io.sort.mb). > I experienced this issue in CDH4.2.1, but am logging the issue here for > greater visibility in case anyone else might run across the issue. > The issue does not exist in 0.21 and beyond due to the implementation of > MAPREDUCE-64. That JIRA significantly changes the way the map output > buffering is done and it looks like the issue has been resolved by those > changes. > I expect this bug will likely be closed / won't fix due to the fact that 0.20 > is obsolete. As stated previously, I am just logging this issue for > visibility in case anyone else is still running something based on 0.20 and > encounters the same problem. > In my situation the issue manifested as an ArrayIndexOutOfBoundsException in > the reduce phase when deserializing a key -- causing the job to fail. > However, I think the problem could manifest in a more dangerous fashion where > the affected job succeeds, but produces corrupt output. The stack trace I > saw was: > 2014-02-13 01:07:34,690 WARN org.apache.hadoop.mapred.Child: Error running > child > java.lang.ArrayIndexOutOfBoundsException: 24 > at > org.apache.avro.io.parsing.Symbol$Alternative.getSymbol(Symbol.java:364) > at > org.apache.avro.io.ResolvingDecoder.doAction(ResolvingDecoder.java:229) > at org.apache.avro.io.parsing.Parser.advance(Parser.java:88) > at > org.apache.avro.io.ResolvingDecoder.readIndex(ResolvingDecoder.java:206) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:148) > at > org.apache.avro.generic.GenericDatumReader.readRecord(GenericDatumReader.java:173) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:144) > at > org.apache.avro.generic.GenericDatumReader.read(GenericDatumReader.java:135) > at > org.apache.crunch.types.avro.SafeAvroSerialization$AvroWrapperDeserializer.deserialize(SafeAvroSerialization.java:86) > at > org.apache.crunch.types.avro.SafeAvroSerialization$AvroWrapperDeserializer.deserialize(SafeAvroSerialization.java:70) > at > org.apache.hadoop.mapreduce.task.ReduceContextImpl.nextKeyValue(ReduceContextImpl.java:135) > at > org.apache.hadoop.mapreduce.task.ReduceContextImpl.nextKey(ReduceContextImpl.java:114) > at > org.apache.hadoop.mapreduce.lib.reduce.WrappedReducer$Context.nextKey(WrappedReducer.java:291) > at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:163) > at > org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:610) > at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:444) > at org.apache.hadoop.mapred.Child$4.run(Child.java:268) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:396) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) > at org.apache.hadoop.mapred.Child.main(Child.java:262) > The problem appears to me to be in > org.apache.hadoop.mapred.MapTask.MapOutputBuffer.Buffer.write(byte[], int, > int). The sequence of events that leads up to the issue is: > * some complete records (cumulative size less than total buffer size) written > to buffer > * large (over io.sort.mb) record starts writing > * [soft buffer limit > exceeded|https://github.com/apache/hadoop-common/blob/release-0.20.1/src/mapred/org/apache/hadoop/mapred/MapTask.java#L1030] > - spill starts > * write of large record continues > * buffer becomes > [full|https://github.com/apache/hadoop-common/blob/release-0.20.1/src/mapred/org/apache/hadoop/mapred/MapTask.java#L1012] > * > [wrap|https://github.com/apache/hadoop-common/blob/release-0.20.1/src/mapred/org/apache/hadoop/mapred/MapTask.java#L1013] > evaluates to true, suggesting the buffer can be safely wrapped > * writing the large record continues until a write occurs such that bufindex > + len == bufstart exactly. When this happens > [buffull|https://github.com/apache/hadoop-common/blob/release-0.20.1/src/mapred/org/apache/hadoop/mapred/MapTask.java#L1018] > evaluates to false, so the data gets written to the buffer without event > * writing of the large value continues with another call to write(), starting > the corruption of the buffer. Buffer full can no longer be detected by the > [buffull > logic|https://github.com/apache/hadoop-common/blob/release-0.20.1/src/mapred/org/apache/hadoop/mapred/MapTask.java#L1012] > that is used when bufindex >= bufstart > The key to this problem occurring is a write where bufindex + len equals > bufstart exactly. > I have titled the issue as having to do with writing large records (over > io.sort.mb), but really I think the issue *could* occur on smaller records if > the serializer generated a write of exactly the right size. For example, if > the buffer is getting close to full, but hasn't exceeded the buffer soft > limit and then a collect() on a new value is called that triggers a write() > such that bufindex + len == bufstart. The size of the write would have to be > relatively large -- greater than the free space offered by the soft limit > (20% of the buffer by default), making the issue occurring that way pretty > unlikely. -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org