Scott Oaks created MAPREDUCE-7338:
-------------------------------------
Summary: ArrayIndexOutOfBoundsException when sort buffer wraps
incorrectly
Key: MAPREDUCE-7338
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7338
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Scott Oaks
Exception generated when writing output from map task:
{noformat}
Error: java.lang.ArrayIndexOutOfBoundsException at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1469)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer$Buffer.write(MapTask.java:1365)
at java.io.DataOutputStream.writeByte(DataOutputStream.java:153) at
org.apache.hadoop.io.WritableUtils.writeVLong(WritableUtils.java:273) at
org.apache.hadoop.io.WritableUtils.writeVInt(WritableUtils.java:253) at
org.apache.hadoop.io.Text.write(Text.java:330) at
org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:98)
at
org.apache.hadoop.io.serializer.WritableSerialization$WritableSerializer.serialize(WritableSerialization.java:82)
at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.collect(MapTask.java:1163)
at org.apache.hadoop.mapred.MapTask$NewOutputCollector.write(MapTask.java:727)
at
org.apache.hadoop.mapreduce.task.TaskInputOutputContextImpl.write(TaskInputOutputContextImpl.java:89)
at
org.apache.hadoop.mapreduce.lib.map.WrappedMapper$Context.write(WrappedMapper.java:112)
at
{noformat}
This happens in our case when the io.sort.mb is set to 1750; if we reduce the
size of the buffer it wraps correctly and no exception is generated.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
