ConfX created MAPREDUCE-7455:
--------------------------------
Summary: org.apache.hadoop.mapred.SpillRecord crashes due to
overflow in buffer size computation
Key: MAPREDUCE-7455
URL: https://issues.apache.org/jira/browse/MAPREDUCE-7455
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 3.3.6
Reporter: ConfX
A large `mapreduce.job.reduces` can cause overflow while computing the byte
buffer in `org.apache.hadoop.mapred.SpillRecord#SpillRecord(int)`, since the
byte buffer size equals to `mapreduce.job.reduces` *
MapTask.MAP_OUTPUT_INDEX_RECORD_LENGTH
To reproduce:
1. set `mapreduce.job.reduces` to 509103844
2. run `mvn surefire:test
-Dtest=org.apache.hadoop.mapred.TestMapTask#testShufflePermissions`
We create a PR that provides a fix by checking the computed buffer size is
positive.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
