Robert Joseph Evans created MAPREDUCE-4760:
----------------------------------------------
Summary: Make a version of Counters that is composit for the job
and stores the counter values in arrays.
Key: MAPREDUCE-4760
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4760
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 0.23.4, 2.0.2-alpha
Reporter: Robert Joseph Evans
Priority: Minor
String interning reduced the size of counters a lot. After that and the fix
for a memory leak in the IPC server a job with 20000 map tasks and 3000
reducers takes about 200MB to store the state of all of the tasks. Looking at
a memory dump of the AM each task attempt has a pointer to a Counters object
that is about 2kb to 3kb in size. That means Counters account for about 56MB
of the 200MB of state. This job only had about 40 task counters in it. Each
counter stores a long value so if we stored them in a long[] instead we should
only be taking up 7MB.
Also assuming that some of the counters only appear in a map task or a reduce
task we should be able to have one CompositCounters for map tasks and one for
reduce tasks so it would reduce the size even further.
NOTE: without this change I would expect to be able to run a 100,000 task job
in the default 1024MB AM heap (875MB/200MB * 2300) I reserved 150MB for IPC
buffers and event data. With this change we could expect to run about 130,000
tasks (875MB/150MB * 2300).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
