[
https://issues.apache.org/jira/browse/HADOOP-7154?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HADOOP-7154:
--------------------------------
Resolution: Fixed
Hadoop Flags: [Reviewed]
Status: Resolved (was: Patch Available)
Committed to trunk and branch-22. Thanks Tom for reviewing.
> Should set MALLOC_ARENA_MAX in hadoop-env.sh
> --------------------------------------------
>
> Key: HADOOP-7154
> URL: https://issues.apache.org/jira/browse/HADOOP-7154
> Project: Hadoop Common
> Issue Type: Improvement
> Components: scripts
> Affects Versions: 0.22.0
> Reporter: Todd Lipcon
> Assignee: Todd Lipcon
> Priority: Minor
> Fix For: 0.22.0
>
> Attachments: hadoop-7154.txt
>
>
> New versions of glibc present in RHEL6 include a new arena allocator design.
> In several clusters we've seen this new allocator cause huge amounts of
> virtual memory to be used, since when multiple threads perform allocations,
> they each get their own memory arena. On a 64-bit system, these arenas are
> 64M mappings, and the maximum number of arenas is 8 times the number of
> cores. We've observed a DN process using 14GB of vmem for only 300M of
> resident set. This causes all kinds of nasty issues for obvious reasons.
> Setting MALLOC_ARENA_MAX to a low number will restrict the number of memory
> arenas and bound the virtual memory, with no noticeable downside in
> performance - we've been recommending MALLOC_ARENA_MAX=4. We should set this
> in hadoop-env.sh to avoid this issue as RHEL6 becomes more and more common.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
