[ https://issues.apache.org/jira/browse/HADOOP-7154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12999255#comment-12999255 ]
Todd Lipcon commented on HADOOP-7154: ------------------------------------- We should also consider adding a "ulimit -u <some high number>" automatically for RHEL6 users, since RHEL6 sets the soft max thread limit to 1024 by default (see red hat bug: https://bugzilla.redhat.com/show_bug.cgi?id=432903) Without this option set, I find I get "Unable to create native thread" errors under any heavy MR load. Clearly we to wrap this stuff so that it won't cause problems on non-RHEL6 operating systems. > Should set MALLOC_ARENA_MAX in hadoop-env.sh > -------------------------------------------- > > Key: HADOOP-7154 > URL: https://issues.apache.org/jira/browse/HADOOP-7154 > Project: Hadoop Common > Issue Type: Improvement > Components: scripts > Affects Versions: 0.22.0 > Reporter: Todd Lipcon > Assignee: Todd Lipcon > Priority: Minor > Fix For: 0.22.0 > > > New versions of glibc present in RHEL6 include a new arena allocator design. > In several clusters we've seen this new allocator cause huge amounts of > virtual memory to be used, since when multiple threads perform allocations, > they each get their own memory arena. On a 64-bit system, these arenas are > 64M mappings, and the maximum number of arenas is 8 times the number of > cores. We've observed a DN process using 14GB of vmem for only 300M of > resident set. This causes all kinds of nasty issues for obvious reasons. > Setting MALLOC_ARENA_MAX to a low number will restrict the number of memory > arenas and bound the virtual memory, with no noticeable downside in > performance - we've been recommending MALLOC_ARENA_MAX=4. We should set this > in hadoop-env.sh to avoid this issue as RHEL6 becomes more and more common. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira