[ https://issues.apache.org/jira/browse/HDFS-15907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17307804#comment-17307804 ]
Stephen O'Donnell commented on HDFS-15907: ------------------------------------------ Thanks for committing [~ayushtkn]. The issue reported in HDFS-15792 is only relevant when parallel image loading is enabled. This was delivered in HDFS-14617, which is only on 3.3, trunk and 2.10. Therefore I think its fine that HDFS-15792 and this one is only on branch-3.3. The branch-2.10 version of HDFS-15792 is different and doesn't use AtomicInt for Java 7 compatibility reasons. > Reduce Memory Overhead of AclFeature by avoiding AtomicInteger > -------------------------------------------------------------- > > Key: HDFS-15907 > URL: https://issues.apache.org/jira/browse/HDFS-15907 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode > Reporter: Stephen O'Donnell > Assignee: Stephen O'Donnell > Priority: Major > Fix For: 3.3.1, 3.4.0 > > Attachments: HDFS-15907.001.patch > > > In HDFS-15792 we made some changes to the AclFeature and ReferenceCountedMap > classes to address a rare bug when loading the FSImage in parallel. > One change we made was to replace an int inside AclFeature with an > AtomicInteger to avoid synchronising the methods in AclFeature. > Discussing this change with [~weichiu], he pointed out that while the > AclFeature cache is intended to reduce the count of AclFeature objects, on a > large cluster, it is possible for there to be many millions of AclFeature > objects. > Previously, the int will have taken 4 bytes of heap. > By moving to a AtomicInteger, we probably have an overhead of: > 4 bytes (or 8 if the heap is over 32GB) for a reference to the atomic long > object > 12 byte overhead for the java object > 4 bytes inside the atomic long to store an int. > > So the total heap overhead has gone from 4 bytes to 20 bytes just to use an > AtomicInteger. > Therefore I think it makes sense to remove the AtomicInteger and just > synchronise the methods of AclFeature where the value is incremented / > decremented / retrieved. -- This message was sent by Atlassian Jira (v8.3.4#803005) --------------------------------------------------------------------- To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org