[
https://issues.apache.org/jira/browse/HBASE-17819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16404562#comment-16404562
]
Anoop Sam John commented on HBASE-17819:
----------------------------------------
Latest patch attached. This does not try to remove the SkipList which is used
to remove blocks by HFile name.
This patch mainly handle the new 2 fields added for off heaping purpose. The
ref count and a boolean. Both together accounts to 21 bytes per entry. We
have BucketEntry class with out these states and after align takes 48 bytes per
instance. That will be used by file based BC. For those we dont need them as
there is no shared memory direct access. For off heap BC, we have a new
UnsafeSharedMemoryBucketEntry which uses directly int primitive and not Atomic
version. The atomic logic been handled within this class. This makes us to get
the same 48 bytes (After align) per entry size. This is based on Unsafe
access. When Unsafe is unavilable, we have a another extension for
BucketEntry which uses AtomicInteger directly. That class will have 64 bytes
as per object size.
Will discuss abt the possible removal of 'blockByHFile' in another issue. This
takes 40 bytes per entry in BC!
> Reduce the heap overhead for BucketCache
> ----------------------------------------
>
> Key: HBASE-17819
> URL: https://issues.apache.org/jira/browse/HBASE-17819
> Project: HBase
> Issue Type: Sub-task
> Components: BucketCache
> Reporter: Anoop Sam John
> Assignee: Anoop Sam John
> Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-17819_V1.patch, HBASE-17819_V2.patch,
> HBASE-17819_V3.patch, HBASE-17819_new_V1.patch
>
>
> We keep Bucket entry map in BucketCache. Below is the math for heapSize for
> the key , value into this map.
> BlockCacheKey
> ---------------
> String hfileName - Ref - 4
> long offset - 8
> BlockType blockType - Ref - 4
> boolean isPrimaryReplicaBlock - 1
> Total = align(12 (Object) + 17 )= 32
> BucketEntry
> ------------
> int offsetBase - 4
> int length - 4
> byte offset1 - 1
> byte deserialiserIndex - 1
> long accessCounter - 8
> BlockPriority priority - Ref - 4
> volatile boolean markedForEvict - 1
> AtomicInteger refCount - 16 + 4
> long cachedTime - 8
> Total = align(12 (Object) + 51) = 64
> ConcurrentHashMap Map.Entry - 40
> blocksByHFile ConcurrentSkipListSet Entry - 40
> Total = 32 + 64 + 80 = 176
> For 10 million blocks we will end up having 1.6GB of heap size.
> This jira aims to reduce this as much as possible
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
