[
https://issues.apache.org/jira/browse/HBASE-7404?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13878249#comment-13878249
]
Lars Hofhansl edited comment on HBASE-7404 at 1/22/14 4:48 AM:
---------------------------------------------------------------
I'm not opposed backporting this if there is interest. I'm wrestling with large
memory machines myself currently. This is in 0.96 and later already; and this
is almost completely new code with very little risk to existing functionality.
[~xieliang007], is your patch the same as the one posted here by [~davelatham]?
If not, mind adding refreshed patch?
If you had some performance numbers that would be great too.
I'd also be curious how this fares performance wise against just using the OS
buffer cache. I.e. how does serialization from the OS cache compare to
serialization from the bucket cache.
And - to preempt any comments to that extend - I realize that the bucket cache
provides more flexibility than the OS buffer cache, which indiscriminately
caches blocks (unless we use fadvise hints, etc)
was (Author: lhofhansl):
I'm not opposed backporting this if there is interest. I'm wrestling with large
memory machines myself currently. This is in 0.96 and later already; and this
is almost completely new code with very little risk to existing functionality.
[~xieliang007], is your patch the same as the posted here by [~davelatham]? If
not, mind add refreshed patch?
If you had some performance numbers that would be great too.
I'd also be curious how this fares performance wise against just using the OS
buffer cache. I.e. how does serialization from the OS cache compare the
serialization from the bucket cache.
And - to preempt any comments to that extend - I realize that the bucket cache
provides more flexibility than the OS buffer cache, which indiscriminately
caches blocks (unless we use fadvise hints, etc)
> Bucket Cache:A solution about CMS,Heap Fragment and Big Cache on HBASE
> ----------------------------------------------------------------------
>
> Key: HBASE-7404
> URL: https://issues.apache.org/jira/browse/HBASE-7404
> Project: HBase
> Issue Type: New Feature
> Affects Versions: 0.94.3
> Reporter: chunhui shen
> Assignee: chunhui shen
> Fix For: 0.95.0
>
> Attachments: 7404-0.94-fixed-lines.txt, 7404-trunk-v10.patch,
> 7404-trunk-v11.patch, 7404-trunk-v12.patch, 7404-trunk-v13.patch,
> 7404-trunk-v13.txt, 7404-trunk-v14.patch, BucketCache.pdf,
> HBASE-7404-backport-0.94.patch, Introduction of Bucket Cache.pdf,
> hbase-7404-94v2.patch, hbase-7404-trunkv2.patch, hbase-7404-trunkv9.patch
>
>
> First, thanks @neil from Fusion-IO share the source code.
> Usage:
> 1.Use bucket cache as main memory cache, configured as the following:
> –"hbase.bucketcache.ioengine" "heap"
> –"hbase.bucketcache.size" 0.4 (size for bucket cache, 0.4 is a percentage of
> max heap size)
> 2.Use bucket cache as a secondary cache, configured as the following:
> –"hbase.bucketcache.ioengine" "file:/disk1/hbase/cache.data"(The file path
> where to store the block data)
> –"hbase.bucketcache.size" 1024 (size for bucket cache, unit is MB, so 1024
> means 1GB)
> –"hbase.bucketcache.combinedcache.enabled" false (default value being true)
> See more configurations from org.apache.hadoop.hbase.io.hfile.CacheConfig and
> org.apache.hadoop.hbase.io.hfile.bucket.BucketCache
> What's Bucket Cache?
> It could greatly decrease CMS and heap fragment by GC
> It support a large cache space for High Read Performance by using high speed
> disk like Fusion-io
> 1.An implementation of block cache like LruBlockCache
> 2.Self manage blocks' storage position through Bucket Allocator
> 3.The cached blocks could be stored in the memory or file system
> 4.Bucket Cache could be used as a mainly block cache(see CombinedBlockCache),
> combined with LruBlockCache to decrease CMS and fragment by GC.
> 5.BucketCache also could be used as a secondary cache(e.g. using Fusionio to
> store block) to enlarge cache space
> How about SlabCache?
> We have studied and test SlabCache first, but the result is bad, because:
> 1.SlabCache use SingleSizeCache, its use ratio of memory is low because kinds
> of block size, especially using DataBlockEncoding
> 2.SlabCache is uesd in DoubleBlockCache, block is cached both in SlabCache
> and LruBlockCache, put the block to LruBlockCache again if hit in SlabCache ,
> it causes CMS and heap fragment don't get any better
> 3.Direct heap performance is not good as heap, and maybe cause OOM, so we
> recommend using "heap" engine
> See more in the attachment and in the patch
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
