Re: 2 bucket caches?
I think you may want to think a bit about this… How far do you want to go with your memory management? 'Off heap' is a new nifty way of saying application level swap and memory management. So what you are basically saying is that I have memory, local persistence, then HDFS persistence. And your local persistence could be anything… (PCIe based flash, UltraDIMMs, RRAM (when it hits the market), SSDs, even raided spinning rust… ) If you’re going in that direction, what is tachyon doing? If you want to do this… and I’m not saying its a bad idea, you’ll want to think a bit more generic. Essentially its a layered hiearchy (memory, p1, p2, …) where p(n) is a pool of devices which have a set of rules on how to propagate pages up or down the hierarchy. On Jun 29, 2015, at 1:20 AM, Jean-Marc Spaggiari jean-m...@spaggiari.org wrote: Hi, Is it possible to have 2 bucket cache on a single region server? Like L2 and L3? I would like to have L2 offheap and block evicted from L2 going into L3 on SSD. So we already have something like that? Or should I open a JIRA? hbase.bucketcache.ioengine can get only one value. Might be nice to have a flume-like approach... hbase.bucketcache=myoffheap,myssddrive hbase.bucketcache.myoffheap.ioengine=offheap hbase.bucketcache.myssddrive.ioengine=file://my_ssd_mnt/there And keep the order specified in hbase.bucketcache, so myoffheap=L2, myssddrive=L3, etc.? Thanks, JM The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental. Use at your own risk. Michael Segel michael_segel (AT) hotmail.com
Re: 2 bucket caches?
Hi Michael, All what you said is exactly what I have in mind. Being able to have a layered hierarchy with different storage engines ((Flash, SSD, Memory, etc.) that you can configure. It's not just offheap, but anything which can offload the BlockCache and which is still faster than going to the spinning drives. JM 2015-06-29 10:43 GMT-04:00 Michael Segel michael_se...@hotmail.com: I think you may want to think a bit about this… How far do you want to go with your memory management? 'Off heap' is a new nifty way of saying application level swap and memory management. So what you are basically saying is that I have memory, local persistence, then HDFS persistence. And your local persistence could be anything… (PCIe based flash, UltraDIMMs, RRAM (when it hits the market), SSDs, even raided spinning rust… ) If you’re going in that direction, what is tachyon doing? If you want to do this… and I’m not saying its a bad idea, you’ll want to think a bit more generic. Essentially its a layered hiearchy (memory, p1, p2, …) where p(n) is a pool of devices which have a set of rules on how to propagate pages up or down the hierarchy. On Jun 29, 2015, at 1:20 AM, Jean-Marc Spaggiari jean-m...@spaggiari.org wrote: Hi, Is it possible to have 2 bucket cache on a single region server? Like L2 and L3? I would like to have L2 offheap and block evicted from L2 going into L3 on SSD. So we already have something like that? Or should I open a JIRA? hbase.bucketcache.ioengine can get only one value. Might be nice to have a flume-like approach... hbase.bucketcache=myoffheap,myssddrive hbase.bucketcache.myoffheap.ioengine=offheap hbase.bucketcache.myssddrive.ioengine=file://my_ssd_mnt/there And keep the order specified in hbase.bucketcache, so myoffheap=L2, myssddrive=L3, etc.? Thanks, JM The opinions expressed here are mine, while they may reflect a cognitive thought, that is purely accidental. Use at your own risk. Michael Segel michael_segel (AT) hotmail.com
Re: 2 bucket caches?
Hi J-M, N-leveled caching is something I've discussed with some folks but it hasn't been done. We already have multi-cache management strategies, such as CombinedBlockCache, so this would be making them more generic and exposing through configuration. Something you'd be interested in taking on? Dunno if you noticed, but we now support caching blocks out into memcached as well. -n On Sun, Jun 28, 2015 at 11:20 PM, Jean-Marc Spaggiari jean-m...@spaggiari.org wrote: Hi, Is it possible to have 2 bucket cache on a single region server? Like L2 and L3? I would like to have L2 offheap and block evicted from L2 going into L3 on SSD. So we already have something like that? Or should I open a JIRA? hbase.bucketcache.ioengine can get only one value. Might be nice to have a flume-like approach... hbase.bucketcache=myoffheap,myssddrive hbase.bucketcache.myoffheap.ioengine=offheap hbase.bucketcache.myssddrive.ioengine=file://my_ssd_mnt/there And keep the order specified in hbase.bucketcache, so myoffheap=L2, myssddrive=L3, etc.? Thanks, JM
2 bucket caches?
Hi, Is it possible to have 2 bucket cache on a single region server? Like L2 and L3? I would like to have L2 offheap and block evicted from L2 going into L3 on SSD. So we already have something like that? Or should I open a JIRA? hbase.bucketcache.ioengine can get only one value. Might be nice to have a flume-like approach... hbase.bucketcache=myoffheap,myssddrive hbase.bucketcache.myoffheap.ioengine=offheap hbase.bucketcache.myssddrive.ioengine=file://my_ssd_mnt/there And keep the order specified in hbase.bucketcache, so myoffheap=L2, myssddrive=L3, etc.? Thanks, JM
