I think you might be misunderstanding what the 'memcache' is, we are calling it 'memstore' now. It is a write buffer, not a cache. It is also memory sensitive, so as you insert more data, hbase will flush the 'memcache' to HDFS. By default memcache is limited to 64MB a store, 40% of Xmx, and we also limit how many outstanding write ahead logs there are, so as these fill up, we flush. The strict limiting factor is the index size. You can control that with block size and compression.
To answer some other questions: 1. The ram used in indexes is the primary limiting factor of how much a regionserver can serve. 2. A region under no write load only occupies the index ram. Only regions actually under write load will have memcache size > 0. This depends on your application. If your application doesn't use random reads, you might consider storing data straight up in HDFS. If you want fast random reads to any part of the dataset, then HBase is your thing. On Mon, Jul 20, 2009 at 12:01 AM, zsongbo<[email protected]> wrote: > Ryan, > > I know we can store more than 250GB in one region ssrver. But how about 3TB, > even 10TB. > Except for the memory usage by indexs, there may have other factors, such as > the memcache. > If there are 5000 regions opened, the total memcache heap will be very > large. > > So, I am thinking two: > 1. What is the key factor of system resource usage in HBase? > 2. Is there any way to make the majority of regions to be static (do not use > too much memory and CPU), and only few regions are active to serve the data > ingesting. > > The above considerations are based on following situations: > 1. Usually, we are processing and managing time-serial dataset. The old data > (e.g. the data of one week or one day ago) can be static and rarely be > queried. > 2. The application of the dataset may be ingesting heavy. But only a few > client to query. > 3. The total dataset is very big (500TB un-compressed), but we cannot run a > 2000-nodes cluster to hold this dataset. Large cluster is so expensive, and > we can only run a cluster with noly 20-50 nodes. > > Schubert > > > On Fri, Jul 10, 2009 at 3:35 PM, Ryan Rawson <[email protected]> wrote: > >> By dedicating more ram to the situation you can achieve more regions >> under a single regionserver. I have noticed that in my own region >> servers, 200-600MB = 1-2MB of index. This value, however, is >> dependent on the size of your keys and values. I have very small keys >> and values. You can also tune the index size with the blocksize >> setting, which directly impacts several things: >> - larger blocks mean less index entries (more efficient ram use). >> - larger blocks mean larger IO units. Can impact random read perf (reduces >> it) >> >> If you have larger data sizes, you can end up with essentially 1 index >> entry per thing (eg: if each thing is 5m, you end up with 1 entry per >> 5m of data). >> >> So depends on the size of your data, some tuning factors, etc. >> >> Putting this all together means, 1 regionserver can most certainly be >> in charge of more than 250GB of stored data. >> >> -ryan >> >> On Fri, Jul 10, 2009 at 12:23 AM, zsongbo<[email protected]> wrote: >> > Ryan, >> > >> > Yes. you are right. >> > But my question is that, even through 1000 regions (250MB)) per >> > regionserver, each regionserver can only support 250GB storage. >> > >> > Please also check this thread "Help needed - Adding HBase to >> architecture", >> > Stack and Andrew have put some talk there. >> > >> > Schubert >> > >> > On Fri, Jul 10, 2009 at 12:55 PM, Ryan Rawson <[email protected]> >> wrote: >> > >> >> That size is not memory-resident, so the total data size is not an >> >> issue. The index size is what limits you with RAM, and its about 1 MB >> >> per region (256MB region). >> >> >> >> -ryan >> >> >> >> On Thu, Jul 9, 2009 at 9:51 PM, zsongbo<[email protected]> wrote: >> >> > Hi Ryan, >> >> > >> >> > Thanks. >> >> > >> >> > If your regionsize is about 250MB, than 400 regions can store 100GB >> data >> >> on >> >> > each regionserver. >> >> > Now, if you have 100TB data, then you need 1000 regionservers. >> >> > We are not google or yahoo who have so many nodes. >> >> > >> >> > Schubert >> >> > >> >> > On Fri, Jul 10, 2009 at 12:29 PM, Ryan Rawson <[email protected]> >> >> wrote: >> >> > >> >> >> re: #2: in fact we don't know that... I know that I ran run 200-400 >> >> >> regions on a regionserver with a heap size of 4-5gb. More even. I >> >> >> bet I could have 1000 regions open on 4gb ram. Each region is ~ 1mb >> >> >> of all the time data, so there we go. >> >> >> >> >> >> As for compactions, they are fairly fast, 0-30s or so depending on a >> >> >> number of factors. Practically speaking it has not been a problem >> for >> >> >> me, and I've put 1200 gb into hbase so far. >> >> >> >> >> >> On Thu, Jul 9, 2009 at 8:58 PM, zsongbo<[email protected]> wrote: >> >> >> > Hi all, >> >> >> > >> >> >> > 1. In this configuration property: >> >> >> > >> >> >> > <property> >> >> >> > <name>hbase.hstore.compactionThreshold</name> >> >> >> > <value>3</value> >> >> >> > <description> >> >> >> > If more than this number of HStoreFiles in any one HStore >> >> >> > (one HStoreFile is written per flush of memcache) then a >> compaction >> >> >> > is run to rewrite all HStoreFiles files as one. Larger numbers >> >> >> > put off compaction but when it runs, it takes longer to >> complete. >> >> >> > During a compaction, updates cannot be flushed to disk. Long >> >> >> > compactions require memory sufficient to carry the logging of >> >> >> > all updates across the duration of the compaction. >> >> >> > If too large, clients timeout during compaction. >> >> >> > </description> >> >> >> > </property> >> >> >> > >> >> >> > >> >> >> > That says "During a compaction, updates cannot be flushed to disk." >> >> >> > Does it mean that, when compaction, the memcache cannot be flushed >> to >> >> >> disk? >> >> >> > I think it is not good. >> >> >> > >> >> >> > 2. We know that HBase cannot serve too many regions on each >> >> regionserver. >> >> >> If >> >> >> > only 200 regions(256MB), only 50GB storage can be used. >> >> >> > I my tested whith have 1.5GB heap and 256MB regionsize, each >> >> regionserver >> >> >> > can support 150 regions, and then OutOfMem. >> >> >> > Can anybody explain more detail here of the reason? >> >> >> > >> >> >> > To use more storage, can I set larger regionsize? such as 1GB, >> 10GB? >> >> >> > I have worry about the compaction time would be long with so large >> >> >> regions. >> >> >> > >> >> >> > Schubert >> >> >> > >> >> >> >> >> > >> >> >> > >> >
