I think you might be misunderstanding what the 'memcache' is, we are
calling it 'memstore' now. It is a write buffer, not a cache. It is
also memory sensitive, so as you insert more data, hbase will flush
the 'memcache' to HDFS. By default memcache is limited to 64MB a
store, 40% of Xmx, and we also limit how many outstanding write ahead
logs there are, so as these fill up, we flush. The strict limiting
factor is the index size. You can control that with block size and
compression.
To answer some other questions:
1. The ram used in indexes is the primary limiting factor of how much
a regionserver can serve.
2. A region under no write load only occupies the index ram. Only
regions actually under write load will have memcache size > 0. This
depends on your application.
If your application doesn't use random reads, you might consider
storing data straight up in HDFS. If you want fast random reads to
any part of the dataset, then HBase is your thing.
On Mon, Jul 20, 2009 at 12:01 AM, zsongbo<[email protected]> wrote:
Ryan,
I know we can store more than 250GB in one region ssrver. But how about
3TB,
even 10TB.
Except for the memory usage by indexs, there may have other factors, such
as
the memcache.
If there are 5000 regions opened, the total memcache heap will be very
large.
So, I am thinking two:
1. What is the key factor of system resource usage in HBase?
2. Is there any way to make the majority of regions to be static (do not
use
too much memory and CPU), and only few regions are active to serve the
data
ingesting.
The above considerations are based on following situations:
1. Usually, we are processing and managing time-serial dataset. The old
data
(e.g. the data of one week or one day ago) can be static and rarely be
queried.
2. The application of the dataset may be ingesting heavy. But only a few
client to query.
3. The total dataset is very big (500TB un-compressed), but we cannot run
a
2000-nodes cluster to hold this dataset. Large cluster is so expensive,
and
we can only run a cluster with noly 20-50 nodes.
Schubert
On Fri, Jul 10, 2009 at 3:35 PM, Ryan Rawson <[email protected]> wrote:
By dedicating more ram to the situation you can achieve more regions
under a single regionserver. I have noticed that in my own region
servers, 200-600MB = 1-2MB of index. This value, however, is
dependent on the size of your keys and values. I have very small keys
and values. You can also tune the index size with the blocksize
setting, which directly impacts several things:
- larger blocks mean less index entries (more efficient ram use).
- larger blocks mean larger IO units. Can impact random read perf
(reduces
it)
If you have larger data sizes, you can end up with essentially 1 index
entry per thing (eg: if each thing is 5m, you end up with 1 entry per
5m of data).
So depends on the size of your data, some tuning factors, etc.
Putting this all together means, 1 regionserver can most certainly be
in charge of more than 250GB of stored data.
-ryan
On Fri, Jul 10, 2009 at 12:23 AM, zsongbo<[email protected]> wrote:
Ryan,
Yes. you are right.
But my question is that, even through 1000 regions (250MB)) per
regionserver, each regionserver can only support 250GB storage.
Please also check this thread "Help needed - Adding HBase to
architecture",
Stack and Andrew have put some talk there.
Schubert
On Fri, Jul 10, 2009 at 12:55 PM, Ryan Rawson <[email protected]>
wrote:
That size is not memory-resident, so the total data size is not an
issue. The index size is what limits you with RAM, and its about 1
MB
per region (256MB region).
-ryan
On Thu, Jul 9, 2009 at 9:51 PM, zsongbo<[email protected]> wrote:
Hi Ryan,
Thanks.
If your regionsize is about 250MB, than 400 regions can store 100GB
data
on
each regionserver.
Now, if you have 100TB data, then you need 1000 regionservers.
We are not google or yahoo who have so many nodes.
Schubert
On Fri, Jul 10, 2009 at 12:29 PM, Ryan Rawson <[email protected]>
wrote:
re: #2: in fact we don't know that... I know that I ran run
200-400
regions on a regionserver with a heap size of 4-5gb. More even.
I
bet I could have 1000 regions open on 4gb ram. Each region is ~
1mb
of all the time data, so there we go.
As for compactions, they are fairly fast, 0-30s or so depending on
a
number of factors. Practically speaking it has not been a problem
for
me, and I've put 1200 gb into hbase so far.
On Thu, Jul 9, 2009 at 8:58 PM, zsongbo<[email protected]> wrote:
Hi all,
1. In this configuration property:
<property>
<name>hbase.hstore.compactionThreshold</name>
<value>3</value>
<description>
If more than this number of HStoreFiles in any one HStore
(one HStoreFile is written per flush of memcache) then a
compaction
is run to rewrite all HStoreFiles files as one. Larger
numbers
put off compaction but when it runs, it takes longer to
complete.
During a compaction, updates cannot be flushed to disk. Long
compactions require memory sufficient to carry the logging of
all updates across the duration of the compaction.
If too large, clients timeout during compaction.
</description>
</property>
That says "During a compaction, updates cannot be flushed to
disk."
Does it mean that, when compaction, the memcache cannot be
flushed
to
disk?
I think it is not good.
2. We know that HBase cannot serve too many regions on each
regionserver.
If
only 200 regions(256MB), only 50GB storage can be used.
I my tested whith have 1.5GB heap and 256MB regionsize, each
regionserver
can support 150 regions, and then OutOfMem.
Can anybody explain more detail here of the reason?
To use more storage, can I set larger regionsize? such as 1GB,
10GB?
I have worry about the compaction time would be long with so
large
regions.
Schubert
