thanks, J.D.
I think I know why the regionserver takes so much memory now,
there are some really big row in my table, 1.2-1.5 GB in size. seems that
the regionserver sometime will try to load the whole region into memory, I
don't know when this will happen, maybe when it does a major compaction or
reassign the region to other regionserver or when it's asked to open/online
a region?.
you question is answered in line.
On Sat, Mar 6, 2010 at 2:15 AM, Jean-Daniel Cryans <[email protected]>wrote:
> On Thu, Mar 4, 2010 at 7:19 PM, steven zhuang
> <[email protected]> wrote:
> > thanks, J.D.
> >
> > I am still not sure about the second question, from the log
> I
> > can see lines like:
> > *org.apache.hadoop.hbase.regionserver.Store: loaded
> > /user/ccenterq/hbase/XXX/1702600912/queries/1289015788537930719,
> > isReference=false, sequence id=1389720128, length=**175533391**,
> > majorCompaction=true (this is the region data, not the index, right?)*
> > I do have some region really big, with millions of columns
> in
> > one column family, but isn't this length a little too big.
>
> The index and the metadata of the files of that Store in that region
> was loaded here.
>
> >
> >
> > About the third one, I am actually not very clear of how memory is
> > used in Hbase, if it's only the few KBs by holding region info, it won't
> > release right?
>
> I don't understand your question. Try an example?
sorry for not be clear, actually I am asking which part of the region has a
length of "175533391" in the following line, I think the index/meta-data
info for a region won't take so much memory.
> *org.apache.hadoop.hbase.regionserver.Store: loaded
> /user/ccenterq/hbase/XXX/1702600912/queries/1289015788537930719,
> isReference=false, sequence id=1389720128, length=**175533391**,
> majorCompaction=true (this is the region data, not the index, right?)*
>
> > Since we adding more regionserver into our cluster, the chance of
> > OOME is much less, but still there are some(we have only 1GB heap for the
> > regoinserver), I wonder which part of Hbase eats most of the heap
> > memory? The region data takes most, right?
>
> We recommend using at least 4GB on a production system. So what
> occupies data is:
>
> - The memstores (we have a global limit per region server).
> - The block cache (global limit too)
> - All the objects used to manage the regions
>
> The block cache is an LRU, so once blocks of data are evicted they
> occupy memory until garbage collected.
>
> > I think there should be some rule to cache/write-back the region
> > data, but now it seems that the Hbase doesn't detect the Memory usage
> status
> > thus cannot release the memory in time.
>
> We do in fact. If you grow your block cache too much, it will start
> evicting even more. If you get to the limit of memstore size, hbase
> will force flush them to free space... then that's all garbage
> collected by java.
>
> >
> >
> > On Fri, Mar 5, 2010 at 10:00 AM, Jean-Daniel Cryans <[email protected]
> >wrote:
> >
> >> Inline.
> >>
> >> J-D
> >>
> >> On Thu, Mar 4, 2010 at 5:53 PM, steven zhuang
> >> <[email protected]> wrote:
> >> > greetings,
> >> >
> >> > I have a cluster which contains a dozens of PCs with
> 4cores
> >> and
> >> > 4GB ram each, at first there were only 5 regionservers in the cluster,
> >> now
> >> > we have added 18 regionservers into the cluseter.
> >> > But after reading the log, I have some questions, hope you
> >> guys
> >> > can help me out.
> >> >
> >> > 1. the cluster will reassign the regions around old
> >> > regionservers and newly added regionservers because of auto-balancing,
> >> > right?
> >>
> >> Yep. It's a bit intense and we plan on making it less disruptive.
> >>
> >> >
> >> > 2. I saw a lot of regions OPEN messages, does this means
> the
> >> > regionserver will load the whole regoin into memory or just portion of
> >> it,
> >> > or just the index(?I don't know what is in the index data)?
> >>
> >> Just the region information and files indexes.
> >>
> >> >
> >> > 3. will the memory be released after the loaded region
> opened
> >> is
> >> > not needed anymore, Coz I don't see anything from the log.
> >>
> >> Which memory? The few KBs used to store the region's information? Or
> >> are you talking about the memory taken by regions that moved to other
> >> region servers? In this case it's really just the java garbage
> >> collector that takes care of it.
> >>
> >
>
