Hi One small observation after giving +1 on the RC. The WAL compression feature causes OOME and causes Full GC.
The problem is, if we have 1500 regions and I need to create recovered.edits for each of the region (I dont have much data in the regions (~300MB)). Now when I try to build the dictionary there is a Node object getting created. Each node object occupies 32 bytes. We have 5 such dictionaries. Initially we create indexToNodes array and its size is 32767. So now we have 32*5*32767 = ~5MB. Now I have 1500 regions. So 5MB*1500 = ~7GB.(Excluding actual data). This seems to a very high initial memory foot print and this never allows me to split the logs and I am not able to make the cluster up at all. Our configured heap size was 8GB, tested in 3 node cluster with 5000 regions, very less data( 1GB in hdfs cluster including replication), some small data is spread evenly across all regions. The formula is 32(Node object size)*5(No of dictionary)*32767(no of node objects)*noofregions. I think this initial memory needs to be documented (documentation should do for now)or has to be fixed with some workarounds. So pls give your thoughts on this. Regards Ram > -----Original Message----- > From: Ramkrishna.S.Vasudevan [mailto:ramkrishna.vasude...@huawei.com] > Sent: Monday, May 14, 2012 11:48 AM > To: dev@hbase.apache.org; 'lars hofhansl' > Subject: RE: ANN: The third hbase 0.94.0 release candidate is available > for download > > Hi > We (it includes the test team here) 0.94 RC and carried out various > operations on it. > Puts, Scans, and all the restart scenarios (using kill -9 also). Even > the > encoding stuffs were tested and carried out our basic test scenarios. > Seems > to work fine. > > Did not test rolling restart with 0.92. By this week we may try to do > some > performance comparison with 0.92. > Also Lars and I agreed for a point release too. > So I am +1 on the RC. > > Regards > Ram > > > -----Original Message----- > > From: lars hofhansl [mailto:lhofha...@yahoo.com] > > Sent: Sunday, May 13, 2012 10:53 PM > > To: dev@hbase.apache.org > > Subject: Re: ANN: The third hbase 0.94.0 release candidate is > available > > for download > > > > OK, I'll change my tactic :) > > > > If there are no -1's by Wed, May 16th, I'll release RC4 as 0.94.0. > > > > -- Lars > > > > > > > > ----- Original Message ----- > > From: Stack <st...@duboce.net> > > To: lars hofhansl <lhofha...@yahoo.com> > > Cc: "dev@hbase.apache.org" <dev@hbase.apache.org> > > Sent: Friday, May 11, 2012 10:39 PM > > Subject: Re: ANN: The third hbase 0.94.0 release candidate is > available > > for download > > > > On Fri, May 11, 2012 at 10:26 PM, lars hofhansl <lhofha...@yahoo.com> > > wrote: > > > Thanks Stack. > > > > > > So that's two +1 (mine doesn't count I guess). And no -1. > > > > Why doesn't yours count? Usually the RMs does, if they +1 it. So, > > that'd be 3x+1 + a non-binding +1. > > > > > I talked to Ram offline, and we'll fix HBase with Hadoop 2.0.0 in a > > 0.94 point release. > > > > > > I would like to see a few more +1's before I declare this the > > official 0.94.0 release. > > > > > > > You might be waiting a while (smile). Fellas seem to be busy... > > > > Good on you Lars, > > St.Ack
