Once the JVM consumes memory it generally will not release it back to the OS (so you will see it near-full).
How much heap are you giving the RS? The issue you're having seems rather odd... Nothing at all changes between the different insert runs? Instantiating a new HTable should not be creating a /hbase node. On Tue, September 1, 2009 8:37 pm, [email protected] wrote: > Hi, > > > I guess Creation of /hbase happen after running table1 = new > HTable("MyTable"); > Right, that warning showed up in the client.There doesn't keep any log in > that region server. When ever I finish to put my data, I would count it by > shell. I put the first 10 files(10M) in five threads. It worked pretty > well. After taking a minutes break, I put next 30 files.It still worked > very quickly. When I count it, I found it became very very slow. > By checking that region server,I found that region server taken so much > RAM(almost on the top), > and seems not to release it. Maybe I should lower the flush threshold.(how > to do it ? ^_^) If I don't take a break and continue to put data, that > warning raised in the client. So, I wonder that region server is too busy > to serve the following client request. If a client can't reach that busy > sever, it would be better to raise some exception that I can catch and try > to handle it. Thanks > > > Fleming > > > > > > > Jonathan Gray > <[email protected] To: [email protected] > m> cc: (bcc: Y_823910/TSMC) Subject: Re: Save > data to HBase 2009/09/02 10:22 > AM > Please respond to > hbase-user > > > > > > > There's no reason to use the write buffer if you are only inserting one > row and then immediately flushing. This code does the exact same thing as > yours: > > > table1 = new HTable("MyTable"); Put p = new Put(...); > p.add(...); table1.put(p); > > That's it. You would only use the write-buffering if you wanted to send > multiple Puts at once. > > As far as the table only having one region, yes this limits all work on > that table to a single node. However with 200MB of data, you are smaller > than the default region size of 256MB. You can lower that split threshold > or you can manually split in the shell or using HBaseAdmin. > > Not sure about that ZooKeeper warning... Creation of /hbase should > happen in the master at boot-time, not in a client running inserts. Do you > have anymore detail? That warning showed up in the client? Anything in > the regionserver? Can you use the shell? > > JG > > > [email protected] wrote: >> Hi, >> >> >> I want to put only 200MB data to a HBase table. >> Just use a single thread to put 1MB data in a time. >> So, there are 200 job in turn. >> It occured WARN zookeeper.ZooKeeperWrapper: Failed to create /hbase, >> while it run to around the 20th job. I found that table has just one >> regin. I wonder whether these tight job will make that region server >> choke. About the region split, I want to know when the table's region >> will be splitted. Is its threshold setting in hbase-default.xml? >> Following are a snippet of my code about putting my data to hbase, >> > anything >> can improve it? >> >> table1 = new HTable("MyTable"); table1.setAutoFlush(false); >> table1.setWriteBufferSize(1024*1024*12); >> >> Put p = new Put(Bytes.toBytes(rs.getString(KeyColumn))); >> p.add(Bytes.toBytes(FamColumn), >> Bytes.toBytes("content"),Bytes.toBytes(json)); >> table1.put(p); >> >> table1.flushCommits(); table1.close(); >> >> My cluster is only 3 PC with (1G,1G,2G RAM ,2 core CPU ) >> 3 region server >> 3 ZK quorum server >> We'll enlarge our cluster to 12PCs with 8G RAM. >> >> >> Thanks >> >> >> Fleming >> >> >> >> >> > >> Amandeep Khurana >> > >> <[email protected] To: >> > [email protected] > >>> cc: (bcc: >>> > Y_823910/TSMC) > > >> Subject: Re: Save data to >> > HBase > > >> 2009/09/02 03:07 >> > >> AM >> > >> Please respond to >> > >> hbase-user > >> > >> > >> >> >> >> >> What do you mean by quick way? You can use the api to put data into it >> through a standalone java program or you can use a MR job to do it.. >> >> >> On Tue, Sep 1, 2009 at 2:48 AM, <[email protected]> wrote: >> >> >>> Hi there, >>> >>> >>> Any quick way that I can save my data(12G) to HBase? >>> Thanks >>> >>> >>> Fleming >>> >>> >> >>> >>> >>> >> >> >> >> >> >> > ------------------------------------------------------------------------- > -- > > >> TSMC PROPERTY >> > >> This email communication (and any attachments) is proprietary >> > information >> for the sole use of its > >> intended recipient. Any unauthorized review, use or distribution by > anyone >> other than the intended > >> recipient is strictly prohibited. If you are not the intended > recipient, >> please notify the sender by > >> replying to this email, and then delete this email and any copies of it >> > >> immediately. Thank you. > >> > ------------------------------------------------------------------------- > -- > > >> >> >> > > > > > ------------------------------------------------------------------------- > -- > TSMC PROPERTY > This email communication (and any attachments) is proprietary information > for the sole use of its intended recipient. Any unauthorized review, use or > distribution by anyone other than the intended recipient is strictly > prohibited. If you are not the intended recipient, please notify the > sender by replying to this email, and then delete this email and any > copies of it immediately. Thank you. > ------------------------------------------------------------------------- > -- > > > > >
