If you aren't using a recent version of HBase branch 0.20 you most likely ran out of available client connections because in HTable, when instantiated, we were creating a new ZooKeeperWrapper every time. So to fix it either update, try not building a HTable for each row or set the hbase.zookeeper.property.maxClientCnxns to something high like 100-200 (you have to reboot HBase after that and make sure all the Zookeeper nodes get the new config).
J-D On Wed, Sep 2, 2009 at 1:52 AM, <[email protected]> wrote: > Hi, > I just use 3 core2 pc as RS with (1G,1G,2G RAM). > Right! I didn't change anything in different insert runs. > These warn point to save2HBaseThread.putFRPF(save2HBaseThread.java:105) > Line 105 is the code table1 = new HTable("MyTable"); > It's still working after following WARN, but if I continue to do the > following jobs. > These warning raised randomly.If you continue to do jobs, it will fail to > put data. > I guess 200MB is not a large dataset,but it took me so much time to put it. > Any suggestion? > Thanks > > Fleming > > WARN zookeeper.ZooKeeperWrapper: Failed to create /hbase -- check quorum > servers, currently=10.42.253.86:2181,10.42.253.182:2181,10.42.254.37:2181 > org.apache.zookeeper.KeeperException$ConnectionLossException: > KeeperErrorCode = ConnectionLoss for /hbase > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:90) > at > org.apache.zookeeper.KeeperException.create(KeeperException.java:42) > at org.apache.zookeeper.ZooKeeper.create(ZooKeeper.java:522) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureExists(ZooKeeperWrapper.java:342) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.ensureParentExists(ZooKeeperWrapper.java:365) > at > org.apache.hadoop.hbase.zookeeper.ZooKeeperWrapper.checkOutOfSafeMode(ZooKeeperWrapper.java:478) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRootRegion(HConnectionManager.java:846) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:515) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:491) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:565) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:524) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:491) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegionInMeta(HConnectionManager.java:565) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:528) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.locateRegion(HConnectionManager.java:491) > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:123) > at org.apache.hadoop.hbase.client.HTable.<init>(HTable.java:78) > at save2HBaseThread.putFRPF(save2HBaseThread.java:105) > at save2HBaseThread.run(save2HBaseThread.java:66) > > > > > > "Jonathan Gray" > <[email protected] To: > [email protected] > m> cc: (bcc: Y_823910/TSMC) > Subject: Re: Save data to HBase > 2009/09/02 01:17 > PM > Please respond to > hbase-user > > > > > > > Once the JVM consumes memory it generally will not release it back to the > OS (so you will see it near-full). > > How much heap are you giving the RS? > > The issue you're having seems rather odd... Nothing at all changes between > the different insert runs? Instantiating a new HTable should not be > creating a /hbase node. > > On Tue, September 1, 2009 8:37 pm, [email protected] wrote: >> Hi, >> >> >> I guess Creation of /hbase happen after running table1 = new >> HTable("MyTable"); >> Right, that warning showed up in the client.There doesn't keep any log in >> that region server. When ever I finish to put my data, I would count it > by >> shell. I put the first 10 files(10M) in five threads. It worked pretty >> well. After taking a minutes break, I put next 30 files.It still worked >> very quickly. When I count it, I found it became very very slow. >> By checking that region server,I found that region server taken so much >> RAM(almost on the top), >> and seems not to release it. Maybe I should lower the flush > threshold.(how >> to do it ? ^_^) If I don't take a break and continue to put data, that >> warning raised in the client. So, I wonder that region server is too busy >> to serve the following client request. If a client can't reach that busy >> sever, it would be better to raise some exception that I can catch and > try >> to handle it. Thanks >> >> >> Fleming >> >> >> >> >> >> >> Jonathan Gray >> <[email protected] To: [email protected] >> m> cc: (bcc: Y_823910/TSMC) Subject: Re: Save >> data to HBase 2009/09/02 10:22 >> AM >> Please respond to >> hbase-user >> >> >> >> >> >> >> There's no reason to use the write buffer if you are only inserting one >> row and then immediately flushing. This code does the exact same thing > as >> yours: >> >> >> table1 = new HTable("MyTable"); Put p = new Put(...); >> p.add(...); table1.put(p); >> >> That's it. You would only use the write-buffering if you wanted to send >> multiple Puts at once. >> >> As far as the table only having one region, yes this limits all work on >> that table to a single node. However with 200MB of data, you are smaller >> than the default region size of 256MB. You can lower that split > threshold >> or you can manually split in the shell or using HBaseAdmin. >> >> Not sure about that ZooKeeper warning... Creation of /hbase should >> happen in the master at boot-time, not in a client running inserts. Do > you >> have anymore detail? That warning showed up in the client? Anything in >> the regionserver? Can you use the shell? >> >> JG >> >> >> [email protected] wrote: >>> Hi, >>> >>> >>> I want to put only 200MB data to a HBase table. >>> Just use a single thread to put 1MB data in a time. >>> So, there are 200 job in turn. >>> It occured WARN zookeeper.ZooKeeperWrapper: Failed to create /hbase, >>> while it run to around the 20th job. I found that table has just one >>> regin. I wonder whether these tight job will make that region server >>> choke. About the region split, I want to know when the table's region >>> will be splitted. Is its threshold setting in hbase-default.xml? >>> Following are a snippet of my code about putting my data to hbase, >>> >> anything >>> can improve it? >>> >>> table1 = new HTable("MyTable"); table1.setAutoFlush(false); >>> table1.setWriteBufferSize(1024*1024*12); >>> >>> Put p = new Put(Bytes.toBytes(rs.getString(KeyColumn))); >>> p.add(Bytes.toBytes(FamColumn), >>> Bytes.toBytes("content"),Bytes.toBytes(json)); >>> table1.put(p); >>> >>> table1.flushCommits(); table1.close(); >>> >>> My cluster is only 3 PC with (1G,1G,2G RAM ,2 core CPU ) >>> 3 region server >>> 3 ZK quorum server >>> We'll enlarge our cluster to 12PCs with 8G RAM. >>> >>> >>> Thanks >>> >>> >>> Fleming >>> >>> >>> >>> >>> >> >>> Amandeep Khurana >>> >> >>> <[email protected] To: >>> >> [email protected] >> >>>> cc: (bcc: >>>> >> Y_823910/TSMC) >> >> >>> Subject: Re: Save data to >>> >> HBase >> >> >>> 2009/09/02 03:07 >>> >> >>> AM >>> >> >>> Please respond to >>> >> >>> hbase-user >> >>> >> >>> >> >>> >>> >>> >>> >>> What do you mean by quick way? You can use the api to put data into it >>> through a standalone java program or you can use a MR job to do it.. >>> >>> >>> On Tue, Sep 1, 2009 at 2:48 AM, <[email protected]> wrote: >>> >>> >>>> Hi there, >>>> >>>> >>>> Any quick way that I can save my data(12G) to HBase? >>>> Thanks >>>> >>>> >>>> Fleming >>>> >>>> >>> >>>> >>>> >>>> >>> >>> >>> >>> >>> >>> >> ------------------------------------------------------------------------- >> -- >> >> >>> TSMC PROPERTY >>> >> >>> This email communication (and any attachments) is proprietary >>> >> information >>> for the sole use of its >> >>> intended recipient. Any unauthorized review, use or distribution by >> anyone >>> other than the intended >> >>> recipient is strictly prohibited. If you are not the intended >> recipient, >>> please notify the sender by >> >>> replying to this email, and then delete this email and any copies of it >>> >> >>> immediately. Thank you. >> >>> >> ------------------------------------------------------------------------- >> -- >> >> >>> >>> >>> >> >> >> >> >> ------------------------------------------------------------------------- >> -- >> TSMC PROPERTY >> This email communication (and any attachments) is proprietary information >> for the sole use of its intended recipient. Any unauthorized review, use > or >> distribution by anyone other than the intended recipient is strictly >> prohibited. If you are not the intended recipient, please notify the >> sender by replying to this email, and then delete this email and any >> copies of it immediately. Thank you. >> ------------------------------------------------------------------------- >> -- >> >> >> >> >> > > > > > > --------------------------------------------------------------------------- > TSMC PROPERTY > This email communication (and any attachments) is proprietary information > for the sole use of its > intended recipient. Any unauthorized review, use or distribution by anyone > other than the intended > recipient is strictly prohibited. If you are not the intended recipient, > please notify the sender by > replying to this email, and then delete this email and any copies of it > immediately. Thank you. > --------------------------------------------------------------------------- > > > >
