Thanks for the brilliant comments, Ryan. For each of this not so graceful close, zookeeper will populate its log file with a WARN record that just likes 2009-08-04 15:15:35,831 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x122e34dd69b00ad due to java.io.IOException: Read error.
It might be confuse some users of HBase, probably we can put some information about this in the documentation. On Tue, Aug 4, 2009 at 2:41 PM, Ryan Rawson<[email protected]> wrote: > We should move the clients to a non-active server API, possibly the > REST one, and avoid using active sessions just for clients. Something > to address in 0.21 I think. > > As for #2, it is recommended now to run a quorum of zookeeper instead > of a single one. This reduces the risk of running out of connections. > > Also the code snippet you listed is a little degenerate, we can never > fully protect ourselves from fork-bomb like code. Your code snippet > suggests that: > - you are creating/closing HTable a lot. Maybe you shouldn't do that? > HTablePool? > - you have 1024+ tables, and need to access them in one client at one time. > > In the mean time, highly consider upgrading to a cluster of 5-7 ZK > hosts. For production, you should consider NOT running them on your > HBase/HDFS/map-reduce nodes. > > Good luck! > -ryan > > On Mon, Aug 3, 2009 at 11:00 PM, Angus He<[email protected]> wrote: >> Hi All, >> >> In HBase 0.20rc, HTable does not explicitly close the connection to >> zookeeper in HTable::close. >> It probably could be better. And in my opinion, it should be for: >> >> 1. It is not well-behaved, although zookeeper is able to detect the >> lost connection after issuing networking I/O operation, . >> 2. It is easy to get zookeeper server stuck with exceptions like "Too >> many connections from /0:0:0:0:0 :0:0:1 - max is 30", when user >> write codes like: >> for (int i = 0; i < 1024; ++i) { >> HTable table = new HTable("foobar"); >> table.close(); >> } >> >> In the current implementation, different HTable instances share the >> same connection to zookeeper if they have same HBaseConfiguration >> instance. For this, we cannot close the connection directly in HTable, >> but probably we could implement HConnection class with >> reference-counting ability. >> >> Any comments? >> >> -- >> Regards >> Angus >> > -- Regards Angus
