HBase has a problem with running out of file handles on machines that act as region servers. From https://issues.apache.org/jira/browse/HADOOP-2577
> Today the rapleaf gave me an lsof listing from a regionserver. Had thousands > of open sockets to datanodes all in ESTABLISHED and CLOSE_WAIT state. On > average they seem to have about ten file descriptors/sockets open per region > (They have 3 column families IIRC. Per family, can have between 1-5 or so > mapfiles open per family – 3 is max... but compacting we open a new one, > etc.). > > They have thousands of regions. 400 regions – ~100G, which is not that much – > takes about 4k open file handles. > > If they want a regionserver to server a decent disk worths – 300-400G – then > thats maybe 1600 regions... 16k file handles. If more than just 3 column > families..... then we are in danger of blowing out limits if they are 32k. One possible solution we've thought of is multiplexing sockets between the DFSClient and the data node. In this case, there would be one socket per client <--> datanode and would run in async mode using select. This would consume far fewer sockets than the current 1 socket / client / datanode / open file. We used a socket multiplexer at Yahoo for the data store I worked on there, the "user data base" (or UDB) which stored all the preference data for all Yahoo pages that could be customized. All the UDB clients each had one socket open for each machine in the UDB server cluster. Similarly, each UDB server had one socket open to talk to all of its clients. When you consider each UDB server had to talk to several thousand clients, and that each server machine ran many server processes to handle load, this was a huge savings in OS overhead. While the 1 socket / client / datanode / open file is a simple model, if we are talking about scaling Hadoop or HBase to thousands of nodes, it seems like socket multiplexing would be a big win in terms of server overhead, especially considering that many of these connections are more idle than in use. Yes, multiplexing a socket is more complicated than having one socket per file, but saving system resources seems like a way to scale. Questions? Comments? Opinions? Flames? --- Jim Kellerman, Senior Engineer; Powerset No virus found in this outgoing message. Checked by AVG Free Edition. Version: 7.5.516 / Virus Database: 269.19.5/1228 - Release Date: 1/16/2008 9:01 AM
