First, I want to thank all on this list who responded to all my questions, and all the HBase developers as well.
After trying both hadoop-0.20-append and Cloudera's latest, along with various settings, we are still leaking both file and socket handles at high ingest rates. The handle usage increases slowly, but linearly, till a process or OS limit is reached. The larger problem is that HBase does not recover from that situation. In fact, if once the file handles were driven high, but less than quota, and the ingest was slackened significantly, there was no recovery. If ingest was completely stopped and restarted, there was recovery. By high ingest rate I mean about 1200 records/sec per node, 2K per record, 50 records per batch put. I truly wish I had the time to dig in to find the answers, or at least indications, but my time is not my own. We will keeping a close eye on HBase, and will likely revisit it before long. Thanks again Thomas Downing
