HI Anze, In word, yes - 0.20.4 is not that stable in my experience, and upgrading to the latest CDH3 beta (which includes HBase 0.89.20100924) should give you a huge improvement in stability.
You'll still need to do a bit of tuning of settings, but once it's well tuned it should be able to hold up under load without crashing. -Todd On Mon, Dec 13, 2010 at 2:41 AM, Anze <anzen...@volja.net> wrote: > Hi all! > > We have been using HBase 0.20.4 (cdh3b1) in production on 2 nodes for a few > months now and we are having constant issues with it. We fell over all > standard traps (like "Too many open files", network configuration > problems,...). All in all, we had about one crash every week or so. > Fortunately we are still using it just for background processing so our > service didn't suffer directly, but we have lost huge amounts of time just > fixing the data errors that resulted from data not being written to permanent > storage. Not to mention fixing the issues. > As you can probably understand, we are very frustrated with this and are > seriously considering moving to another bigtable. > > Right now, HBase crashes whenever we run very intensive rebuild of secondary > index (normal table, but we use it as secondary index) to a huge table. I have > found this: > http://wiki.apache.org/hadoop/Hbase/Troubleshooting > (see problem 9) > One of the lines read: > "Make sure you give plenty of RAM (in hbase-env.sh), the default of 1GB won't > be able to sustain long running imports." > > So, if I understand correctly, no matter how HBase is set up, if I run an > intensive enough application, it will choke? I would expect it to be slower > when under (too much) pressure, but not to crash. > > Of course, we will somehow solve this issue (working on it), but... :( > > What are your experiences with HBase? Is it stable? Is it just us and the way > we set it up? > > Also, would upgrading to 0.89 (cdh3b3) help? > > Thanks, > > Anze > > -- Todd Lipcon Software Engineer, Cloudera
