Thanks for posting config. The reason I suggest removing i-cms is because of this paragraph (pointed to from the hbase performance page up on wiki): http://java.sun.com/javase/technologies/hotspot/gc/gc_tuning_6.html#icms
Are you sure this works in your hbase-env.sh: ulimit -n 32000 Check the first line in your master or regionserver log after startup. It'll print out the ulimit that hbase sees. Make sure its 32k. Yes, xceivers is important, especially when loading gets up to where you are. I took a look at the regionserver log you posted. Please enable DEBUG going forward (See FAQ in wiki for how). What are you tasktrackers doing? Are they doing the hbase loading? You might try turning down how many task run concurrently on each tasktracker. The running tasktracker may be sucking resources from hdfs (and thus by association, from hbase): i.e. mapred.map.tasks and mapred.reduce.tasks (Pardon me if this advice has been given previous and you've already acted on it). St.Ack On Tue, Jan 12, 2010 at 5:36 AM, Dmitriy Lyfar <[email protected]> wrote: > Hi Stack, > > I have following configuration: http://pastebin.com/m3a471fab > so I'll remove incremental mode option. > > 2010/1/11 stack <[email protected]> > > > What do you have for your HBASE_OPTS in conf/hbase-env.sh Dmitry? > > > > Remove this: > > > > -XX:+CMSIncrementalMode > > > > ... if its present on your HBASE_OPTS. > > > > Change your hbase.zookeeper.property.tickTime from 2 to 3 so that zk > > session > > goes for longer (See the comment in the head of the 0.20 branch for > > explaination). > > > > Ok, I'll change. > > > > > > What else is running on the machines where the regionserver times out its > > session w/ zk? Anything? The uploader? > > > > Nothing, just hadoop datanode, tasktracker and regionserver (no zk). > > > > > > You've set the ulimit > 1024 and xceivers? Right (I don't see that in > your > > old messages) > > > > As for ulimit I have 32K. But did not change xceivers, I will try and > report > about results, thanks. > Btw, is 2Gb heap size is enough for hadoop in such configuration? > > -- > Regards, Lyfar Dmitriy >
