Sorry, forgot to insert link to DEBUG regionserver logs: http://pastebin.com/m70f01f36
2010/1/13 Dmitriy Lyfar <[email protected]> > Hi Stack, > > Thank you for you help. I set xceivers in hdfs xml config like: > > <property> > <name>dfs.datanode.max.xcievers</name> > <value>8192</value> > </property> > > And ulimit is 32K for sure. I turned off DEBUG logging level for hbase and > here is log for one of regionservers after I have inserted 200K records > (each row is 25Kb). > Speed still the same (about 1K rows per second). > Random ints plays a role of row keys now (i.e. uniform random distribution > on (0, 100 * 1000)). > What do you think is 5GB for hbase and 2GB for hdfs enough? > > >> What are you tasktrackers doing? Are they doing the hbase loading? You >> might try turning down how many task run concurrently on each tasktracker. >> The running tasktracker may be sucking resources from hdfs (and thus by >> association, from hbase): i.e. mapred.map.tasks and mapred.reduce.tasks >> (Pardon me if this advice has been given previous and you've already acted >> on it). > > > Tasktrackers is not used now (I planned them for future use in statistical > analysis). So I turned them off for last tests. Data uploader is several > clients which run simultaneously on name node and each of them inserts 100K > records. > > -- > Regards, Lyfar Dmitriy > -- Regards, Lyfar Dmitriy mailto: [email protected] jabber: [email protected]
