Getting RetriesExhaustedWithDetailsException due to NSRE means that it took forever for a region server to close or split a region, what you pasted from the region server talks a region closing but that also happens during split.
I'd suggest digging more in those region server logs using this guide: http://hbase.apache.org/book/trouble.html Also make sure you review this http://hbase.apache.org/book/performance.html Finally giving a 1GB heap to HBase while inserting a lot of data is like making a malnourished child work in a coal mine, it's not very nice of you. J-D On Wed, Jun 22, 2011 at 11:06 PM, Sam Seigal <[email protected]> wrote: > Hi, > > I am loading data into my HBase cluster and running into two issues - > > During my import, I received the following exception -> > > org.apache.hadoop.hbase.client.RetriesExhaustedWithDetailsException: Failed > 53484 actions: servers with issues: spock7001:60020, > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatch(HConnectionManager.java:1220) > at > org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.processBatchOfPuts(HConnectionManager.java:1234) > at > org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:819) > at org.apache.hadoop.hbase.client.HTable.doPut(HTable.java:675) > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:660) > > May have cluster issues => true > Cause 0 > > When I check the logs on the regions server, the last thrown exception is > the following => > > Thu Jun 23 05:16:18 2011 GMT regionserver 10460-0@spock7001:0 [DEBUG] (IPC > Server handler 7 on 60020) > { org.apache.hadoop.hbase.NotServingRegionException: > hbaseTable,,1308805558566.5aefc6c2b9599f55f8b40351a61db03c. is closing > Thu Jun 23 05:22:18 2011 GMT regionserver 10460-0@spock7001:0 [DEBUG] > (regionserver60020.logRoller) org.apache.hadoop.conf.Configuration: > java.io.IOException: config() > > On running status 'detailed' in the shell , I get => > > 0 regionsInTransition > 3 live servers > spock7001:60020 1308805454136 > requests=0, regions=0, usedHeap=470, maxHeap=910 > spock6002:60020 1308805434201 > requests=0, regions=1, usedHeap=550, maxHeap=910 > hbaseTable,,1308805558566.5aefc6c2b9599f55f8b40351a61db03c. > stores=1, storefiles=2, storefileSizeMB=383, memstoreSizeMB=0, > storefileIndexSizeMB=1 > spock6001:60020 1308805268507 > requests=0, regions=2, usedHeap=90, maxHeap=910 > -ROOT-,,0 > stores=1, storefiles=1, storefileSizeMB=0, memstoreSizeMB=0, > storefileIndexSizeMB=0 > .META.,,1 > stores=1, storefiles=0, storefileSizeMB=0, memstoreSizeMB=0, > storefileIndexSizeMB=0 > 0 dead servers > > > I am issuing a checkAndPut() to insert records into HBase. Is this a bug ? > > Secondly, I have followed the instructions in the HBase book to increase > write throughput. I have the following settings for my hbase table: > > config = HBaseConfiguration.create(); > table = new HTable (config, "hbaseTable"); > table.setAutoFlush(false); > table.setWriteBufferSize(104857600); > > However, according to my logs, each checkAndPut() call takes on an average > of 5 milliseconds. Is this unavoidable overhead due to locking ? > > All of my HBase daemons are running with -Xmx1g of heapsize. > > Any help is appreciated. > > Thank you, > > Sam >
