If you setAutoFlush(true) then you don't have to flushCommits() ;) Also HBase isn't meant for the heavy upload of one just 1 node, and when importing data you have to be more careful since it's not a "normal" usage pattern.
Make sure of to follow the instructions in this link if you want to have more chance of succeeding: http://wiki.apache.org/hadoop/PerformanceTuning Makre you use LZO, that you give at least 3GB of RAM and that the machine doesn't swap. Check conf/hbase-env.sh to change the default 1000MB heap. WRT the exact reason your region server died, you pasted way too little information from its log. J-D On Wed, Nov 11, 2009 at 1:45 AM, 梁景明 <[email protected]> wrote: > Hi , i have a large data sized 27G to insert into hbase whith one > regionserver. > and i used mapreduce to insert the data. > Whatever how much maps i used, how maps' thread to sleep to control the > speed, > it just worked before 20% data inserted ,then failed , and hbase can't > start again. > it's weird. > > is there some pool to store the data for insert,if there is too much than > its size ,will caused error. > so , will .flushCommits() clear that pool? thanks for any help. > > *here is my code to insert in a map process*. > > ============================ > table = new HTable(conf, tablename.getBytes()); > table.setAutoFlush(true); > Put p = new Put(Bytes.toBytes(obj.getKey())); > HashMap cols = obj.getColumns(); > ....... > table.put(p); > table.flushCommits(); > ================================ > > *here is my logs* > > Hadoop insert log > ====================================================== > org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to contact > region server 192.168.1.116:60020 for region > chapter,b74054c6fba7f1f072c6a3a4fc3d329a,1257926538767, row > 'b7e49883d0380b4194025170f8f9cb7f', but failed after 10 attempts. > Exceptions: > java.net.ConnectException: Connection refused at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1001) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.doCall(HConnectionManager.java:1192) > > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.process(HConnectionManager.java:1114) > at > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1200) > at org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:605) > at org.apache.hadoop.hbase.client.HTable.put(HTable.java:470) > at com.soko.hbase.tool.HbaseUtil.insertData(HbaseUtil.java:118) > at com.soko.mr.HbaseFictionMR$Map.map(HbaseFictionMR.java:50) > at com.soko.mr.HbaseFictionMR$Map.map(HbaseFictionMR.java:1) > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) > at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) > at org.apache.hadoop.mapred.Child.main(Child.java:170) > ========================================================================================= > > hbase-futureha-regionserver-ubuntu5.log > ================================================================= > 2009-11-11 16:44:06,998 INFO org.apache.hadoop.hbase.regionserver.HRegion: > Closed chapter,90c0011cb6287924c818d371a27e145f,1257924043800 > 2009-11-11 16:44:06,998 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server at: > 192.168.1.116:60020 > 2009-11-11 16:44:09,091 INFO org.apache.hadoop.hbase.Leases: > regionserver/192.168.1.116:60020.leaseChecker closing leases > 2009-11-11 16:44:09,091 INFO org.apache.hadoop.hbase.Leases: > regionserver/192.168.1.116:60020.leaseChecker closed leases > 2009-11-11 16:44:10,931 INFO > org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread exiting > 2009-11-11 16:44:10,932 INFO org.apache.zookeeper.ZooKeeper: Closing > session: 0x124e11cbae00001 > 2009-11-11 16:44:10,932 INFO org.apache.zookeeper.ClientCnxn: Closing > ClientCnxn for session: 0x124e11cbae00001 > 2009-11-11 16:44:10,968 INFO org.apache.zookeeper.ClientCnxn: Exception > while closing send thread for session 0x124e11cbae00001 : Read error rc = -1 > java.nio.DirectByteBuffer[pos=0 lim=4 cap=4] > ================================================================ > > Master > ---------------------------------------- > 2009-11-11 16:59:38,792 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server > handler 2 on 60000: exiting > 2009-11-11 16:59:38,791 INFO org.apache.zookeeper.ZooKeeper: Closing > session: 0x124e11cbae00000 > 2009-11-11 16:59:38,793 INFO org.apache.zookeeper.ClientCnxn: Closing > ClientCnxn for session: 0x124e11cbae00000 > 2009-11-11 16:59:38,795 INFO org.apache.zookeeper.ClientCnxn: Exception > while closing send thread for session 0x124e11cbae00000 : Read error rc = -1 > java.nio.DirectByteBuffer[pos=0 lim=4 cap=4] > ----------------------------------------- > > ZooKeeper > ============================= > 2009-11-11 16:59:38,219 INFO org.apache.zookeeper.server.NIOServerCnxn: > Creating new session 0x124e11cbae0007e > 2009-11-11 16:59:38,229 INFO org.apache.zookeeper.server.NIOServerCnxn: > Finished init of 0x124e11cbae0007e valid:true > 2009-11-11 16:59:38,494 WARN > org.apache.zookeeper.server.PrepRequestProcessor: Got exception when > processing sessionid:0x124e11cbae00000 type:create cxid:0x10 > zxid:0xfffffffffffffffe txntype:unknown n/a > org.apache.zookeeper.KeeperException$NodeExistsException: KeeperErrorCode = > NodeExists > at > org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:245) > at > org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:114) > 2009-11-11 16:59:38,793 INFO > org.apache.zookeeper.server.PrepRequestProcessor: Processed session > termination request for id: 0x124e11cbae00000 > ============================ >
