thanks very much. i think , yes, i should change hbase to mysql in my case, if mysql is not enough i will change it to hbase. like chinese word ,i used chopper to kill the chicken ,may i should use knife. :) thanks any way.
2009/11/12 Jean-Daniel Cryans <[email protected]> > Ouch! Well the region server needs at least one cpu, if you have to > tasks writing at the same time AND if you are reading from HDFS then I > can only see trouble. At least set a maximum number of map and reduce > slots to 1 and give 1200MB to HBase, the rest to the rest of the > stack... even then I can't guarantee anything. > > J-D > > On Wed, Nov 11, 2009 at 6:21 PM, 梁景明 <[email protected]> wrote: > > Intel CORE2 E4500 2.2GHz 64bit > > 2G DDR2 667 > > 160GB SATA > > > > u mean it will use all memory when the data is too large? > > > > > > 2009/11/12 Jean-Daniel Cryans <[email protected]> > > > >> What I mean is that compression will put less stress on IO so less > >> writing to the disk, etc. > >> > >> With 1GB of RAM it doesn't give much space for garbage collection so I > >> guess you have huge GC pauses. By default we use the CMS garbage > >> collector and it needs memory in order to be more efficient. 2GB is > >> still underpowered if you only have a single machine. > >> > >> What kind of machine do you have by the way? CPU, RAM, etc > >> > >> Thx, > >> > >> J-D > >> > >> On Wed, Nov 11, 2009 at 5:34 PM, 梁景明 <[email protected]> wrote: > >> > hi, thanks. > >> > first u mean to use compression > >> > second u mean to use more regionserver ,right? > >> > third u introduce a way to improve hbase performance. > >> > > >> > To use one region server is base on i only have one serverhost in > >> internet, > >> > and i want to support hbase server for my web like a scalable > database. > >> > > >> > Does it mean hbase used limited just by one server , so if i use > >> > compression, > >> > it maybe also happen to fail for inputing a large data ,right? > >> > > >> > in this case , how large data does one region server work about 2g > ram . > >> > > >> > and the logs is too large ,i just pick up the same part to sent , i am > >> > sorry, > >> > but i guess it must be some thing to limit the data to input ,because > i > >> > inputed 1G data to it , it works ok, when it was up to 27G ,i > normally > >> stop > >> > about 20% ,about 5-6G data, it died. > >> > > >> > > >> > 2009/11/12 Jean-Daniel Cryans <[email protected]> > >> > > >> >> If you setAutoFlush(true) then you don't have to flushCommits() ;) > >> >> > >> >> Also HBase isn't meant for the heavy upload of one just 1 node, and > >> >> when importing data you have to be more careful since it's not a > >> >> "normal" usage pattern. > >> >> > >> >> Make sure of to follow the instructions in this link if you want to > >> >> have more chance of succeeding: > >> >> http://wiki.apache.org/hadoop/PerformanceTuning > >> >> Makre you use LZO, that you give at least 3GB of RAM and that the > >> >> machine doesn't swap. Check conf/hbase-env.sh to change the default > >> >> 1000MB heap. > >> >> > >> >> WRT the exact reason your region server died, you pasted way too > >> >> little information from its log. > >> >> > >> >> J-D > >> >> > >> >> On Wed, Nov 11, 2009 at 1:45 AM, 梁景明 <[email protected]> wrote: > >> >> > Hi , i have a large data sized 27G to insert into hbase whith one > >> >> > regionserver. > >> >> > and i used mapreduce to insert the data. > >> >> > Whatever how much maps i used, how maps' thread to sleep to control > >> the > >> >> > speed, > >> >> > it just worked before 20% data inserted ,then failed , and hbase > >> can't > >> >> > start again. > >> >> > it's weird. > >> >> > > >> >> > is there some pool to store the data for insert,if there is too > much > >> than > >> >> > its size ,will caused error. > >> >> > so , will .flushCommits() clear that pool? thanks for any help. > >> >> > > >> >> > *here is my code to insert in a map process*. > >> >> > > >> >> > ============================ > >> >> > table = new HTable(conf, tablename.getBytes()); > >> >> > table.setAutoFlush(true); > >> >> > Put p = new Put(Bytes.toBytes(obj.getKey())); > >> >> > HashMap cols = obj.getColumns(); > >> >> > ....... > >> >> > table.put(p); > >> >> > table.flushCommits(); > >> >> > ================================ > >> >> > > >> >> > *here is my logs* > >> >> > > >> >> > Hadoop insert log > >> >> > ====================================================== > >> >> > org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to > >> >> contact > >> >> > region server 192.168.1.116:60020 for region > >> >> > chapter,b74054c6fba7f1f072c6a3a4fc3d329a,1257926538767, row > >> >> > 'b7e49883d0380b4194025170f8f9cb7f', but failed after 10 attempts. > >> >> > Exceptions: > >> >> > java.net.ConnectException: Connection refused at > >> >> > > >> >> > >> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.getRegionServerWithRetries(HConnectionManager.java:1001) > >> >> > at > >> >> > > >> >> > >> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers$2.doCall(HConnectionManager.java:1192) > >> >> > > >> >> > at > >> >> > >> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers$Batch.process(HConnectionManager.java:1114) > >> >> > at > >> >> > >> > org.apache.hadoop.hbase.client.HConnectionManager$TableServers.processBatchOfRows(HConnectionManager.java:1200) > >> >> > at > >> >> org.apache.hadoop.hbase.client.HTable.flushCommits(HTable.java:605) > >> >> > at > org.apache.hadoop.hbase.client.HTable.put(HTable.java:470) > >> >> > at > com.soko.hbase.tool.HbaseUtil.insertData(HbaseUtil.java:118) > >> >> > at > com.soko.mr.HbaseFictionMR$Map.map(HbaseFictionMR.java:50) > >> >> > at com.soko.mr.HbaseFictionMR$Map.map(HbaseFictionMR.java:1) > >> >> > at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50) > >> >> > at > >> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:358) > >> >> > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:307) > >> >> > at org.apache.hadoop.mapred.Child.main(Child.java:170) > >> >> > > >> >> > >> > ========================================================================================= > >> >> > > >> >> > hbase-futureha-regionserver-ubuntu5.log > >> >> > ================================================================= > >> >> > 2009-11-11 16:44:06,998 INFO > >> >> org.apache.hadoop.hbase.regionserver.HRegion: > >> >> > Closed chapter,90c0011cb6287924c818d371a27e145f,1257924043800 > >> >> > 2009-11-11 16:44:06,998 INFO > >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: aborting server > >> at: > >> >> > 192.168.1.116:60020 > >> >> > 2009-11-11 16:44:09,091 INFO org.apache.hadoop.hbase.Leases: > >> >> > regionserver/192.168.1.116:60020.leaseChecker closing leases > >> >> > 2009-11-11 16:44:09,091 INFO org.apache.hadoop.hbase.Leases: > >> >> > regionserver/192.168.1.116:60020.leaseChecker closed leases > >> >> > 2009-11-11 16:44:10,931 INFO > >> >> > org.apache.hadoop.hbase.regionserver.HRegionServer: worker thread > >> exiting > >> >> > 2009-11-11 16:44:10,932 INFO org.apache.zookeeper.ZooKeeper: > Closing > >> >> > session: 0x124e11cbae00001 > >> >> > 2009-11-11 16:44:10,932 INFO org.apache.zookeeper.ClientCnxn: > Closing > >> >> > ClientCnxn for session: 0x124e11cbae00001 > >> >> > 2009-11-11 16:44:10,968 INFO org.apache.zookeeper.ClientCnxn: > >> Exception > >> >> > while closing send thread for session 0x124e11cbae00001 : Read > error > >> rc = > >> >> -1 > >> >> > java.nio.DirectByteBuffer[pos=0 lim=4 cap=4] > >> >> > ================================================================ > >> >> > > >> >> > Master > >> >> > ---------------------------------------- > >> >> > 2009-11-11 16:59:38,792 INFO org.apache.hadoop.ipc.HBaseServer: IPC > >> >> Server > >> >> > handler 2 on 60000: exiting > >> >> > 2009-11-11 16:59:38,791 INFO org.apache.zookeeper.ZooKeeper: > Closing > >> >> > session: 0x124e11cbae00000 > >> >> > 2009-11-11 16:59:38,793 INFO org.apache.zookeeper.ClientCnxn: > Closing > >> >> > ClientCnxn for session: 0x124e11cbae00000 > >> >> > 2009-11-11 16:59:38,795 INFO org.apache.zookeeper.ClientCnxn: > >> Exception > >> >> > while closing send thread for session 0x124e11cbae00000 : Read > error > >> rc = > >> >> -1 > >> >> > java.nio.DirectByteBuffer[pos=0 lim=4 cap=4] > >> >> > ----------------------------------------- > >> >> > > >> >> > ZooKeeper > >> >> > ============================= > >> >> > 2009-11-11 16:59:38,219 INFO > >> org.apache.zookeeper.server.NIOServerCnxn: > >> >> > Creating new session 0x124e11cbae0007e > >> >> > 2009-11-11 16:59:38,229 INFO > >> org.apache.zookeeper.server.NIOServerCnxn: > >> >> > Finished init of 0x124e11cbae0007e valid:true > >> >> > 2009-11-11 16:59:38,494 WARN > >> >> > org.apache.zookeeper.server.PrepRequestProcessor: Got exception > when > >> >> > processing sessionid:0x124e11cbae00000 type:create cxid:0x10 > >> >> > zxid:0xfffffffffffffffe txntype:unknown n/a > >> >> > org.apache.zookeeper.KeeperException$NodeExistsException: > >> KeeperErrorCode > >> >> = > >> >> > NodeExists > >> >> > at > >> >> > > >> >> > >> > org.apache.zookeeper.server.PrepRequestProcessor.pRequest(PrepRequestProcessor.java:245) > >> >> > at > >> >> > > >> >> > >> > org.apache.zookeeper.server.PrepRequestProcessor.run(PrepRequestProcessor.java:114) > >> >> > 2009-11-11 16:59:38,793 INFO > >> >> > org.apache.zookeeper.server.PrepRequestProcessor: Processed session > >> >> > termination request for id: 0x124e11cbae00000 > >> >> > ============================ > >> >> > > >> >> > >> > > >> > > >
