hi colleagues,
I am doing batup insert on a 20 nodes cluster. Unfortunately the data are
inserted to the last region (sequential insert),
I have already applied the Hbase-1058 patch before the batch insert. But I
got a "big" region with 4450 HStoreFile after
a long run. Now I want to start the hbase, but always got a OOME on
compaction. And the OOME will infect to other regionServer
if one server is OOME.
How to save my data?
My Configuration:
1. hadoop 0.19 branch head
2. hbase 0.19.1 with Hbase-1058 patch
3. 8GRAM (reserved 6GRAM for Hbase in hbase-env.sh)
4. hbase.hstore.compaction.max = 5 (probaly of no use)
2009-05-12 13:55:12,520 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Dump of metrics:
request=0.0, regions=316, stores=316, storefiles=4450,
storefileIndexSize=251, memcacheSize=0, usedHeap=3941, maxHeap=5333
2009-05-12 13:55:12,520 FATAL
org.apache.hadoop.hbase.regionserver.HRegionServer: Set stop flag in
regionserver/0:0:0:0:0:0:0:0:62020.compactor
java.lang.OutOfMemoryError: Java heap space
at java.util.Arrays.copyOf(Arrays.java:2760)
at java.util.Arrays.copyOf(Arrays.java:2734)
at java.util.ArrayList.ensureCapacity(ArrayList.java:167)
at java.util.ArrayList.add(ArrayList.java:351)
at
org.apache.hadoop.hbase.io.MapFile$Reader.readIndex(MapFile.java:370)
at
org.apache.hadoop.hbase.io.MapFile$Reader.seekInternal(MapFile.java:462)
at
org.apache.hadoop.hbase.io.MapFile$Reader.getClosest(MapFile.java:586)
at
org.apache.hadoop.hbase.io.MapFile$Reader.getClosest(MapFile.java:569)
at
org.apache.hadoop.hbase.io.BloomFilterMapFile$Reader.getClosest(BloomFilterMapFile.java:115)
at
org.apache.hadoop.hbase.io.HalfMapFileReader.getClosest(HalfMapFileReader.java:152)
at
org.apache.hadoop.hbase.io.HalfMapFileReader.next(HalfMapFileReader.java:190)
at
org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1133)
at
org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:936)
at
org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:727)
at
org.apache.hadoop.hbase.regionserver.HRegion.compactStores(HRegion.java:684)
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitThread.java:105)
2009-05-12 13:55:13,937 INFO org.apache.hadoop.ipc.HBaseServer: Stopping
server on 62020
2009-05-12 13:55:14,003 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 7 on 62020: exiting
2009-05-12 13:55:14,503 INFO
org.apache.hadoop.hbase.regionserver.HRegionServer: Stopping infoServer
2009-05-12 13:55:15,627 INFO org.apache.hadoop.ipc.HBaseServer: IPC Server
handler 100 on 62020: exiting
Any suggestions are welcomed! Thanks!
