Hmmmm, I should have sent to hbase mailing list. Please ignore this. Sorry for the spam.
On Fri, Oct 17, 2008 at 12:30 AM, Jim Kellerman (POWERSET) < [EMAIL PROTECTED]> wrote: > In the future, you will get a more timely response for hbase > questions if you post them on the [EMAIL PROTECTED] > mailing list. > > In order to address your question, it would be helpful to > know your hardware configuration (memory, # of cores), > any changes you have made to hbase-site.xml, how many > file handles are allocated per process, what else is > running on the same machine as the region server and > what versions of hadoop and hbase you are running. > > --- > Jim Kellerman, Powerset (Live Search, Microsoft Corporation) > > > -----Original Message----- > > From: Rui Xing [mailto:[EMAIL PROTECTED] > > Sent: Thursday, October 16, 2008 4:52 AM > > To: core-user@hadoop.apache.org > > Subject: out of memory error > > > > Hello List, > > > > We encountered an out-of-memory error in data loading. We have 5 data > > nodes > > and 1 name node distributed on 6 machines. Block-level compression was > > used. > > Following is the log output. Seems the problem was caused in compression. > > Is > > there anybody who ever experienced such error? Any helps or clues are > > appreciated. > > > > 2008-10-15 21:44:33,069 FATAL > > [regionserver/0:0:0:0:0:0:0:0:60020.compactor] > > regionserver.HRegionServer$1(579): Set stop flag in > > regionserver/0:0:0:0:0:0:0:0:60020.compactor > > java.lang.OutOfMemoryError > > at sun.misc.Unsafe.allocateMemory(Native Method) > > at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:99) > > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288) > > at > > > org.apache.hadoop.io.compress.zlib.ZlibDecompressor.<init>(ZlibDecompresso > > r.java:108) > > at > > > org.apache.hadoop.io.compress.zlib.ZlibDecompressor.<init>(ZlibDecompresso > > r.java:115) > > at > > > org.apache.hadoop.io.compress.zlib.ZlibFactory.getZlibDecompressor(ZlibFac > > tory.java:104) > > at > > > org.apache.hadoop.io.compress.DefaultCodec.createDecompressor(DefaultCodec > > .java:80) > > at > > > org.apache.hadoop.io.SequenceFile$Reader.getPooledOrNewDecompressor(Sequen > > ceFile.java:1458) > > at > > org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1543) > > at > > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1442) > > at > > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1431) > > at > > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1426) > > at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:292) > > at > > > org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.< > > init>(HStoreFile.java:635) > > at > > > org.apache.hadoop.hbase.regionserver.HStoreFile$BloomFilterMapFile$Reader. > > <init>(HStoreFile.java:717) > > at > > > org.apache.hadoop.hbase.regionserver.HStoreFile$HalfMapFileReader.<init>(H > > StoreFile.java:915) > > at > > > org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java: > > 408) > > at > > org.apache.hadoop.hbase.regionserver.HStore.<init>(HStore.java:263) > > at > > > org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.jav > > a:1698) > > at > > org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:481) > > at > > org.apache.hadoop.hbase.regionserver.HRegion.<init>(HRegion.java:421) > > at > > > org.apache.hadoop.hbase.regionserver.HRegion.splitRegion(HRegion.java:815) > > at > > > org.apache.hadoop.hbase.regionserver.CompactSplitThread.split(CompactSplit > > Thread.java:133) > > at > > > org.apache.hadoop.hbase.regionserver.CompactSplitThread.run(CompactSplitTh > > read.java:86) > > 2008-10-15 21:44:33,661 FATAL > > [regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher] > > regionserver.Flusher(183): > > Replay of hlog required. Forcing server restart > > org.apache.hadoop.hbase.DroppedSnapshotException: region: > > p4p_test,,1224072139042 > > at > > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.ja > > va:1087) > > at > > org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:985) > > at > > > org.apache.hadoop.hbase.regionserver.Flusher.flushRegion(Flusher.java:174) > > at > > org.apache.hadoop.hbase.regionserver.Flusher.run(Flusher.java:91) > > Caused by: java.lang.OutOfMemoryError > > at sun.misc.Unsafe.allocateMemory(Native Method) > > at java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:99) > > at java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:288) > > at > > > org.apache.hadoop.io.compress.zlib.ZlibDecompressor.<init>(ZlibDecompresso > > r.java:107) > > at > > > org.apache.hadoop.io.compress.zlib.ZlibDecompressor.<init>(ZlibDecompresso > > r.java:115) > > at > > > org.apache.hadoop.io.compress.zlib.ZlibFactory.getZlibDecompressor(ZlibFac > > tory.java:104) > > at > > > org.apache.hadoop.io.compress.DefaultCodec.createDecompressor(DefaultCodec > > .java:80) > > at > > > org.apache.hadoop.io.SequenceFile$Reader.getPooledOrNewDecompressor(Sequen > > ceFile.java:1458) > > at > > org.apache.hadoop.io.SequenceFile$Reader.init(SequenceFile.java:1555) > > at > > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1442) > > at > > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1431) > > at > > org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1426) > > at org.apache.hadoop.io.MapFile$Reader.open(MapFile.java:292) > > at > > > org.apache.hadoop.hbase.regionserver.HStoreFile$HbaseMapFile$HbaseReader.< > > init>(HStoreFile.java:635) > > at > > > org.apache.hadoop.hbase.regionserver.HStoreFile$BloomFilterMapFile$Reader. > > <init>(HStoreFile.java:717) > > at > > > org.apache.hadoop.hbase.regionserver.HStoreFile.getReader(HStoreFile.java: > > 413) > > at > > > org.apache.hadoop.hbase.regionserver.HStore.updateReaders(HStore.java:665) > > at > > > org.apache.hadoop.hbase.regionserver.HStore.internalFlushCache(HStore.java > > :640) > > at > > org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:577) > > at > > > org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.ja > > va:1074) > > ... 3 more > > 2008-10-15 21:44:33,661 INFO > > [regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher] > > regionserver.Flusher(109): > > regionserver/0:0:0:0:0:0:0:0:60020.cacheFlusher exiting > > 2008-10-15 21:44:33,665 DEBUG > > [regionserver/0:0:0:0:0:0:0:0:60020.logRoller] > > regionserver.HLog(236): Closing current log writer > > /hbase/log_172.19.139.3_1224070931944_60020/hlog.dat.1224078265898 > > 2008-10-15 21:44:33,667 INFO > > [regionserver/0:0:0:0:0:0:0:0:60020.logRoller] > > regionserver.HLog(249): New log writer created at > > /hbase/log_172.19.139.3_1224070931944_60020/hlog.dat.1224078273665 > > 2008-10-15 21:44:33,667 INFO > > [regionserver/0:0:0:0:0:0:0:0:60020.logRoller] > > regionserver.LogRoller(93): LogRoller exiting. > > 2008-10-15 21:44:34,910 DEBUG [regionserver/0:0:0:0:0:0:0:0:60020] > > hbase.RegionHistorian(316): Offlined > > 2008-10-15 21:44:34,911 INFO [regionserver/0:0:0:0:0:0:0:0:60020] > > regionserver.HRegionServer(432): Stopping infoServer > > 2008-10-15 21:44:34,911 INFO [Acceptor ServerSocket[addr= > > 0.0.0.0/0.0.0.0,port=0,localport=60030]] > > util.ThreadedServer$Acceptor(656): > > Stopping Acceptor > > ServerSocket[addr=0.0.0.0/0.0.0.0,port=0,localport=60030] > > 2008-10-15 21:44:34,914 INFO [regionserver/0:0:0:0:0:0:0:0:60020] > > http.SocketListener(212): Stopped SocketListener on 0.0.0.0:60030 > > > > Thanks, > > -Ray >
