[
https://issues.apache.org/jira/browse/HBASE-1415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12708953#action_12708953
]
stack commented on HBASE-1415:
------------------------------
No worries Andrew. I can reproduce easy enough. Let me see if I can figure a
patch for you and J-D to look at.
> Stuck on memcache flush
> -----------------------
>
> Key: HBASE-1415
> URL: https://issues.apache.org/jira/browse/HBASE-1415
> Project: Hadoop HBase
> Issue Type: Bug
> Reporter: stack
> Assignee: stack
> Fix For: 0.20.0
>
>
> Latest trunk, regionserver gets stuck doing below:
> {code}
> 2009-05-13 03:05:28,126 [regionserver/0:0:0:0:0:0:0:0:60021.logRoller] INFO
> org.apache.hadoop.hbase.regionserver.HLog: Closed
> hdfs://aa0-000-12.u.powerset.com:9000/hbasetrunk2/.logs/aa0-000-13.u.powerset.com_1242178588540_60021/hlog.dat.1242183899413,
> entries=10051
> 2009-05-13 03:05:28,126 [regionserver/0:0:0:0:0:0:0:0:60021.logRoller] DEBUG
> org.apache.hadoop.hbase.regionserver.HLog: Found 0 logs to remove out of
> total 53; oldest outstanding seqnum is 6838308 from region
> TestTable,0842399686,1242180442452
> 2009-05-13 03:05:28,737 [IPC Server handler 9 on 60021] INFO
> org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Forced flushing of
> TestTable,0107641622,1242181965182 because global memcache limit of 497.8m
> exceeded; currently 513.3m and flushing till 311.1m
> 2009-05-13 03:05:28,737 [IPC Server handler 9 on 60021] DEBUG
> org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Too many store files in
> store info: 5, waiting
> 2009-05-13 03:05:28,737 [IPC Server handler 9 on 60021] DEBUG
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction requested
> for region TestTable,0107641622,1242181965182/614435386 because:
> regionserver/0:0:0:0:0:0:0:0:60021.cacheFlusher
> 2009-05-13 03:06:07,143 [regionserver/0:0:0:0:0:0:0:0:60021.compactor] DEBUG
> org.apache.hadoop.hbase.regionserver.Store: Completed compaction of info;
> store size is 168.0m
> 2009-05-13 03:06:07,143 [regionserver/0:0:0:0:0:0:0:0:60021.compactor] INFO
> org.apache.hadoop.hbase.regionserver.HRegion: compaction completed on region
> TestTable,0092188538,1242183375224 in 3mins, 58sec
> 2009-05-13 03:06:58,757 [IPC Server handler 9 on 60021] DEBUG
> org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Too many store files in
> store info: 5, waiting
> 2009-05-13 03:06:58,757 [IPC Server handler 9 on 60021] DEBUG
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction requested
> for region TestTable,0107641622,1242181965182/614435386 because:
> regionserver/0:0:0:0:0:0:0:0:60021.cacheFlusher
> 2009-05-13 03:08:28,767 [IPC Server handler 9 on 60021] DEBUG
> org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Too many store files in
> store info: 5, waiting
> 2009-05-13 03:08:28,767 [IPC Server handler 9 on 60021] DEBUG
> org.apache.hadoop.hbase.regionserver.CompactSplitThread: Compaction requested
> for region TestTable,0107641622,1242181965182/614435386 because:
> regionserver/0:0:0:0:0:0:0:0:60021.cacheFlusher
> 2009-05-13 03:09:58,777 [IPC Server handler 9 on 60021] DEBUG
> org.apache.hadoop.hbase.regionserver.MemcacheFlusher: Too many store files in
> store info: 5, waiting
> ...
> {code}
> Thread dumping I see a few threads blocked here:
> {code}
> "IPC Server handler 4 on 60021" daemon prio=10 tid=0x00007f42982ee000
> nid=0x599b waiting for monitor entry [0x000000004380b000..0x000000004380bb00]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hbase.regionserver.MemcacheFlusher.reclaimMemcacheMemory(MemcacheFlusher.java:294)
> - waiting to lock <0x00007f42aac8cbd8> (a
> org.apache.hadoop.hbase.regionserver.MemcacheFlusher)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.batchUpdates(HRegionServer.java:1800)
> at sun.reflect.GeneratedMethodAccessor6.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.hbase.ipc.HBaseRPC$Server.call(HBaseRPC.java:642)
> at
> org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:911)
> ..
> {code}
> And this is a good while after my MR upload failed.
> Assigning myself since I can reproduce easy enough.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
