Can you take jstack of the region server next time this happens ? Btw please consider upgrading. 0.96.x is too old.
> On Aug 3, 2015, at 5:18 AM, Chang Chen <baibaic...@gmail.com> wrote: > > Hi All > > We use HBase 0.96.1.1, and find a very strange issue which looks > like CompactSplitThread is waiting for something! The following logs are > from problematic RegionServer: > > > 2015-07-29 10:15:39,702 INFO > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Completed > compaction: Request = > regionName=realtime_rec,901943112,1426251108696.3a3c8d1c888f98518ce42f94b4033a39., > storeName=usr, fileCount=3, fileSize=2.4 M, priority=12, > time=21663559757818028; duration=0sec > > 2015-07-29 19:37:38,566 INFO > org.apache.hadoop.hbase.regionserver.CompactSplitThread: Aborted > compaction: Request = > regionName=full_user_profile,4U5BN9EUHMZXZA58NX811Q64XRBQMNZF2KH2,1406419093595.e850bec0a113f4fb6c7ed5a136bc630e., > storeName=cat, fileCount=3, fileSize=55.2 M (31.6 M, 11.5 M, 12.1 M), > priority=12, time=21697278778022407; duration=0sec > > > Between 10:15 and 19:37, I didn't find any logs generated by > CompactSplitThread, and we restarted HBase cluster at time of 19:37. The > waiting of compaction thread made compaction queue size constantly > increase, and eventually slow down HTable.get > > Our major and minor compaction thread are set to 1, and I guess > CompactSplitThread may be blocked in selectCompaction, since the aborted > compaction only spent 0 seconds! > > Is there any clue which can help me to find more information? > > Thanks > Chang