[
https://issues.apache.org/jira/browse/HADOOP-2587?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jim Kellerman updated HADOOP-2587:
----------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
Tests passed. Committed.
> Splits getting blocked by compactions causeing region to be offline for the
> length of the compaction 10-15 mins
> ---------------------------------------------------------------------------------------------------------------
>
> Key: HADOOP-2587
> URL: https://issues.apache.org/jira/browse/HADOOP-2587
> Project: Hadoop
> Issue Type: Bug
> Components: contrib/hbase
> Environment: hadoop subversion 611087
> Reporter: Billy Pearson
> Assignee: Jim Kellerman
> Fix For: 0.16.0
>
> Attachments: hbase-root-regionserver-PE1750-3.log, patch.txt
>
>
> The below is cut out of one of my region servers logs full log attached
> What is happening is there is one region on a this region server and its is
> under heave insert load so compaction are back to back one one finishes a new
> one starts the problem starts when its time to split the region.
> A compaction starts just millsecs before the split starts blocking the split
> but the split closes the region before the compaction is finished. Causing
> the region to be offline until the compaction is done. Once the compaction is
> done the split finishes and all is returned to normal but this is a big
> problem for production if the region is offline for 10-15 mins.
> The solution would be not to let the split thread to issue the below line
> while a compaction on that region is happening.
> 2008-01-11 16:22:01,247 DEBUG org.apache.hadoop.hbase.HRegionServer:
> webdata,,1200085987488 closing (Adding to retiringRegions)
> The only time I have seen this bug is when there is only one region on a
> region server because if more then one then the compaction happens to the
> other region(s) after the first one is done compaction and the split can do
> what it needs on the first region with out getting blocked.
> {code}
> 2008-01-11 16:22:01,020 INFO org.apache.hadoop.hbase.HRegion: compaction
> completed on region webdata,,1200085987488. Took 16mins, 10sec
> 2008-01-11 16:22:01,020 DEBUG org.apache.hadoop.hbase.HStore: compaction for
> HStore webdata,,1200085987488/size needed.
> 2008-01-11 16:22:01,020 DEBUG org.apache.hadoop.hbase.HRegion:
> 1773667150/size needs compaction
> 2008-01-11 16:22:01,021 INFO org.apache.hadoop.hbase.HRegion: starting
> compaction on region webdata,,1200085987488
> 2008-01-11 16:22:01,021 DEBUG org.apache.hadoop.hbase.HStore: started
> compaction of 14 files using
> /gfs_storage/hadoop-root/hbase/hregion_1773667150/compaction.dir/hregion_1773667150/size
> for webdata,,1200085987488/size
> 2008-01-11 16:22:01,123 DEBUG org.apache.hadoop.hbase.HRegion: Started
> memcache flush for region webdata,,1200085987488. Size 31.2m
> 2008-01-11 16:22:01,232 INFO org.apache.hadoop.hbase.HRegion: Splitting
> webdata,,1200085987488 because largest aggregate size is 100.7m and desired
> size is 64.0m
> 2008-01-11 16:22:01,247 DEBUG org.apache.hadoop.hbase.HRegionServer:
> webdata,,1200085987488 closing (Adding to retiringRegions)
> ...
> lots of NotServingRegionException's
> ...
> 2008-01-11 16:32:59,876 INFO org.apache.hadoop.hbase.HRegion: compaction
> completed on region webdata,,1200085987488. Took 10mins, 58sec
> ...
> 2008-01-11 16:33:02,193 DEBUG org.apache.hadoop.hbase.HRegion: Cleaned up
> /gfs_storage/hadoop-root/hbase/hregion_1773667150/splits true
> 2008-01-11 16:33:02,194 INFO org.apache.hadoop.hbase.HRegion: Region split of
> webdata,,1200085987488 complete; new regions: webdata,,1200090121237,
> webdata,com.tom.ent/2008-01-04/0PGM/09034104.html:http,1200090121239. Split
> took 11mins, 0sec
> 2008-01-11 16:33:02,227 DEBUG
> org.apache.hadoop.hbase.HConnectionManager$TableServers: No servers for
> .META.. Doing a find...
> 2008-01-11 16:33:02,283 DEBUG
> org.apache.hadoop.hbase.HConnectionManager$TableServers: Found 1 region(s)
> for .META. at address: 10.0.0.4:60020, regioninfo: regionname: -ROOT-,,0,
> startKey: <>, encodedName(70236052) tableDesc: {name: -ROOT-, families:
> {info:={name: info, max versions: 1, compression: NONE, in memory: false, max
> length: 2147483647, bloom filter: none}}}
> 2008-01-11 16:33:02,284 INFO org.apache.hadoop.hbase.HRegionServer: Updating
> .META. with region split info
> 2008-01-11 16:33:02,290 DEBUG org.apache.hadoop.hbase.HRegionServer:
> Reporting region split to master
> 2008-01-11 16:33:02,291 INFO org.apache.hadoop.hbase.HRegionServer: region
> split, META update, and report to master all successful. Old
> region=webdata,,1200085987488, new regions: webdata,,1200090121237,
> webdata,com.tom.ent/2008-01-04/0PGM/09034104.html:http,1200090121239
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
