[
https://issues.apache.org/jira/browse/HBASE-674?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12609705#action_12609705
]
Billy Pearson commented on HBASE-674:
-------------------------------------
I thank this should be a blocker for 2.0 as it causes problems on long running
jobs and region server that have been online for a while.
What I am seeing is the last line of the flush like the below log lines has the
memcache size on the end after the flush. this grows with every flush that has
data in it.
The problem comes up with the region server thanks the memcache is > then
hbase.hregion.memcache.flush.size then it starts flushing back to back with
little or nothing
to flush when it starts doing this on my servers the region server hangs and is
needed to be restarted after killing it with a kill -9 pid.
Also a side note I do see flushes back to back on region server that only have
one or two regions I see a memcache flush then 1-3 more of the same region back
to back then its ok but this seams to go away after the region server host more
then 4 region.
Logs from a few days ago:
{code}
2008-06-25 21:02:20,632 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214427652459 in 10604ms, sequence
id=1973821, 17.0m
2008-06-25 21:02:20,633 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214427652459 in 1ms, sequence
id=1973822, 17.0m
2008-06-25 21:02:21,478 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214427652459 in 842ms, sequence
id=1973832, 17.4m
2008-06-25 21:02:22,896 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214427652459 in 1408ms, sequence
id=1976711, 17.8m
2008-06-25 21:11:20,979 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214427652459 in 16988ms, sequence
id=3827578, 52.0m
hbase restarted
2008-06-25 21:16:18,004 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214427652459 in 10336ms, sequence
id=4817365, 17.0m
2008-06-25 21:16:18,408 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214427652459 in 404ms, sequence
id=4817378, 17.0m
2008-06-25 21:16:19,838 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214427652459 in 1421ms, sequence
id=4817952, 17.7m
2008-06-25 21:19:01,512 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214427652459 in 9692ms, sequence
id=5661153, 32.8m
2008-06-25 21:19:01,821 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214427652459 in 308ms, sequence
id=5661169, 32.8m
2008-06-25 21:19:04,261 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214427652459 in 2439ms, sequence
id=5661523, 33.4m
2008-06-25 21:19:05,317 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214427652459 in 1052ms, sequence
id=5666391, 33.5m
2008-06-25 21:49:21,616 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214427652459 in 14214ms, sequence
id=7415697, 64.7m
2008-06-25 21:49:22,369 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214427652459 in 738ms, sequence
id=7415798, 64.7m
2008-06-25 21:49:22,373 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214427652459 in 4ms, sequence
id=7415800, 64.7m
hbase restarted
2008-06-25 22:54:33,665 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214448561806 in 11168ms, sequence
id=9238125, 19.5m
2008-06-25 22:54:34,248 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214448561806 in 582ms, sequence
id=9241589, 19.7m
2008-06-25 22:54:34,989 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214448561806 in 740ms, sequence
id=9242882, 19.8m
2008-06-25 22:54:35,749 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214448561806 in 759ms, sequence
id=9244773, 19.8m
2008-06-25 22:54:36,619 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214448561806 in 869ms, sequence
id=9247214, 19.8m
2008-06-25 22:54:37,657 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214448561806 in 1031ms, sequence
id=9249946, 20.0m
2008-06-25 23:14:29,857 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214448561806 in 13549ms, sequence
id=12776517, 53.0m
hbase restart
2008-06-25 23:21:25,013 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214448561806 in 13706ms, sequence
id=14457037, 20.2m
2008-06-25 23:21:25,508 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214448561806 in 494ms, sequence
id=14474826, 20.2m
2008-06-25 23:21:26,353 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214448561806 in 844ms, sequence
id=14475821, 20.3m
2008-06-25 23:21:27,208 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214448561806 in 854ms, sequence
id=14478027, 20.3m
2008-06-25 23:21:27,816 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214448561806 in 607ms, sequence
id=14481138, 20.3m
2008-06-25 23:21:28,609 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214448561806 in 791ms, sequence
id=14483523, 20.3m
2008-06-25 23:21:29,234 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214448561806 in 623ms, sequence
id=14485701, 20.3m
2008-06-25 23:21:31,459 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214448561806 in 2223ms, sequence
id=14487619, 20.6m
2008-06-25 23:21:32,546 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214448561806 in 1079ms, sequence
id=14496978, 20.8m
2008-06-25 23:24:36,701 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214448561806 in 3432ms, sequence
id=15103902, 27.4m
hbase restarted
2008-06-25 23:28:18,288 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214448561806 in 1ms, sequence
id=15103936, 0.0
2008-06-25 23:28:19,151 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214454498285 in 0ms, sequence
id=15103937, 0.0
2008-06-25 23:45:43,021 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214454498285 in 14039ms, sequence
id=19210706, 23.6m
2008-06-25 23:45:43,285 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214454498285 in 264ms, sequence
id=19226471, 23.6m
2008-06-25 23:45:44,016 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214454498285 in 730ms, sequence
id=19227141, 23.6m
2008-06-25 23:45:45,481 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214454498285 in 1464ms, sequence
id=19229633, 23.7m
2008-06-25 23:45:46,474 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214454498285 in 990ms, sequence
id=19234660, 23.8m
2008-06-25 23:47:56,208 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region
webdata,com.wallbuilders.www%2FLIBissuesArticles.asp%3Fid%3D45%3Ahttp467123 in
759ms, sequence id=19961148, 21.7m
2008-06-25 23:55:07,552 DEBUG org.apache.hadoop.hbase.regionserver.HRegion:
Finished memcache flush for region webdata,,1214454498285 in 5795ms, sequence
id=22333011, 41.4m
hbase restarted
{code}
> memcache size unreliable
> ------------------------
>
> Key: HBASE-674
> URL: https://issues.apache.org/jira/browse/HBASE-674
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.1.2
> Reporter: stack
> Attachments: 674-v2.patch, 674.patch
>
>
> Multiple updates against same row/column/ts will be seen as increments to
> cache size on insert but when we then play the memcache at flush time, we'll
> only see the most recent entry and decrement the memcache size by whatever
> its size; memcache will be off.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
