[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973143#comment-14973143 ] Hudson commented on HBASE-14624: SUCCESS: Integrated in HBase-1.3-IT #271 (See [https://builds.apache.org/job/HBase-1.3-IT/271/]) HBASE-14624 BucketCache.freeBlock is too expensive (tedyu: rev 40887c94b715f58b016b12d0b2d9c80032d0262b) * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt, 14624-v2.txt, 14624-v3.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973116#comment-14973116 ] Hudson commented on HBASE-14624: FAILURE: Integrated in HBase-0.98-on-Hadoop-1.1 #1121 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/1121/]) HBASE-14624 BucketCache.freeBlock is too expensive (tedyu: rev f2150149669fc236e122e1e165584e9561714177) * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt, 14624-v2.txt, 14624-v3.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973053#comment-14973053 ] Hudson commented on HBASE-14624: SUCCESS: Integrated in HBase-TRUNK #6957 (See [https://builds.apache.org/job/HBase-TRUNK/6957/]) HBASE-14624 BucketCache.freeBlock is too expensive (tedyu: rev 95978477f8565defb3c4792f901e833abf63f768) * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt, 14624-v2.txt, 14624-v3.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973018#comment-14973018 ] Hudson commented on HBASE-14624: SUCCESS: Integrated in HBase-1.2-IT #242 (See [https://builds.apache.org/job/HBase-1.2-IT/242/]) HBASE-14624 BucketCache.freeBlock is too expensive (tedyu: rev 5958692892ec89653c7b6326f5f35eb60abef714) * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt, 14624-v2.txt, 14624-v3.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973012#comment-14973012 ] Hudson commented on HBASE-14624: FAILURE: Integrated in HBase-1.3 #305 (See [https://builds.apache.org/job/HBase-1.3/305/]) HBASE-14624 BucketCache.freeBlock is too expensive (tedyu: rev 40887c94b715f58b016b12d0b2d9c80032d0262b) * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt, 14624-v2.txt, 14624-v3.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14973004#comment-14973004 ] Hudson commented on HBASE-14624: FAILURE: Integrated in HBase-1.1 #722 (See [https://builds.apache.org/job/HBase-1.1/722/]) HBASE-14624 BucketCache.freeBlock is too expensive (tedyu: rev 5625bd01298d90ba69f295219824c5b3c4abcd2b) * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt, 14624-v2.txt, 14624-v3.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14972986#comment-14972986 ] Hudson commented on HBASE-14624: FAILURE: Integrated in HBase-1.0 #1099 (See [https://builds.apache.org/job/HBase-1.0/1099/]) HBASE-14624 BucketCache.freeBlock is too expensive (tedyu: rev e7a4e9fa63512e9f9dac4ed5e1886074d077aa4e) * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt, 14624-v2.txt, 14624-v3.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14972985#comment-14972985 ] Hudson commented on HBASE-14624: FAILURE: Integrated in HBase-0.98 #1167 (See [https://builds.apache.org/job/HBase-0.98/1167/]) HBASE-14624 BucketCache.freeBlock is too expensive (tedyu: rev f2150149669fc236e122e1e165584e9561714177) * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt, 14624-v2.txt, 14624-v3.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14972983#comment-14972983 ] Hudson commented on HBASE-14624: FAILURE: Integrated in HBase-1.2 #304 (See [https://builds.apache.org/job/HBase-1.2/304/]) HBASE-14624 BucketCache.freeBlock is too expensive (tedyu: rev 5958692892ec89653c7b6326f5f35eb60abef714) * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketAllocator.java > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt, 14624-v2.txt, 14624-v3.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14972860#comment-14972860 ] Andrew Purtell commented on HBASE-14624: +1 > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox >Assignee: Ted Yu > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt, 14624-v2.txt, 14624-v3.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[
https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14972166#comment-14972166
]
Hadoop QA commented on HBASE-14624:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12768392/14624-v3.txt
against master branch at commit f34011860e04fd1261b3b30cab0265016a22faa2.
ATTACHMENT ID: 12768392
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16201//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16201//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16201//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16201//console
This message is automatically generated.
> BucketCache.freeBlock is too expensive
> --
>
> Key: HBASE-14624
> URL: https://issues.apache.org/jira/browse/HBASE-14624
> Project: HBase
> Issue Type: Bug
> Components: BlockCache
>Affects Versions: 1.0.0
>Reporter: Randy Fox
>Assignee: Ted Yu
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: 14624-v1.txt, 14624-v2.txt, 14624-v3.txt
>
>
> Moving regions is unacceptably slow when using bucket cache, as it takes too
> long to free all the blocks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971618#comment-14971618 ] Andrew Purtell commented on HBASE-14624: Thanks for the note about ArrayDeque. Note I didn't suggest that specifically. I'm sure you can find a way to do this without allocating an iterator just to remove a head or tail element. That seems to me the one outstanding point (from myself and Anoop), then we're good here > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt, 14624-v2.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971608#comment-14971608 ] Ted Yu commented on HBASE-14624: w.r.t. Deque, we have the following choices: java.util.ArrayDeque java.util.concurrent.ConcurrentLinkedDeque >From http://docs.oracle.com/javase/7/docs/api/java/util/ArrayDeque.html : Most ArrayDeque operations run in amortized constant time. Exceptions include remove, removeFirstOccurrence, removeLastOccurrence, contains, iterator.remove(), and the bulk operations, all of which run in linear time. So ArrayDeque can be ruled out. > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt, 14624-v2.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971587#comment-14971587 ] Randy Fox commented on HBASE-14624: --- As a baseline (and to install the patch) I did a rolling restart of one RS and it took 7 minutes to move the regions. i waited a few hours for bucket cache to fill up and then moved all the regions off and it took 20 seconds or so and I verified the buckets freed up. > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt, 14624-v2.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971571#comment-14971571 ] Andrew Purtell commented on HBASE-14624: Please don't commit yet Ted. > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt, 14624-v2.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971569#comment-14971569 ] Andrew Purtell commented on HBASE-14624: bq. Seems we can not avoid an extra object creation (Iterator) with every call ! Noticed this also. Use an alternative to Set where you can pull elements from the head or tail without allocating a throwaway iterator just to effectively do the same. Maybe one of the implementations of Deque? > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt, 14624-v2.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971549#comment-14971549 ] stack commented on HBASE-14624: --- Any measurement on improvement? How we know this better especially given it is doing: b = freeBuckets.iterator().next(); > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt, 14624-v2.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14971539#comment-14971539 ] Ted Yu commented on HBASE-14624: Any more comment ? If not, planning to commit later today. > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt, 14624-v2.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970847#comment-14970847 ] Anoop Sam John commented on HBASE-14624: Seems we can not avoid an extra object creation (Iterator) with every call ! > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt, 14624-v2.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970845#comment-14970845 ] Anoop Sam John commented on HBASE-14624: I see.. from freeBuckets we were getting last entry and down from completelyFreeBuckets the 1st one.. Did not go into details and this diff got my mind doubtful. +1 then. > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt, 14624-v2.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970671#comment-14970671 ] Ted Yu commented on HBASE-14624: I agree with Yu and Chunhui's comments above. > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt, 14624-v2.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970577#comment-14970577 ] Yu Li commented on HBASE-14624: --- Agree. There will be a succeeding remove from the free buckets after block allocated, and previously get from last index should be a cost optimization for remove op in ArrayList. Now that we change to use set, the order no longer matters. > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt, 14624-v2.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[
https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970567#comment-14970567
]
chunhui shen commented on HBASE-14624:
--
{code}
- b = freeBuckets.get(freeBuckets.size() - 1);
+ b = freeBuckets.iterator().next();
{code}
In the context, the first entry and the last entry are both ok.
> BucketCache.freeBlock is too expensive
> --
>
> Key: HBASE-14624
> URL: https://issues.apache.org/jira/browse/HBASE-14624
> Project: HBase
> Issue Type: Bug
> Components: BlockCache
>Affects Versions: 1.0.0
>Reporter: Randy Fox
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: 14624-v1.txt, 14624-v2.txt
>
>
> Moving regions is unacceptably slow when using bucket cache, as it takes too
> long to free all the blocks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970561#comment-14970561 ] Hiroshi Ikeda commented on HBASE-14624: --- LinkedMap in commons collection for now? > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt, 14624-v2.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970495#comment-14970495 ] Anoop Sam John commented on HBASE-14624: Yep > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt, 14624-v2.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[
https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970481#comment-14970481
]
Hadoop QA commented on HBASE-14624:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12768198/14624-v2.txt
against master branch at commit eb4f9b8b32cf0551088a342cfa9b46b20593ac27.
ATTACHMENT ID: 12768198
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 checkstyle{color}. The applied patch does not increase the
total number of checkstyle errors
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn post-site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16188//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16188//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16188//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16188//console
This message is automatically generated.
> BucketCache.freeBlock is too expensive
> --
>
> Key: HBASE-14624
> URL: https://issues.apache.org/jira/browse/HBASE-14624
> Project: HBase
> Issue Type: Bug
> Components: BlockCache
>Affects Versions: 1.0.0
>Reporter: Randy Fox
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: 14624-v1.txt, 14624-v2.txt
>
>
> Moving regions is unacceptably slow when using bucket cache, as it takes too
> long to free all the blocks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970478#comment-14970478 ] stack commented on HBASE-14624: --- Sounds like we need a test (to ensure order preserved?) > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt, 14624-v2.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970470#comment-14970470 ] Anoop Sam John commented on HBASE-14624: LinkedHashSet can preserve order which we want to use here. Can that be used for the patch? > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt, 14624-v2.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970434#comment-14970434 ] Ted Yu commented on HBASE-14624: There is no reverse iterator provided by HashSet Do you have suggestion(s) ? Thanks > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt, 14624-v2.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[
https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970426#comment-14970426
]
Anoop Sam John commented on HBASE-14624:
LinkedList changed to HashSet
{code}
- b = freeBuckets.get(freeBuckets.size() - 1);
+ b = freeBuckets.iterator().next();
{code}
Is this order correct.. We were getting the last added entry and here it wont
be that? How you handle the order?
> BucketCache.freeBlock is too expensive
> --
>
> Key: HBASE-14624
> URL: https://issues.apache.org/jira/browse/HBASE-14624
> Project: HBase
> Issue Type: Bug
> Components: BlockCache
>Affects Versions: 1.0.0
>Reporter: Randy Fox
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: 14624-v1.txt, 14624-v2.txt
>
>
> Moving regions is unacceptably slow when using bucket cache, as it takes too
> long to free all the blocks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[
https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970302#comment-14970302
]
Hadoop QA commented on HBASE-14624:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12768019/14624-v1.txt
against master branch at commit 36d4a5a650ceead3be5012cdd25dc35efa383755.
ATTACHMENT ID: 12768019
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop versions{color}. The patch compiles with all
supported hadoop versions (2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.0 2.6.1 2.7.0
2.7.1)
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 protoc{color}. The applied patch does not increase the
total number of protoc compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:red}-1 checkstyle{color}. The applied patch generated
1734 checkstyle errors (more than the master's current 1732 errors).
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn post-site goal
to fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/16185//testReport/
Release Findbugs (version 2.0.3)warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/16185//artifact/patchprocess/newFindbugsWarnings.html
Checkstyle Errors:
https://builds.apache.org/job/PreCommit-HBASE-Build/16185//artifact/patchprocess/checkstyle-aggregate.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/16185//console
This message is automatically generated.
> BucketCache.freeBlock is too expensive
> --
>
> Key: HBASE-14624
> URL: https://issues.apache.org/jira/browse/HBASE-14624
> Project: HBase
> Issue Type: Bug
> Components: BlockCache
>Affects Versions: 1.0.0
>Reporter: Randy Fox
> Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16
>
> Attachments: 14624-v1.txt
>
>
> Moving regions is unacceptably slow when using bucket cache, as it takes too
> long to free all the blocks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970138#comment-14970138 ] Randy Fox commented on HBASE-14624: --- yes > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970128#comment-14970128 ] Elliott Clark commented on HBASE-14624: --- Is that with evict on close set to true ? > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Bug > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox > Fix For: 2.0.0, 1.2.0, 1.3.0, 1.0.3, 1.1.3, 0.98.16 > > Attachments: 14624-v1.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14970070#comment-14970070 ] Randy Fox commented on HBASE-14624: --- The test passed with flying colors. Regions close super fast with this change. > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Improvement > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox > Attachments: 14624-v1.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14969419#comment-14969419 ] Randy Fox commented on HBASE-14624: --- The region that took 3 minutes to close was 1.4G compressed. The evict blocks on close was not working (https://issues.apache.org/jira/browse/HBASE-14663). The patch for it seems to solve the problem. I can give the above change a try. > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Improvement > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox > Attachments: 14624-v1.txt > > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14968699#comment-14968699 ] Anoop Sam John commented on HBASE-14624: [~randy.fox] Any chance for trying out a patch with above suggestion and test? > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Improvement > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[
https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14968599#comment-14968599
]
chunhui shen commented on HBASE-14624:
--
What 's the size of region ?
You could set EVICT_BLOCKS_ON_CLOSE to false, thus freeBlocks won't be called
when moving region.
About code improvement：
final class BucketSizeInfo {
private List bucketList, freeBuckets, completelyFreeBuckets;
The above Lists shoule be changed as HashSet to improve performance, since
contains and removes operation is expensive for List
> BucketCache.freeBlock is too expensive
> --
>
> Key: HBASE-14624
> URL: https://issues.apache.org/jira/browse/HBASE-14624
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache
>Affects Versions: 1.0.0
>Reporter: Randy Fox
>
> Moving regions is unacceptably slow when using bucket cache, as it takes too
> long to free all the blocks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14963477#comment-14963477 ] Randy Fox commented on HBASE-14624: --- There are no logs for that. Deduced it from the code and that fact that when i added a table to cache it also started taking along time to move when it was previously quick. Vladimir Rodionov suggested i open this Jira after discussions on the maillist. > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Improvement > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14963471#comment-14963471 ] Anoop Sam John commented on HBASE-14624: Not able to see from log that bucket cache free call takes time.. Missing some logs? > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Improvement > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14963452#comment-14963452 ] Randy Fox commented on HBASE-14624: --- The region was about 1.4G compressed and it took about 2.5 minutes to move. The bucketcache is configed at 72G, but was not close to full. The bucket sizes: 9216,17408,33792,66560 2015-10-15 08:34:28,510 INFO org.apache.hadoop.hbase.regionserver.RSRpcServices: Close dad6c71ed395df19220ef1056a110086, moving to hb17.prod1.connexity.net,60020,125924021 2015-10-15 08:34:28,510 DEBUG org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler: Processing close of Wildfire_graph3,\x00)"y\x1B\xF0\x13-,1434402364692.dad6c71ed395df19220ef1056a110086. 2015-10-15 08:34:28,511 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Closing Wildfire_graph3,\x00)"y\x1B\xF0\x13-,1434402364692.dad6c71ed395df19220ef1056a110086.: disabling compactions & flushes 2015-10-15 08:34:28,511 INFO org.apache.hadoop.hbase.regionserver.HRegion: Running close preflush of Wildfire_graph3,\x00)"y\x1B\xF0\x13-,1434402364692.dad6c71ed395df19220ef1056a110086. 2015-10-15 08:34:28,511 INFO org.apache.hadoop.hbase.regionserver.HRegion: Started memstore flush for Wildfire_graph3,\x00)"y\x1B\xF0\x13-,1434402364692.dad6c71ed395df19220ef1056a110086., current region memstore size 44.89 MB, and 1/1 column families' memstores are being flushed. 2015-10-15 08:34:28,511 WARN org.apache.hadoop.hbase.regionserver.wal.FSHLog: Couldn't find oldest seqNum for the region we are about to flush: [dad6c71ed395df19220ef1056a110086] 2015-10-15 08:34:29,137 INFO org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher: Flushed, sequenceid=70839856845, memsize=44.9 M, hasBloomFilter=false, into tmp file hdfs://woz/hbase/data/default/Wildfire_graph3/dad6c71ed395df19220ef1056a110086/.tmp/3600b47839a945de9733cf17581458e0 2015-10-15 08:34:29,144 DEBUG org.apache.hadoop.hbase.regionserver.HRegionFileSystem: Committing store file hdfs://woz/hbase/data/default/Wildfire_graph3/dad6c71ed395df19220ef1056a110086/.tmp/3600b47839a945de9733cf17581458e0 as hdfs://woz/hbase/data/default/Wildfire_graph3/dad6c71ed395df19220ef1056a110086/L/3600b47839a945de9733cf17581458e0 2015-10-15 08:34:29,150 INFO org.apache.hadoop.hbase.regionserver.HStore: Added hdfs://woz/hbase/data/default/Wildfire_graph3/dad6c71ed395df19220ef1056a110086/L/3600b47839a945de9733cf17581458e0, entries=190131, sequenceid=70839856845, filesize=3.2 M 2015-10-15 08:34:29,151 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~44.89 MB/47066832, currentsize=22.39 KB/22928 for region Wildfire_graph3,\x00)"y\x1B\xF0\x13-,1434402364692.dad6c71ed395df19220ef1056a110086. in 640ms, sequenceid=70839856845, compaction requested=false 2015-10-15 08:34:29,152 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: Updates disabled for region Wildfire_graph3,\x00)"y\x1B\xF0\x13-,1434402364692.dad6c71ed395df19220ef1056a110086. 2015-10-15 08:34:29,152 INFO org.apache.hadoop.hbase.regionserver.HRegion: Started memstore flush for Wildfire_graph3,\x00)"y\x1B\xF0\x13-,1434402364692.dad6c71ed395df19220ef1056a110086., current region memstore size 22.39 KB, and 1/1 column families' memstores are being flushed. 2015-10-15 08:34:29,152 WARN org.apache.hadoop.hbase.regionserver.wal.FSHLog: Couldn't find oldest seqNum for the region we are about to flush: [dad6c71ed395df19220ef1056a110086] 2015-10-15 08:34:29,225 INFO org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher: Flushed, sequenceid=70839856876, memsize=22.4 K, hasBloomFilter=false, into tmp file hdfs://woz/hbase/data/default/Wildfire_graph3/dad6c71ed395df19220ef1056a110086/.tmp/a8a74549c7a44174b105cbed23cfaea1 2015-10-15 08:34:29,231 DEBUG org.apache.hadoop.hbase.regionserver.HRegionFileSystem: Committing store file hdfs://woz/hbase/data/default/Wildfire_graph3/dad6c71ed395df19220ef1056a110086/.tmp/a8a74549c7a44174b105cbed23cfaea1 as hdfs://woz/hbase/data/default/Wildfire_graph3/dad6c71ed395df19220ef1056a110086/L/a8a74549c7a44174b105cbed23cfaea1 2015-10-15 08:34:29,279 INFO org.apache.hadoop.hbase.regionserver.HStore: Added hdfs://woz/hbase/data/default/Wildfire_graph3/dad6c71ed395df19220ef1056a110086/L/a8a74549c7a44174b105cbed23cfaea1, entries=128, sequenceid=70839856876, filesize=2.6 K 2015-10-15 08:34:29,280 INFO org.apache.hadoop.hbase.regionserver.HRegion: Finished memstore flush of ~22.39 KB/22928, currentsize=0 B/0 for region Wildfire_graph3,\x00)"y\x1B\xF0\x13-,1434402364692.dad6c71ed395df19220ef1056a110086. in 128ms, sequenceid=70839856876, compaction requested=true ... 2015-10-15 08:37:02,945 INFO org.apache.hadoop.hbase.regionserver.HStore: Closed L 2015-10-15 08:37:02,954 DEBUG org.apache.hadoop.hbase.wal.WALSplitter: Wrote region seqId=hdfs://woz/hbase/data/default/Wildfire_graph3/dad6c71ed395df19220ef1056a110086/recovered.edits/7083985
[jira] [Commented] (HBASE-14624) BucketCache.freeBlock is too expensive
[ https://issues.apache.org/jira/browse/HBASE-14624?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14959734#comment-14959734 ] stack commented on HBASE-14624: --- Can we have more detail [~randy.fox]? What you see in log? How big the bucketcache if we are to repro? Thanks. > BucketCache.freeBlock is too expensive > -- > > Key: HBASE-14624 > URL: https://issues.apache.org/jira/browse/HBASE-14624 > Project: HBase > Issue Type: Improvement > Components: BlockCache >Affects Versions: 1.0.0 >Reporter: Randy Fox > > Moving regions is unacceptably slow when using bucket cache, as it takes too > long to free all the blocks. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
