[jira] [Updated] (HBASE-7160) Improve IdLock and remove its minor defects
[ https://issues.apache.org/jira/browse/HBASE-7160?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hiroshi Ikeda updated HBASE-7160: - Attachment: HBASE-7160-V2.patch Patch v2 from review board. > Improve IdLock and remove its minor defects > --- > > Key: HBASE-7160 > URL: https://issues.apache.org/jira/browse/HBASE-7160 > Project: HBase > Issue Type: Improvement >Reporter: Hiroshi Ikeda >Assignee: Hiroshi Ikeda >Priority: Minor > Attachments: HBASE-7160.patch, HBASE-7160-V2.patch > > > Combination of synchronizations and concurrent collections complicates the > code, and it is hard to trace the code and to confirm its correctness. We > should re-create the class and make it more understandable. > In the current code, I find the following minor defects: > (1) In the case that there is a waiting thread for a lock in getLockEntry() > and another thread is releasing the lock by calling releaseLockEntry(), > trying to get the lock with a 3rd thread by calling getLockEntry() falls into > a busy loop until the waiting thread wakes up and gets the lock. > Even if notify() wakes up the blocked thread and causes a context switch to > the waked thread immediately, synchronization might block the waked thread > and cause another context switch, and the busy loop might continue for a > while. > (2) In the same case as (1), since releasing the lock is merely notifying > without removing the lock-entry from the map, interrupting the waiting thread > might leave an unused lock-entry (entry.numWaiters == 0) in the map. This is > a memory leak unless the id of the lock is used again. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7183) print WARN message if hbase.replication.sizeOfLogQueue is too big
Sho Shimauchi created HBASE-7183: Summary: print WARN message if hbase.replication.sizeOfLogQueue is too big Key: HBASE-7183 URL: https://issues.apache.org/jira/browse/HBASE-7183 Project: HBase Issue Type: Improvement Components: Replication Reporter: Sho Shimauchi A metric hbase.replication.sizeOfLogQueue may become big when replication is delaying. It would be useful if HBase prints WARN log which tells hbase.replication.sizeOfLogQueue is too big. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7160) Improve IdLock and remove its minor defects
[
https://issues.apache.org/jira/browse/HBASE-7160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1344#comment-1344
]
Hiroshi Ikeda commented on HBASE-7160:
--
{quote}
Looking at Entry.release():
+public void release() {
+ if (released.compareAndSet(false, true)) {
Would this do the trick ?
{quote}
Calling compareAndSet() in the Entry.release() is a mere safety net.
Otherwise multiple calling Entry.release() can break the exclusiveness of
IdLock.
> Improve IdLock and remove its minor defects
> ---
>
> Key: HBASE-7160
> URL: https://issues.apache.org/jira/browse/HBASE-7160
> Project: HBase
> Issue Type: Improvement
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: HBASE-7160.patch
>
>
> Combination of synchronizations and concurrent collections complicates the
> code, and it is hard to trace the code and to confirm its correctness. We
> should re-create the class and make it more understandable.
> In the current code, I find the following minor defects:
> (1) In the case that there is a waiting thread for a lock in getLockEntry()
> and another thread is releasing the lock by calling releaseLockEntry(),
> trying to get the lock with a 3rd thread by calling getLockEntry() falls into
> a busy loop until the waiting thread wakes up and gets the lock.
> Even if notify() wakes up the blocked thread and causes a context switch to
> the waked thread immediately, synchronization might block the waked thread
> and cause another context switch, and the busy loop might continue for a
> while.
> (2) In the same case as (1), since releasing the lock is merely notifying
> without removing the lock-entry from the map, interrupting the waiting thread
> might leave an unused lock-entry (entry.numWaiters == 0) in the map. This is
> a memory leak unless the id of the lock is used again.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7160) Improve IdLock and remove its minor defects
[
https://issues.apache.org/jira/browse/HBASE-7160?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1343#comment-1343
]
Hiroshi Ikeda commented on HBASE-7160:
--
bq. Hiroshi Ikeda This is interesting. There is indeed busy waiting while an
Entry is being unlocked. On the other hand I find it hard to prove by visual
inspection that the new code has no such busy waiting.
The following pattern is commonly used for Atomic* for the purpose of
non-blocking:
{code}
while(true) {
value = valueReference.get();
// some light code
if(valueReference.compareAndSet(value, nextValue)) {
break;
}
}
{code}
and the above "if" condition is true and breaks the infinite loop if other
threads don't interfere in the valueReference while executing "some light code".
If other threads interfere we have to start over the loop, (and we cannot
prevent it because of non-blocking even if the "some light code" is very
small,) but in practice we can ignore the situation that other threads continue
to interfere and we have to loop many times.
The new code merely combines sort of the above pattern.
bq. Are you planning some multithreaded performance testing with the patch to
demonstrate that it is better?
Sorry, I have no idea how to do it.
> Improve IdLock and remove its minor defects
> ---
>
> Key: HBASE-7160
> URL: https://issues.apache.org/jira/browse/HBASE-7160
> Project: HBase
> Issue Type: Improvement
>Reporter: Hiroshi Ikeda
>Assignee: Hiroshi Ikeda
>Priority: Minor
> Attachments: HBASE-7160.patch
>
>
> Combination of synchronizations and concurrent collections complicates the
> code, and it is hard to trace the code and to confirm its correctness. We
> should re-create the class and make it more understandable.
> In the current code, I find the following minor defects:
> (1) In the case that there is a waiting thread for a lock in getLockEntry()
> and another thread is releasing the lock by calling releaseLockEntry(),
> trying to get the lock with a 3rd thread by calling getLockEntry() falls into
> a busy loop until the waiting thread wakes up and gets the lock.
> Even if notify() wakes up the blocked thread and causes a context switch to
> the waked thread immediately, synchronization might block the waked thread
> and cause another context switch, and the busy loop might continue for a
> while.
> (2) In the same case as (1), since releasing the lock is merely notifying
> without removing the lock-entry from the map, interrupting the waiting thread
> might leave an unused lock-entry (entry.numWaiters == 0) in the map. This is
> a memory leak unless the id of the lock is used again.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6787) Convert RowProcessorProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499983#comment-13499983 ] Devaraj Das commented on HBASE-6787: Test TestRegionServerCoprocessorExceptionWithAbort passes locally. > Convert RowProcessorProtocol to protocol buffer service > --- > > Key: HBASE-6787 > URL: https://issues.apache.org/jira/browse/HBASE-6787 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Gary Helmling >Assignee: Devaraj Das > Fix For: 0.96.0 > > Attachments: 6787-1.patch, 6787-2.patch, 6787-3.patch, 6787-4.patch, > 6787-5.patch, 6787-6.patch > > > With coprocessor endpoints now exposed as protobuf defined services, we > should convert over all of our built-in endpoints to PB services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6787) Convert RowProcessorProtocol to protocol buffer service
[
https://issues.apache.org/jira/browse/HBASE-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499981#comment-13499981
]
Hadoop QA commented on HBASE-6787:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12554075/6787-6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
98 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 24 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3362//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3362//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3362//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3362//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3362//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3362//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3362//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3362//console
This message is automatically generated.
> Convert RowProcessorProtocol to protocol buffer service
> ---
>
> Key: HBASE-6787
> URL: https://issues.apache.org/jira/browse/HBASE-6787
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors
>Reporter: Gary Helmling
>Assignee: Devaraj Das
> Fix For: 0.96.0
>
> Attachments: 6787-1.patch, 6787-2.patch, 6787-3.patch, 6787-4.patch,
> 6787-5.patch, 6787-6.patch
>
>
> With coprocessor endpoints now exposed as protobuf defined services, we
> should convert over all of our built-in endpoints to PB services.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
[
https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499973#comment-13499973
]
Hadoop QA commented on HBASE-6580:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12554094/HBASE-6580_v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
99 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 24 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestCheckTestClasses
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3360//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3360//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3360//console
This message is automatically generated.
> New HTable pool, based on HBase(byte[], HConnection, ExecutorService)
> constructor
> -
>
> Key: HBASE-6580
> URL: https://issues.apache.org/jira/browse/HBASE-6580
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 0.92.2, 0.94.2
>Reporter: Lars Hofhansl
>Priority: Minor
> Attachments: HBASE-6580_v1.patch
>
>
> Here I propose a very simple TablePool.
> It could be called LightHTablePool (or something - if you have a better name).
> Internally it would maintain an HConnection and an Executor service and each
> invocation of getTable(...) would create a new HTable and close() would just
> close it.
> In testing I find this more light weight than HTablePool and easier to
> monitor in terms of resources used.
> It would hardly be more than a few dozen lines of code.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6423) Writes should not block reads on blocking updates to memstores
[
https://issues.apache.org/jira/browse/HBASE-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499972#comment-13499972
]
Hadoop QA commented on HBASE-6423:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12554076/trunk-6423.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
99 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 22 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.io.TestHeapSize
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3361//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3361//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3361//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3361//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3361//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3361//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3361//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3361//console
This message is automatically generated.
> Writes should not block reads on blocking updates to memstores
> --
>
> Key: HBASE-6423
> URL: https://issues.apache.org/jira/browse/HBASE-6423
> Project: HBase
> Issue Type: Bug
>Reporter: Karthik Ranganathan
>Assignee: Amitanand Aiyer
> Attachments: trunk-6423.patch
>
>
> We have a big data use case where we turn off WAL and have a ton of reads and
> writes. We found that:
> 1. flushing a memstore takes a while (GZIP compression)
> 2. incoming writes cause the new memstore to grow in an unbounded fashion
> 3. this triggers blocking memstore updates
> 4. in turn, this causes all the RPC handler threads to block on writes to
> that memstore
> 5. we are not able to read during this time as RPC handlers are blocked
> At a higher level, we should not hold up the RPC threads while blocking
> updates, and we should build in some sort of rate control.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6580) New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499961#comment-13499961 ] Lars Hofhansl commented on HBASE-6580: -- That would work. Ideally I'd like to get HTablePool out of the mix. It's heavy weight and not needed (and in fact just adds overhead) if the both the connection and thread pool are shared between HTables. > New HTable pool, based on HBase(byte[], HConnection, ExecutorService) > constructor > - > > Key: HBASE-6580 > URL: https://issues.apache.org/jira/browse/HBASE-6580 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.92.2, 0.94.2 >Reporter: Lars Hofhansl >Priority: Minor > Attachments: HBASE-6580_v1.patch > > > Here I propose a very simple TablePool. > It could be called LightHTablePool (or something - if you have a better name). > Internally it would maintain an HConnection and an Executor service and each > invocation of getTable(...) would create a new HTable and close() would just > close it. > In testing I find this more light weight than HTablePool and easier to > monitor in terms of resources used. > It would hardly be more than a few dozen lines of code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7108) Don't use legal family name for system folder at region level
[ https://issues.apache.org/jira/browse/HBASE-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499948#comment-13499948 ] stack commented on HBASE-7108: -- Todd's suggestion sounds good to me. > Don't use legal family name for system folder at region level > - > > Key: HBASE-7108 > URL: https://issues.apache.org/jira/browse/HBASE-7108 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.92.2, 0.94.2, 0.96.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 0.96.0 > > Attachments: HBASE-7108-v0.patch > > > CHANGED, was: Don't allow "recovered.edits" as legal family name > Region directories can contain folders called "recovered.edits", log > splitting related. > But there's nothing that prevent a user to create a family with that name... > HLog.RECOVERED_EDITS_DIR = "recovered.edits"; > HRegion.MERGEDIR = "merges"; // fixed with HBASE-6158 > SplitTransaction.SPLITDIR = "splits"; // fixed with HBASE-6158 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6580) New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrian Muraru updated HBASE-6580: - Attachment: HBASE-6580_v1.patch > New HTable pool, based on HBase(byte[], HConnection, ExecutorService) > constructor > - > > Key: HBASE-6580 > URL: https://issues.apache.org/jira/browse/HBASE-6580 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.92.2, 0.94.2 >Reporter: Lars Hofhansl >Priority: Minor > Attachments: HBASE-6580_v1.patch > > > Here I propose a very simple TablePool. > It could be called LightHTablePool (or something - if you have a better name). > Internally it would maintain an HConnection and an Executor service and each > invocation of getTable(...) would create a new HTable and close() would just > close it. > In testing I find this more light weight than HTablePool and easier to > monitor in terms of resources used. > It would hardly be more than a few dozen lines of code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4805) Allow better control of resource consumption in HTable
[ https://issues.apache.org/jira/browse/HBASE-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrian Muraru updated HBASE-4805: - Attachment: (was: HBASE-6580_v1.patch) > Allow better control of resource consumption in HTable > -- > > Key: HBASE-4805 > URL: https://issues.apache.org/jira/browse/HBASE-4805 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.92.0, 0.94.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.92.0, 0.94.0 > > Attachments: 4805.txt, 4805-v2.txt, 4805-v3.txt, 4805-v4.txt, > 4805-v5.txt > > > From some internal discussions at Salesforce we concluded that we need better > control over the resources (mostly threads) consumed by HTable when used in a > AppServer with many client threads. > Since HTable is not thread safe, the only options are cache them (in a custom > thread local or using HTablePool) or to create them on-demand. > I propose a simple change: Add a new constructor to HTable that takes an > optional ExecutorService and HConnection instance. That would make HTable a > pretty lightweight object and we would manage the ES and HC separately. > I'll upload a patch a soon to get some feedback. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4805) Allow better control of resource consumption in HTable
[ https://issues.apache.org/jira/browse/HBASE-4805?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrian Muraru updated HBASE-4805: - Attachment: HBASE-6580_v1.patch > Allow better control of resource consumption in HTable > -- > > Key: HBASE-4805 > URL: https://issues.apache.org/jira/browse/HBASE-4805 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.92.0, 0.94.0 >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.92.0, 0.94.0 > > Attachments: 4805.txt, 4805-v2.txt, 4805-v3.txt, 4805-v4.txt, > 4805-v5.txt, HBASE-6580_v1.patch > > > From some internal discussions at Salesforce we concluded that we need better > control over the resources (mostly threads) consumed by HTable when used in a > AppServer with many client threads. > Since HTable is not thread safe, the only options are cache them (in a custom > thread local or using HTablePool) or to create them on-demand. > I propose a simple change: Add a new constructor to HTable that takes an > optional ExecutorService and HConnection instance. That would make HTable a > pretty lightweight object and we would manage the ES and HC separately. > I'll upload a patch a soon to get some feedback. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6580) New HTable pool, based on HBase(byte[], HConnection, ExecutorService) constructor
[ https://issues.apache.org/jira/browse/HBASE-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adrian Muraru updated HBASE-6580: - Affects Version/s: 0.92.2 0.94.2 Release Note: New HTableInterfaceFactory implementation using shared ExecutorService for all HTable instances managed by HTablePool Status: Patch Available (was: Open) This is something we need in production as well. Digging deeper I see the underlying HConnection is already shared in the current HTablePool impl. (i.e there is a single, private Configuration instance passed when HTablePool is initialised and used for all HTable instances built by pool). What is missing is a shared ExecutorService for all htable's so I called it SharedExecutorHTableFactory Patch attached (it's on hbase-trunk but I can easily port to 0.92 and 0.94) > New HTable pool, based on HBase(byte[], HConnection, ExecutorService) > constructor > - > > Key: HBASE-6580 > URL: https://issues.apache.org/jira/browse/HBASE-6580 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.94.2, 0.92.2 >Reporter: Lars Hofhansl >Priority: Minor > > Here I propose a very simple TablePool. > It could be called LightHTablePool (or something - if you have a better name). > Internally it would maintain an HConnection and an Executor service and each > invocation of getTable(...) would create a new HTable and close() would just > close it. > In testing I find this more light weight than HTablePool and easier to > monitor in terms of resources used. > It would hardly be more than a few dozen lines of code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7180) RegionScannerImpl.next() is inefficient.
[ https://issues.apache.org/jira/browse/HBASE-7180?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499911#comment-13499911 ] Lars Hofhansl commented on HBASE-7180: -- With this patch a RegionObserver wrapping a RegionScanner in preScannerOpen can handle the readpoint and start/close region operation itself and then call the cheaper next() on the wrapped scanner when needed. > RegionScannerImpl.next() is inefficient. > > > Key: HBASE-7180 > URL: https://issues.apache.org/jira/browse/HBASE-7180 > Project: HBase > Issue Type: Bug >Reporter: Lars Hofhansl > Attachments: 7180-0.94-SKETCH.txt > > > We just came across a special scenario. > For our Phoenix project (SQL runtime for HBase), we push a lot of work into > HBase via coprocessors. One method is to wrap RegionScanner in coprocessor > hooks and then do processing in the hook to avoid returning a lot of data to > the client unnecessarily. > In this specific case this is pretty bad. Since the wrapped RegionScanner's > next() does not "know" that it is called this way is still does all of this > on each invocation: > # Starts a RegionOperation > # Increments the request count > # set the current read point on a thread local (because generally each call > could come from a different thread) > # Finally does the next on its StoreScanner(s) > # Ends the RegionOperation > When this is done in a tight loop millions of times (as is the case for us) > it starts to become significant. > Not sure what to do about this, really. Opening this issue for discussion. > One way is to extend the RegionScanner with an "internal" next() method of > sorts, so that all this overhead can be avoided. The coprocessor could call > the regular next() methods once and then just call the cheaper internal > version. > Are there better/cleaner ways? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4210) Allow coprocessor to interact with batches per region sent from a client(?)
[
https://issues.apache.org/jira/browse/HBASE-4210?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499868#comment-13499868
]
Asaf Mesika commented on HBASE-4210:
One perfect use case for this is
[https://issues.apache.org/jira/browse/HBASE-3434].
You can have a co-processor hook on preIncrementValue which converts a list of
Increment to a list of Put, and then have a co-processor hook on preGet which
aggregates them and serve them as one result. On the preCompact hook you can
aggregate them and increment the original KV.
Without the preMulti you are forced to write each Put to the WAL, thus not
enjoying the write once to WAL on a list of Increment. All caused by the fact
that you are only seeing one Increment at a time
> Allow coprocessor to interact with batches per region sent from a client(?)
> ---
>
> Key: HBASE-4210
> URL: https://issues.apache.org/jira/browse/HBASE-4210
> Project: HBase
> Issue Type: New Feature
>Reporter: Lars Hofhansl
>Priority: Minor
>
> Currently the coprocessor write hooks - {pre|post}{Put|Delete} - are strictly
> one row|cell operations.
> It might be a good idea to allow a coprocessor to deal with batches of puts
> and deletes as they arrive from the client.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6423) Writes should not block reads on blocking updates to memstores
[
https://issues.apache.org/jira/browse/HBASE-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499867#comment-13499867
]
Ted Yu commented on HBASE-6423:
---
Patch looks good.
{code}
+ if (timeToWait < 0L) {
+this.updatesBlockedMs.add(now - startTime);
+LOG.info("Failed to unblocking updates for region " + this + " '"
+ + Thread.currentThread().getName() + "' in time. The region is still
busy.");
+throw new RegionTooBusyException("region is flushing");
{code}
typo: to unblocking updates -> to unblock updates
Should we include -timeToWait, the amount of time we are unable to wait in the
exception message ?
> Writes should not block reads on blocking updates to memstores
> --
>
> Key: HBASE-6423
> URL: https://issues.apache.org/jira/browse/HBASE-6423
> Project: HBase
> Issue Type: Bug
>Reporter: Karthik Ranganathan
>Assignee: Amitanand Aiyer
> Attachments: trunk-6423.patch
>
>
> We have a big data use case where we turn off WAL and have a ton of reads and
> writes. We found that:
> 1. flushing a memstore takes a while (GZIP compression)
> 2. incoming writes cause the new memstore to grow in an unbounded fashion
> 3. this triggers blocking memstore updates
> 4. in turn, this causes all the RPC handler threads to block on writes to
> that memstore
> 5. we are not able to read during this time as RPC handlers are blocked
> At a higher level, we should not hold up the RPC threads while blocking
> updates, and we should build in some sort of rate control.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6787) Convert RowProcessorProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499857#comment-13499857 ] Devaraj Das commented on HBASE-6787: bq. Yeah.. that's true. Should we move this work to a different jira (where we make the relevant changes for the other coprocessor implementations as well). Raised HBASE-7182 for the interface addition. > Convert RowProcessorProtocol to protocol buffer service > --- > > Key: HBASE-6787 > URL: https://issues.apache.org/jira/browse/HBASE-6787 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Gary Helmling >Assignee: Devaraj Das > Fix For: 0.96.0 > > Attachments: 6787-1.patch, 6787-2.patch, 6787-3.patch, 6787-4.patch, > 6787-5.patch, 6787-6.patch > > > With coprocessor endpoints now exposed as protobuf defined services, we > should convert over all of our built-in endpoints to PB services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6423) Writes should not block reads on blocking updates to memstores
[ https://issues.apache.org/jira/browse/HBASE-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499856#comment-13499856 ] Jimmy Xiang commented on HBASE-6423: I attached a patch: trunk-6423.patch. It is consistent with current behavior. However, you can adjust the waiting time. > Writes should not block reads on blocking updates to memstores > -- > > Key: HBASE-6423 > URL: https://issues.apache.org/jira/browse/HBASE-6423 > Project: HBase > Issue Type: Bug >Reporter: Karthik Ranganathan >Assignee: Amitanand Aiyer > Attachments: trunk-6423.patch > > > We have a big data use case where we turn off WAL and have a ton of reads and > writes. We found that: > 1. flushing a memstore takes a while (GZIP compression) > 2. incoming writes cause the new memstore to grow in an unbounded fashion > 3. this triggers blocking memstore updates > 4. in turn, this causes all the RPC handler threads to block on writes to > that memstore > 5. we are not able to read during this time as RPC handlers are blocked > At a higher level, we should not hold up the RPC threads while blocking > updates, and we should build in some sort of rate control. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6423) Writes should not block reads on blocking updates to memstores
[ https://issues.apache.org/jira/browse/HBASE-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-6423: --- Status: Patch Available (was: Open) > Writes should not block reads on blocking updates to memstores > -- > > Key: HBASE-6423 > URL: https://issues.apache.org/jira/browse/HBASE-6423 > Project: HBase > Issue Type: Bug >Reporter: Karthik Ranganathan >Assignee: Amitanand Aiyer > Attachments: trunk-6423.patch > > > We have a big data use case where we turn off WAL and have a ton of reads and > writes. We found that: > 1. flushing a memstore takes a while (GZIP compression) > 2. incoming writes cause the new memstore to grow in an unbounded fashion > 3. this triggers blocking memstore updates > 4. in turn, this causes all the RPC handler threads to block on writes to > that memstore > 5. we are not able to read during this time as RPC handlers are blocked > At a higher level, we should not hold up the RPC threads while blocking > updates, and we should build in some sort of rate control. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6423) Writes should not block reads on blocking updates to memstores
[ https://issues.apache.org/jira/browse/HBASE-6423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jimmy Xiang updated HBASE-6423: --- Attachment: trunk-6423.patch > Writes should not block reads on blocking updates to memstores > -- > > Key: HBASE-6423 > URL: https://issues.apache.org/jira/browse/HBASE-6423 > Project: HBase > Issue Type: Bug >Reporter: Karthik Ranganathan >Assignee: Amitanand Aiyer > Attachments: trunk-6423.patch > > > We have a big data use case where we turn off WAL and have a ton of reads and > writes. We found that: > 1. flushing a memstore takes a while (GZIP compression) > 2. incoming writes cause the new memstore to grow in an unbounded fashion > 3. this triggers blocking memstore updates > 4. in turn, this causes all the RPC handler threads to block on writes to > that memstore > 5. we are not able to read during this time as RPC handlers are blocked > At a higher level, we should not hold up the RPC threads while blocking > updates, and we should build in some sort of rate control. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6787) Convert RowProcessorProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-6787: --- Attachment: 6787-6.patch Missed some getResult changes in the earlier patch. This patch removes the default implementation of getResult (that would return null), and instead adds getResult to the concrete rowprocessor implementations. Please review the patch. Once this patch is committed, I'll do similar PBfying of user code in the other coprocessor protocols. > Convert RowProcessorProtocol to protocol buffer service > --- > > Key: HBASE-6787 > URL: https://issues.apache.org/jira/browse/HBASE-6787 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Gary Helmling >Assignee: Devaraj Das > Fix For: 0.96.0 > > Attachments: 6787-1.patch, 6787-2.patch, 6787-3.patch, 6787-4.patch, > 6787-5.patch, 6787-6.patch > > > With coprocessor endpoints now exposed as protobuf defined services, we > should convert over all of our built-in endpoints to PB services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7181) Merge documentation improvement
[ https://issues.apache.org/jira/browse/HBASE-7181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-7181: --- Status: Patch Available (was: Open) > Merge documentation improvement > --- > > Key: HBASE-7181 > URL: https://issues.apache.org/jira/browse/HBASE-7181 > Project: HBase > Issue Type: Improvement > Components: documentation >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari >Priority: Minor > Labels: documentation, merge > Attachments: HBASE-7181.patch > > > There is one small issue on the documentation regarding the merge. The class > name is not correct. Also, it might be usefull to give an example of the > region format. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7181) Merge documentation improvement
[ https://issues.apache.org/jira/browse/HBASE-7181?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-7181: --- Attachment: HBASE-7181.patch Patch proposed for documentation update. > Merge documentation improvement > --- > > Key: HBASE-7181 > URL: https://issues.apache.org/jira/browse/HBASE-7181 > Project: HBase > Issue Type: Improvement > Components: documentation >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari >Priority: Minor > Labels: documentation, merge > Attachments: HBASE-7181.patch > > > There is one small issue on the documentation regarding the merge. The class > name is not correct. Also, it might be usefull to give an example of the > region format. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7181) Merge documentation improvement
Jean-Marc Spaggiari created HBASE-7181: -- Summary: Merge documentation improvement Key: HBASE-7181 URL: https://issues.apache.org/jira/browse/HBASE-7181 Project: HBase Issue Type: Improvement Components: documentation Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Minor There is one small issue on the documentation regarding the merge. The class name is not correct. Also, it might be usefull to give an example of the region format. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7108) Don't use legal family name for system folder at region level
[ https://issues.apache.org/jira/browse/HBASE-7108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-7108: --- Description: CHANGED, was: Don't allow "recovered.edits" as legal family name Region directories can contain folders called "recovered.edits", log splitting related. But there's nothing that prevent a user to create a family with that name... HLog.RECOVERED_EDITS_DIR = "recovered.edits"; HRegion.MERGEDIR = "merges"; // fixed with HBASE-6158 SplitTransaction.SPLITDIR = "splits"; // fixed with HBASE-6158 was: Region directories can contain a folder called "recovered.edits", log splitting related. But there's nothing that prevent a user to create a family with that name... Summary: Don't use legal family name for system folder at region level (was: Don't allow "recovered.edits" as legal family name) > Don't use legal family name for system folder at region level > - > > Key: HBASE-7108 > URL: https://issues.apache.org/jira/browse/HBASE-7108 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.92.2, 0.94.2, 0.96.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 0.96.0 > > Attachments: HBASE-7108-v0.patch > > > CHANGED, was: Don't allow "recovered.edits" as legal family name > Region directories can contain folders called "recovered.edits", log > splitting related. > But there's nothing that prevent a user to create a family with that name... > HLog.RECOVERED_EDITS_DIR = "recovered.edits"; > HRegion.MERGEDIR = "merges"; // fixed with HBASE-6158 > SplitTransaction.SPLITDIR = "splits"; // fixed with HBASE-6158 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6787) Convert RowProcessorProtocol to protocol buffer service
[
https://issues.apache.org/jira/browse/HBASE-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499653#comment-13499653
]
Hadoop QA commented on HBASE-6787:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12553949/6787-5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 7 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
98 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 24 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestDrainingServer
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3358//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3358//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3358//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3358//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3358//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3358//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3358//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3358//console
This message is automatically generated.
> Convert RowProcessorProtocol to protocol buffer service
> ---
>
> Key: HBASE-6787
> URL: https://issues.apache.org/jira/browse/HBASE-6787
> Project: HBase
> Issue Type: Sub-task
> Components: Coprocessors
>Reporter: Gary Helmling
>Assignee: Devaraj Das
> Fix For: 0.96.0
>
> Attachments: 6787-1.patch, 6787-2.patch, 6787-3.patch, 6787-4.patch,
> 6787-5.patch
>
>
> With coprocessor endpoints now exposed as protobuf defined services, we
> should convert over all of our built-in endpoints to PB services.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6787) Convert RowProcessorProtocol to protocol buffer service
[ https://issues.apache.org/jira/browse/HBASE-6787?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-6787: --- Attachment: 6787-5.patch This addresses Ted's comments except for the comments to do with 'Type'. This is because I removed the code that infers the type of the result (in the last patch, I used to check if the result was null, and if so, send an empty result of the desired type). That seemed undesirable when I thought about it (and could hide usercode bugs). Instead, the implementations of the rowprocessors should always provide a getResult implementation to return the correct result type instance (albeit in some cases it may be empty). > Convert RowProcessorProtocol to protocol buffer service > --- > > Key: HBASE-6787 > URL: https://issues.apache.org/jira/browse/HBASE-6787 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Reporter: Gary Helmling >Assignee: Devaraj Das > Fix For: 0.96.0 > > Attachments: 6787-1.patch, 6787-2.patch, 6787-3.patch, 6787-4.patch, > 6787-5.patch > > > With coprocessor endpoints now exposed as protobuf defined services, we > should convert over all of our built-in endpoints to PB services. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
