[jira] [Commented] (HBASE-10000) Initiate lease recovery for outstanding WAL files at the very beginning of recovery
[
https://issues.apache.org/jira/browse/HBASE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845154#comment-13845154
]
Hadoop QA commented on HBASE-1:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618190/1-recover-ts-with-pb-8.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 25 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8133//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8133//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8133//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8133//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8133//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8133//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8133//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8133//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8133//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8133//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8133//console
This message is automatically generated.
> Initiate lease recovery for outstanding WAL files at the very beginning of
> recovery
> ---
>
> Key: HBASE-1
> URL: https://issues.apache.org/jira/browse/HBASE-1
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.98.1
>
> Attachments: 1-0.96-v5.txt, 1-0.96-v6.txt,
> 1-recover-ts-with-pb-2.txt, 1-recover-ts-with-pb-3.txt,
> 1-recover-ts-with-pb-4.txt, 1-recover-ts-with-pb-5.txt,
> 1-recover-ts-with-pb-6.txt, 1-recover-ts-with-pb-7.txt,
> 1-recover-ts-with-pb-8.txt, 1-recover-ts-with-pb-8.txt, 1-v4.txt,
> 1-v5.txt, 1-v6.txt
>
>
> At the beginning of recovery, master can send lease recovery requests
> concurrently for outstanding WAL files using a thread pool.
> Each split worker would first check whether the WAL file it processes is
> closed.
> Thanks to Nicolas Liochon and Jeffery discussion with whom gave rise to this
> idea.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HBASE-5349) Automagically tweak global memstore and block cache sizes based on workload
[ https://issues.apache.org/jira/browse/HBASE-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-5349: -- Resolution: Fixed Release Note: Both memstore heap size and on heap block cache sizes can be tuned automatically within the RS life time. The algorithm to be used for this tuning is also pluggable. This feature adds the below new config hbase.regionserver.global.memstore.size.max.range and hbase.regionserver.global.memstore.size.min.range using which one can specify the total heap % within which the memstore size can vary. hfile.block.cache.size.max.range and hfile.block.cache.size.max.range using which one can specify the total heap % within which the memstore size can vary. Using hbase.regionserver.heapmemory.tuner.class one can plugin an impl for the tuner algorithm. Pass the FQCN of the tuner impl class which implements org.apache.hadoop.hbase.regionserver.HeapMemoryTuner The period within which the tuner checks for a possible tune can be adjusted with hbase.regionserver.heapmemory.tuner.period. This defaults to 30 (5 mins) The tuner algorithm recieves a TunerContext within which we pass the number of block cache evictions happened within that time and the memstore flushes happened (Forced due to global heap preasssure and normal flushes as seperate items) Config changes -- Changed the config name hbase.regionserver.global.memstore.upperLimit to hbase.regionserver.global.memstore.size This will be the intial memstore size allocate with default value as 40% Also changed hbase.regionserver.global.memstore.lowerLimit to hbase.regionserver.global.memstore.size.lower.limit There is a semantic change also here. "hbase.regionserver.global.memstore.lowerLimit" represented the total heap % upon which we start force flushes for memstores. Which defaults to 38% of total heap space. The new one is the % of memstore max heap size (Represented by hbase.regionserver.global.memstore.size). This defaults to 95% of hbase.regionserver.global.memstore.size value. Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to Trunk. Thanks all for the reviews.. > Automagically tweak global memstore and block cache sizes based on workload > --- > > Key: HBASE-5349 > URL: https://issues.apache.org/jira/browse/HBASE-5349 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-5349_V2.patch, HBASE-5349_V3.patch, > HBASE-5349_V4.patch, HBASE-5349_V5.patch, WIP_HBASE-5349.patch > > > Hypertable does a neat thing where it changes the size given to the CellCache > (our MemStores) and Block Cache based on the workload. If you need an image, > scroll down at the bottom of this link: > http://www.hypertable.com/documentation/architecture/ > That'd be one less thing to configure. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-5349) Automagically tweak global memstore and block cache sizes based on workload
[ https://issues.apache.org/jira/browse/HBASE-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845130#comment-13845130 ] ramkrishna.s.vasudevan commented on HBASE-5349: --- +1 on commit. > Automagically tweak global memstore and block cache sizes based on workload > --- > > Key: HBASE-5349 > URL: https://issues.apache.org/jira/browse/HBASE-5349 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-5349_V2.patch, HBASE-5349_V3.patch, > HBASE-5349_V4.patch, HBASE-5349_V5.patch, WIP_HBASE-5349.patch > > > Hypertable does a neat thing where it changes the size given to the CellCache > (our MemStores) and Block Cache based on the workload. If you need an image, > scroll down at the bottom of this link: > http://www.hypertable.com/documentation/architecture/ > That'd be one less thing to configure. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HBASE-10000) Initiate lease recovery for outstanding WAL files at the very beginning of recovery
[ https://issues.apache.org/jira/browse/HBASE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-1: --- Attachment: 1-recover-ts-with-pb-8.txt > Initiate lease recovery for outstanding WAL files at the very beginning of > recovery > --- > > Key: HBASE-1 > URL: https://issues.apache.org/jira/browse/HBASE-1 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.98.1 > > Attachments: 1-0.96-v5.txt, 1-0.96-v6.txt, > 1-recover-ts-with-pb-2.txt, 1-recover-ts-with-pb-3.txt, > 1-recover-ts-with-pb-4.txt, 1-recover-ts-with-pb-5.txt, > 1-recover-ts-with-pb-6.txt, 1-recover-ts-with-pb-7.txt, > 1-recover-ts-with-pb-8.txt, 1-recover-ts-with-pb-8.txt, 1-v4.txt, > 1-v5.txt, 1-v6.txt > > > At the beginning of recovery, master can send lease recovery requests > concurrently for outstanding WAL files using a thread pool. > Each split worker would first check whether the WAL file it processes is > closed. > Thanks to Nicolas Liochon and Jeffery discussion with whom gave rise to this > idea. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HBASE-10000) Initiate lease recovery for outstanding WAL files at the very beginning of recovery
[ https://issues.apache.org/jira/browse/HBASE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-1: --- Attachment: (was: 1-recover-ts-with-pb-7.txt) > Initiate lease recovery for outstanding WAL files at the very beginning of > recovery > --- > > Key: HBASE-1 > URL: https://issues.apache.org/jira/browse/HBASE-1 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.98.1 > > Attachments: 1-0.96-v5.txt, 1-0.96-v6.txt, > 1-recover-ts-with-pb-2.txt, 1-recover-ts-with-pb-3.txt, > 1-recover-ts-with-pb-4.txt, 1-recover-ts-with-pb-5.txt, > 1-recover-ts-with-pb-6.txt, 1-recover-ts-with-pb-7.txt, > 1-recover-ts-with-pb-8.txt, 1-recover-ts-with-pb-8.txt, 1-v4.txt, > 1-v5.txt, 1-v6.txt > > > At the beginning of recovery, master can send lease recovery requests > concurrently for outstanding WAL files using a thread pool. > Each split worker would first check whether the WAL file it processes is > closed. > Thanks to Nicolas Liochon and Jeffery discussion with whom gave rise to this > idea. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10000) Initiate lease recovery for outstanding WAL files at the very beginning of recovery
[
https://issues.apache.org/jira/browse/HBASE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845114#comment-13845114
]
Ted Yu commented on HBASE-1:
The test failure (taking place around 2013-12-11 05:11:49) was not related to
patch.
For testTaskResigned() :
{code}
int version = ZKUtil.checkExists(zkw, tasknode);
// Could be small race here.
if (tot_mgr_resubmit.get() == 0) waitForCounter(tot_mgr_resubmit, 0, 1,
to/2);
{code}
There was no log similar to the following (corresponding to waitForCounter()
call above):
{code}
2013-12-10 21:23:54,905 INFO [main] hbase.Waiter(174): Waiting up to [3,200]
milli-secs(wait.for.ratio=[1])
{code}
Meaning, the version (2) retrieved corresponded to resubmitted task. version1
retrieved same value, leading to assertion failure.
I placed breakpoints at the beginning of splitLogDistributed() and
recoverDFSFileLease() - none of them got hit.
> Initiate lease recovery for outstanding WAL files at the very beginning of
> recovery
> ---
>
> Key: HBASE-1
> URL: https://issues.apache.org/jira/browse/HBASE-1
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.98.1
>
> Attachments: 1-0.96-v5.txt, 1-0.96-v6.txt,
> 1-recover-ts-with-pb-2.txt, 1-recover-ts-with-pb-3.txt,
> 1-recover-ts-with-pb-4.txt, 1-recover-ts-with-pb-5.txt,
> 1-recover-ts-with-pb-6.txt, 1-recover-ts-with-pb-7.txt,
> 1-recover-ts-with-pb-7.txt, 1-recover-ts-with-pb-8.txt, 1-v4.txt,
> 1-v5.txt, 1-v6.txt
>
>
> At the beginning of recovery, master can send lease recovery requests
> concurrently for outstanding WAL files using a thread pool.
> Each split worker would first check whether the WAL file it processes is
> closed.
> Thanks to Nicolas Liochon and Jeffery discussion with whom gave rise to this
> idea.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-8755) A new write thread model for HLog to improve the overall HBase write throughput
[ https://issues.apache.org/jira/browse/HBASE-8755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845106#comment-13845106 ] stack commented on HBASE-8755: -- I'm running some tests local just to make sure. Will report back... > A new write thread model for HLog to improve the overall HBase write > throughput > --- > > Key: HBASE-8755 > URL: https://issues.apache.org/jira/browse/HBASE-8755 > Project: HBase > Issue Type: Improvement > Components: Performance, wal >Reporter: Feng Honghua >Assignee: stack >Priority: Critical > Attachments: 8755-syncer.patch, 8755trunkV2.txt, > HBASE-8755-0.94-V0.patch, HBASE-8755-0.94-V1.patch, HBASE-8755-0.96-v0.patch, > HBASE-8755-trunk-V0.patch, HBASE-8755-trunk-V1.patch, > HBASE-8755-trunk-v4.patch, HBASE-8755-trunk-v6.patch, > HBASE-8755-trunk-v7.patch, HBASE-8755-v5.patch > > > In current write model, each write handler thread (executing put()) will > individually go through a full 'append (hlog local buffer) => HLog writer > append (write to hdfs) => HLog writer sync (sync hdfs)' cycle for each write, > which incurs heavy race condition on updateLock and flushLock. > The only optimization where checking if current syncTillHere > txid in > expectation for other thread help write/sync its own txid to hdfs and > omitting the write/sync actually help much less than expectation. > Three of my colleagues(Ye Hangjun / Wu Zesheng / Zhang Peng) at Xiaomi > proposed a new write thread model for writing hdfs sequence file and the > prototype implementation shows a 4X improvement for throughput (from 17000 to > 7+). > I apply this new write thread model in HLog and the performance test in our > test cluster shows about 3X throughput improvement (from 12150 to 31520 for 1 > RS, from 22000 to 7 for 5 RS), the 1 RS write throughput (1K row-size) > even beats the one of BigTable (Precolator published in 2011 says Bigtable's > write throughput then is 31002). I can provide the detailed performance test > results if anyone is interested. > The change for new write thread model is as below: > 1> All put handler threads append the edits to HLog's local pending buffer; > (it notifies AsyncWriter thread that there is new edits in local buffer) > 2> All put handler threads wait in HLog.syncer() function for underlying > threads to finish the sync that contains its txid; > 3> An single AsyncWriter thread is responsible for retrieve all the buffered > edits in HLog's local pending buffer and write to the hdfs > (hlog.writer.append); (it notifies AsyncFlusher thread that there is new > writes to hdfs that needs a sync) > 4> An single AsyncFlusher thread is responsible for issuing a sync to hdfs > to persist the writes by AsyncWriter; (it notifies the AsyncNotifier thread > that sync watermark increases) > 5> An single AsyncNotifier thread is responsible for notifying all pending > put handler threads which are waiting in the HLog.syncer() function > 6> No LogSyncer thread any more (since there is always > AsyncWriter/AsyncFlusher threads do the same job it does) -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-8755) A new write thread model for HLog to improve the overall HBase write throughput
[
https://issues.apache.org/jira/browse/HBASE-8755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845103#comment-13845103
]
Hadoop QA commented on HBASE-8755:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618171/HBASE-8755-trunk-v7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8132//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8132//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8132//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8132//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8132//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8132//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8132//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8132//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8132//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8132//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8132//console
This message is automatically generated.
> A new write thread model for HLog to improve the overall HBase write
> throughput
> ---
>
> Key: HBASE-8755
> URL: https://issues.apache.org/jira/browse/HBASE-8755
> Project: HBase
> Issue Type: Improvement
> Components: Performance, wal
>Reporter: Feng Honghua
>Assignee: stack
>Priority: Critical
> Attachments: 8755-syncer.patch, 8755trunkV2.txt,
> HBASE-8755-0.94-V0.patch, HBASE-8755-0.94-V1.patch, HBASE-8755-0.96-v0.patch,
> HBASE-8755-trunk-V0.patch, HBASE-8755-trunk-V1.patch,
> HBASE-8755-trunk-v4.patch, HBASE-8755-trunk-v6.patch,
> HBASE-8755-trunk-v7.patch, HBASE-8755-v5.patch
>
>
> In current write model, each write handler thread (executing put()) will
> individually go through a full 'append (hlog local buffer) => HLog writer
> append (write to hdfs) => HLog writer sync (sync hdfs)' cycle for each write,
> which incurs heavy race condition on updateLock and flushLock.
> The only optimization where checking if current syncTillHere > txid in
> expectation for other thread help write/sync its own txid to hdfs and
> omitting the write/sync actually help much less than expectation.
> Three of my colleagues(Ye Hangjun / Wu Zesheng / Zhang Peng) at Xiaomi
> proposed a new write thread model for writing hdfs sequence file and the
> prototype implementation shows a 4X improvement for throughput (from 17000 to
> 7+).
> I apply this new write thread model in HLog and the performance test in our
> test cluster shows about 3X throughput improvement (from 12150 to 31520 for 1
> RS, from 22000 to 7 for 5 RS), the 1 RS write throughput (1K row-size)
> even beats the one of BigTable (Precolator published in 2011 says Bigtable's
> write throughput then is 31002). I can provide the detai
[jira] [Commented] (HBASE-10124) Make Sub Classes Static When Possible
[ https://issues.apache.org/jira/browse/HBASE-10124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845091#comment-13845091 ] Jonathan Hsieh commented on HBASE-10124: Any of the now public static classes that livein non test classes need a @InterfaceAudience.Public vs @InterfaceAudience.Private consideration. I didn't look carefully but my guess is all are likely supposed to be private. > Make Sub Classes Static When Possible > - > > Key: HBASE-10124 > URL: https://issues.apache.org/jira/browse/HBASE-10124 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.98.0, 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Labels: noob > Attachments: HBASE-10124-0.patch > > > Coverity has found a few places where it's possible to change a private inner > class to static with out any other code changes. This will be a small perf > win. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-5349) Automagically tweak global memstore and block cache sizes based on workload
[ https://issues.apache.org/jira/browse/HBASE-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845086#comment-13845086 ] Lars Hofhansl commented on HBASE-5349: -- Nice milestone towards 1.0. :) +1 > Automagically tweak global memstore and block cache sizes based on workload > --- > > Key: HBASE-5349 > URL: https://issues.apache.org/jira/browse/HBASE-5349 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-5349_V2.patch, HBASE-5349_V3.patch, > HBASE-5349_V4.patch, HBASE-5349_V5.patch, WIP_HBASE-5349.patch > > > Hypertable does a neat thing where it changes the size given to the CellCache > (our MemStores) and Block Cache based on the workload. If you need an image, > scroll down at the bottom of this link: > http://www.hypertable.com/documentation/architecture/ > That'd be one less thing to configure. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10000) Initiate lease recovery for outstanding WAL files at the very beginning of recovery
[
https://issues.apache.org/jira/browse/HBASE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845078#comment-13845078
]
Hadoop QA commented on HBASE-1:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618166/1-recover-ts-with-pb-8.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 25 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestSplitLogManager
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8131//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8131//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8131//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8131//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8131//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8131//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8131//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8131//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8131//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8131//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8131//console
This message is automatically generated.
> Initiate lease recovery for outstanding WAL files at the very beginning of
> recovery
> ---
>
> Key: HBASE-1
> URL: https://issues.apache.org/jira/browse/HBASE-1
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.98.1
>
> Attachments: 1-0.96-v5.txt, 1-0.96-v6.txt,
> 1-recover-ts-with-pb-2.txt, 1-recover-ts-with-pb-3.txt,
> 1-recover-ts-with-pb-4.txt, 1-recover-ts-with-pb-5.txt,
> 1-recover-ts-with-pb-6.txt, 1-recover-ts-with-pb-7.txt,
> 1-recover-ts-with-pb-7.txt, 1-recover-ts-with-pb-8.txt, 1-v4.txt,
> 1-v5.txt, 1-v6.txt
>
>
> At the beginning of recovery, master can send lease recovery requests
> concurrently for outstanding WAL files using a thread pool.
> Each split worker would first check whether the WAL file it processes is
> closed.
> Thanks to Nicolas Liochon and Jeffery discussion with whom gave rise to this
> idea.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10122) Fix EncodedSeekPerformanceTest
[ https://issues.apache.org/jira/browse/HBASE-10122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845077#comment-13845077 ] Jonathan Hsieh commented on HBASE-10122: I have a rough patch that adds a hbase-perf module and with a little modification this could be a fine microbenchmark. To fix this, I think we'd add methods that create properly encoded data and report its performance. If you want, as long as the current version compiles, you can assign it to me and I'll take care of it. > Fix EncodedSeekPerformanceTest > -- > > Key: HBASE-10122 > URL: https://issues.apache.org/jira/browse/HBASE-10122 > Project: HBase > Issue Type: Test > Components: test >Reporter: Jimmy Xiang >Priority: Minor > Attachments: trunk-10122.patch > > > After HBASE-9870, the encoding in cache is always the same as that on disk. > EncodedSeekPerformanceTest relies that the two encodings are different. So > now, it is not working as expected. We need to fix it if the tool is still > needed. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10117) Avoid synchronization in HRegionScannerImpl.isFilterDone
[
https://issues.apache.org/jira/browse/HBASE-10117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845074#comment-13845074
]
Lars Hofhansl commented on HBASE-10117:
---
Numbers... 20m rows, 1 col. simple {{select count(1) ...}}
||without patch|| with patch || with patch + Phoenix patch||
|8.8s|8.1s|7.4s|
> Avoid synchronization in HRegionScannerImpl.isFilterDone
>
>
> Key: HBASE-10117
> URL: https://issues.apache.org/jira/browse/HBASE-10117
> Project: HBase
> Issue Type: Bug
> Components: Performance
>Reporter: Lars Hofhansl
>Assignee: Lars Hofhansl
> Fix For: 0.94.15, 0.96.2, 0.98.1, 0.99.0
>
> Attachments: 10117-0.94-v2.txt, 10117-0.94-v3.txt, 10117-0.94.txt,
> 10117-trunk-v2.txt, 10117-trunk.txt
>
>
> A while ago I introduced HRegoinScannerImpl.nextRaw() to allow coprocessors
> and scanners with caching > 1 to avoid repeated synchronization during
> scanning (which puts up memory fences, which in turn slows things down on
> multi core machines).
> Looking at the code again I see that isFilterDone() is called from nextRaw()
> and isFilterDone() is synchronized.
> The caller of nextRaw is required to ensure single threaded access to
> nextRaw() anyway, we can call an unsynchronized internal version of
> isFilterDone().
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-5349) Automagically tweak global memstore and block cache sizes based on workload
[ https://issues.apache.org/jira/browse/HBASE-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845075#comment-13845075 ] stack commented on HBASE-5349: -- [~anoop.hbase] I am good on commit to trunk. Go for it. > Automagically tweak global memstore and block cache sizes based on workload > --- > > Key: HBASE-5349 > URL: https://issues.apache.org/jira/browse/HBASE-5349 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-5349_V2.patch, HBASE-5349_V3.patch, > HBASE-5349_V4.patch, HBASE-5349_V5.patch, WIP_HBASE-5349.patch > > > Hypertable does a neat thing where it changes the size given to the CellCache > (our MemStores) and Block Cache based on the workload. If you need an image, > scroll down at the bottom of this link: > http://www.hypertable.com/documentation/architecture/ > That'd be one less thing to configure. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-9870) HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
[
https://issues.apache.org/jira/browse/HBASE-9870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845069#comment-13845069
]
Lars Hofhansl commented on HBASE-9870:
--
[~jxiang], Don't we need the same fix in 0.94?
> HFileDataBlockEncoderImpl#diskToCacheFormat uses wrong format
> -
>
> Key: HBASE-9870
> URL: https://issues.apache.org/jira/browse/HBASE-9870
> Project: HBase
> Issue Type: Bug
>Reporter: Jimmy Xiang
>Assignee: Jimmy Xiang
> Fix For: 0.98.0, 0.96.1
>
> Attachments: trunk-9870.patch, trunk-9870_v2.patch,
> trunk-9870_v3.patch
>
>
> In this method, we have
> {code}
> if (block.getBlockType() == BlockType.ENCODED_DATA) {
> if (block.getDataBlockEncodingId() == onDisk.getId()) {
> // The block is already in the desired in-cache encoding.
> return block;
> }
> {code}
> This assumes onDisk encoding is the same as that of inCache. This is not
> true when we change the encoding of a CF. This could be one of the reasons I
> got data loss with online encoding change?
> If I make sure onDisk == inCache all the time, my ITBLL with online encoding
> change worked once for me.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HBASE-10114) _scan_internal() in table.rb should accept argument that specifies reverse scan
[
https://issues.apache.org/jira/browse/HBASE-10114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-10114:
---
Release Note:
HBase shell supports Reversed Scan.
User can specify REVERSED => true along with other scan parameters.
> _scan_internal() in table.rb should accept argument that specifies reverse
> scan
> ---
>
> Key: HBASE-10114
> URL: https://issues.apache.org/jira/browse/HBASE-10114
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: chunhui shen
> Fix For: 0.98.0, 0.99.0
>
> Attachments: hbase-10114-trunk.patch
>
>
> Reversed Scan usage on Shell:
> hbase> scan 't1', {REVERSED => true}
> Other parameter could also be added with 'REVERSED', like LIMIT,COLUMNS.
> Examples:
> hbase(main):019:0* scan 't1'
> ROW COLUMN+CELL
>
>
> 001 column=f1:q1,
> timestamp=1386644670350, value=v1
>
> 002 column=f1:q2,
> timestamp=1386644675585, value=v1
>
> 003 column=f1:q1,
> timestamp=1386644680090, value=v1
>
> 003 column=f1:q2,
> timestamp=1386644682749, value=v1
>
> 004 column=f1:q1,
> timestamp=1386644692589, value=v1
>
> 4 row(s) in 0.0130 seconds
> hbase(main):022:0> scan 't1',{REVERSED => true}
> ROW COLUMN+CELL
>
>
> 004 column=f1:q1,
> timestamp=1386644692589, value=v1
>
> 003 column=f1:q1,
> timestamp=1386644680090, value=v1
>
> 003 column=f1:q2,
> timestamp=1386644682749, value=v1
>
> 002 column=f1:q2,
> timestamp=1386644675585, value=v1
>
> 001 column=f1:q1,
> timestamp=1386644670350, value=v1
> hbase(main):023:0> scan 't1',{REVERSED => true,COLUMNS=>'f1:q2'}
> ROW COLUMN+CELL
>
>
> 003 column=f1:q2,
> timestamp=1386644682749, value=v1
>
> 002 column=f1:q2,
> timestamp=1386644675585, value=v1
>
> 2 row(s) in 0.0100 seconds
> Here is how boolean argument is accepted:
> {code}
> cache_blocks = args["CACHE_BLOCKS"] || true
> {code}
> Flag for reverse scan should be supported in similar manner
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10125) KeyValue(Cell c) copy constructor doesn't copy tags data
[
https://issues.apache.org/jira/browse/HBASE-10125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845053#comment-13845053
]
Hadoop QA commented on HBASE-10125:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618165/hbase-10125.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8130//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8130//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8130//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8130//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8130//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8130//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8130//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8130//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8130//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8130//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8130//console
This message is automatically generated.
> KeyValue(Cell c) copy constructor doesn't copy tags data
>
>
> Key: HBASE-10125
> URL: https://issues.apache.org/jira/browse/HBASE-10125
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
> Fix For: 0.98.0
>
> Attachments: hbase-10125.patch
>
>
> I happened to come across this:
> {code}
> public KeyValue(Cell c) {
> this(c.getRowArray(), c.getRowOffset(), (int)c.getRowLength(),
> c.getFamilyArray(), c.getFamilyOffset(), (int)c.getFamilyLength(),
> c.getQualifierArray(), c.getQualifierOffset(), (int)
> c.getQualifierLength(),
> c.getTimestamp(), Type.codeToType(c.getTypeByte()),
> c.getValueArray(), c.getValueOffset(), c.getValueLength());
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10124) Make Sub Classes Static When Possible
[
https://issues.apache.org/jira/browse/HBASE-10124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845043#comment-13845043
]
Hadoop QA commented on HBASE-10124:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618151/HBASE-10124-0.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 11 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8129//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8129//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8129//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8129//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8129//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8129//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8129//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8129//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8129//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8129//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8129//console
This message is automatically generated.
> Make Sub Classes Static When Possible
> -
>
> Key: HBASE-10124
> URL: https://issues.apache.org/jira/browse/HBASE-10124
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 0.98.0, 0.96.0
>Reporter: Elliott Clark
>Assignee: Elliott Clark
> Labels: noob
> Attachments: HBASE-10124-0.patch
>
>
> Coverity has found a few places where it's possible to change a private inner
> class to static with out any other code changes. This will be a small perf
> win.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Resolved] (HBASE-10096) make BulkDeleteResponse implement Writable to reduce the transfer size for rpc
[ https://issues.apache.org/jira/browse/HBASE-10096?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John resolved HBASE-10096. Resolution: Won't Fix > make BulkDeleteResponse implement Writable to reduce the transfer size for rpc > -- > > Key: HBASE-10096 > URL: https://issues.apache.org/jira/browse/HBASE-10096 > Project: HBase > Issue Type: Improvement > Components: Coprocessors, Performance >Reporter: cuijianwei > Attachments: HBASE-10096-0.94-v1.patch > > > As [https://issues.apache.org/jira/browse/HBASE-6942] introduced, > BulkDeleteEndpoint provides 'delete' method which performs like sql : "delete > from table where ...". BulkDeleteEndpoint is efficient because it can > complete scan and delete in one rpc and also could be implemented parallelly > in different regions using coprocessor. BulkDeleteResponse is represents the > result of BulkDeleteEndpoint.delete and will be serialized using a standard > java serializable way. However, the serialized length of BulkDeleteResponse > will be longer than one hundred byte length and may be not efficient enough > to pass on the network. Therefore, is it better to make BulkDeleteResponse > implement Writable interface and provide more efficient serialize method? -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Comment Edited] (HBASE-10096) make BulkDeleteResponse implement Writable to reduce the transfer size for rpc
[ https://issues.apache.org/jira/browse/HBASE-10096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845036#comment-13845036 ] Anoop Sam John edited comment on HBASE-10096 at 12/11/13 3:34 AM: -- Yes 96+ versions there is no problems. Also for the BC we will keep 0.94 as it is. I will close this issue as 'Wont Fix'. If you can guarentee the use of changed version at client and server side then can appy the patch internally and use. (The perf gain might be less only. Stil ) Pls keep the patch attached to this jira so that some one wanted can use that.. Because of BC we can not apply this to 0.94. Thanks for the finding and patch [~cuijianwei] was (Author: anoop.hbase): Yes 96+ versions there is no problems. Also for the BC we will keep 0.94 as it as. I will close this issue as Wont Fix. If you can guarentee the use of changed version at client and server side then can appy the patch internally and use. (The perf gain might be less only. Stil ) Pls keep the patch attached to this jira so that some one wanted can use that.. Bacause of BC we can not apply this to 0.94. Thanks for the finding and patch [~cuijianwei] > make BulkDeleteResponse implement Writable to reduce the transfer size for rpc > -- > > Key: HBASE-10096 > URL: https://issues.apache.org/jira/browse/HBASE-10096 > Project: HBase > Issue Type: Improvement > Components: Coprocessors, Performance >Reporter: cuijianwei > Attachments: HBASE-10096-0.94-v1.patch > > > As [https://issues.apache.org/jira/browse/HBASE-6942] introduced, > BulkDeleteEndpoint provides 'delete' method which performs like sql : "delete > from table where ...". BulkDeleteEndpoint is efficient because it can > complete scan and delete in one rpc and also could be implemented parallelly > in different regions using coprocessor. BulkDeleteResponse is represents the > result of BulkDeleteEndpoint.delete and will be serialized using a standard > java serializable way. However, the serialized length of BulkDeleteResponse > will be longer than one hundred byte length and may be not efficient enough > to pass on the network. Therefore, is it better to make BulkDeleteResponse > implement Writable interface and provide more efficient serialize method? -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-8755) A new write thread model for HLog to improve the overall HBase write throughput
[ https://issues.apache.org/jira/browse/HBASE-8755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845038#comment-13845038 ] Feng Honghua commented on HBASE-8755: - [~v.himanshu]: Looking forward to your performance comparison test result (trunk, with-this-patch, with-syncer-only) using latest HLogPE with new 'appendWithoutSync + sync' logic. And I'll also try to do the same test for double comparison/confirm. Performance improvement is the first and foremost goal of this patch, code cleanup is a just by-the-way side-effect, so we want to see this patch accepted/checked-in because of the performance improvement it brings, but because of the code cleanup it does :-) > A new write thread model for HLog to improve the overall HBase write > throughput > --- > > Key: HBASE-8755 > URL: https://issues.apache.org/jira/browse/HBASE-8755 > Project: HBase > Issue Type: Improvement > Components: Performance, wal >Reporter: Feng Honghua >Assignee: stack >Priority: Critical > Attachments: 8755-syncer.patch, 8755trunkV2.txt, > HBASE-8755-0.94-V0.patch, HBASE-8755-0.94-V1.patch, HBASE-8755-0.96-v0.patch, > HBASE-8755-trunk-V0.patch, HBASE-8755-trunk-V1.patch, > HBASE-8755-trunk-v4.patch, HBASE-8755-trunk-v6.patch, > HBASE-8755-trunk-v7.patch, HBASE-8755-v5.patch > > > In current write model, each write handler thread (executing put()) will > individually go through a full 'append (hlog local buffer) => HLog writer > append (write to hdfs) => HLog writer sync (sync hdfs)' cycle for each write, > which incurs heavy race condition on updateLock and flushLock. > The only optimization where checking if current syncTillHere > txid in > expectation for other thread help write/sync its own txid to hdfs and > omitting the write/sync actually help much less than expectation. > Three of my colleagues(Ye Hangjun / Wu Zesheng / Zhang Peng) at Xiaomi > proposed a new write thread model for writing hdfs sequence file and the > prototype implementation shows a 4X improvement for throughput (from 17000 to > 7+). > I apply this new write thread model in HLog and the performance test in our > test cluster shows about 3X throughput improvement (from 12150 to 31520 for 1 > RS, from 22000 to 7 for 5 RS), the 1 RS write throughput (1K row-size) > even beats the one of BigTable (Precolator published in 2011 says Bigtable's > write throughput then is 31002). I can provide the detailed performance test > results if anyone is interested. > The change for new write thread model is as below: > 1> All put handler threads append the edits to HLog's local pending buffer; > (it notifies AsyncWriter thread that there is new edits in local buffer) > 2> All put handler threads wait in HLog.syncer() function for underlying > threads to finish the sync that contains its txid; > 3> An single AsyncWriter thread is responsible for retrieve all the buffered > edits in HLog's local pending buffer and write to the hdfs > (hlog.writer.append); (it notifies AsyncFlusher thread that there is new > writes to hdfs that needs a sync) > 4> An single AsyncFlusher thread is responsible for issuing a sync to hdfs > to persist the writes by AsyncWriter; (it notifies the AsyncNotifier thread > that sync watermark increases) > 5> An single AsyncNotifier thread is responsible for notifying all pending > put handler threads which are waiting in the HLog.syncer() function > 6> No LogSyncer thread any more (since there is always > AsyncWriter/AsyncFlusher threads do the same job it does) -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10096) make BulkDeleteResponse implement Writable to reduce the transfer size for rpc
[ https://issues.apache.org/jira/browse/HBASE-10096?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845036#comment-13845036 ] Anoop Sam John commented on HBASE-10096: Yes 96+ versions there is no problems. Also for the BC we will keep 0.94 as it as. I will close this issue as Wont Fix. If you can guarentee the use of changed version at client and server side then can appy the patch internally and use. (The perf gain might be less only. Stil ) Pls keep the patch attached to this jira so that some one wanted can use that.. Bacause of BC we can not apply this to 0.94. Thanks for the finding and patch [~cuijianwei] > make BulkDeleteResponse implement Writable to reduce the transfer size for rpc > -- > > Key: HBASE-10096 > URL: https://issues.apache.org/jira/browse/HBASE-10096 > Project: HBase > Issue Type: Improvement > Components: Coprocessors, Performance >Reporter: cuijianwei > Attachments: HBASE-10096-0.94-v1.patch > > > As [https://issues.apache.org/jira/browse/HBASE-6942] introduced, > BulkDeleteEndpoint provides 'delete' method which performs like sql : "delete > from table where ...". BulkDeleteEndpoint is efficient because it can > complete scan and delete in one rpc and also could be implemented parallelly > in different regions using coprocessor. BulkDeleteResponse is represents the > result of BulkDeleteEndpoint.delete and will be serialized using a standard > java serializable way. However, the serialized length of BulkDeleteResponse > will be longer than one hundred byte length and may be not efficient enough > to pass on the network. Therefore, is it better to make BulkDeleteResponse > implement Writable interface and provide more efficient serialize method? -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HBASE-8755) A new write thread model for HLog to improve the overall HBase write throughput
[ https://issues.apache.org/jira/browse/HBASE-8755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Feng Honghua updated HBASE-8755: Attachment: HBASE-8755-trunk-v7.patch > A new write thread model for HLog to improve the overall HBase write > throughput > --- > > Key: HBASE-8755 > URL: https://issues.apache.org/jira/browse/HBASE-8755 > Project: HBase > Issue Type: Improvement > Components: Performance, wal >Reporter: Feng Honghua >Assignee: stack >Priority: Critical > Attachments: 8755-syncer.patch, 8755trunkV2.txt, > HBASE-8755-0.94-V0.patch, HBASE-8755-0.94-V1.patch, HBASE-8755-0.96-v0.patch, > HBASE-8755-trunk-V0.patch, HBASE-8755-trunk-V1.patch, > HBASE-8755-trunk-v4.patch, HBASE-8755-trunk-v6.patch, > HBASE-8755-trunk-v7.patch, HBASE-8755-v5.patch > > > In current write model, each write handler thread (executing put()) will > individually go through a full 'append (hlog local buffer) => HLog writer > append (write to hdfs) => HLog writer sync (sync hdfs)' cycle for each write, > which incurs heavy race condition on updateLock and flushLock. > The only optimization where checking if current syncTillHere > txid in > expectation for other thread help write/sync its own txid to hdfs and > omitting the write/sync actually help much less than expectation. > Three of my colleagues(Ye Hangjun / Wu Zesheng / Zhang Peng) at Xiaomi > proposed a new write thread model for writing hdfs sequence file and the > prototype implementation shows a 4X improvement for throughput (from 17000 to > 7+). > I apply this new write thread model in HLog and the performance test in our > test cluster shows about 3X throughput improvement (from 12150 to 31520 for 1 > RS, from 22000 to 7 for 5 RS), the 1 RS write throughput (1K row-size) > even beats the one of BigTable (Precolator published in 2011 says Bigtable's > write throughput then is 31002). I can provide the detailed performance test > results if anyone is interested. > The change for new write thread model is as below: > 1> All put handler threads append the edits to HLog's local pending buffer; > (it notifies AsyncWriter thread that there is new edits in local buffer) > 2> All put handler threads wait in HLog.syncer() function for underlying > threads to finish the sync that contains its txid; > 3> An single AsyncWriter thread is responsible for retrieve all the buffered > edits in HLog's local pending buffer and write to the hdfs > (hlog.writer.append); (it notifies AsyncFlusher thread that there is new > writes to hdfs that needs a sync) > 4> An single AsyncFlusher thread is responsible for issuing a sync to hdfs > to persist the writes by AsyncWriter; (it notifies the AsyncNotifier thread > that sync watermark increases) > 5> An single AsyncNotifier thread is responsible for notifying all pending > put handler threads which are waiting in the HLog.syncer() function > 6> No LogSyncer thread any more (since there is always > AsyncWriter/AsyncFlusher threads do the same job it does) -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-8755) A new write thread model for HLog to improve the overall HBase write throughput
[ https://issues.apache.org/jira/browse/HBASE-8755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845035#comment-13845035 ] Feng Honghua commented on HBASE-8755: - Thanks [~jmhsieh], I've made and attached a new patch based on your comment. Thanks everyone again for your valuable comment and feedback. > A new write thread model for HLog to improve the overall HBase write > throughput > --- > > Key: HBASE-8755 > URL: https://issues.apache.org/jira/browse/HBASE-8755 > Project: HBase > Issue Type: Improvement > Components: Performance, wal >Reporter: Feng Honghua >Assignee: stack >Priority: Critical > Attachments: 8755-syncer.patch, 8755trunkV2.txt, > HBASE-8755-0.94-V0.patch, HBASE-8755-0.94-V1.patch, HBASE-8755-0.96-v0.patch, > HBASE-8755-trunk-V0.patch, HBASE-8755-trunk-V1.patch, > HBASE-8755-trunk-v4.patch, HBASE-8755-trunk-v6.patch, > HBASE-8755-trunk-v7.patch, HBASE-8755-v5.patch > > > In current write model, each write handler thread (executing put()) will > individually go through a full 'append (hlog local buffer) => HLog writer > append (write to hdfs) => HLog writer sync (sync hdfs)' cycle for each write, > which incurs heavy race condition on updateLock and flushLock. > The only optimization where checking if current syncTillHere > txid in > expectation for other thread help write/sync its own txid to hdfs and > omitting the write/sync actually help much less than expectation. > Three of my colleagues(Ye Hangjun / Wu Zesheng / Zhang Peng) at Xiaomi > proposed a new write thread model for writing hdfs sequence file and the > prototype implementation shows a 4X improvement for throughput (from 17000 to > 7+). > I apply this new write thread model in HLog and the performance test in our > test cluster shows about 3X throughput improvement (from 12150 to 31520 for 1 > RS, from 22000 to 7 for 5 RS), the 1 RS write throughput (1K row-size) > even beats the one of BigTable (Precolator published in 2011 says Bigtable's > write throughput then is 31002). I can provide the detailed performance test > results if anyone is interested. > The change for new write thread model is as below: > 1> All put handler threads append the edits to HLog's local pending buffer; > (it notifies AsyncWriter thread that there is new edits in local buffer) > 2> All put handler threads wait in HLog.syncer() function for underlying > threads to finish the sync that contains its txid; > 3> An single AsyncWriter thread is responsible for retrieve all the buffered > edits in HLog's local pending buffer and write to the hdfs > (hlog.writer.append); (it notifies AsyncFlusher thread that there is new > writes to hdfs that needs a sync) > 4> An single AsyncFlusher thread is responsible for issuing a sync to hdfs > to persist the writes by AsyncWriter; (it notifies the AsyncNotifier thread > that sync watermark increases) > 5> An single AsyncNotifier thread is responsible for notifying all pending > put handler threads which are waiting in the HLog.syncer() function > 6> No LogSyncer thread any more (since there is always > AsyncWriter/AsyncFlusher threads do the same job it does) -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HBASE-10120) start-hbase.sh doesn't respect --config in non-distributed mode
[ https://issues.apache.org/jira/browse/HBASE-10120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk updated HBASE-10120: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to all four branches. Thanks for the quick review. > start-hbase.sh doesn't respect --config in non-distributed mode > --- > > Key: HBASE-10120 > URL: https://issues.apache.org/jira/browse/HBASE-10120 > Project: HBase > Issue Type: Bug > Components: scripts >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Trivial > Fix For: 0.98.0, 0.94.15, 0.96.2, 0.99.0 > > Attachments: HBASE-10120.00.patch > > > A custom value for --config is not passed along to hbase-daemon.sh by > start-hbase.sh when invoked in local mode (hbase.cluster.distributed=false). > When --config is specified, variables from hbase-env.sh are applied to the > runtime, but not hbase-site.xml. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-5349) Automagically tweak global memstore and block cache sizes based on workload
[ https://issues.apache.org/jira/browse/HBASE-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845030#comment-13845030 ] Anoop Sam John commented on HBASE-5349: --- The pending comment from Ram is a minor one. I will fix it on commit Ram. > Automagically tweak global memstore and block cache sizes based on workload > --- > > Key: HBASE-5349 > URL: https://issues.apache.org/jira/browse/HBASE-5349 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-5349_V2.patch, HBASE-5349_V3.patch, > HBASE-5349_V4.patch, HBASE-5349_V5.patch, WIP_HBASE-5349.patch > > > Hypertable does a neat thing where it changes the size given to the CellCache > (our MemStores) and Block Cache based on the workload. If you need an image, > scroll down at the bottom of this link: > http://www.hypertable.com/documentation/architecture/ > That'd be one less thing to configure. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-5349) Automagically tweak global memstore and block cache sizes based on workload
[ https://issues.apache.org/jira/browse/HBASE-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845029#comment-13845029 ] Anoop Sam John commented on HBASE-5349: --- Thanks for the reviews. I have to make a next version addressing few comments from Ram. Sure Ted I will add the release notes. Any more comments [~stack] ? > Automagically tweak global memstore and block cache sizes based on workload > --- > > Key: HBASE-5349 > URL: https://issues.apache.org/jira/browse/HBASE-5349 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-5349_V2.patch, HBASE-5349_V3.patch, > HBASE-5349_V4.patch, HBASE-5349_V5.patch, WIP_HBASE-5349.patch > > > Hypertable does a neat thing where it changes the size given to the CellCache > (our MemStores) and Block Cache based on the workload. If you need an image, > scroll down at the bottom of this link: > http://www.hypertable.com/documentation/architecture/ > That'd be one less thing to configure. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10071) support data type for get/scan in hbase shell
[
https://issues.apache.org/jira/browse/HBASE-10071?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845018#comment-13845018
]
cuijianwei commented on HBASE-10071:
[~stack] Thanks for your comment:). I go through the scan.rb in trunk. The way
to specify custom formatting by column is more general and easy to use. The
formatter can also be stipulated by custom class. Maybe, we can backport this
feature into 0.94?
> support data type for get/scan in hbase shell
> -
>
> Key: HBASE-10071
> URL: https://issues.apache.org/jira/browse/HBASE-10071
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Affects Versions: 0.94.14
>Reporter: cuijianwei
> Attachments: HBASE-10071-0.94-v1.patch
>
>
> Users tend to run hbase shell to query hbase quickly. The result will be
> shown as binary format which may not look clear enough when users write
> columns using specified types, such as long/int/short. Therefore, it may be
> helpful if the results could be shown as specified format. We make a patch to
> extend get/scan in hbase shell in which user could specify the data type in
> get/scan for each column as:
> {code}
> scan 'table', {COLUMNS=>['CF:QF:long']}
> get 'table', 'r0', {COLUMN=>'CF:QF:long'}
> {code}
> Then, the result will be shown as Long type. The result of above get will be:
> {code}
> COLUMNCELL
>
>
> CF:QFtimestamp=24311261, value=24311229
> {code}
> This extended format is compatible with previous format, if users do not
> specify the data type, the command will also work and output binary format.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10125) KeyValue(Cell c) copy constructor doesn't copy tags data
[
https://issues.apache.org/jira/browse/HBASE-10125?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13845016#comment-13845016
]
Ted Yu commented on HBASE-10125:
lgtm
Nice finding, Jeff.
> KeyValue(Cell c) copy constructor doesn't copy tags data
>
>
> Key: HBASE-10125
> URL: https://issues.apache.org/jira/browse/HBASE-10125
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
> Fix For: 0.98.0
>
> Attachments: hbase-10125.patch
>
>
> I happened to come across this:
> {code}
> public KeyValue(Cell c) {
> this(c.getRowArray(), c.getRowOffset(), (int)c.getRowLength(),
> c.getFamilyArray(), c.getFamilyOffset(), (int)c.getFamilyLength(),
> c.getQualifierArray(), c.getQualifierOffset(), (int)
> c.getQualifierLength(),
> c.getTimestamp(), Type.codeToType(c.getTypeByte()),
> c.getValueArray(), c.getValueOffset(), c.getValueLength());
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HBASE-10000) Initiate lease recovery for outstanding WAL files at the very beginning of recovery
[ https://issues.apache.org/jira/browse/HBASE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-1: --- Attachment: 1-recover-ts-with-pb-8.txt Forgot an equal sign in the comparison. Looped TestFSHDFSUtils 8 times locally which all passed. > Initiate lease recovery for outstanding WAL files at the very beginning of > recovery > --- > > Key: HBASE-1 > URL: https://issues.apache.org/jira/browse/HBASE-1 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.98.1 > > Attachments: 1-0.96-v5.txt, 1-0.96-v6.txt, > 1-recover-ts-with-pb-2.txt, 1-recover-ts-with-pb-3.txt, > 1-recover-ts-with-pb-4.txt, 1-recover-ts-with-pb-5.txt, > 1-recover-ts-with-pb-6.txt, 1-recover-ts-with-pb-7.txt, > 1-recover-ts-with-pb-7.txt, 1-recover-ts-with-pb-8.txt, 1-v4.txt, > 1-v5.txt, 1-v6.txt > > > At the beginning of recovery, master can send lease recovery requests > concurrently for outstanding WAL files using a thread pool. > Each split worker would first check whether the WAL file it processes is > closed. > Thanks to Nicolas Liochon and Jeffery discussion with whom gave rise to this > idea. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HBASE-10125) KeyValue(Cell c) copy constructor doesn't copy tags data
[
https://issues.apache.org/jira/browse/HBASE-10125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeffrey Zhong updated HBASE-10125:
--
Attachment: hbase-10125.patch
> KeyValue(Cell c) copy constructor doesn't copy tags data
>
>
> Key: HBASE-10125
> URL: https://issues.apache.org/jira/browse/HBASE-10125
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
> Fix For: 0.98.0
>
> Attachments: hbase-10125.patch
>
>
> I happened to come across this:
> {code}
> public KeyValue(Cell c) {
> this(c.getRowArray(), c.getRowOffset(), (int)c.getRowLength(),
> c.getFamilyArray(), c.getFamilyOffset(), (int)c.getFamilyLength(),
> c.getQualifierArray(), c.getQualifierOffset(), (int)
> c.getQualifierLength(),
> c.getTimestamp(), Type.codeToType(c.getTypeByte()),
> c.getValueArray(), c.getValueOffset(), c.getValueLength());
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HBASE-10125) KeyValue(Cell c) copy constructor doesn't copy tags data
[
https://issues.apache.org/jira/browse/HBASE-10125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jeffrey Zhong updated HBASE-10125:
--
Status: Patch Available (was: Open)
> KeyValue(Cell c) copy constructor doesn't copy tags data
>
>
> Key: HBASE-10125
> URL: https://issues.apache.org/jira/browse/HBASE-10125
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
> Fix For: 0.98.0
>
> Attachments: hbase-10125.patch
>
>
> I happened to come across this:
> {code}
> public KeyValue(Cell c) {
> this(c.getRowArray(), c.getRowOffset(), (int)c.getRowLength(),
> c.getFamilyArray(), c.getFamilyOffset(), (int)c.getFamilyLength(),
> c.getQualifierArray(), c.getQualifierOffset(), (int)
> c.getQualifierLength(),
> c.getTimestamp(), Type.codeToType(c.getTypeByte()),
> c.getValueArray(), c.getValueOffset(), c.getValueLength());
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Resolved] (HBASE-10119) Allow HBase coprocessors to clean up when they fail
[
https://issues.apache.org/jira/browse/HBASE-10119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell resolved HBASE-10119.
Resolution: Fixed
Fix Version/s: 0.94.15
Hadoop Flags: Reviewed
Committed to 0.94 also.
Thanks for the patch Benoît.
> Allow HBase coprocessors to clean up when they fail
> ---
>
> Key: HBASE-10119
> URL: https://issues.apache.org/jira/browse/HBASE-10119
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 0.96.0
>Reporter: Benoit Sigoure
>Assignee: Benoit Sigoure
> Fix For: 0.98.0, 0.96.1, 0.94.15, 0.99.0
>
> Attachments: HBASE-10119-0.94.patch, HBASE-10119.patch
>
>
> In the thread [Giving a chance to buggy coprocessors to clean
> up|http://osdir.com/ml/general/2013-12/msg17334.html] I brought up the issue
> that coprocessors currently don't have a chance to release their own
> resources (be they internal resources within the JVM, or external resources
> elsewhere) when they get forcefully removed due to an uncaught exception
> escaping.
> It would be nice to fix that, either by adding an API called by the
> {{CoprocessorHost}} when killing a faulty coprocessor, or by guaranteeing
> that the coprocessor's {{stop()}} method will be invoked then.
> This feature request is actually pretty important due to bug HBASE-9046,
> which means that it's not possible to properly clean up a coprocessor without
> restarting the RegionServer (!!).
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Created] (HBASE-10125) KeyValue(Cell c) copy constructor doesn't copy tags data
Jeffrey Zhong created HBASE-10125:
-
Summary: KeyValue(Cell c) copy constructor doesn't copy tags data
Key: HBASE-10125
URL: https://issues.apache.org/jira/browse/HBASE-10125
Project: HBase
Issue Type: Bug
Affects Versions: 0.98.0
Reporter: Jeffrey Zhong
Assignee: Jeffrey Zhong
Fix For: 0.98.0
I happened to come across this:
{code}
public KeyValue(Cell c) {
this(c.getRowArray(), c.getRowOffset(), (int)c.getRowLength(),
c.getFamilyArray(), c.getFamilyOffset(), (int)c.getFamilyLength(),
c.getQualifierArray(), c.getQualifierOffset(), (int)
c.getQualifierLength(),
c.getTimestamp(), Type.codeToType(c.getTypeByte()),
c.getValueArray(), c.getValueOffset(), c.getValueLength());
}
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10111) Verify that a snapshot is not corrupted before restoring it
[ https://issues.apache.org/jira/browse/HBASE-10111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844993#comment-13844993 ] Hudson commented on HBASE-10111: SUCCESS: Integrated in hbase-0.96 #221 (See [https://builds.apache.org/job/hbase-0.96/221/]) HBASE-10111 Verify that a snapshot is not corrupted before restoring it (mbertozzi: rev 1549819) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileInfo.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotReferenceUtil.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java > Verify that a snapshot is not corrupted before restoring it > --- > > Key: HBASE-10111 > URL: https://issues.apache.org/jira/browse/HBASE-10111 > Project: HBase > Issue Type: Bug > Components: snapshots >Affects Versions: 0.98.0, 0.96.0, 0.94.14 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15 > > Attachments: HBASE-10111-v0.patch, HBASE-10111-v1.patch > > > To avoid assigning/opening regions with missing files, verify that the > snapshot is not corrupted before restoring/cloning it. > In 96 a corrupted file in a region is "not a problem" since the assignment > will give up after awhile. > In 94 having a corrupted file in a region means looping forever, on "enable", > until manual intervention. (Easy way to test this is create a table, disable > it, add a corrupted reference file and enable the table to start looping: you > can use echo "foo" > > .) -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10110) Fix Potential Resource Leak in StoreFlusher
[ https://issues.apache.org/jira/browse/HBASE-10110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844995#comment-13844995 ] Hudson commented on HBASE-10110: SUCCESS: Integrated in hbase-0.96 #221 (See [https://builds.apache.org/job/hbase-0.96/221/]) HBASE-10110 Fix Potential Resource Leak in StoreFlusher (eclark: rev 1549929) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFlusher.java > Fix Potential Resource Leak in StoreFlusher > --- > > Key: HBASE-10110 > URL: https://issues.apache.org/jira/browse/HBASE-10110 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.98.0, 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Labels: noob > Fix For: 0.98.0, 0.96.2, 0.99.0 > > Attachments: HBASE-10110-0.patch, HBASE-10110-0.patch > > -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10119) Allow HBase coprocessors to clean up when they fail
[
https://issues.apache.org/jira/browse/HBASE-10119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844990#comment-13844990
]
Hudson commented on HBASE-10119:
SUCCESS: Integrated in hbase-0.96 #221 (See
[https://builds.apache.org/job/hbase-0.96/221/])
HBASE-10119. Allow HBase coprocessors to clean up when they fail (Benoit
Sigoure) (apurtell: rev 1549976)
*
/hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
> Allow HBase coprocessors to clean up when they fail
> ---
>
> Key: HBASE-10119
> URL: https://issues.apache.org/jira/browse/HBASE-10119
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 0.96.0
>Reporter: Benoit Sigoure
>Assignee: Benoit Sigoure
> Fix For: 0.98.0, 0.96.1, 0.99.0
>
> Attachments: HBASE-10119-0.94.patch, HBASE-10119.patch
>
>
> In the thread [Giving a chance to buggy coprocessors to clean
> up|http://osdir.com/ml/general/2013-12/msg17334.html] I brought up the issue
> that coprocessors currently don't have a chance to release their own
> resources (be they internal resources within the JVM, or external resources
> elsewhere) when they get forcefully removed due to an uncaught exception
> escaping.
> It would be nice to fix that, either by adding an API called by the
> {{CoprocessorHost}} when killing a faulty coprocessor, or by guaranteeing
> that the coprocessor's {{stop()}} method will be invoked then.
> This feature request is actually pretty important due to bug HBASE-9046,
> which means that it's not possible to properly clean up a coprocessor without
> restarting the RegionServer (!!).
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10000) Initiate lease recovery for outstanding WAL files at the very beginning of recovery
[
https://issues.apache.org/jira/browse/HBASE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844997#comment-13844997
]
Hadoop QA commented on HBASE-1:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618148/1-recover-ts-with-pb-7.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 25 new
or modified tests.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.util.TestFSHDFSUtils
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8128//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8128//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8128//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8128//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8128//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8128//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8128//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8128//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8128//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8128//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8128//console
This message is automatically generated.
> Initiate lease recovery for outstanding WAL files at the very beginning of
> recovery
> ---
>
> Key: HBASE-1
> URL: https://issues.apache.org/jira/browse/HBASE-1
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Fix For: 0.98.1
>
> Attachments: 1-0.96-v5.txt, 1-0.96-v6.txt,
> 1-recover-ts-with-pb-2.txt, 1-recover-ts-with-pb-3.txt,
> 1-recover-ts-with-pb-4.txt, 1-recover-ts-with-pb-5.txt,
> 1-recover-ts-with-pb-6.txt, 1-recover-ts-with-pb-7.txt,
> 1-recover-ts-with-pb-7.txt, 1-v4.txt, 1-v5.txt, 1-v6.txt
>
>
> At the beginning of recovery, master can send lease recovery requests
> concurrently for outstanding WAL files using a thread pool.
> Each split worker would first check whether the WAL file it processes is
> closed.
> Thanks to Nicolas Liochon and Jeffery discussion with whom gave rise to this
> idea.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-9966) Create IntegrationTest for Online Bloom Filter Change
[ https://issues.apache.org/jira/browse/HBASE-9966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844988#comment-13844988 ] Hudson commented on HBASE-9966: --- SUCCESS: Integrated in hbase-0.96 #221 (See [https://builds.apache.org/job/hbase-0.96/221/]) HBASE-9966. Create integration test for online Bloom Filter change (Aleksandr Shulman) (apurtell: rev 1549990) * /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/ChangeBloomFilterAction.java * /hbase/branches/0.96/hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/factories/SlowDeterministicMonkeyFactory.java > Create IntegrationTest for Online Bloom Filter Change > - > > Key: HBASE-9966 > URL: https://issues.apache.org/jira/browse/HBASE-9966 > Project: HBase > Issue Type: Sub-task > Components: HFile, test >Affects Versions: 0.98.0, 0.96.1 >Reporter: Aleksandr Shulman >Assignee: Aleksandr Shulman > Labels: online_schema_change > Fix For: 0.96.1, 0.98.1, 0.99.0 > > Attachments: HBASE-9966-96.patch, HBASE-9966-98.patch, > HBASE-9966-trunk.patch > > > For online schema change, a user is perfectly with her rights to modify the > compression algorithm used, or the bloom filter. > Therefore, we should add these actions to our ChaosMonkey tests to ensure > that they do not introduce instability. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10112) Hbase rest query params for maxVersions and maxValues are not parsed
[ https://issues.apache.org/jira/browse/HBASE-10112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844994#comment-13844994 ] Hudson commented on HBASE-10112: SUCCESS: Integrated in hbase-0.96 #221 (See [https://builds.apache.org/job/hbase-0.96/221/]) HBASE-10112. REST query params for maxVersions and maxValues are not parsed (Jean-Marc Spaggiari) (apurtell: rev 1549914) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/RowSpec.java > Hbase rest query params for maxVersions and maxValues are not parsed > > > Key: HBASE-10112 > URL: https://issues.apache.org/jira/browse/HBASE-10112 > Project: HBase > Issue Type: Bug > Components: REST >Affects Versions: 0.98.0, 0.96.1, 0.94.14, 0.99.0 >Reporter: Koert Kuipers >Assignee: Jean-Marc Spaggiari >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15, 0.99.0 > > Attachments: HBASE-10112-v0-0.94.patch, HBASE-10112-v0-0.96.patch, > HBASE-10112-v0-0.98.patch, HBASE-10112-v0-trunk.patch > > > i am trying to use maxValues with a "globbed" row resource in stargate. from > looking at the source code one has to do something like > table/row/column(s)/timestamp(s)/?n=1 > (except the ?n=1 piece must be urlencoded) > however i cannot get the n=1 piece to work. i get this stacktrace: > > Problem accessing > /some_table_name/93%2B002%2B*/cf:tx_CUST_NAME/1,13862892906600/%3Fn%3D1. > Reason: > String index out of range: 50Caused > by:java.lang.StringIndexOutOfBoundsException: String index out of > range: 50 > at java.lang.AbstractStringBuilder.charAt(AbstractStringBuilder.java:174) > at java.lang.StringBuilder.charAt(StringBuilder.java:55) > at org.apache.hadoop.hbase.rest.RowSpec.parseQueryParams(RowSpec.java:260) > at org.apache.hadoop.hbase.rest.RowSpec.(RowSpec.java:59) > at > org.apache.hadoop.hbase.rest.RowResource. (RowResource.java:74) > at > org.apache.hadoop.hbase.rest.TableResource.getRowResource(TableResource.java:90) > the offending line is (260 in RowSpec): > c = query.charAt(i); > i think this should be > c = query.charAt(j); > same for line 248 (which handles the maxVersions) > i have not been able to test this (never tried to build hbase myself). -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10116) SlabCache metrics improvements
[ https://issues.apache.org/jira/browse/HBASE-10116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844991#comment-13844991 ] Hudson commented on HBASE-10116: SUCCESS: Integrated in hbase-0.96 #221 (See [https://builds.apache.org/job/hbase-0.96/221/]) HBASE-10116 SlabCache metrics improvements (ndimiduk: rev 1549937) * /hbase/branches/0.96/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/ServerMetricsTmpl.jamon * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CombinedBlockCache.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperImpl.java * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java > SlabCache metrics improvements > -- > > Key: HBASE-10116 > URL: https://issues.apache.org/jira/browse/HBASE-10116 > Project: HBase > Issue Type: Improvement > Components: metrics, regionserver >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 0.98.0, 0.96.2, 0.99.0 > > Attachments: HBASE-10116.00.patch > > > SlabCache incorrectly reports block count and could do a better job reporting > free size. MetricsRegionServerWrapperImpl incorrectly reports block cache > count. Expose Direct Memory stats on RegionServer WebUI. > Also clean up javadoc on BlockCache interface methods, and whitespace in > cache-related classes. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10103) TestNodeHealthCheckChore#testRSHealthChore: Stoppable must have been stopped
[
https://issues.apache.org/jira/browse/HBASE-10103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844992#comment-13844992
]
Hudson commented on HBASE-10103:
SUCCESS: Integrated in hbase-0.96 #221 (See
[https://builds.apache.org/job/hbase-0.96/221/])
Amend HBASE-10103. Fix testHealthCheckerTimeout and add default test timeouts
(apurtell: rev 1549996)
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/TestNodeHealthCheckChore.java
HBASE-10103. TestNodeHealthCheckChore#testRSHealthChore: Stoppable must have
been stopped (apurtell: rev 1549932)
*
/hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/TestNodeHealthCheckChore.java
> TestNodeHealthCheckChore#testRSHealthChore: Stoppable must have been stopped
>
>
> Key: HBASE-10103
> URL: https://issues.apache.org/jira/browse/HBASE-10103
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.99.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Fix For: 0.98.0, 0.96.1, 0.99.0
>
> Attachments: 10103-amend-1.patch, 10103.patch, 10103.patch,
> 10103.patch
>
>
> {noformat}
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 623.639 sec
> <<< FAILURE!
> testRSHealthChore(org.apache.hadoop.hbase.TestNodeHealthCheckChore) Time
> elapsed: 0.001 sec <<< FAILURE!
> java.lang.AssertionError: Stoppable must have been stopped.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hbase.TestNodeHealthCheckChore.testRSHealthChore(TestNodeHealthCheckChore.java:108)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10117) Avoid synchronization in HRegionScannerImpl.isFilterDone
[ https://issues.apache.org/jira/browse/HBASE-10117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844989#comment-13844989 ] Hudson commented on HBASE-10117: SUCCESS: Integrated in hbase-0.96 #221 (See [https://builds.apache.org/job/hbase-0.96/221/]) HBASE-10117 Avoid synchronization in HRegionScannerImpl.isFilterDone (larsh: rev 1549922) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > Avoid synchronization in HRegionScannerImpl.isFilterDone > > > Key: HBASE-10117 > URL: https://issues.apache.org/jira/browse/HBASE-10117 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.94.15, 0.96.2, 0.98.1, 0.99.0 > > Attachments: 10117-0.94-v2.txt, 10117-0.94-v3.txt, 10117-0.94.txt, > 10117-trunk-v2.txt, 10117-trunk.txt > > > A while ago I introduced HRegoinScannerImpl.nextRaw() to allow coprocessors > and scanners with caching > 1 to avoid repeated synchronization during > scanning (which puts up memory fences, which in turn slows things down on > multi core machines). > Looking at the code again I see that isFilterDone() is called from nextRaw() > and isFilterDone() is synchronized. > The caller of nextRaw is required to ensure single threaded access to > nextRaw() anyway, we can call an unsynchronized internal version of > isFilterDone(). -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10120) start-hbase.sh doesn't respect --config in non-distributed mode
[ https://issues.apache.org/jira/browse/HBASE-10120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844974#comment-13844974 ] Lars Hofhansl commented on HBASE-10120: --- +1 for simple bug fix. Generally use your discretion for things like this :) > start-hbase.sh doesn't respect --config in non-distributed mode > --- > > Key: HBASE-10120 > URL: https://issues.apache.org/jira/browse/HBASE-10120 > Project: HBase > Issue Type: Bug > Components: scripts >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Trivial > Fix For: 0.98.0, 0.94.15, 0.96.2, 0.99.0 > > Attachments: HBASE-10120.00.patch > > > A custom value for --config is not passed along to hbase-daemon.sh by > start-hbase.sh when invoked in local mode (hbase.cluster.distributed=false). > When --config is specified, variables from hbase-env.sh are applied to the > runtime, but not hbase-site.xml. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10119) Allow HBase coprocessors to clean up when they fail
[
https://issues.apache.org/jira/browse/HBASE-10119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844972#comment-13844972
]
Lars Hofhansl commented on HBASE-10119:
---
The new behavior is less surprising than the old. So +1 for 0.94
> Allow HBase coprocessors to clean up when they fail
> ---
>
> Key: HBASE-10119
> URL: https://issues.apache.org/jira/browse/HBASE-10119
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 0.96.0
>Reporter: Benoit Sigoure
>Assignee: Benoit Sigoure
> Fix For: 0.98.0, 0.96.1, 0.99.0
>
> Attachments: HBASE-10119-0.94.patch, HBASE-10119.patch
>
>
> In the thread [Giving a chance to buggy coprocessors to clean
> up|http://osdir.com/ml/general/2013-12/msg17334.html] I brought up the issue
> that coprocessors currently don't have a chance to release their own
> resources (be they internal resources within the JVM, or external resources
> elsewhere) when they get forcefully removed due to an uncaught exception
> escaping.
> It would be nice to fix that, either by adding an API called by the
> {{CoprocessorHost}} when killing a faulty coprocessor, or by guaranteeing
> that the coprocessor's {{stop()}} method will be invoked then.
> This feature request is actually pretty important due to bug HBASE-9046,
> which means that it's not possible to properly clean up a coprocessor without
> restarting the RegionServer (!!).
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10069) Potential duplicate calls to log#appendNoSync() in HRegion#doMiniBatchMutation()
[
https://issues.apache.org/jira/browse/HBASE-10069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844970#comment-13844970
]
Hadoop QA commented on HBASE-10069:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618145/HBASE-10069.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop1.0{color}. The patch compiles against the hadoop
1.0 profile.
{color:green}+1 hadoop1.1{color}. The patch compiles against the hadoop
1.1 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 3 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/8127//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8127//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8127//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8127//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8127//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8127//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8127//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8127//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8127//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/8127//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/8127//console
This message is automatically generated.
> Potential duplicate calls to log#appendNoSync() in
> HRegion#doMiniBatchMutation()
>
>
> Key: HBASE-10069
> URL: https://issues.apache.org/jira/browse/HBASE-10069
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Sergey Shelukhin
>Priority: Minor
> Attachments: HBASE-10069.patch
>
>
> In HRegion#doMiniBatchMutation():
> {code}
> if (nonceGroup != currentNonceGroup || nonce != currentNonce) {
> if (walEdit.size() > 0) {
> assert isInReplay;
> txid = this.log.appendNoSync(this.getRegionInfo(),
> htableDescriptor.getTableName(),
> walEdit, m.getClusterIds(), now, htableDescriptor,
> this.sequenceId, true,
> currentNonceGroup, currentNonce);
> hasWalAppends = true;
> }
> currentNonceGroup = nonceGroup;
> currentNonce = nonce;
> }
> // Add WAL edits by CP
> WALEdit fromCP = batchOp.walEditsFromCoprocessors[i];
> if (fromCP != null) {
> for (KeyValue kv : fromCP.getKeyValues()) {
> walEdit.add(kv);
> }
> }
> ...
> Mutation mutation = batchOp.getMutation(firstIndex);
> if (walEdit.size() > 0) {
> txid = this.log.appendNoSync(this.getRegionInfo(),
> this.htableDescriptor.getTableName(),
> walEdit, mutation.getClusterIds(), now, this.htableDescriptor,
> this.sequenceId,
> true, currentNonceGroup, currentNonce);
> hasWalAppends = true;
> }
> {code}
> If fromCP is null, th
[jira] [Commented] (HBASE-9047) Tool to handle finishing replication when the cluster is offline
[
https://issues.apache.org/jira/browse/HBASE-9047?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844966#comment-13844966
]
Demai Ni commented on HBASE-9047:
-
[~lhofhansl],
I went through the findbug issues, and didn't identify any introduced by this
patch. The only one related with replications is
{code}
org.apache.hadoop.hbase.replication.regionserver.ReplicationSource.processEndOfFile()
might ignore java.io.IOException
{code}
this is an existing one
> Tool to handle finishing replication when the cluster is offline
>
>
> Key: HBASE-9047
> URL: https://issues.apache.org/jira/browse/HBASE-9047
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 0.96.0
>Reporter: Jean-Daniel Cryans
>Assignee: Demai Ni
> Fix For: 0.98.0
>
> Attachments: HBASE-9047-0.94.9-v0.PATCH, HBASE-9047-trunk-v0.patch,
> HBASE-9047-trunk-v1.patch, HBASE-9047-trunk-v2.patch,
> HBASE-9047-trunk-v3.patch, HBASE-9047-trunk-v4.patch,
> HBASE-9047-trunk-v4.patch, HBASE-9047-trunk-v5.patch,
> HBASE-9047-trunk-v6.patch, HBASE-9047-trunk-v7.patch
>
>
> We're having a discussion on the mailing list about replicating the data on a
> cluster that was shut down in an offline fashion. The motivation could be
> that you don't want to bring HBase back up but still need that data on the
> slave.
> So I have this idea of a tool that would be running on the master cluster
> while it is down, although it could also run at any time. Basically it would
> be able to read the replication state of each master region server, finish
> replicating what's missing to all the slave, and then clear that state in
> zookeeper.
> The code that handles replication does most of that already, see
> ReplicationSourceManager and ReplicationSource. Basically when
> ReplicationSourceManager.init() is called, it will check all the queues in ZK
> and try to grab those that aren't attached to a region server. If the whole
> cluster is down, it will grab all of them.
> The beautiful thing here is that you could start that tool on all your
> machines and the load will be spread out, but that might not be a big concern
> if replication wasn't lagging since it would take a few seconds to finish
> replicating the missing data for each region server.
> I'm guessing when starting ReplicationSourceManager you'd give it a fake
> region server ID, and you'd tell it not to start its own source.
> FWIW the main difference in how replication is handled between Apache's HBase
> and Facebook's is that the latter is always done separately of HBase itself.
> This jira isn't about doing that.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10112) Hbase rest query params for maxVersions and maxValues are not parsed
[ https://issues.apache.org/jira/browse/HBASE-10112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844962#comment-13844962 ] Hudson commented on HBASE-10112: SUCCESS: Integrated in HBase-0.98 #7 (See [https://builds.apache.org/job/HBase-0.98/7/]) HBASE-10112 Hbase rest query params for maxVersions and maxValues are not parsed (tedyu: rev 1549900) * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/RowSpec.java > Hbase rest query params for maxVersions and maxValues are not parsed > > > Key: HBASE-10112 > URL: https://issues.apache.org/jira/browse/HBASE-10112 > Project: HBase > Issue Type: Bug > Components: REST >Affects Versions: 0.98.0, 0.96.1, 0.94.14, 0.99.0 >Reporter: Koert Kuipers >Assignee: Jean-Marc Spaggiari >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15, 0.99.0 > > Attachments: HBASE-10112-v0-0.94.patch, HBASE-10112-v0-0.96.patch, > HBASE-10112-v0-0.98.patch, HBASE-10112-v0-trunk.patch > > > i am trying to use maxValues with a "globbed" row resource in stargate. from > looking at the source code one has to do something like > table/row/column(s)/timestamp(s)/?n=1 > (except the ?n=1 piece must be urlencoded) > however i cannot get the n=1 piece to work. i get this stacktrace: > > Problem accessing > /some_table_name/93%2B002%2B*/cf:tx_CUST_NAME/1,13862892906600/%3Fn%3D1. > Reason: > String index out of range: 50Caused > by:java.lang.StringIndexOutOfBoundsException: String index out of > range: 50 > at java.lang.AbstractStringBuilder.charAt(AbstractStringBuilder.java:174) > at java.lang.StringBuilder.charAt(StringBuilder.java:55) > at org.apache.hadoop.hbase.rest.RowSpec.parseQueryParams(RowSpec.java:260) > at org.apache.hadoop.hbase.rest.RowSpec.(RowSpec.java:59) > at > org.apache.hadoop.hbase.rest.RowResource. (RowResource.java:74) > at > org.apache.hadoop.hbase.rest.TableResource.getRowResource(TableResource.java:90) > the offending line is (260 in RowSpec): > c = query.charAt(i); > i think this should be > c = query.charAt(j); > same for line 248 (which handles the maxVersions) > i have not been able to test this (never tried to build hbase myself). -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10110) Fix Potential Resource Leak in StoreFlusher
[ https://issues.apache.org/jira/browse/HBASE-10110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844963#comment-13844963 ] Hudson commented on HBASE-10110: SUCCESS: Integrated in HBase-0.98 #7 (See [https://builds.apache.org/job/HBase-0.98/7/]) HBASE-10110 Fix Potential Resource Leak in StoreFlusher (eclark: rev 1549928) * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFlusher.java > Fix Potential Resource Leak in StoreFlusher > --- > > Key: HBASE-10110 > URL: https://issues.apache.org/jira/browse/HBASE-10110 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.98.0, 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Labels: noob > Fix For: 0.98.0, 0.96.2, 0.99.0 > > Attachments: HBASE-10110-0.patch, HBASE-10110-0.patch > > -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10103) TestNodeHealthCheckChore#testRSHealthChore: Stoppable must have been stopped
[
https://issues.apache.org/jira/browse/HBASE-10103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844960#comment-13844960
]
Hudson commented on HBASE-10103:
SUCCESS: Integrated in HBase-0.98 #7 (See
[https://builds.apache.org/job/HBase-0.98/7/])
Amend HBASE-10103. Fix testHealthCheckerTimeout and add default test timeouts
(apurtell: rev 1549995)
*
/hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/TestNodeHealthCheckChore.java
HBASE-10103. TestNodeHealthCheckChore#testRSHealthChore: Stoppable must have
been stopped (apurtell: rev 1549931)
*
/hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/TestNodeHealthCheckChore.java
> TestNodeHealthCheckChore#testRSHealthChore: Stoppable must have been stopped
>
>
> Key: HBASE-10103
> URL: https://issues.apache.org/jira/browse/HBASE-10103
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.99.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Fix For: 0.98.0, 0.96.1, 0.99.0
>
> Attachments: 10103-amend-1.patch, 10103.patch, 10103.patch,
> 10103.patch
>
>
> {noformat}
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 623.639 sec
> <<< FAILURE!
> testRSHealthChore(org.apache.hadoop.hbase.TestNodeHealthCheckChore) Time
> elapsed: 0.001 sec <<< FAILURE!
> java.lang.AssertionError: Stoppable must have been stopped.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hbase.TestNodeHealthCheckChore.testRSHealthChore(TestNodeHealthCheckChore.java:108)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10069) Potential duplicate calls to log#appendNoSync() in HRegion#doMiniBatchMutation()
[
https://issues.apache.org/jira/browse/HBASE-10069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844965#comment-13844965
]
Sergey Shelukhin commented on HBASE-10069:
--
will commit if tests pass
> Potential duplicate calls to log#appendNoSync() in
> HRegion#doMiniBatchMutation()
>
>
> Key: HBASE-10069
> URL: https://issues.apache.org/jira/browse/HBASE-10069
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Sergey Shelukhin
>Priority: Minor
> Attachments: HBASE-10069.patch
>
>
> In HRegion#doMiniBatchMutation():
> {code}
> if (nonceGroup != currentNonceGroup || nonce != currentNonce) {
> if (walEdit.size() > 0) {
> assert isInReplay;
> txid = this.log.appendNoSync(this.getRegionInfo(),
> htableDescriptor.getTableName(),
> walEdit, m.getClusterIds(), now, htableDescriptor,
> this.sequenceId, true,
> currentNonceGroup, currentNonce);
> hasWalAppends = true;
> }
> currentNonceGroup = nonceGroup;
> currentNonce = nonce;
> }
> // Add WAL edits by CP
> WALEdit fromCP = batchOp.walEditsFromCoprocessors[i];
> if (fromCP != null) {
> for (KeyValue kv : fromCP.getKeyValues()) {
> walEdit.add(kv);
> }
> }
> ...
> Mutation mutation = batchOp.getMutation(firstIndex);
> if (walEdit.size() > 0) {
> txid = this.log.appendNoSync(this.getRegionInfo(),
> this.htableDescriptor.getTableName(),
> walEdit, mutation.getClusterIds(), now, this.htableDescriptor,
> this.sequenceId,
> true, currentNonceGroup, currentNonce);
> hasWalAppends = true;
> }
> {code}
> If fromCP is null, there may not be new edits added to walEdit.
> But log#appendNoSync() would be called one more time at line 2368.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10119) Allow HBase coprocessors to clean up when they fail
[
https://issues.apache.org/jira/browse/HBASE-10119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844958#comment-13844958
]
Hudson commented on HBASE-10119:
SUCCESS: Integrated in HBase-0.98 #7 (See
[https://builds.apache.org/job/HBase-0.98/7/])
HBASE-10119. Allow HBase coprocessors to clean up when they fail (Benoit
Sigoure) (apurtell: rev 1549975)
*
/hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
> Allow HBase coprocessors to clean up when they fail
> ---
>
> Key: HBASE-10119
> URL: https://issues.apache.org/jira/browse/HBASE-10119
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 0.96.0
>Reporter: Benoit Sigoure
>Assignee: Benoit Sigoure
> Fix For: 0.98.0, 0.96.1, 0.99.0
>
> Attachments: HBASE-10119-0.94.patch, HBASE-10119.patch
>
>
> In the thread [Giving a chance to buggy coprocessors to clean
> up|http://osdir.com/ml/general/2013-12/msg17334.html] I brought up the issue
> that coprocessors currently don't have a chance to release their own
> resources (be they internal resources within the JVM, or external resources
> elsewhere) when they get forcefully removed due to an uncaught exception
> escaping.
> It would be nice to fix that, either by adding an API called by the
> {{CoprocessorHost}} when killing a faulty coprocessor, or by guaranteeing
> that the coprocessor's {{stop()}} method will be invoked then.
> This feature request is actually pretty important due to bug HBASE-9046,
> which means that it's not possible to properly clean up a coprocessor without
> restarting the RegionServer (!!).
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10117) Avoid synchronization in HRegionScannerImpl.isFilterDone
[ https://issues.apache.org/jira/browse/HBASE-10117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844957#comment-13844957 ] Hudson commented on HBASE-10117: SUCCESS: Integrated in HBase-0.98 #7 (See [https://builds.apache.org/job/HBase-0.98/7/]) HBASE-10117 Avoid synchronization in HRegionScannerImpl.isFilterDone (larsh: rev 1549921) * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > Avoid synchronization in HRegionScannerImpl.isFilterDone > > > Key: HBASE-10117 > URL: https://issues.apache.org/jira/browse/HBASE-10117 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.94.15, 0.96.2, 0.98.1, 0.99.0 > > Attachments: 10117-0.94-v2.txt, 10117-0.94-v3.txt, 10117-0.94.txt, > 10117-trunk-v2.txt, 10117-trunk.txt > > > A while ago I introduced HRegoinScannerImpl.nextRaw() to allow coprocessors > and scanners with caching > 1 to avoid repeated synchronization during > scanning (which puts up memory fences, which in turn slows things down on > multi core machines). > Looking at the code again I see that isFilterDone() is called from nextRaw() > and isFilterDone() is synchronized. > The caller of nextRaw is required to ensure single threaded access to > nextRaw() anyway, we can call an unsynchronized internal version of > isFilterDone(). -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10116) SlabCache metrics improvements
[ https://issues.apache.org/jira/browse/HBASE-10116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844959#comment-13844959 ] Hudson commented on HBASE-10116: SUCCESS: Integrated in HBase-0.98 #7 (See [https://builds.apache.org/job/HBase-0.98/7/]) HBASE-10116 SlabCache metrics improvements (ndimiduk: rev 1549936) * /hbase/branches/0.98/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/ServerMetricsTmpl.jamon * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CombinedBlockCache.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperImpl.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java > SlabCache metrics improvements > -- > > Key: HBASE-10116 > URL: https://issues.apache.org/jira/browse/HBASE-10116 > Project: HBase > Issue Type: Improvement > Components: metrics, regionserver >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 0.98.0, 0.96.2, 0.99.0 > > Attachments: HBASE-10116.00.patch > > > SlabCache incorrectly reports block count and could do a better job reporting > free size. MetricsRegionServerWrapperImpl incorrectly reports block cache > count. Expose Direct Memory stats on RegionServer WebUI. > Also clean up javadoc on BlockCache interface methods, and whitespace in > cache-related classes. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10111) Verify that a snapshot is not corrupted before restoring it
[ https://issues.apache.org/jira/browse/HBASE-10111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844961#comment-13844961 ] Hudson commented on HBASE-10111: SUCCESS: Integrated in HBase-0.98 #7 (See [https://builds.apache.org/job/HBase-0.98/7/]) HBASE-10111 Verify that a snapshot is not corrupted before restoring it (mbertozzi: rev 1549818) * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileInfo.java * /hbase/branches/0.98/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotReferenceUtil.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java * /hbase/branches/0.98/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java > Verify that a snapshot is not corrupted before restoring it > --- > > Key: HBASE-10111 > URL: https://issues.apache.org/jira/browse/HBASE-10111 > Project: HBase > Issue Type: Bug > Components: snapshots >Affects Versions: 0.98.0, 0.96.0, 0.94.14 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15 > > Attachments: HBASE-10111-v0.patch, HBASE-10111-v1.patch > > > To avoid assigning/opening regions with missing files, verify that the > snapshot is not corrupted before restoring/cloning it. > In 96 a corrupted file in a region is "not a problem" since the assignment > will give up after awhile. > In 94 having a corrupted file in a region means looping forever, on "enable", > until manual intervention. (Easy way to test this is create a table, disable > it, add a corrupted reference file and enable the table to start looping: you > can use echo "foo" > > .) -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-9966) Create IntegrationTest for Online Bloom Filter Change
[ https://issues.apache.org/jira/browse/HBASE-9966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844956#comment-13844956 ] Hudson commented on HBASE-9966: --- SUCCESS: Integrated in HBase-0.98 #7 (See [https://builds.apache.org/job/HBase-0.98/7/]) HBASE-9966. Create integration test for online Bloom Filter change (Aleksandr Shulman) (apurtell: rev 1549989) * /hbase/branches/0.98/hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/ChangeBloomFilterAction.java * /hbase/branches/0.98/hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/factories/SlowDeterministicMonkeyFactory.java > Create IntegrationTest for Online Bloom Filter Change > - > > Key: HBASE-9966 > URL: https://issues.apache.org/jira/browse/HBASE-9966 > Project: HBase > Issue Type: Sub-task > Components: HFile, test >Affects Versions: 0.98.0, 0.96.1 >Reporter: Aleksandr Shulman >Assignee: Aleksandr Shulman > Labels: online_schema_change > Fix For: 0.96.1, 0.98.1, 0.99.0 > > Attachments: HBASE-9966-96.patch, HBASE-9966-98.patch, > HBASE-9966-trunk.patch > > > For online schema change, a user is perfectly with her rights to modify the > compression algorithm used, or the bloom filter. > Therefore, we should add these actions to our ChaosMonkey tests to ensure > that they do not introduce instability. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10114) _scan_internal() in table.rb should accept argument that specifies reverse scan
[
https://issues.apache.org/jira/browse/HBASE-10114?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844952#comment-13844952
]
chunhui shen commented on HBASE-10114:
--
Committed to trunk and 0.98.
Thanks Ted.
> _scan_internal() in table.rb should accept argument that specifies reverse
> scan
> ---
>
> Key: HBASE-10114
> URL: https://issues.apache.org/jira/browse/HBASE-10114
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
> Fix For: 0.98.0, 0.99.0
>
> Attachments: hbase-10114-trunk.patch
>
>
> Reversed Scan usage on Shell:
> hbase> scan 't1', {REVERSED => true}
> Other parameter could also be added with 'REVERSED', like LIMIT,COLUMNS.
> Examples:
> hbase(main):019:0* scan 't1'
> ROW COLUMN+CELL
>
>
> 001 column=f1:q1,
> timestamp=1386644670350, value=v1
>
> 002 column=f1:q2,
> timestamp=1386644675585, value=v1
>
> 003 column=f1:q1,
> timestamp=1386644680090, value=v1
>
> 003 column=f1:q2,
> timestamp=1386644682749, value=v1
>
> 004 column=f1:q1,
> timestamp=1386644692589, value=v1
>
> 4 row(s) in 0.0130 seconds
> hbase(main):022:0> scan 't1',{REVERSED => true}
> ROW COLUMN+CELL
>
>
> 004 column=f1:q1,
> timestamp=1386644692589, value=v1
>
> 003 column=f1:q1,
> timestamp=1386644680090, value=v1
>
> 003 column=f1:q2,
> timestamp=1386644682749, value=v1
>
> 002 column=f1:q2,
> timestamp=1386644675585, value=v1
>
> 001 column=f1:q1,
> timestamp=1386644670350, value=v1
> hbase(main):023:0> scan 't1',{REVERSED => true,COLUMNS=>'f1:q2'}
> ROW COLUMN+CELL
>
>
> 003 column=f1:q2,
> timestamp=1386644682749, value=v1
>
> 002 column=f1:q2,
> timestamp=1386644675585, value=v1
>
> 2 row(s) in 0.0100 seconds
> Here is how boolean argument is accepted:
> {code}
> cache_blocks = args["CACHE_BLOCKS"] || true
> {code}
> Flag for reverse scan should be supported in similar manner
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HBASE-10114) _scan_internal() in table.rb should accept argument that specifies reverse scan
[
https://issues.apache.org/jira/browse/HBASE-10114?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
chunhui shen updated HBASE-10114:
-
Resolution: Fixed
Assignee: chunhui shen
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> _scan_internal() in table.rb should accept argument that specifies reverse
> scan
> ---
>
> Key: HBASE-10114
> URL: https://issues.apache.org/jira/browse/HBASE-10114
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: chunhui shen
> Fix For: 0.98.0, 0.99.0
>
> Attachments: hbase-10114-trunk.patch
>
>
> Reversed Scan usage on Shell:
> hbase> scan 't1', {REVERSED => true}
> Other parameter could also be added with 'REVERSED', like LIMIT,COLUMNS.
> Examples:
> hbase(main):019:0* scan 't1'
> ROW COLUMN+CELL
>
>
> 001 column=f1:q1,
> timestamp=1386644670350, value=v1
>
> 002 column=f1:q2,
> timestamp=1386644675585, value=v1
>
> 003 column=f1:q1,
> timestamp=1386644680090, value=v1
>
> 003 column=f1:q2,
> timestamp=1386644682749, value=v1
>
> 004 column=f1:q1,
> timestamp=1386644692589, value=v1
>
> 4 row(s) in 0.0130 seconds
> hbase(main):022:0> scan 't1',{REVERSED => true}
> ROW COLUMN+CELL
>
>
> 004 column=f1:q1,
> timestamp=1386644692589, value=v1
>
> 003 column=f1:q1,
> timestamp=1386644680090, value=v1
>
> 003 column=f1:q2,
> timestamp=1386644682749, value=v1
>
> 002 column=f1:q2,
> timestamp=1386644675585, value=v1
>
> 001 column=f1:q1,
> timestamp=1386644670350, value=v1
> hbase(main):023:0> scan 't1',{REVERSED => true,COLUMNS=>'f1:q2'}
> ROW COLUMN+CELL
>
>
> 003 column=f1:q2,
> timestamp=1386644682749, value=v1
>
> 002 column=f1:q2,
> timestamp=1386644675585, value=v1
>
> 2 row(s) in 0.0100 seconds
> Here is how boolean argument is accepted:
> {code}
> cache_blocks = args["CACHE_BLOCKS"] || true
> {code}
> Flag for reverse scan should be supported in similar manner
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10124) Make Sub Classes Static When Possible
[ https://issues.apache.org/jira/browse/HBASE-10124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844945#comment-13844945 ] stack commented on HBASE-10124: --- +1 > Make Sub Classes Static When Possible > - > > Key: HBASE-10124 > URL: https://issues.apache.org/jira/browse/HBASE-10124 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.98.0, 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Labels: noob > Attachments: HBASE-10124-0.patch > > > Coverity has found a few places where it's possible to change a private inner > class to static with out any other code changes. This will be a small perf > win. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10110) Fix Potential Resource Leak in StoreFlusher
[ https://issues.apache.org/jira/browse/HBASE-10110?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844941#comment-13844941 ] Hudson commented on HBASE-10110: SUCCESS: Integrated in HBase-TRUNK #4719 (See [https://builds.apache.org/job/HBase-TRUNK/4719/]) HBASE-10110 Fix Potential Resource Leak in StoreFlusher (eclark: rev 1549927) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFlusher.java > Fix Potential Resource Leak in StoreFlusher > --- > > Key: HBASE-10110 > URL: https://issues.apache.org/jira/browse/HBASE-10110 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.98.0, 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Labels: noob > Fix For: 0.98.0, 0.96.2, 0.99.0 > > Attachments: HBASE-10110-0.patch, HBASE-10110-0.patch > > -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10112) Hbase rest query params for maxVersions and maxValues are not parsed
[ https://issues.apache.org/jira/browse/HBASE-10112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844940#comment-13844940 ] Hudson commented on HBASE-10112: SUCCESS: Integrated in HBase-TRUNK #4719 (See [https://builds.apache.org/job/HBase-TRUNK/4719/]) HBASE-10112 Hbase rest query params for maxVersions and maxValues are not parsed (tedyu: rev 1549899) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/rest/RowSpec.java > Hbase rest query params for maxVersions and maxValues are not parsed > > > Key: HBASE-10112 > URL: https://issues.apache.org/jira/browse/HBASE-10112 > Project: HBase > Issue Type: Bug > Components: REST >Affects Versions: 0.98.0, 0.96.1, 0.94.14, 0.99.0 >Reporter: Koert Kuipers >Assignee: Jean-Marc Spaggiari >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15, 0.99.0 > > Attachments: HBASE-10112-v0-0.94.patch, HBASE-10112-v0-0.96.patch, > HBASE-10112-v0-0.98.patch, HBASE-10112-v0-trunk.patch > > > i am trying to use maxValues with a "globbed" row resource in stargate. from > looking at the source code one has to do something like > table/row/column(s)/timestamp(s)/?n=1 > (except the ?n=1 piece must be urlencoded) > however i cannot get the n=1 piece to work. i get this stacktrace: > > Problem accessing > /some_table_name/93%2B002%2B*/cf:tx_CUST_NAME/1,13862892906600/%3Fn%3D1. > Reason: > String index out of range: 50Caused > by:java.lang.StringIndexOutOfBoundsException: String index out of > range: 50 > at java.lang.AbstractStringBuilder.charAt(AbstractStringBuilder.java:174) > at java.lang.StringBuilder.charAt(StringBuilder.java:55) > at org.apache.hadoop.hbase.rest.RowSpec.parseQueryParams(RowSpec.java:260) > at org.apache.hadoop.hbase.rest.RowSpec.(RowSpec.java:59) > at > org.apache.hadoop.hbase.rest.RowResource. (RowResource.java:74) > at > org.apache.hadoop.hbase.rest.TableResource.getRowResource(TableResource.java:90) > the offending line is (260 in RowSpec): > c = query.charAt(i); > i think this should be > c = query.charAt(j); > same for line 248 (which handles the maxVersions) > i have not been able to test this (never tried to build hbase myself). -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10117) Avoid synchronization in HRegionScannerImpl.isFilterDone
[ https://issues.apache.org/jira/browse/HBASE-10117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844935#comment-13844935 ] Hudson commented on HBASE-10117: SUCCESS: Integrated in HBase-TRUNK #4719 (See [https://builds.apache.org/job/HBase-TRUNK/4719/]) HBASE-10117 Avoid synchronization in HRegionScannerImpl.isFilterDone (larsh: rev 1549920) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > Avoid synchronization in HRegionScannerImpl.isFilterDone > > > Key: HBASE-10117 > URL: https://issues.apache.org/jira/browse/HBASE-10117 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.94.15, 0.96.2, 0.98.1, 0.99.0 > > Attachments: 10117-0.94-v2.txt, 10117-0.94-v3.txt, 10117-0.94.txt, > 10117-trunk-v2.txt, 10117-trunk.txt > > > A while ago I introduced HRegoinScannerImpl.nextRaw() to allow coprocessors > and scanners with caching > 1 to avoid repeated synchronization during > scanning (which puts up memory fences, which in turn slows things down on > multi core machines). > Looking at the code again I see that isFilterDone() is called from nextRaw() > and isFilterDone() is synchronized. > The caller of nextRaw is required to ensure single threaded access to > nextRaw() anyway, we can call an unsynchronized internal version of > isFilterDone(). -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10111) Verify that a snapshot is not corrupted before restoring it
[ https://issues.apache.org/jira/browse/HBASE-10111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844939#comment-13844939 ] Hudson commented on HBASE-10111: SUCCESS: Integrated in HBase-TRUNK #4719 (See [https://builds.apache.org/job/HBase-TRUNK/4719/]) HBASE-10111 Verify that a snapshot is not corrupted before restoring it (mbertozzi: rev 1549817) * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreFileInfo.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotReferenceUtil.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java > Verify that a snapshot is not corrupted before restoring it > --- > > Key: HBASE-10111 > URL: https://issues.apache.org/jira/browse/HBASE-10111 > Project: HBase > Issue Type: Bug > Components: snapshots >Affects Versions: 0.98.0, 0.96.0, 0.94.14 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15 > > Attachments: HBASE-10111-v0.patch, HBASE-10111-v1.patch > > > To avoid assigning/opening regions with missing files, verify that the > snapshot is not corrupted before restoring/cloning it. > In 96 a corrupted file in a region is "not a problem" since the assignment > will give up after awhile. > In 94 having a corrupted file in a region means looping forever, on "enable", > until manual intervention. (Easy way to test this is create a table, disable > it, add a corrupted reference file and enable the table to start looping: you > can use echo "foo" > > .) -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10103) TestNodeHealthCheckChore#testRSHealthChore: Stoppable must have been stopped
[
https://issues.apache.org/jira/browse/HBASE-10103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844938#comment-13844938
]
Hudson commented on HBASE-10103:
SUCCESS: Integrated in HBase-TRUNK #4719 (See
[https://builds.apache.org/job/HBase-TRUNK/4719/])
Amend HBASE-10103. Fix testHealthCheckerTimeout and add default test timeouts
(apurtell: rev 1549994)
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestNodeHealthCheckChore.java
HBASE-10103. TestNodeHealthCheckChore#testRSHealthChore: Stoppable must have
been stopped (apurtell: rev 1549930)
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/TestNodeHealthCheckChore.java
> TestNodeHealthCheckChore#testRSHealthChore: Stoppable must have been stopped
>
>
> Key: HBASE-10103
> URL: https://issues.apache.org/jira/browse/HBASE-10103
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.99.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Fix For: 0.98.0, 0.96.1, 0.99.0
>
> Attachments: 10103-amend-1.patch, 10103.patch, 10103.patch,
> 10103.patch
>
>
> {noformat}
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 623.639 sec
> <<< FAILURE!
> testRSHealthChore(org.apache.hadoop.hbase.TestNodeHealthCheckChore) Time
> elapsed: 0.001 sec <<< FAILURE!
> java.lang.AssertionError: Stoppable must have been stopped.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hbase.TestNodeHealthCheckChore.testRSHealthChore(TestNodeHealthCheckChore.java:108)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10116) SlabCache metrics improvements
[ https://issues.apache.org/jira/browse/HBASE-10116?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844937#comment-13844937 ] Hudson commented on HBASE-10116: SUCCESS: Integrated in HBase-TRUNK #4719 (See [https://builds.apache.org/job/HBase-TRUNK/4719/]) HBASE-10116 SlabCache metrics improvements (ndimiduk: rev 1549935) * /hbase/trunk/hbase-server/src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/ServerMetricsTmpl.jamon * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/CombinedBlockCache.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerWrapperImpl.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/DirectMemoryUtils.java > SlabCache metrics improvements > -- > > Key: HBASE-10116 > URL: https://issues.apache.org/jira/browse/HBASE-10116 > Project: HBase > Issue Type: Improvement > Components: metrics, regionserver >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Minor > Fix For: 0.98.0, 0.96.2, 0.99.0 > > Attachments: HBASE-10116.00.patch > > > SlabCache incorrectly reports block count and could do a better job reporting > free size. MetricsRegionServerWrapperImpl incorrectly reports block cache > count. Expose Direct Memory stats on RegionServer WebUI. > Also clean up javadoc on BlockCache interface methods, and whitespace in > cache-related classes. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10119) Allow HBase coprocessors to clean up when they fail
[
https://issues.apache.org/jira/browse/HBASE-10119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844936#comment-13844936
]
Hudson commented on HBASE-10119:
SUCCESS: Integrated in HBase-TRUNK #4719 (See
[https://builds.apache.org/job/HBase-TRUNK/4719/])
HBASE-10119. Allow HBase coprocessors to clean up when they fail (Benoit
Sigoure) (apurtell: rev 1549974)
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/CoprocessorHost.java
> Allow HBase coprocessors to clean up when they fail
> ---
>
> Key: HBASE-10119
> URL: https://issues.apache.org/jira/browse/HBASE-10119
> Project: HBase
> Issue Type: New Feature
>Affects Versions: 0.96.0
>Reporter: Benoit Sigoure
>Assignee: Benoit Sigoure
> Fix For: 0.98.0, 0.96.1, 0.99.0
>
> Attachments: HBASE-10119-0.94.patch, HBASE-10119.patch
>
>
> In the thread [Giving a chance to buggy coprocessors to clean
> up|http://osdir.com/ml/general/2013-12/msg17334.html] I brought up the issue
> that coprocessors currently don't have a chance to release their own
> resources (be they internal resources within the JVM, or external resources
> elsewhere) when they get forcefully removed due to an uncaught exception
> escaping.
> It would be nice to fix that, either by adding an API called by the
> {{CoprocessorHost}} when killing a faulty coprocessor, or by guaranteeing
> that the coprocessor's {{stop()}} method will be invoked then.
> This feature request is actually pretty important due to bug HBASE-9046,
> which means that it's not possible to properly clean up a coprocessor without
> restarting the RegionServer (!!).
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-9966) Create IntegrationTest for Online Bloom Filter Change
[ https://issues.apache.org/jira/browse/HBASE-9966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844934#comment-13844934 ] Hudson commented on HBASE-9966: --- SUCCESS: Integrated in HBase-TRUNK #4719 (See [https://builds.apache.org/job/HBase-TRUNK/4719/]) HBASE-9966. Create integration test for online Bloom Filter change (Aleksandr Shulman) (apurtell: rev 1549986) * /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/actions/ChangeBloomFilterAction.java * /hbase/trunk/hbase-it/src/test/java/org/apache/hadoop/hbase/chaos/factories/SlowDeterministicMonkeyFactory.java > Create IntegrationTest for Online Bloom Filter Change > - > > Key: HBASE-9966 > URL: https://issues.apache.org/jira/browse/HBASE-9966 > Project: HBase > Issue Type: Sub-task > Components: HFile, test >Affects Versions: 0.98.0, 0.96.1 >Reporter: Aleksandr Shulman >Assignee: Aleksandr Shulman > Labels: online_schema_change > Fix For: 0.96.1, 0.98.1, 0.99.0 > > Attachments: HBASE-9966-96.patch, HBASE-9966-98.patch, > HBASE-9966-trunk.patch > > > For online schema change, a user is perfectly with her rights to modify the > compression algorithm used, or the bloom filter. > Therefore, we should add these actions to our ChaosMonkey tests to ensure > that they do not introduce instability. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HBASE-10124) Make Sub Classes Static When Possible
[ https://issues.apache.org/jira/browse/HBASE-10124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-10124: -- Attachment: HBASE-10124-0.patch > Make Sub Classes Static When Possible > - > > Key: HBASE-10124 > URL: https://issues.apache.org/jira/browse/HBASE-10124 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.98.0, 0.96.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Labels: noob > Attachments: HBASE-10124-0.patch > > > Coverity has found a few places where it's possible to change a private inner > class to static with out any other code changes. This will be a small perf > win. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HBASE-10124) Make Sub Classes Static When Possible
[ https://issues.apache.org/jira/browse/HBASE-10124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-10124: -- Status: Patch Available (was: Open) > Make Sub Classes Static When Possible > - > > Key: HBASE-10124 > URL: https://issues.apache.org/jira/browse/HBASE-10124 > Project: HBase > Issue Type: Sub-task >Affects Versions: 0.96.0, 0.98.0 >Reporter: Elliott Clark >Assignee: Elliott Clark > Labels: noob > Attachments: HBASE-10124-0.patch > > > Coverity has found a few places where it's possible to change a private inner > class to static with out any other code changes. This will be a small perf > win. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HBASE-10000) Initiate lease recovery for outstanding WAL files at the very beginning of recovery
[ https://issues.apache.org/jira/browse/HBASE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-1: --- Attachment: 1-recover-ts-with-pb-7.txt > Initiate lease recovery for outstanding WAL files at the very beginning of > recovery > --- > > Key: HBASE-1 > URL: https://issues.apache.org/jira/browse/HBASE-1 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.98.1 > > Attachments: 1-0.96-v5.txt, 1-0.96-v6.txt, > 1-recover-ts-with-pb-2.txt, 1-recover-ts-with-pb-3.txt, > 1-recover-ts-with-pb-4.txt, 1-recover-ts-with-pb-5.txt, > 1-recover-ts-with-pb-6.txt, 1-recover-ts-with-pb-7.txt, > 1-recover-ts-with-pb-7.txt, 1-v4.txt, 1-v5.txt, 1-v6.txt > > > At the beginning of recovery, master can send lease recovery requests > concurrently for outstanding WAL files using a thread pool. > Each split worker would first check whether the WAL file it processes is > closed. > Thanks to Nicolas Liochon and Jeffery discussion with whom gave rise to this > idea. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10117) Avoid synchronization in HRegionScannerImpl.isFilterDone
[ https://issues.apache.org/jira/browse/HBASE-10117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844880#comment-13844880 ] Hudson commented on HBASE-10117: SUCCESS: Integrated in HBase-0.94-security #356 (See [https://builds.apache.org/job/HBase-0.94-security/356/]) HBASE-10117 Avoid synchronization in HRegionScannerImpl.isFilterDone (larsh: rev 1549917) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > Avoid synchronization in HRegionScannerImpl.isFilterDone > > > Key: HBASE-10117 > URL: https://issues.apache.org/jira/browse/HBASE-10117 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.94.15, 0.96.2, 0.98.1, 0.99.0 > > Attachments: 10117-0.94-v2.txt, 10117-0.94-v3.txt, 10117-0.94.txt, > 10117-trunk-v2.txt, 10117-trunk.txt > > > A while ago I introduced HRegoinScannerImpl.nextRaw() to allow coprocessors > and scanners with caching > 1 to avoid repeated synchronization during > scanning (which puts up memory fences, which in turn slows things down on > multi core machines). > Looking at the code again I see that isFilterDone() is called from nextRaw() > and isFilterDone() is synchronized. > The caller of nextRaw is required to ensure single threaded access to > nextRaw() anyway, we can call an unsynchronized internal version of > isFilterDone(). -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10111) Verify that a snapshot is not corrupted before restoring it
[ https://issues.apache.org/jira/browse/HBASE-10111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844881#comment-13844881 ] Hudson commented on HBASE-10111: SUCCESS: Integrated in HBase-0.94-security #356 (See [https://builds.apache.org/job/HBase-0.94-security/356/]) HBASE-10111 Verify that a snapshot is not corrupted before restoring it (mbertozzi: rev 1549820) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotReferenceUtil.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java > Verify that a snapshot is not corrupted before restoring it > --- > > Key: HBASE-10111 > URL: https://issues.apache.org/jira/browse/HBASE-10111 > Project: HBase > Issue Type: Bug > Components: snapshots >Affects Versions: 0.98.0, 0.96.0, 0.94.14 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15 > > Attachments: HBASE-10111-v0.patch, HBASE-10111-v1.patch > > > To avoid assigning/opening regions with missing files, verify that the > snapshot is not corrupted before restoring/cloning it. > In 96 a corrupted file in a region is "not a problem" since the assignment > will give up after awhile. > In 94 having a corrupted file in a region means looping forever, on "enable", > until manual intervention. (Easy way to test this is create a table, disable > it, add a corrupted reference file and enable the table to start looping: you > can use echo "foo" > > .) -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10112) Hbase rest query params for maxVersions and maxValues are not parsed
[ https://issues.apache.org/jira/browse/HBASE-10112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844882#comment-13844882 ] Hudson commented on HBASE-10112: SUCCESS: Integrated in HBase-0.94-security #356 (See [https://builds.apache.org/job/HBase-0.94-security/356/]) HBASE-10112. REST query params for maxVersions and maxValues are not parsed (Jean-Marc Spaggiari) (apurtell: rev 1549916) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/RowSpec.java > Hbase rest query params for maxVersions and maxValues are not parsed > > > Key: HBASE-10112 > URL: https://issues.apache.org/jira/browse/HBASE-10112 > Project: HBase > Issue Type: Bug > Components: REST >Affects Versions: 0.98.0, 0.96.1, 0.94.14, 0.99.0 >Reporter: Koert Kuipers >Assignee: Jean-Marc Spaggiari >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15, 0.99.0 > > Attachments: HBASE-10112-v0-0.94.patch, HBASE-10112-v0-0.96.patch, > HBASE-10112-v0-0.98.patch, HBASE-10112-v0-trunk.patch > > > i am trying to use maxValues with a "globbed" row resource in stargate. from > looking at the source code one has to do something like > table/row/column(s)/timestamp(s)/?n=1 > (except the ?n=1 piece must be urlencoded) > however i cannot get the n=1 piece to work. i get this stacktrace: > > Problem accessing > /some_table_name/93%2B002%2B*/cf:tx_CUST_NAME/1,13862892906600/%3Fn%3D1. > Reason: > String index out of range: 50Caused > by:java.lang.StringIndexOutOfBoundsException: String index out of > range: 50 > at java.lang.AbstractStringBuilder.charAt(AbstractStringBuilder.java:174) > at java.lang.StringBuilder.charAt(StringBuilder.java:55) > at org.apache.hadoop.hbase.rest.RowSpec.parseQueryParams(RowSpec.java:260) > at org.apache.hadoop.hbase.rest.RowSpec.(RowSpec.java:59) > at > org.apache.hadoop.hbase.rest.RowResource. (RowResource.java:74) > at > org.apache.hadoop.hbase.rest.TableResource.getRowResource(TableResource.java:90) > the offending line is (260 in RowSpec): > c = query.charAt(i); > i think this should be > c = query.charAt(j); > same for line 248 (which handles the maxVersions) > i have not been able to test this (never tried to build hbase myself). -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10069) Potential duplicate calls to log#appendNoSync() in HRegion#doMiniBatchMutation()
[
https://issues.apache.org/jira/browse/HBASE-10069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844878#comment-13844878
]
Ted Yu commented on HBASE-10069:
Your call, Sergey.
> Potential duplicate calls to log#appendNoSync() in
> HRegion#doMiniBatchMutation()
>
>
> Key: HBASE-10069
> URL: https://issues.apache.org/jira/browse/HBASE-10069
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Sergey Shelukhin
>Priority: Minor
> Attachments: HBASE-10069.patch
>
>
> In HRegion#doMiniBatchMutation():
> {code}
> if (nonceGroup != currentNonceGroup || nonce != currentNonce) {
> if (walEdit.size() > 0) {
> assert isInReplay;
> txid = this.log.appendNoSync(this.getRegionInfo(),
> htableDescriptor.getTableName(),
> walEdit, m.getClusterIds(), now, htableDescriptor,
> this.sequenceId, true,
> currentNonceGroup, currentNonce);
> hasWalAppends = true;
> }
> currentNonceGroup = nonceGroup;
> currentNonce = nonce;
> }
> // Add WAL edits by CP
> WALEdit fromCP = batchOp.walEditsFromCoprocessors[i];
> if (fromCP != null) {
> for (KeyValue kv : fromCP.getKeyValues()) {
> walEdit.add(kv);
> }
> }
> ...
> Mutation mutation = batchOp.getMutation(firstIndex);
> if (walEdit.size() > 0) {
> txid = this.log.appendNoSync(this.getRegionInfo(),
> this.htableDescriptor.getTableName(),
> walEdit, mutation.getClusterIds(), now, this.htableDescriptor,
> this.sequenceId,
> true, currentNonceGroup, currentNonce);
> hasWalAppends = true;
> }
> {code}
> If fromCP is null, there may not be new edits added to walEdit.
> But log#appendNoSync() would be called one more time at line 2368.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10069) Potential duplicate calls to log#appendNoSync() in HRegion#doMiniBatchMutation()
[
https://issues.apache.org/jira/browse/HBASE-10069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844876#comment-13844876
]
Sergey Shelukhin commented on HBASE-10069:
--
it;s better for the assertion to fire, exception is just for the case if they
are disabled.
> Potential duplicate calls to log#appendNoSync() in
> HRegion#doMiniBatchMutation()
>
>
> Key: HBASE-10069
> URL: https://issues.apache.org/jira/browse/HBASE-10069
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Sergey Shelukhin
>Priority: Minor
> Attachments: HBASE-10069.patch
>
>
> In HRegion#doMiniBatchMutation():
> {code}
> if (nonceGroup != currentNonceGroup || nonce != currentNonce) {
> if (walEdit.size() > 0) {
> assert isInReplay;
> txid = this.log.appendNoSync(this.getRegionInfo(),
> htableDescriptor.getTableName(),
> walEdit, m.getClusterIds(), now, htableDescriptor,
> this.sequenceId, true,
> currentNonceGroup, currentNonce);
> hasWalAppends = true;
> }
> currentNonceGroup = nonceGroup;
> currentNonce = nonce;
> }
> // Add WAL edits by CP
> WALEdit fromCP = batchOp.walEditsFromCoprocessors[i];
> if (fromCP != null) {
> for (KeyValue kv : fromCP.getKeyValues()) {
> walEdit.add(kv);
> }
> }
> ...
> Mutation mutation = batchOp.getMutation(firstIndex);
> if (walEdit.size() > 0) {
> txid = this.log.appendNoSync(this.getRegionInfo(),
> this.htableDescriptor.getTableName(),
> walEdit, mutation.getClusterIds(), now, this.htableDescriptor,
> this.sequenceId,
> true, currentNonceGroup, currentNonce);
> hasWalAppends = true;
> }
> {code}
> If fromCP is null, there may not be new edits added to walEdit.
> But log#appendNoSync() would be called one more time at line 2368.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10069) Potential duplicate calls to log#appendNoSync() in HRegion#doMiniBatchMutation()
[
https://issues.apache.org/jira/browse/HBASE-10069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844871#comment-13844871
]
Ted Yu commented on HBASE-10069:
Thanks for the fix, Sergey.
lgtm
{code}
assert isInReplay;
+if (!isInReplay) {
{code}
nit: the assertion is no longer needed.
> Potential duplicate calls to log#appendNoSync() in
> HRegion#doMiniBatchMutation()
>
>
> Key: HBASE-10069
> URL: https://issues.apache.org/jira/browse/HBASE-10069
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Sergey Shelukhin
>Priority: Minor
> Attachments: HBASE-10069.patch
>
>
> In HRegion#doMiniBatchMutation():
> {code}
> if (nonceGroup != currentNonceGroup || nonce != currentNonce) {
> if (walEdit.size() > 0) {
> assert isInReplay;
> txid = this.log.appendNoSync(this.getRegionInfo(),
> htableDescriptor.getTableName(),
> walEdit, m.getClusterIds(), now, htableDescriptor,
> this.sequenceId, true,
> currentNonceGroup, currentNonce);
> hasWalAppends = true;
> }
> currentNonceGroup = nonceGroup;
> currentNonce = nonce;
> }
> // Add WAL edits by CP
> WALEdit fromCP = batchOp.walEditsFromCoprocessors[i];
> if (fromCP != null) {
> for (KeyValue kv : fromCP.getKeyValues()) {
> walEdit.add(kv);
> }
> }
> ...
> Mutation mutation = batchOp.getMutation(firstIndex);
> if (walEdit.size() > 0) {
> txid = this.log.appendNoSync(this.getRegionInfo(),
> this.htableDescriptor.getTableName(),
> walEdit, mutation.getClusterIds(), now, this.htableDescriptor,
> this.sequenceId,
> true, currentNonceGroup, currentNonce);
> hasWalAppends = true;
> }
> {code}
> If fromCP is null, there may not be new edits added to walEdit.
> But log#appendNoSync() would be called one more time at line 2368.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10117) Avoid synchronization in HRegionScannerImpl.isFilterDone
[ https://issues.apache.org/jira/browse/HBASE-10117?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844862#comment-13844862 ] Hudson commented on HBASE-10117: SUCCESS: Integrated in HBase-0.94 #1223 (See [https://builds.apache.org/job/HBase-0.94/1223/]) HBASE-10117 Avoid synchronization in HRegionScannerImpl.isFilterDone (larsh: rev 1549917) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java > Avoid synchronization in HRegionScannerImpl.isFilterDone > > > Key: HBASE-10117 > URL: https://issues.apache.org/jira/browse/HBASE-10117 > Project: HBase > Issue Type: Bug > Components: Performance >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.94.15, 0.96.2, 0.98.1, 0.99.0 > > Attachments: 10117-0.94-v2.txt, 10117-0.94-v3.txt, 10117-0.94.txt, > 10117-trunk-v2.txt, 10117-trunk.txt > > > A while ago I introduced HRegoinScannerImpl.nextRaw() to allow coprocessors > and scanners with caching > 1 to avoid repeated synchronization during > scanning (which puts up memory fences, which in turn slows things down on > multi core machines). > Looking at the code again I see that isFilterDone() is called from nextRaw() > and isFilterDone() is synchronized. > The caller of nextRaw is required to ensure single threaded access to > nextRaw() anyway, we can call an unsynchronized internal version of > isFilterDone(). -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10111) Verify that a snapshot is not corrupted before restoring it
[ https://issues.apache.org/jira/browse/HBASE-10111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844863#comment-13844863 ] Hudson commented on HBASE-10111: SUCCESS: Integrated in HBase-0.94 #1223 (See [https://builds.apache.org/job/HBase-0.94/1223/]) HBASE-10111 Verify that a snapshot is not corrupted before restoring it (mbertozzi: rev 1549820) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotReferenceUtil.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestRestoreSnapshotFromClient.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/SnapshotTestingUtils.java > Verify that a snapshot is not corrupted before restoring it > --- > > Key: HBASE-10111 > URL: https://issues.apache.org/jira/browse/HBASE-10111 > Project: HBase > Issue Type: Bug > Components: snapshots >Affects Versions: 0.98.0, 0.96.0, 0.94.14 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15 > > Attachments: HBASE-10111-v0.patch, HBASE-10111-v1.patch > > > To avoid assigning/opening regions with missing files, verify that the > snapshot is not corrupted before restoring/cloning it. > In 96 a corrupted file in a region is "not a problem" since the assignment > will give up after awhile. > In 94 having a corrupted file in a region means looping forever, on "enable", > until manual intervention. (Easy way to test this is create a table, disable > it, add a corrupted reference file and enable the table to start looping: you > can use echo "foo" > > .) -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10112) Hbase rest query params for maxVersions and maxValues are not parsed
[ https://issues.apache.org/jira/browse/HBASE-10112?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844864#comment-13844864 ] Hudson commented on HBASE-10112: SUCCESS: Integrated in HBase-0.94 #1223 (See [https://builds.apache.org/job/HBase-0.94/1223/]) HBASE-10112. REST query params for maxVersions and maxValues are not parsed (Jean-Marc Spaggiari) (apurtell: rev 1549916) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/rest/RowSpec.java > Hbase rest query params for maxVersions and maxValues are not parsed > > > Key: HBASE-10112 > URL: https://issues.apache.org/jira/browse/HBASE-10112 > Project: HBase > Issue Type: Bug > Components: REST >Affects Versions: 0.98.0, 0.96.1, 0.94.14, 0.99.0 >Reporter: Koert Kuipers >Assignee: Jean-Marc Spaggiari >Priority: Minor > Fix For: 0.98.0, 0.96.1, 0.94.15, 0.99.0 > > Attachments: HBASE-10112-v0-0.94.patch, HBASE-10112-v0-0.96.patch, > HBASE-10112-v0-0.98.patch, HBASE-10112-v0-trunk.patch > > > i am trying to use maxValues with a "globbed" row resource in stargate. from > looking at the source code one has to do something like > table/row/column(s)/timestamp(s)/?n=1 > (except the ?n=1 piece must be urlencoded) > however i cannot get the n=1 piece to work. i get this stacktrace: > > Problem accessing > /some_table_name/93%2B002%2B*/cf:tx_CUST_NAME/1,13862892906600/%3Fn%3D1. > Reason: > String index out of range: 50Caused > by:java.lang.StringIndexOutOfBoundsException: String index out of > range: 50 > at java.lang.AbstractStringBuilder.charAt(AbstractStringBuilder.java:174) > at java.lang.StringBuilder.charAt(StringBuilder.java:55) > at org.apache.hadoop.hbase.rest.RowSpec.parseQueryParams(RowSpec.java:260) > at org.apache.hadoop.hbase.rest.RowSpec.(RowSpec.java:59) > at > org.apache.hadoop.hbase.rest.RowResource. (RowResource.java:74) > at > org.apache.hadoop.hbase.rest.TableResource.getRowResource(TableResource.java:90) > the offending line is (260 in RowSpec): > c = query.charAt(i); > i think this should be > c = query.charAt(j); > same for line 248 (which handles the maxVersions) > i have not been able to test this (never tried to build hbase myself). -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HBASE-10069) Potential duplicate calls to log#appendNoSync() in HRegion#doMiniBatchMutation()
[
https://issues.apache.org/jira/browse/HBASE-10069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HBASE-10069:
-
Assignee: Sergey Shelukhin
Status: Patch Available (was: Open)
> Potential duplicate calls to log#appendNoSync() in
> HRegion#doMiniBatchMutation()
>
>
> Key: HBASE-10069
> URL: https://issues.apache.org/jira/browse/HBASE-10069
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Assignee: Sergey Shelukhin
>Priority: Minor
> Attachments: HBASE-10069.patch
>
>
> In HRegion#doMiniBatchMutation():
> {code}
> if (nonceGroup != currentNonceGroup || nonce != currentNonce) {
> if (walEdit.size() > 0) {
> assert isInReplay;
> txid = this.log.appendNoSync(this.getRegionInfo(),
> htableDescriptor.getTableName(),
> walEdit, m.getClusterIds(), now, htableDescriptor,
> this.sequenceId, true,
> currentNonceGroup, currentNonce);
> hasWalAppends = true;
> }
> currentNonceGroup = nonceGroup;
> currentNonce = nonce;
> }
> // Add WAL edits by CP
> WALEdit fromCP = batchOp.walEditsFromCoprocessors[i];
> if (fromCP != null) {
> for (KeyValue kv : fromCP.getKeyValues()) {
> walEdit.add(kv);
> }
> }
> ...
> Mutation mutation = batchOp.getMutation(firstIndex);
> if (walEdit.size() > 0) {
> txid = this.log.appendNoSync(this.getRegionInfo(),
> this.htableDescriptor.getTableName(),
> walEdit, mutation.getClusterIds(), now, this.htableDescriptor,
> this.sequenceId,
> true, currentNonceGroup, currentNonce);
> hasWalAppends = true;
> }
> {code}
> If fromCP is null, there may not be new edits added to walEdit.
> But log#appendNoSync() would be called one more time at line 2368.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HBASE-10069) Potential duplicate calls to log#appendNoSync() in HRegion#doMiniBatchMutation()
[
https://issues.apache.org/jira/browse/HBASE-10069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HBASE-10069:
-
Attachment: HBASE-10069.patch
> Potential duplicate calls to log#appendNoSync() in
> HRegion#doMiniBatchMutation()
>
>
> Key: HBASE-10069
> URL: https://issues.apache.org/jira/browse/HBASE-10069
> Project: HBase
> Issue Type: Task
>Reporter: Ted Yu
>Priority: Minor
> Attachments: HBASE-10069.patch
>
>
> In HRegion#doMiniBatchMutation():
> {code}
> if (nonceGroup != currentNonceGroup || nonce != currentNonce) {
> if (walEdit.size() > 0) {
> assert isInReplay;
> txid = this.log.appendNoSync(this.getRegionInfo(),
> htableDescriptor.getTableName(),
> walEdit, m.getClusterIds(), now, htableDescriptor,
> this.sequenceId, true,
> currentNonceGroup, currentNonce);
> hasWalAppends = true;
> }
> currentNonceGroup = nonceGroup;
> currentNonce = nonce;
> }
> // Add WAL edits by CP
> WALEdit fromCP = batchOp.walEditsFromCoprocessors[i];
> if (fromCP != null) {
> for (KeyValue kv : fromCP.getKeyValues()) {
> walEdit.add(kv);
> }
> }
> ...
> Mutation mutation = batchOp.getMutation(firstIndex);
> if (walEdit.size() > 0) {
> txid = this.log.appendNoSync(this.getRegionInfo(),
> this.htableDescriptor.getTableName(),
> walEdit, mutation.getClusterIds(), now, this.htableDescriptor,
> this.sequenceId,
> true, currentNonceGroup, currentNonce);
> hasWalAppends = true;
> }
> {code}
> If fromCP is null, there may not be new edits added to walEdit.
> But log#appendNoSync() would be called one more time at line 2368.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HBASE-10069) Potential duplicate calls to log#appendNoSync() in HRegion#doMiniBatchMutation()
[
https://issues.apache.org/jira/browse/HBASE-10069?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sergey Shelukhin updated HBASE-10069:
-
Issue Type: Bug (was: Task)
> Potential duplicate calls to log#appendNoSync() in
> HRegion#doMiniBatchMutation()
>
>
> Key: HBASE-10069
> URL: https://issues.apache.org/jira/browse/HBASE-10069
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Sergey Shelukhin
>Priority: Minor
> Attachments: HBASE-10069.patch
>
>
> In HRegion#doMiniBatchMutation():
> {code}
> if (nonceGroup != currentNonceGroup || nonce != currentNonce) {
> if (walEdit.size() > 0) {
> assert isInReplay;
> txid = this.log.appendNoSync(this.getRegionInfo(),
> htableDescriptor.getTableName(),
> walEdit, m.getClusterIds(), now, htableDescriptor,
> this.sequenceId, true,
> currentNonceGroup, currentNonce);
> hasWalAppends = true;
> }
> currentNonceGroup = nonceGroup;
> currentNonce = nonce;
> }
> // Add WAL edits by CP
> WALEdit fromCP = batchOp.walEditsFromCoprocessors[i];
> if (fromCP != null) {
> for (KeyValue kv : fromCP.getKeyValues()) {
> walEdit.add(kv);
> }
> }
> ...
> Mutation mutation = batchOp.getMutation(firstIndex);
> if (walEdit.size() > 0) {
> txid = this.log.appendNoSync(this.getRegionInfo(),
> this.htableDescriptor.getTableName(),
> walEdit, mutation.getClusterIds(), now, this.htableDescriptor,
> this.sequenceId,
> true, currentNonceGroup, currentNonce);
> hasWalAppends = true;
> }
> {code}
> If fromCP is null, there may not be new edits added to walEdit.
> But log#appendNoSync() would be called one more time at line 2368.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-8755) A new write thread model for HLog to improve the overall HBase write throughput
[ https://issues.apache.org/jira/browse/HBASE-8755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844859#comment-13844859 ] Himanshu Vashishtha commented on HBASE-8755: Thanks for the explanation [~fenghh]; and it pretty much answers all my questions. Also, looking more, getting rid of LogSyncer thread eases out the locking semantics of rolling. Yes, I reran the above experiments on a more standard environment (4 DNs with HLogPE running on a DN, and log level set to INFO instead of Debug), and got mixed results this time. Varied threads from 2 to 100 and didn't get a clear winner. Given the current state of this patch and the cleanup it does, I am +1 for committing this. Looking forward to it. Thanks. > A new write thread model for HLog to improve the overall HBase write > throughput > --- > > Key: HBASE-8755 > URL: https://issues.apache.org/jira/browse/HBASE-8755 > Project: HBase > Issue Type: Improvement > Components: Performance, wal >Reporter: Feng Honghua >Assignee: stack >Priority: Critical > Attachments: 8755-syncer.patch, 8755trunkV2.txt, > HBASE-8755-0.94-V0.patch, HBASE-8755-0.94-V1.patch, HBASE-8755-0.96-v0.patch, > HBASE-8755-trunk-V0.patch, HBASE-8755-trunk-V1.patch, > HBASE-8755-trunk-v4.patch, HBASE-8755-trunk-v6.patch, HBASE-8755-v5.patch > > > In current write model, each write handler thread (executing put()) will > individually go through a full 'append (hlog local buffer) => HLog writer > append (write to hdfs) => HLog writer sync (sync hdfs)' cycle for each write, > which incurs heavy race condition on updateLock and flushLock. > The only optimization where checking if current syncTillHere > txid in > expectation for other thread help write/sync its own txid to hdfs and > omitting the write/sync actually help much less than expectation. > Three of my colleagues(Ye Hangjun / Wu Zesheng / Zhang Peng) at Xiaomi > proposed a new write thread model for writing hdfs sequence file and the > prototype implementation shows a 4X improvement for throughput (from 17000 to > 7+). > I apply this new write thread model in HLog and the performance test in our > test cluster shows about 3X throughput improvement (from 12150 to 31520 for 1 > RS, from 22000 to 7 for 5 RS), the 1 RS write throughput (1K row-size) > even beats the one of BigTable (Precolator published in 2011 says Bigtable's > write throughput then is 31002). I can provide the detailed performance test > results if anyone is interested. > The change for new write thread model is as below: > 1> All put handler threads append the edits to HLog's local pending buffer; > (it notifies AsyncWriter thread that there is new edits in local buffer) > 2> All put handler threads wait in HLog.syncer() function for underlying > threads to finish the sync that contains its txid; > 3> An single AsyncWriter thread is responsible for retrieve all the buffered > edits in HLog's local pending buffer and write to the hdfs > (hlog.writer.append); (it notifies AsyncFlusher thread that there is new > writes to hdfs that needs a sync) > 4> An single AsyncFlusher thread is responsible for issuing a sync to hdfs > to persist the writes by AsyncWriter; (it notifies the AsyncNotifier thread > that sync watermark increases) > 5> An single AsyncNotifier thread is responsible for notifying all pending > put handler threads which are waiting in the HLog.syncer() function > 6> No LogSyncer thread any more (since there is always > AsyncWriter/AsyncFlusher threads do the same job it does) -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-5349) Automagically tweak global memstore and block cache sizes based on workload
[ https://issues.apache.org/jira/browse/HBASE-5349?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844857#comment-13844857 ] Ted Yu commented on HBASE-5349: --- Anoop: Can you update release notes ? Thanks > Automagically tweak global memstore and block cache sizes based on workload > --- > > Key: HBASE-5349 > URL: https://issues.apache.org/jira/browse/HBASE-5349 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.92.0 >Reporter: Jean-Daniel Cryans >Assignee: Anoop Sam John > Fix For: 0.99.0 > > Attachments: HBASE-5349_V2.patch, HBASE-5349_V3.patch, > HBASE-5349_V4.patch, HBASE-5349_V5.patch, WIP_HBASE-5349.patch > > > Hypertable does a neat thing where it changes the size given to the CellCache > (our MemStores) and Block Cache based on the workload. If you need an image, > scroll down at the bottom of this link: > http://www.hypertable.com/documentation/architecture/ > That'd be one less thing to configure. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10103) TestNodeHealthCheckChore#testRSHealthChore: Stoppable must have been stopped
[
https://issues.apache.org/jira/browse/HBASE-10103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844854#comment-13844854
]
Elliott Clark commented on HBASE-10103:
---
Thanks [~apurtell]
> TestNodeHealthCheckChore#testRSHealthChore: Stoppable must have been stopped
>
>
> Key: HBASE-10103
> URL: https://issues.apache.org/jira/browse/HBASE-10103
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.99.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Fix For: 0.98.0, 0.96.1, 0.99.0
>
> Attachments: 10103-amend-1.patch, 10103.patch, 10103.patch,
> 10103.patch
>
>
> {noformat}
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 623.639 sec
> <<< FAILURE!
> testRSHealthChore(org.apache.hadoop.hbase.TestNodeHealthCheckChore) Time
> elapsed: 0.001 sec <<< FAILURE!
> java.lang.AssertionError: Stoppable must have been stopped.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hbase.TestNodeHealthCheckChore.testRSHealthChore(TestNodeHealthCheckChore.java:108)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Comment Edited] (HBASE-8755) A new write thread model for HLog to improve the overall HBase write throughput
[
https://issues.apache.org/jira/browse/HBASE-8755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844839#comment-13844839
]
Jonathan Hsieh edited comment on HBASE-8755 at 12/10/13 11:59 PM:
--
I did most of a review of v4 last week -- here are a few nits:
nit: (fix on commit)
{code}
+ //up and holds the lock
+ // NOTE! can't hold 'upateLock' here since rollWriter will pend
+ // on 'sync()' with 'updateLock', but 'sync()' will wait for
+ // AsyncWriter/AsyncSyncer/AsyncNotifier series. without upateLock
+ // can leads to pendWrites more than pendingTxid, but not problem
{code}
spelling 2x: upate -> update
This can go in a follow up issue -- and please add a description of the threads
/ queues / invariants and how a wal writes happens in the class javadoc. An
updated version of the 1-6 list in the description would be great.
Good stuff [~fenghh]!
was (Author: jmhsieh):
I did most of a review of v4 last week -- here are a few nits:
nit: (fix on commit)
{code}
+ //up and holds the lock
+ // NOTE! can't hold 'upateLock' here since rollWriter will pend
+ // on 'sync()' with 'updateLock', but 'sync()' will wait for
+ // AsyncWriter/AsyncSyncer/AsyncNotifier series. without upateLock
+ // can leads to pendWrites more than pendingTxid, but not problem
{code}
spelling: upate -> update
This can go in a follow up issue -- and please add a description of the threads
/ queues / invariants and how a wal writes happens in the class javadoc. An
updated version of the 1-6 list in the description would be great.
Good stuff [~fenghh]!
> A new write thread model for HLog to improve the overall HBase write
> throughput
> ---
>
> Key: HBASE-8755
> URL: https://issues.apache.org/jira/browse/HBASE-8755
> Project: HBase
> Issue Type: Improvement
> Components: Performance, wal
>Reporter: Feng Honghua
>Assignee: stack
>Priority: Critical
> Attachments: 8755-syncer.patch, 8755trunkV2.txt,
> HBASE-8755-0.94-V0.patch, HBASE-8755-0.94-V1.patch, HBASE-8755-0.96-v0.patch,
> HBASE-8755-trunk-V0.patch, HBASE-8755-trunk-V1.patch,
> HBASE-8755-trunk-v4.patch, HBASE-8755-trunk-v6.patch, HBASE-8755-v5.patch
>
>
> In current write model, each write handler thread (executing put()) will
> individually go through a full 'append (hlog local buffer) => HLog writer
> append (write to hdfs) => HLog writer sync (sync hdfs)' cycle for each write,
> which incurs heavy race condition on updateLock and flushLock.
> The only optimization where checking if current syncTillHere > txid in
> expectation for other thread help write/sync its own txid to hdfs and
> omitting the write/sync actually help much less than expectation.
> Three of my colleagues(Ye Hangjun / Wu Zesheng / Zhang Peng) at Xiaomi
> proposed a new write thread model for writing hdfs sequence file and the
> prototype implementation shows a 4X improvement for throughput (from 17000 to
> 7+).
> I apply this new write thread model in HLog and the performance test in our
> test cluster shows about 3X throughput improvement (from 12150 to 31520 for 1
> RS, from 22000 to 7 for 5 RS), the 1 RS write throughput (1K row-size)
> even beats the one of BigTable (Precolator published in 2011 says Bigtable's
> write throughput then is 31002). I can provide the detailed performance test
> results if anyone is interested.
> The change for new write thread model is as below:
> 1> All put handler threads append the edits to HLog's local pending buffer;
> (it notifies AsyncWriter thread that there is new edits in local buffer)
> 2> All put handler threads wait in HLog.syncer() function for underlying
> threads to finish the sync that contains its txid;
> 3> An single AsyncWriter thread is responsible for retrieve all the buffered
> edits in HLog's local pending buffer and write to the hdfs
> (hlog.writer.append); (it notifies AsyncFlusher thread that there is new
> writes to hdfs that needs a sync)
> 4> An single AsyncFlusher thread is responsible for issuing a sync to hdfs
> to persist the writes by AsyncWriter; (it notifies the AsyncNotifier thread
> that sync watermark increases)
> 5> An single AsyncNotifier thread is responsible for notifying all pending
> put handler threads which are waiting in the HLog.syncer() function
> 6> No LogSyncer thread any more (since there is always
> AsyncWriter/AsyncFlusher threads do the same job it does)
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-8755) A new write thread model for HLog to improve the overall HBase write throughput
[
https://issues.apache.org/jira/browse/HBASE-8755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844839#comment-13844839
]
Jonathan Hsieh commented on HBASE-8755:
---
I did most of a review of v4 last week -- here are a few nits:
nit: (fix on commit)
{code}
+ //up and holds the lock
+ // NOTE! can't hold 'upateLock' here since rollWriter will pend
+ // on 'sync()' with 'updateLock', but 'sync()' will wait for
+ // AsyncWriter/AsyncSyncer/AsyncNotifier series. without upateLock
+ // can leads to pendWrites more than pendingTxid, but not problem
{code}
spelling: upate -> update
This can go in a follow up issue -- and please add a description of the threads
/ queues / invariants and how a wal writes happens in the class javadoc. An
updated version of the 1-6 list in the description would be great.
Good stuff [~fenghh]!
> A new write thread model for HLog to improve the overall HBase write
> throughput
> ---
>
> Key: HBASE-8755
> URL: https://issues.apache.org/jira/browse/HBASE-8755
> Project: HBase
> Issue Type: Improvement
> Components: Performance, wal
>Reporter: Feng Honghua
>Assignee: stack
>Priority: Critical
> Attachments: 8755-syncer.patch, 8755trunkV2.txt,
> HBASE-8755-0.94-V0.patch, HBASE-8755-0.94-V1.patch, HBASE-8755-0.96-v0.patch,
> HBASE-8755-trunk-V0.patch, HBASE-8755-trunk-V1.patch,
> HBASE-8755-trunk-v4.patch, HBASE-8755-trunk-v6.patch, HBASE-8755-v5.patch
>
>
> In current write model, each write handler thread (executing put()) will
> individually go through a full 'append (hlog local buffer) => HLog writer
> append (write to hdfs) => HLog writer sync (sync hdfs)' cycle for each write,
> which incurs heavy race condition on updateLock and flushLock.
> The only optimization where checking if current syncTillHere > txid in
> expectation for other thread help write/sync its own txid to hdfs and
> omitting the write/sync actually help much less than expectation.
> Three of my colleagues(Ye Hangjun / Wu Zesheng / Zhang Peng) at Xiaomi
> proposed a new write thread model for writing hdfs sequence file and the
> prototype implementation shows a 4X improvement for throughput (from 17000 to
> 7+).
> I apply this new write thread model in HLog and the performance test in our
> test cluster shows about 3X throughput improvement (from 12150 to 31520 for 1
> RS, from 22000 to 7 for 5 RS), the 1 RS write throughput (1K row-size)
> even beats the one of BigTable (Precolator published in 2011 says Bigtable's
> write throughput then is 31002). I can provide the detailed performance test
> results if anyone is interested.
> The change for new write thread model is as below:
> 1> All put handler threads append the edits to HLog's local pending buffer;
> (it notifies AsyncWriter thread that there is new edits in local buffer)
> 2> All put handler threads wait in HLog.syncer() function for underlying
> threads to finish the sync that contains its txid;
> 3> An single AsyncWriter thread is responsible for retrieve all the buffered
> edits in HLog's local pending buffer and write to the hdfs
> (hlog.writer.append); (it notifies AsyncFlusher thread that there is new
> writes to hdfs that needs a sync)
> 4> An single AsyncFlusher thread is responsible for issuing a sync to hdfs
> to persist the writes by AsyncWriter; (it notifies the AsyncNotifier thread
> that sync watermark increases)
> 5> An single AsyncNotifier thread is responsible for notifying all pending
> put handler threads which are waiting in the HLog.syncer() function
> 6> No LogSyncer thread any more (since there is always
> AsyncWriter/AsyncFlusher threads do the same job it does)
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-8755) A new write thread model for HLog to improve the overall HBase write throughput
[ https://issues.apache.org/jira/browse/HBASE-8755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844838#comment-13844838 ] Ted Yu commented on HBASE-8755: --- +1 from me. > A new write thread model for HLog to improve the overall HBase write > throughput > --- > > Key: HBASE-8755 > URL: https://issues.apache.org/jira/browse/HBASE-8755 > Project: HBase > Issue Type: Improvement > Components: Performance, wal >Reporter: Feng Honghua >Assignee: stack >Priority: Critical > Attachments: 8755-syncer.patch, 8755trunkV2.txt, > HBASE-8755-0.94-V0.patch, HBASE-8755-0.94-V1.patch, HBASE-8755-0.96-v0.patch, > HBASE-8755-trunk-V0.patch, HBASE-8755-trunk-V1.patch, > HBASE-8755-trunk-v4.patch, HBASE-8755-trunk-v6.patch, HBASE-8755-v5.patch > > > In current write model, each write handler thread (executing put()) will > individually go through a full 'append (hlog local buffer) => HLog writer > append (write to hdfs) => HLog writer sync (sync hdfs)' cycle for each write, > which incurs heavy race condition on updateLock and flushLock. > The only optimization where checking if current syncTillHere > txid in > expectation for other thread help write/sync its own txid to hdfs and > omitting the write/sync actually help much less than expectation. > Three of my colleagues(Ye Hangjun / Wu Zesheng / Zhang Peng) at Xiaomi > proposed a new write thread model for writing hdfs sequence file and the > prototype implementation shows a 4X improvement for throughput (from 17000 to > 7+). > I apply this new write thread model in HLog and the performance test in our > test cluster shows about 3X throughput improvement (from 12150 to 31520 for 1 > RS, from 22000 to 7 for 5 RS), the 1 RS write throughput (1K row-size) > even beats the one of BigTable (Precolator published in 2011 says Bigtable's > write throughput then is 31002). I can provide the detailed performance test > results if anyone is interested. > The change for new write thread model is as below: > 1> All put handler threads append the edits to HLog's local pending buffer; > (it notifies AsyncWriter thread that there is new edits in local buffer) > 2> All put handler threads wait in HLog.syncer() function for underlying > threads to finish the sync that contains its txid; > 3> An single AsyncWriter thread is responsible for retrieve all the buffered > edits in HLog's local pending buffer and write to the hdfs > (hlog.writer.append); (it notifies AsyncFlusher thread that there is new > writes to hdfs that needs a sync) > 4> An single AsyncFlusher thread is responsible for issuing a sync to hdfs > to persist the writes by AsyncWriter; (it notifies the AsyncNotifier thread > that sync watermark increases) > 5> An single AsyncNotifier thread is responsible for notifying all pending > put handler threads which are waiting in the HLog.syncer() function > 6> No LogSyncer thread any more (since there is always > AsyncWriter/AsyncFlusher threads do the same job it does) -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-8755) A new write thread model for HLog to improve the overall HBase write throughput
[ https://issues.apache.org/jira/browse/HBASE-8755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844827#comment-13844827 ] stack commented on HBASE-8755: -- All sounds good above [~fenghh] I'm running a few tests here on cluster to see it basically works. Any other comments by anyone else? Otherwise, was planning on committing. We can work on further speedup in new issues; e.g. see if we can do less threads as per [~himan...@cloudera.com] > A new write thread model for HLog to improve the overall HBase write > throughput > --- > > Key: HBASE-8755 > URL: https://issues.apache.org/jira/browse/HBASE-8755 > Project: HBase > Issue Type: Improvement > Components: Performance, wal >Reporter: Feng Honghua >Assignee: stack >Priority: Critical > Attachments: 8755-syncer.patch, 8755trunkV2.txt, > HBASE-8755-0.94-V0.patch, HBASE-8755-0.94-V1.patch, HBASE-8755-0.96-v0.patch, > HBASE-8755-trunk-V0.patch, HBASE-8755-trunk-V1.patch, > HBASE-8755-trunk-v4.patch, HBASE-8755-trunk-v6.patch, HBASE-8755-v5.patch > > > In current write model, each write handler thread (executing put()) will > individually go through a full 'append (hlog local buffer) => HLog writer > append (write to hdfs) => HLog writer sync (sync hdfs)' cycle for each write, > which incurs heavy race condition on updateLock and flushLock. > The only optimization where checking if current syncTillHere > txid in > expectation for other thread help write/sync its own txid to hdfs and > omitting the write/sync actually help much less than expectation. > Three of my colleagues(Ye Hangjun / Wu Zesheng / Zhang Peng) at Xiaomi > proposed a new write thread model for writing hdfs sequence file and the > prototype implementation shows a 4X improvement for throughput (from 17000 to > 7+). > I apply this new write thread model in HLog and the performance test in our > test cluster shows about 3X throughput improvement (from 12150 to 31520 for 1 > RS, from 22000 to 7 for 5 RS), the 1 RS write throughput (1K row-size) > even beats the one of BigTable (Precolator published in 2011 says Bigtable's > write throughput then is 31002). I can provide the detailed performance test > results if anyone is interested. > The change for new write thread model is as below: > 1> All put handler threads append the edits to HLog's local pending buffer; > (it notifies AsyncWriter thread that there is new edits in local buffer) > 2> All put handler threads wait in HLog.syncer() function for underlying > threads to finish the sync that contains its txid; > 3> An single AsyncWriter thread is responsible for retrieve all the buffered > edits in HLog's local pending buffer and write to the hdfs > (hlog.writer.append); (it notifies AsyncFlusher thread that there is new > writes to hdfs that needs a sync) > 4> An single AsyncFlusher thread is responsible for issuing a sync to hdfs > to persist the writes by AsyncWriter; (it notifies the AsyncNotifier thread > that sync watermark increases) > 5> An single AsyncNotifier thread is responsible for notifying all pending > put handler threads which are waiting in the HLog.syncer() function > 6> No LogSyncer thread any more (since there is always > AsyncWriter/AsyncFlusher threads do the same job it does) -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HBASE-10103) TestNodeHealthCheckChore#testRSHealthChore: Stoppable must have been stopped
[
https://issues.apache.org/jira/browse/HBASE-10103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell updated HBASE-10103:
---
Attachment: 10103-amend-1.patch
> TestNodeHealthCheckChore#testRSHealthChore: Stoppable must have been stopped
>
>
> Key: HBASE-10103
> URL: https://issues.apache.org/jira/browse/HBASE-10103
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.99.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Fix For: 0.98.0, 0.96.1, 0.99.0
>
> Attachments: 10103-amend-1.patch, 10103.patch, 10103.patch,
> 10103.patch
>
>
> {noformat}
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 623.639 sec
> <<< FAILURE!
> testRSHealthChore(org.apache.hadoop.hbase.TestNodeHealthCheckChore) Time
> elapsed: 0.001 sec <<< FAILURE!
> java.lang.AssertionError: Stoppable must have been stopped.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hbase.TestNodeHealthCheckChore.testRSHealthChore(TestNodeHealthCheckChore.java:108)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Resolved] (HBASE-10103) TestNodeHealthCheckChore#testRSHealthChore: Stoppable must have been stopped
[
https://issues.apache.org/jira/browse/HBASE-10103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell resolved HBASE-10103.
Resolution: Fixed
> TestNodeHealthCheckChore#testRSHealthChore: Stoppable must have been stopped
>
>
> Key: HBASE-10103
> URL: https://issues.apache.org/jira/browse/HBASE-10103
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.99.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Fix For: 0.98.0, 0.96.1, 0.99.0
>
> Attachments: 10103-amend-1.patch, 10103.patch, 10103.patch,
> 10103.patch
>
>
> {noformat}
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 623.639 sec
> <<< FAILURE!
> testRSHealthChore(org.apache.hadoop.hbase.TestNodeHealthCheckChore) Time
> elapsed: 0.001 sec <<< FAILURE!
> java.lang.AssertionError: Stoppable must have been stopped.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hbase.TestNodeHealthCheckChore.testRSHealthChore(TestNodeHealthCheckChore.java:108)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10103) TestNodeHealthCheckChore#testRSHealthChore: Stoppable must have been stopped
[
https://issues.apache.org/jira/browse/HBASE-10103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844791#comment-13844791
]
Andrew Purtell commented on HBASE-10103:
Attached committed amendment.
> TestNodeHealthCheckChore#testRSHealthChore: Stoppable must have been stopped
>
>
> Key: HBASE-10103
> URL: https://issues.apache.org/jira/browse/HBASE-10103
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.99.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Fix For: 0.98.0, 0.96.1, 0.99.0
>
> Attachments: 10103-amend-1.patch, 10103.patch, 10103.patch,
> 10103.patch
>
>
> {noformat}
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 623.639 sec
> <<< FAILURE!
> testRSHealthChore(org.apache.hadoop.hbase.TestNodeHealthCheckChore) Time
> elapsed: 0.001 sec <<< FAILURE!
> java.lang.AssertionError: Stoppable must have been stopped.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hbase.TestNodeHealthCheckChore.testRSHealthChore(TestNodeHealthCheckChore.java:108)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10103) TestNodeHealthCheckChore#testRSHealthChore: Stoppable must have been stopped
[
https://issues.apache.org/jira/browse/HBASE-10103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844783#comment-13844783
]
Andrew Purtell commented on HBASE-10103:
Whoops, yes I did something dumb at the last minute.
> TestNodeHealthCheckChore#testRSHealthChore: Stoppable must have been stopped
>
>
> Key: HBASE-10103
> URL: https://issues.apache.org/jira/browse/HBASE-10103
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.99.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Fix For: 0.98.0, 0.96.1, 0.99.0
>
> Attachments: 10103.patch, 10103.patch, 10103.patch
>
>
> {noformat}
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 623.639 sec
> <<< FAILURE!
> testRSHealthChore(org.apache.hadoop.hbase.TestNodeHealthCheckChore) Time
> elapsed: 0.001 sec <<< FAILURE!
> java.lang.AssertionError: Stoppable must have been stopped.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hbase.TestNodeHealthCheckChore.testRSHealthChore(TestNodeHealthCheckChore.java:108)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-9966) Create IntegrationTest for Online Bloom Filter Change
[ https://issues.apache.org/jira/browse/HBASE-9966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844770#comment-13844770 ] Aleksandr Shulman commented on HBASE-9966: -- Thanks Andrew! > Create IntegrationTest for Online Bloom Filter Change > - > > Key: HBASE-9966 > URL: https://issues.apache.org/jira/browse/HBASE-9966 > Project: HBase > Issue Type: Sub-task > Components: HFile, test >Affects Versions: 0.98.0, 0.96.1 >Reporter: Aleksandr Shulman >Assignee: Aleksandr Shulman > Labels: online_schema_change > Fix For: 0.96.1, 0.98.1, 0.99.0 > > Attachments: HBASE-9966-96.patch, HBASE-9966-98.patch, > HBASE-9966-trunk.patch > > > For online schema change, a user is perfectly with her rights to modify the > compression algorithm used, or the bloom filter. > Therefore, we should add these actions to our ChaosMonkey tests to ensure > that they do not introduce instability. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Resolved] (HBASE-9966) Create IntegrationTest for Online Bloom Filter Change
[ https://issues.apache.org/jira/browse/HBASE-9966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell resolved HBASE-9966. --- Resolution: Fixed Hadoop Flags: Reviewed Committed test only change to trunk, 0.98, 0.96. Ran TestIngestTest IT with the slow deterministic monkey and this appeared to work as intended (saw messages from HFile that files had bloomfilter type X but column family specifies type Y) > Create IntegrationTest for Online Bloom Filter Change > - > > Key: HBASE-9966 > URL: https://issues.apache.org/jira/browse/HBASE-9966 > Project: HBase > Issue Type: Sub-task > Components: HFile, test >Affects Versions: 0.98.0, 0.96.1 >Reporter: Aleksandr Shulman >Assignee: Aleksandr Shulman > Labels: online_schema_change > Fix For: 0.96.1, 0.98.1, 0.99.0 > > Attachments: HBASE-9966-96.patch, HBASE-9966-98.patch, > HBASE-9966-trunk.patch > > > For online schema change, a user is perfectly with her rights to modify the > compression algorithm used, or the bloom filter. > Therefore, we should add these actions to our ChaosMonkey tests to ensure > that they do not introduce instability. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Comment Edited] (HBASE-9966) Create IntegrationTest for Online Bloom Filter Change
[ https://issues.apache.org/jira/browse/HBASE-9966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844766#comment-13844766 ] Andrew Purtell edited comment on HBASE-9966 at 12/10/13 10:41 PM: -- Committed test only change to trunk, 0.98, 0.96. Ran IntegrationIngestTest with the slow deterministic monkey and this appeared to work as intended (saw messages from HFile that files had bloomfilter type X but column family specifies type Y) was (Author: apurtell): Committed test only change to trunk, 0.98, 0.96. Ran TestIngestTest IT with the slow deterministic monkey and this appeared to work as intended (saw messages from HFile that files had bloomfilter type X but column family specifies type Y) > Create IntegrationTest for Online Bloom Filter Change > - > > Key: HBASE-9966 > URL: https://issues.apache.org/jira/browse/HBASE-9966 > Project: HBase > Issue Type: Sub-task > Components: HFile, test >Affects Versions: 0.98.0, 0.96.1 >Reporter: Aleksandr Shulman >Assignee: Aleksandr Shulman > Labels: online_schema_change > Fix For: 0.96.1, 0.98.1, 0.99.0 > > Attachments: HBASE-9966-96.patch, HBASE-9966-98.patch, > HBASE-9966-trunk.patch > > > For online schema change, a user is perfectly with her rights to modify the > compression algorithm used, or the bloom filter. > Therefore, we should add these actions to our ChaosMonkey tests to ensure > that they do not introduce instability. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Reopened] (HBASE-10103) TestNodeHealthCheckChore#testRSHealthChore: Stoppable must have been stopped
[
https://issues.apache.org/jira/browse/HBASE-10103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell reopened HBASE-10103:
> TestNodeHealthCheckChore#testRSHealthChore: Stoppable must have been stopped
>
>
> Key: HBASE-10103
> URL: https://issues.apache.org/jira/browse/HBASE-10103
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.99.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Fix For: 0.98.0, 0.96.1, 0.99.0
>
> Attachments: 10103.patch, 10103.patch, 10103.patch
>
>
> {noformat}
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 623.639 sec
> <<< FAILURE!
> testRSHealthChore(org.apache.hadoop.hbase.TestNodeHealthCheckChore) Time
> elapsed: 0.001 sec <<< FAILURE!
> java.lang.AssertionError: Stoppable must have been stopped.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hbase.TestNodeHealthCheckChore.testRSHealthChore(TestNodeHealthCheckChore.java:108)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HBASE-10000) Initiate lease recovery for outstanding WAL files at the very beginning of recovery
[ https://issues.apache.org/jira/browse/HBASE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844756#comment-13844756 ] Ted Yu commented on HBASE-1: TestNodeHealthCheckChore failure was not related to patch v7. Patch has been updated on reviewboard. > Initiate lease recovery for outstanding WAL files at the very beginning of > recovery > --- > > Key: HBASE-1 > URL: https://issues.apache.org/jira/browse/HBASE-1 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Fix For: 0.98.1 > > Attachments: 1-0.96-v5.txt, 1-0.96-v6.txt, > 1-recover-ts-with-pb-2.txt, 1-recover-ts-with-pb-3.txt, > 1-recover-ts-with-pb-4.txt, 1-recover-ts-with-pb-5.txt, > 1-recover-ts-with-pb-6.txt, 1-recover-ts-with-pb-7.txt, 1-v4.txt, > 1-v5.txt, 1-v6.txt > > > At the beginning of recovery, master can send lease recovery requests > concurrently for outstanding WAL files using a thread pool. > Each split worker would first check whether the WAL file it processes is > closed. > Thanks to Nicolas Liochon and Jeffery discussion with whom gave rise to this > idea. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HBASE-10103) TestNodeHealthCheckChore#testRSHealthChore: Stoppable must have been stopped
[
https://issues.apache.org/jira/browse/HBASE-10103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844757#comment-13844757
]
Andrew Purtell commented on HBASE-10103:
Will amend to do both, coming up...
> TestNodeHealthCheckChore#testRSHealthChore: Stoppable must have been stopped
>
>
> Key: HBASE-10103
> URL: https://issues.apache.org/jira/browse/HBASE-10103
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.98.0, 0.99.0
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
> Fix For: 0.98.0, 0.96.1, 0.99.0
>
> Attachments: 10103.patch, 10103.patch, 10103.patch
>
>
> {noformat}
> Tests run: 4, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 623.639 sec
> <<< FAILURE!
> testRSHealthChore(org.apache.hadoop.hbase.TestNodeHealthCheckChore) Time
> elapsed: 0.001 sec <<< FAILURE!
> java.lang.AssertionError: Stoppable must have been stopped.
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at
> org.apache.hadoop.hbase.TestNodeHealthCheckChore.testRSHealthChore(TestNodeHealthCheckChore.java:108)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Comment Edited] (HBASE-10103) TestNodeHealthCheckChore#testRSHealthChore: Stoppable must have been stopped
[
https://issues.apache.org/jira/browse/HBASE-10103?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13844754#comment-13844754
]
Elliott Clark edited comment on HBASE-10103 at 12/10/13 10:29 PM:
--
I'm seeing TestNodeHealthCheckChore hang (or run slower than the timeout) on
trunk on my jenkins box (twice in a row). Running just
TestNodeHealthCheckChore locally passes though.
Here's the code stack that I saw one of the two times:
{code}
"pool-1-thread-1" prio=10 tid=0x7f1f106d1800 nid=0x57b1 runnable
[0x7f1f14676000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:220)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
- locked <0xfbac8c28> (a java.io.BufferedInputStream)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
- locked <0xfb8a6e00> (a java.io.InputStreamReader)
at java.io.InputStreamReader.read(InputStreamReader.java:167)
at java.io.BufferedReader.fill(BufferedReader.java:136)
at java.io.BufferedReader.read1(BufferedReader.java:187)
at java.io.BufferedReader.read(BufferedReader.java:261)
- locked <0xfb8a6e00> (a java.io.InputStreamReader)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.parseExecResult(Shell.java:602)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:446)
at org.apache.hadoop.util.Shell.run(Shell.java:379)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:589)
at
org.apache.hadoop.hbase.HealthChecker.checkHealth(HealthChecker.java:76)
at
org.apache.hadoop.hbase.TestNodeHealthCheckChore.healthCheckerTest(TestNodeHealthCheckChore.java:88)
at
org.apache.hadoop.hbase.TestNodeHealthCheckChore.testHealthCheckerTimeout(TestNodeHealthCheckChore.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238)
{code}
When I reverted this I got past the test. Does the timeout need to be adjusted
or does this need to be moved into a medium test ?
was (Author: eclark):
I'm seeing TestNodeHealthCheckChore hang (or run slower than the timeout) on
trunk on my jenkins box (twice in a row).
Here's the code stack that I saw one of the two times:
{code}
"pool-1-thread-1" prio=10 tid=0x7f1f106d1800 nid=0x57b1 runnable
[0x7f1f14676000]
java.lang.Thread.State: RUNNABLE
at java.io.FileInputStream.readBytes(Native Method)
at java.io.FileInputStream.read(FileInputStream.java:220)
at java.io.BufferedInputStream.read1(BufferedInputStream.java:256)
at java.io.BufferedInputStream.read(BufferedInputStream.java:317)
- locked <0xfbac8c28> (a java.io.BufferedInputStream)
at sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:264)
at sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:306)
at sun.nio.cs.StreamDecoder.read(StreamDecoder.java:158)
- locked <0xfb8a6e00> (a java.io.InputStreamReader)
at java.io.InputStreamReader.read(InputStreamReader.java:167)
at java.io.BufferedReader.fill(BufferedReader.java:136)
at java.io.BufferedReader.read1(BufferedReader.java:187)
at java.io.BufferedReader.read(BufferedReader.java:261)
- locked <0xfb8a6e00> (a java.io.InputStreamReader)
at
org.apache.hadoop.util.Shell$ShellCommandExecutor.parseExecResult(Shell.java:602)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:446)
at org.apache.hadoop.util.She
