[jira] [Commented] (HBASE-17997) jruby-complete-1.6.8.jar is in cached_classpath.txt
[
https://issues.apache.org/jira/browse/HBASE-17997?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020711#comment-16020711
]
Xiang Li commented on HBASE-17997:
--
Summarize the logic here:
1. In hbase-assembly/pom.xml, the output file is specified for Maven dependency
plug-in, so that all dependency jars in dev environment are written into the
file specified.
{code}
${project.parent.basedir}/target/cached_classpath.txt
{code}
2. bin/hbase will make good use of that file by reading all jars out and then
appending them into classpath for dev environment
{code}
add_maven_deps_to_classpath() {
f="${HBASE_HOME}/target/cached_classpath.txt"
if [ ! -f "${f}" ]
then
echo "As this is a development environment, we need ${f} to be generated
from maven (command: mvn install -DskipTests)"
exit 1
fi
CLASSPATH=${CLASSPATH}:`cat "${f}"` <-- here
}
#Add the development env class path stuff
if $in_dev_env; then
add_maven_deps_to_classpath
fi
{code}
So that is the reason why jruby-complete jar, being as a dependency of hbase
shell, appears in the classpath of dev environment and leading to the error as
described in the "Description"
> jruby-complete-1.6.8.jar is in cached_classpath.txt
> ---
>
> Key: HBASE-17997
> URL: https://issues.apache.org/jira/browse/HBASE-17997
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Xiang Li
>
> HBASE-15199 moves jruby-complete-1.6.8.jar to lib/ruby directory.
> However, jruby-complete-1.6.8.jar still appears in cached_classpath.txt
> This means that user would see exception similar to the following when
> starting hbase in standalone mode with s3a as rootdir :
> {code}
> 2017-05-04 16:41:32,854 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=38659]
> internal.S3MetadataResponseHandler: Unable to parse last modified date: Thu,
> 04 May 2017 16:27:09 GMT
> java.lang.IllegalStateException: Joda-time 2.2 or later version is required,
> but found version: null
> at com.amazonaws.util.DateUtils.handleException(DateUtils.java:149)
> at com.amazonaws.util.DateUtils.parseRFC822Date(DateUtils.java:195)
> at
> com.amazonaws.services.s3.internal.ServiceUtils.parseRfc822Date(ServiceUtils.java:78)
> at
> com.amazonaws.services.s3.internal.AbstractS3ResponseHandler.populateObjectMetadata(AbstractS3ResponseHandler.java:115)
> at
> com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:52)
> at
> com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:30)
> at
> com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1072)
> at
> com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:746)
> at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
> at
> com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1191)
> at org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:148)
> at org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:281)
> at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:364)
> at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:75)
> at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.positionalReadWithExtra(HFileBlock.java:722)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1420)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1677)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1504)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:439)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:904)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:267)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:169)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:363)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:217)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2132)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2122)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5687)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(H
[jira] [Updated] (HBASE-18008) Any HColumnDescriptor we give out should be immutable
[ https://issues.apache.org/jira/browse/HBASE-18008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-18008: --- Status: Patch Available (was: Open) > Any HColumnDescriptor we give out should be immutable > - > > Key: HBASE-18008 > URL: https://issues.apache.org/jira/browse/HBASE-18008 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai > Fix For: 2.0.0 > > Attachments: HBASE-18008.v0.patch, HBASE-18008.v1.patch, > HBASE-18008.v2.patch, HBASE-18008.v3.patch > > > This is similar to HBASE-15583, but we should move up on to the > ColumnFamilyDescriptor rather than ColumnDescriptor. > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18008) Any HColumnDescriptor we give out should be immutable
[ https://issues.apache.org/jira/browse/HBASE-18008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-18008: --- Attachment: HBASE-18008.v3.patch 03 -- # rebase # all failed tests pass > Any HColumnDescriptor we give out should be immutable > - > > Key: HBASE-18008 > URL: https://issues.apache.org/jira/browse/HBASE-18008 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai > Fix For: 2.0.0 > > Attachments: HBASE-18008.v0.patch, HBASE-18008.v1.patch, > HBASE-18008.v2.patch, HBASE-18008.v3.patch > > > This is similar to HBASE-15583, but we should move up on to the > ColumnFamilyDescriptor rather than ColumnDescriptor. > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18008) Any HColumnDescriptor we give out should be immutable
[ https://issues.apache.org/jira/browse/HBASE-18008?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chia-Ping Tsai updated HBASE-18008: --- Status: Open (was: Patch Available) > Any HColumnDescriptor we give out should be immutable > - > > Key: HBASE-18008 > URL: https://issues.apache.org/jira/browse/HBASE-18008 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.0.0 >Reporter: Chia-Ping Tsai >Assignee: Chia-Ping Tsai > Fix For: 2.0.0 > > Attachments: HBASE-18008.v0.patch, HBASE-18008.v1.patch, > HBASE-18008.v2.patch, HBASE-18008.v3.patch > > > This is similar to HBASE-15583, but we should move up on to the > ColumnFamilyDescriptor rather than ColumnDescriptor. > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HBASE-17997) jruby-complete-1.6.8.jar is in cached_classpath.txt
[
https://issues.apache.org/jira/browse/HBASE-17997?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiang Li reassigned HBASE-17997:
Assignee: Xiang Li
> jruby-complete-1.6.8.jar is in cached_classpath.txt
> ---
>
> Key: HBASE-17997
> URL: https://issues.apache.org/jira/browse/HBASE-17997
> Project: HBase
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Xiang Li
>
> HBASE-15199 moves jruby-complete-1.6.8.jar to lib/ruby directory.
> However, jruby-complete-1.6.8.jar still appears in cached_classpath.txt
> This means that user would see exception similar to the following when
> starting hbase in standalone mode with s3a as rootdir :
> {code}
> 2017-05-04 16:41:32,854 WARN
> [RpcServer.FifoWFPBQ.priority.handler=18,queue=0,port=38659]
> internal.S3MetadataResponseHandler: Unable to parse last modified date: Thu,
> 04 May 2017 16:27:09 GMT
> java.lang.IllegalStateException: Joda-time 2.2 or later version is required,
> but found version: null
> at com.amazonaws.util.DateUtils.handleException(DateUtils.java:149)
> at com.amazonaws.util.DateUtils.parseRFC822Date(DateUtils.java:195)
> at
> com.amazonaws.services.s3.internal.ServiceUtils.parseRfc822Date(ServiceUtils.java:78)
> at
> com.amazonaws.services.s3.internal.AbstractS3ResponseHandler.populateObjectMetadata(AbstractS3ResponseHandler.java:115)
> at
> com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:52)
> at
> com.amazonaws.services.s3.internal.S3ObjectResponseHandler.handle(S3ObjectResponseHandler.java:30)
> at
> com.amazonaws.http.AmazonHttpClient.handleResponse(AmazonHttpClient.java:1072)
> at
> com.amazonaws.http.AmazonHttpClient.executeOneRequest(AmazonHttpClient.java:746)
> at
> com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.java:489)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:310)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:3785)
> at
> com.amazonaws.services.s3.AmazonS3Client.getObject(AmazonS3Client.java:1191)
> at org.apache.hadoop.fs.s3a.S3AInputStream.reopen(S3AInputStream.java:148)
> at org.apache.hadoop.fs.s3a.S3AInputStream.lazySeek(S3AInputStream.java:281)
> at org.apache.hadoop.fs.s3a.S3AInputStream.read(S3AInputStream.java:364)
> at org.apache.hadoop.fs.FSInputStream.read(FSInputStream.java:75)
> at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:92)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock.positionalReadWithExtra(HFileBlock.java:722)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1420)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockDataInternal(HFileBlock.java:1677)
> at
> org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderImpl.readBlockData(HFileBlock.java:1504)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:439)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.seekTo(HFileReaderV2.java:904)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:267)
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:169)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekScanners(StoreScanner.java:363)
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:217)
> at
> org.apache.hadoop.hbase.regionserver.HStore.createScanner(HStore.java:2132)
> at org.apache.hadoop.hbase.regionserver.HStore.getScanner(HStore.java:2122)
> at
> org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:5687)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:2679)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2665)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:2647)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6906)
> at org.apache.hadoop.hbase.regionserver.HRegion.get(HRegion.java:6885)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.get(RSRpcServices.java:2007)
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18013) Write response directly instead of creating a fake call when setup connection
[
https://issues.apache.org/jira/browse/HBASE-18013?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020689#comment-16020689
]
Hadoop QA commented on HBASE-18013:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 44s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
32m 22s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 133m 45s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 180m 50s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12869389/HBASE-18013-v1.patch |
| JIRA Issue | HBASE-18013 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 2cf5518bfa40 3.13.0-105-generic #152-Ubuntu SMP Fri Dec 2
15:37:11 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 28d619b |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6890/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6890/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Write response directly instead of creating a fake call when setup connection
> -
>
> Key: HBASE-18013
> URL: https://issues.apache.org/jira/browse/HBASE-18013
> Project: HBase
> Issue Type: Sub-task
> Compon
[jira] [Commented] (HBASE-18085) Prevent parallel purge in ObjectPool
[
https://issues.apache.org/jira/browse/HBASE-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020685#comment-16020685
]
Yu Li commented on HBASE-18085:
---
bq. In ur test method for tryLock, there is no logic other than just try lock
and release. If that is been removed as dead code by compiler?
Ok got your point now.
Have changed the testLock method to use Blackhole like:
{code}
@Benchmark
public void testLock(Blackhole bh) {
if (purgeLock.tryLock()) {
try {
// do purge
bh.consume(0);
return;
} finally {
purgeLock.unlock();
}
}
bh.consume(1);
}
{code}
and the result turned out:
{noformat}
Benchmark Mode Cnt Score
Error Units
AtomicLockBenchmark.testAtomicBooleanthrpt 10047677994.547 ±
1295778.841 ops/s
AtomicLockBenchmark.testLock thrpt 100 2277364613.072 ±
49185975.985 ops/s
AtomicLockBenchmark.testVolatileBoolean thrpt 10044386308.984 ±
644407.794 ops/s
{noformat}
Checking the compiled class:
{noformat}
public void testLock(org.openjdk.jmh.infra.Blackhole);
Code:
0: aload_0
1: getfield #7 // Field
purgeLock:Ljava/util/concurrent/locks/Lock;
4: invokeinterface #11, 1 // InterfaceMethod
java/util/concurrent/locks/Lock.tryLock:()Z
9: ifeq 39
12: aload_1
13: iconst_0
14: invokevirtual #12 // Method
org/openjdk/jmh/infra/Blackhole.consume:(I)V
17: aload_0
18: getfield #7 // Field
purgeLock:Ljava/util/concurrent/locks/Lock;
21: invokeinterface #13, 1 // InterfaceMethod
java/util/concurrent/locks/Lock.unlock:()V
26: return
27: astore_2
28: aload_0
29: getfield #7 // Field
purgeLock:Ljava/util/concurrent/locks/Lock;
32: invokeinterface #13, 1 // InterfaceMethod
java/util/concurrent/locks/Lock.unlock:()V
37: aload_2
38: athrow
39: aload_1
40: iconst_1
41: invokevirtual #12 // Method
org/openjdk/jmh/infra/Blackhole.consume:(I)V
44: return
Exception table:
fromto target type
121727 any
{noformat}
Let me know your thoughts sir [~anoop.hbase]
> Prevent parallel purge in ObjectPool
>
>
> Key: HBASE-18085
> URL: https://issues.apache.org/jira/browse/HBASE-18085
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: e89l05465.st3.jstack, HBASE-18085.patch
>
>
> Parallel purge in ObjectPool is meaningless and will cause contention issue
> since {{ReferenceQueue#poll}} has synchronization (source code shown below)
> {code}
> public Reference poll() {
> if (head == null)
> return null;
> synchronized (lock) {
> return reallyPoll();
> }
> }
> {code}
> We observed threads blocking on the purge method while using offheap bucket
> cache, and we could easily reproduce this by testing the 100% cache hit case
> in bucket cache with enough reading threads.
> We propose to add a purgeLock and use tryLock to avoid parallel purge.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18085) Prevent parallel purge in ObjectPool
[
https://issues.apache.org/jira/browse/HBASE-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020668#comment-16020668
]
Anoop Sam John commented on HBASE-18085:
In ur test method for tryLock, there is no logic other than just try lock and
release. If that is been removed as dead code by compiler? One way to avoid
that is using the return value. See BlackHole in JMH and its usage.. The diff
in numbers that reported by JMH benchmark is so huge! The impl of tryLock is
having a volatile read and all. So this huge diff in numbers looks strange
no? That was my doubt.
> Prevent parallel purge in ObjectPool
>
>
> Key: HBASE-18085
> URL: https://issues.apache.org/jira/browse/HBASE-18085
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: e89l05465.st3.jstack, HBASE-18085.patch
>
>
> Parallel purge in ObjectPool is meaningless and will cause contention issue
> since {{ReferenceQueue#poll}} has synchronization (source code shown below)
> {code}
> public Reference poll() {
> if (head == null)
> return null;
> synchronized (lock) {
> return reallyPoll();
> }
> }
> {code}
> We observed threads blocking on the purge method while using offheap bucket
> cache, and we could easily reproduce this by testing the 100% cache hit case
> in bucket cache with enough reading threads.
> We propose to add a purgeLock and use tryLock to avoid parallel purge.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-16961) FileSystem Quotas
[ https://issues.apache.org/jira/browse/HBASE-16961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020666#comment-16020666 ] Chia-Ping Tsai commented on HBASE-16961: Bravo!! > FileSystem Quotas > - > > Key: HBASE-16961 > URL: https://issues.apache.org/jira/browse/HBASE-16961 > Project: HBase > Issue Type: New Feature >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 2.0.0 > > Attachments: hbase-quota-test.sh > > > Umbrella issue for tracking the filesystem utilization of HBase data, > defining quotas on that utilization, and enforcement when utilization exceeds > the limits of the quota. > At a high level: we can define quotas on tables and namespaces. Region size > is computed by RegionServers and sent to the Master. The Master inspects the > sizes of Regions, rolling up to table and namespace sizes. Defined quotas in > the quota table are evaluated given the computed sizes, and, for those > tables/namespaces violating the quota, RegionServers are informed to take > some action to limit any further filesystem growth by that table/namespace. > Discuss: > https://lists.apache.org/thread.html/66a4b0c3725b5cbdd61dd6111c43847adaeef7b7da5f4cd045df30ef@%3Cdev.hbase.apache.org%3E > Design Doc: > http://home.apache.org/~elserj/hbase/FileSystemQuotasforApacheHBase.pdf or > https://docs.google.com/document/d/1VtLWDkB2tpwc_zgCNPE1ulZOeecF-YA2FYSK3TSs_bw/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17850) Backup system repair utility
[
https://issues.apache.org/jira/browse/HBASE-17850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020659#comment-16020659
]
Hudson commented on HBASE-17850:
SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #3060 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/3060/])
HBASE-17850 Backup system repair utility (Vladimir Rodionov) (tedyu: rev
28d619b22bec1b93259a96bfaba3ec1aecb026a2)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/impl/BackupAdminImpl.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/impl/TableBackupClient.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestBackupBase.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/BackupRestoreConstants.java
* (add)
hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestIncrementalBackupWithFailures.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestFullBackupWithFailures.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/impl/BackupCommands.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/BackupDriver.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/BackupInfo.java
* (add)
hbase-server/src/test/java/org/apache/hadoop/hbase/backup/TestBackupRepair.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/backup/impl/IncrementalTableBackupClient.java
> Backup system repair utility
>
>
> Key: HBASE-17850
> URL: https://issues.apache.org/jira/browse/HBASE-17850
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-17850-v2.patch, HBASE-17850-v3.patch,
> HBASE-17850-v4.patch, HBASE-17850-v5.patch, HBASE-17850-v6.patch,
> HBASE-17850-v7.patch
>
>
> Backup repair tool restores integrity of a backup system table and removes
> artefacts of a failed backup session from a file system(s)
> This is command-line tool. To run the tool:
> {code}
> hbase backup repair
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18093) Overloading the meaning of 'enabled' in Quota Manager to indicate either quota disabled or quota manager not ready is not good
[
https://issues.apache.org/jira/browse/HBASE-18093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020651#comment-16020651
]
Hadoop QA commented on HBASE-18093:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 30s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
45s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 38s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
41s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
59m 43s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 228m 58s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
13s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 320m 53s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestAsyncProcedureAdminApi |
| | hadoop.hbase.client.TestAsyncRegionAdminApi |
| | hadoop.hbase.client.TestMultiRespectsLimits |
| | hadoop.hbase.client.TestAsyncBalancerAdminApi |
| | hadoop.hbase.client.TestMobSnapshotCloneIndependence |
| | hadoop.hbase.client.TestAsyncTableAdminApi |
| | hadoop.hbase.client.TestBlockEvictionFromClient |
| | hadoop.hbase.regionserver.TestPerColumnFamilyFlush |
| | hadoop.hbase.util.TestHBaseFsckReplicas |
| | hadoop.hbase.client.TestAsyncSnapshotAdminApi |
| Timed out junit tests | org.apache.hadoop.hbase.backup.TestFullBackup |
| | org.apache.hadoop.hbase.replication.regionserver.TestWALEntryStream |
| | org.apache.hadoop.hbase.backup.TestRestoreBoundaryTests |
| | org.apache.hadoop.hbase.backup.TestBackupBoundaryTests |
| | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12869361/HBASE-18093.v3-master.patch
|
| JIRA Issue | HBASE-18093 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux cd48a58f4558 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-su
[jira] [Commented] (HBASE-18005) read replica: handle the case that region server hosting both primary replica and meta region is down
[
https://issues.apache.org/jira/browse/HBASE-18005?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020633#comment-16020633
]
Hadoop QA commented on HBASE-18005:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 31s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 24s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 14s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 7s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 0s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 17s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
33m 17s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 56s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 110m 21s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
51s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 169m 29s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.mapreduce.TestMultiTableSnapshotInputFormat
|
| | hadoop.hbase.snapshot.TestMobRestoreFlushSnapshotFromClient |
| | hadoop.hbase.mapreduce.TestLoadIncrementalHFilesSplitRecovery |
| | hadoop.hbase.mapreduce.TestSyncTable |
| | hadoop.hbase.mapreduce.TestImportTSVWithTTLs |
| | hadoop.hbase.mapreduce.TestSecureLoadIncrementalHFilesSplitRecovery |
| | hadoop.hbase.mapreduce.TestTableInputFormat |
| | hadoop.hbase.snapshot.TestMobFlushSnapshotFromClient |
| | hadoop.hbase.snapshot.TestFlushSnapshotFromClient |
| | hadoop.hbase.mapreduce.TestTableInputFormatScan2 |
| | hadoop.hbase.quotas.TestQuotaThrottle |
| Timed out junit tests |
org.apache.hadoop.hbase.master.procedure.TestSafemodeBringsDownMaster |
| | org.apache.hadoop.hbase.mapreduce.TestTableMapReduce |
| | org.apache.hadoop.hbase.snapshot.TestSecureExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestMobExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestExportSnapshot |
| | org.apache.hadoop.hbase.mapreduce.Te
[jira] [Commented] (HBASE-18066) Get with closest_row_before on "hbase:meta" can return empty Cell during region merge/split
[ https://issues.apache.org/jira/browse/HBASE-18066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020631#comment-16020631 ] stack commented on HBASE-18066: --- bq. Yes, and I think , better to support getClosestRowBefore in master , otherwise request from branch-1.x hbase-client to master branch server will fail. Interesting [~openinx]. Yeah, we removed the code already but for example, asynchbase does not current run against hbase2 because of the missing getClosestRowBefore (which messes up my being able to use opentsdb studying hbase2 in operation). > Get with closest_row_before on "hbase:meta" can return empty Cell during > region merge/split > --- > > Key: HBASE-18066 > URL: https://issues.apache.org/jira/browse/HBASE-18066 > Project: HBase > Issue Type: Bug > Components: hbase, regionserver >Affects Versions: 1.3.1 > Environment: Linux (16.04.2), MacOS 10.11.6. > Standalone and distributed HBase setup. >Reporter: Andrey Elenskiy >Assignee: Zheng Hu > Attachments: TestGetWithClosestRowBeforeWhenSplit.java > > > During region split/merge there's a brief period of time where doing a "Get" > with "closest_row_before=true" on "hbase:meta" may return empty > "GetResponse.result.cell" field even though parent, splitA and splitB regions > are all in "hbase:meta". Both gohbase (https://github.com/tsuna/gohbase) and > AsyncHBase (https://github.com/OpenTSDB/asynchbase) interprets this as > "TableDoesNotExist", which is returned to the client. > Here's a gist that reproduces this problem: > https://gist.github.com/Timoha/c7a236b768be9220e85e53e1ca53bf96. Note that > you have to use older HTable client (I used 1.2.4) as current versions ignore > `Get.setClosestRowBefore(bool)` option. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18066) Get with closest_row_before on "hbase:meta" can return empty Cell during region merge/split
[ https://issues.apache.org/jira/browse/HBASE-18066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020628#comment-16020628 ] Zheng Hu commented on HBASE-18066: -- bq. And for master I think we have already removed the related code already? Yes, and I think , better to support getClosestRowBefore in master , otherwise request from branch-1.x hbase-client to master branch server will fail. > Get with closest_row_before on "hbase:meta" can return empty Cell during > region merge/split > --- > > Key: HBASE-18066 > URL: https://issues.apache.org/jira/browse/HBASE-18066 > Project: HBase > Issue Type: Bug > Components: hbase, regionserver >Affects Versions: 1.3.1 > Environment: Linux (16.04.2), MacOS 10.11.6. > Standalone and distributed HBase setup. >Reporter: Andrey Elenskiy >Assignee: Zheng Hu > Attachments: TestGetWithClosestRowBeforeWhenSplit.java > > > During region split/merge there's a brief period of time where doing a "Get" > with "closest_row_before=true" on "hbase:meta" may return empty > "GetResponse.result.cell" field even though parent, splitA and splitB regions > are all in "hbase:meta". Both gohbase (https://github.com/tsuna/gohbase) and > AsyncHBase (https://github.com/OpenTSDB/asynchbase) interprets this as > "TableDoesNotExist", which is returned to the client. > Here's a gist that reproduces this problem: > https://gist.github.com/Timoha/c7a236b768be9220e85e53e1ca53bf96. Note that > you have to use older HTable client (I used 1.2.4) as current versions ignore > `Get.setClosestRowBefore(bool)` option. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HBASE-18066) Get with closest_row_before on "hbase:meta" can return empty Cell during region merge/split
[ https://issues.apache.org/jira/browse/HBASE-18066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zheng Hu reassigned HBASE-18066: Assignee: Zheng Hu > Get with closest_row_before on "hbase:meta" can return empty Cell during > region merge/split > --- > > Key: HBASE-18066 > URL: https://issues.apache.org/jira/browse/HBASE-18066 > Project: HBase > Issue Type: Bug > Components: hbase, regionserver >Affects Versions: 1.3.1 > Environment: Linux (16.04.2), MacOS 10.11.6. > Standalone and distributed HBase setup. >Reporter: Andrey Elenskiy >Assignee: Zheng Hu > Attachments: TestGetWithClosestRowBeforeWhenSplit.java > > > During region split/merge there's a brief period of time where doing a "Get" > with "closest_row_before=true" on "hbase:meta" may return empty > "GetResponse.result.cell" field even though parent, splitA and splitB regions > are all in "hbase:meta". Both gohbase (https://github.com/tsuna/gohbase) and > AsyncHBase (https://github.com/OpenTSDB/asynchbase) interprets this as > "TableDoesNotExist", which is returned to the client. > Here's a gist that reproduces this problem: > https://gist.github.com/Timoha/c7a236b768be9220e85e53e1ca53bf96. Note that > you have to use older HTable client (I used 1.2.4) as current versions ignore > `Get.setClosestRowBefore(bool)` option. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18013) Write response directly instead of creating a fake call when setup connection
[
https://issues.apache.org/jira/browse/HBASE-18013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-18013:
--
Attachment: HBASE-18013-v1.patch
Fix findbugs warnings.
> Write response directly instead of creating a fake call when setup connection
> -
>
> Key: HBASE-18013
> URL: https://issues.apache.org/jira/browse/HBASE-18013
> Project: HBase
> Issue Type: Sub-task
> Components: IPC/RPC
>Affects Versions: 2.0.0
>Reporter: Duo Zhang
>Assignee: Duo Zhang
> Fix For: 2.0.0
>
> Attachments: HBASE-18013.patch, HBASE-18013-v1.patch
>
>
> We do not execute the 'call' with CallRunner, so it is not necessary to
> create it. We can remove several fields in RpcServer.Connection, and we can
> also remove {{setSaslTokenResponse}} and {{setConnectionHeaderResponse}} in
> RpcServer.Call.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18084) Improve CleanerChore to clean from directory which consumes more disk space
[ https://issues.apache.org/jira/browse/HBASE-18084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020591#comment-16020591 ] Yu Li commented on HBASE-18084: --- Ok, at a second thought I think I got your point [~huaxiang], but here we need to return a result opposite to Long.compare. And I think a direct line here would be better for readability. > Improve CleanerChore to clean from directory which consumes more disk space > --- > > Key: HBASE-18084 > URL: https://issues.apache.org/jira/browse/HBASE-18084 > Project: HBase > Issue Type: Bug >Reporter: Yu Li >Assignee: Yu Li > Attachments: HBASE-18084.patch, HBASE-18084.v2.patch > > > Currently CleanerChore cleans the directory in dictionary order, rather than > from the directory with largest space usage. And when data abnormally > accumulated to some huge volume in archive directory, the cleaning speed > might not be enough. > This proposal is another improvement working together with HBASE-18083 to > resolve our online issue (archive dir consumed more than 1.8PB SSD space) -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18085) Prevent parallel purge in ObjectPool
[
https://issues.apache.org/jira/browse/HBASE-18085?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020588#comment-16020588
]
Yu Li commented on HBASE-18085:
---
bq. Can we use the return value of purgeLock.tryLock() passed to BlackHole or
so?
Sorry, don't quite catch you sir, mind clarify?
While writing the JMH case, I felt that using AtomicBoolean or volatile boolean
we're actually re-implementing the tryLock logic. And I won't be surprised if
JIT has any optimization for its native implementation (smile).
> Prevent parallel purge in ObjectPool
>
>
> Key: HBASE-18085
> URL: https://issues.apache.org/jira/browse/HBASE-18085
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: e89l05465.st3.jstack, HBASE-18085.patch
>
>
> Parallel purge in ObjectPool is meaningless and will cause contention issue
> since {{ReferenceQueue#poll}} has synchronization (source code shown below)
> {code}
> public Reference poll() {
> if (head == null)
> return null;
> synchronized (lock) {
> return reallyPoll();
> }
> }
> {code}
> We observed threads blocking on the purge method while using offheap bucket
> cache, and we could easily reproduce this by testing the 100% cache hit case
> in bucket cache with enough reading threads.
> We propose to add a purgeLock and use tryLock to avoid parallel purge.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18066) Get with closest_row_before on "hbase:meta" can return empty Cell during region merge/split
[ https://issues.apache.org/jira/browse/HBASE-18066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020586#comment-16020586 ] Duo Zhang commented on HBASE-18066: --- I would suggest we implement it with reverse scan at server side. And for master I think we have already removed the related code already? > Get with closest_row_before on "hbase:meta" can return empty Cell during > region merge/split > --- > > Key: HBASE-18066 > URL: https://issues.apache.org/jira/browse/HBASE-18066 > Project: HBase > Issue Type: Bug > Components: hbase, regionserver >Affects Versions: 1.3.1 > Environment: Linux (16.04.2), MacOS 10.11.6. > Standalone and distributed HBase setup. >Reporter: Andrey Elenskiy > Attachments: TestGetWithClosestRowBeforeWhenSplit.java > > > During region split/merge there's a brief period of time where doing a "Get" > with "closest_row_before=true" on "hbase:meta" may return empty > "GetResponse.result.cell" field even though parent, splitA and splitB regions > are all in "hbase:meta". Both gohbase (https://github.com/tsuna/gohbase) and > AsyncHBase (https://github.com/OpenTSDB/asynchbase) interprets this as > "TableDoesNotExist", which is returned to the client. > Here's a gist that reproduces this problem: > https://gist.github.com/Timoha/c7a236b768be9220e85e53e1ca53bf96. Note that > you have to use older HTable client (I used 1.2.4) as current versions ignore > `Get.setClosestRowBefore(bool)` option. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18066) Get with closest_row_before on "hbase:meta" can return empty Cell during region merge/split
[ https://issues.apache.org/jira/browse/HBASE-18066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020583#comment-16020583 ] Zheng Hu commented on HBASE-18066: -- bq. My question is why we implement the function with two steps? I mean first get the row key and then use this row key to get the row. Is it only an implementation compromise? Or intentional? [~Apache9], If implement the function with one step(scan backward) , then the implementation would be similar to reverse scan which was not implemented yet (Maybe a lot of work was needed for reverse scan at that time), so we provided the method getClosestRowBefore to locate the region in hbase:meta, I guess. For compatibility, we should keep getClosestRowBefore in branch-1.x & master branch at server side, so we should fix the bug at server side. There are two way to fix the bug, In my opinion: 1.Just implement getClosestRowBefore by reverse scan at server side. 2.Still two steps, but we get the readpoint firstly, and then pass the readpoint in both getting rowkey step and getting row step. Let me have a try. > Get with closest_row_before on "hbase:meta" can return empty Cell during > region merge/split > --- > > Key: HBASE-18066 > URL: https://issues.apache.org/jira/browse/HBASE-18066 > Project: HBase > Issue Type: Bug > Components: hbase, regionserver >Affects Versions: 1.3.1 > Environment: Linux (16.04.2), MacOS 10.11.6. > Standalone and distributed HBase setup. >Reporter: Andrey Elenskiy > Attachments: TestGetWithClosestRowBeforeWhenSplit.java > > > During region split/merge there's a brief period of time where doing a "Get" > with "closest_row_before=true" on "hbase:meta" may return empty > "GetResponse.result.cell" field even though parent, splitA and splitB regions > are all in "hbase:meta". Both gohbase (https://github.com/tsuna/gohbase) and > AsyncHBase (https://github.com/OpenTSDB/asynchbase) interprets this as > "TableDoesNotExist", which is returned to the client. > Here's a gist that reproduces this problem: > https://gist.github.com/Timoha/c7a236b768be9220e85e53e1ca53bf96. Note that > you have to use older HTable client (I used 1.2.4) as current versions ignore > `Get.setClosestRowBefore(bool)` option. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18084) Improve CleanerChore to clean from directory which consumes more disk space
[
https://issues.apache.org/jira/browse/HBASE-18084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020580#comment-16020580
]
Yu Li commented on HBASE-18084:
---
bq. Replace it with Long.compare(long x, long y)?
What's the reason please? I could find quite a lot primitive long comparison
such as {{HRegion#isFlushSize}}
{code}
private boolean isFlushSize(final long size) {
return size > this.memstoreFlushSize;
}
{code}
Thanks.
Will wait for another day to commit this one, but to be honest, I think we have
spent too much time here for a simple enough change.
> Improve CleanerChore to clean from directory which consumes more disk space
> ---
>
> Key: HBASE-18084
> URL: https://issues.apache.org/jira/browse/HBASE-18084
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-18084.patch, HBASE-18084.v2.patch
>
>
> Currently CleanerChore cleans the directory in dictionary order, rather than
> from the directory with largest space usage. And when data abnormally
> accumulated to some huge volume in archive directory, the cleaning speed
> might not be enough.
> This proposal is another improvement working together with HBASE-18083 to
> resolve our online issue (archive dir consumed more than 1.8PB SSD space)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18042) Client Compatibility breaks between versions 1.2 and 1.3
[
https://issues.apache.org/jira/browse/HBASE-18042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Duo Zhang updated HBASE-18042:
--
Affects Version/s: 1.4.0
1.2.5
1.1.10
Priority: Critical (was: Major)
Fix Version/s: 1.1.11
1.3.2
1.2.6
1.4.0
> Client Compatibility breaks between versions 1.2 and 1.3
>
>
> Key: HBASE-18042
> URL: https://issues.apache.org/jira/browse/HBASE-18042
> Project: HBase
> Issue Type: Bug
>Affects Versions: 1.4.0, 1.3.1, 1.2.5, 1.1.10
>Reporter: Karan Mehta
>Assignee: Karan Mehta
>Priority: Critical
> Fix For: 1.4.0, 1.2.6, 1.3.2, 1.1.11
>
>
> OpenTSDB uses AsyncHBase as its client, rather than using the traditional
> HBase Client. From version 1.2 to 1.3, the {{ClientProtos}} have been
> changed. Newer fields are added to {{ScanResponse}} proto.
> For a typical Scan request in 1.2, would require caller to make an
> OpenScanner Request, GetNextRows Request and a CloseScanner Request, based on
> {{more_rows}} boolean field in the {{ScanResponse}} proto.
> However, from 1.3, new parameter {{more_results_in_region}} was added, which
> limits the results per region. Therefore the client has to now manage sending
> all the requests for each region. Further more, if the results are exhausted
> from a particular region, the {{ScanResponse}} will set
> {{more_results_in_region}} to false, but {{more_results}} can still be true.
> Whenever the former is set to false, the {{RegionScanner}} will also be
> closed.
> OpenTSDB makes an OpenScanner Request and receives all its results in the
> first {{ScanResponse}} itself, thus creating a condition as described in
> above paragraph. Since {{more_rows}} is true, it will proceed to send next
> request at which point the {{RSRpcServices}} will throw
> {{UnknownScannerException}}. The protobuf client compatibility is maintained
> but expected behavior is modified.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18068) Fix flaky test TestAsyncSnapshotAdminApi
[
https://issues.apache.org/jira/browse/HBASE-18068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020562#comment-16020562
]
Zheng Hu commented on HBASE-18068:
--
Sorry for the late rely. The patch looks good and thanks for the fix.
> Fix flaky test TestAsyncSnapshotAdminApi
>
>
> Key: HBASE-18068
> URL: https://issues.apache.org/jira/browse/HBASE-18068
> Project: HBase
> Issue Type: Sub-task
> Components: Client
>Reporter: Appy
>Assignee: Appy
> Fix For: 2.0.0
>
> Attachments: HBASE-18068.master.001.patch
>
>
> Test failures
> {noformat}
> org.apache.hadoop.hbase.client.TestAsyncSnapshotAdminApi.testRestoreSnapshot
>
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotException:
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotException: Restore already in
> progress on the table=testRestoreSnapshot
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.restoreSnapshot(SnapshotManager.java:854)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.restoreSnapshot(SnapshotManager.java:818)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.restoreOrCloneSnapshot(SnapshotManager.java:780)
> at org.apache.hadoop.hbase.master.HMaster$14.run(HMaster.java:2324)
> at
> org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:133)
> at org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2320)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.restoreSnapshot(MasterRpcServices.java:1224)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:257)
>
> org.apache.hadoop.hbase.client.TestAsyncSnapshotAdminApi.testDeleteSnapshots
>
> org.apache.hadoop.hbase.snapshot.SnapshotCreationException:
> org.apache.hadoop.hbase.snapshot.SnapshotCreationException: Rejected taking {
> ss=snapshotName1 table=testDeleteSnapshots type=FLUSH } because we are
> already running another snapshot on the same table { ss=snapshotName1
> table=testDeleteSnapshots type=FLUSH }
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.prepareToTakeSnapshot(SnapshotManager.java:440)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.snapshotEnabledTable(SnapshotManager.java:497)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.takeSnapshot(SnapshotManager.java:598)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.snapshot(MasterRpcServices.java:1299)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:257)
>
> org.apache.hadoop.hbase.client.TestAsyncSnapshotAdminApi.testListSnapshots
>
> org.apache.hadoop.hbase.snapshot.SnapshotDoesNotExistException: Snapshot
> 'snapshotName2' doesn't exist on the filesystem
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.deleteSnapshot(SnapshotManager.java:289)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.deleteSnapshot(MasterRpcServices.java:461)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:257)
>
> {noformat}
> https://builds.apache.org/job/HBASE-Flaky-Tests/16152/
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-11544) [Ergonomics] hbase.client.scanner.caching is dogged and will try to return batch even if it means OOME
[
https://issues.apache.org/jira/browse/HBASE-11544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020554#comment-16020554
]
Karan Mehta commented on HBASE-11544:
-
[~jonathan.lawlor]
Can the server return multiple partial rows?
If yes, why does client side code assume that only the last result is partial
in {{ClientScanner}} ?
If not, why do we have a repeated boolean value for {{partial_flag_per_result}}
in {{Client.protos}} ?
> [Ergonomics] hbase.client.scanner.caching is dogged and will try to return
> batch even if it means OOME
> --
>
> Key: HBASE-11544
> URL: https://issues.apache.org/jira/browse/HBASE-11544
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: Jonathan Lawlor
>Priority: Critical
> Fix For: 2.0.0, 1.1.0
>
> Attachments: Allocation_Hot_Spots.html, gc.j.png,
> HBASE-11544-addendum-v1.patch, HBASE-11544-addendum-v2.patch,
> HBASE-11544-branch_1_0-v1.patch, HBASE-11544-branch_1_0-v2.patch,
> HBASE-11544-v1.patch, HBASE-11544-v2.patch, HBASE-11544-v3.patch,
> HBASE-11544-v4.patch, HBASE-11544-v5.patch, HBASE-11544-v6.patch,
> HBASE-11544-v6.patch, HBASE-11544-v6.patch, HBASE-11544-v7.patch,
> HBASE-11544-v8-branch-1.patch, HBASE-11544-v8.patch, hits.j.png, h.png,
> mean.png, m.png, net.j.png, q (2).png
>
>
> Running some tests, I set hbase.client.scanner.caching=1000. Dataset has
> large cells. I kept OOME'ing.
> Serverside, we should measure how much we've accumulated and return to the
> client whatever we've gathered once we pass out a certain size threshold
> rather than keep accumulating till we OOME.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18005) read replica: handle the case that region server hosting both primary replica and meta region is down
[ https://issues.apache.org/jira/browse/HBASE-18005?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] huaxiang sun updated HBASE-18005: - Attachment: HBASE-18005-master-004.patch v4 patch. Changed based on HBASE-18035, enabled useMetaReplica for the client in unittest case. Fixed one issue that the meta replica returns DoNotRetryIOException when it calls getRegionLocations(). > read replica: handle the case that region server hosting both primary replica > and meta region is down > - > > Key: HBASE-18005 > URL: https://issues.apache.org/jira/browse/HBASE-18005 > Project: HBase > Issue Type: Bug >Reporter: huaxiang sun >Assignee: huaxiang sun > Attachments: HBASE-18005-master-001.patch, > HBASE-18005-master-002.patch, HBASE-18005-master-003.patch, > HBASE-18005-master-004.patch > > > Identified one corner case in testing that when the region server hosting > both primary replica and the meta region is down, the client tries to reload > the primary replica location from meta table, it is supposed to clean up only > the cached location for specific replicaId, but it clears caches for all > replicas. Please see > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java#L813 > Since it takes some time for regions to be reassigned (including meta > region), the following may throw exception > https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/client/RpcRetryingCallerWithReadReplicas.java#L173 > This exception needs to be caught and it needs to get cached location (in > this case, the primary replica's location is not available). If there are > cached locations for other replicas, it can still go ahead to get stale > values from secondary replicas. > With meta replica, it still helps to not clean up the caches for all replicas > as the info from primary meta replica is up-to-date. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18027) Replication should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020514#comment-16020514 ] Andrew Purtell commented on HBASE-18027: Again, maybe the right patch this time > Replication should respect RPC size limits when batching edits > -- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch, > HBASE-18027.patch, HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry in the replication context is then placed in order by hash of > encoded region name into one of these N lists. Each of the N lists is then > sent all at once in one replication RPC. We do not test if the sum of data in > each N list will exceed RPC size limits. This code presumes each individual > edit is reasonably small. Not checking for aggregate size while assembling > the lists into RPCs is an oversight and can lead to replication failure when > that assumption is violated. > We can fix this by generating as many replication RPC calls as we need to > drain a list, keeping each RPC under limit, instead of assuming the whole > list will fit in one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18027) Replication should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18027: --- Attachment: HBASE-18027.patch > Replication should respect RPC size limits when batching edits > -- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch, > HBASE-18027.patch, HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry in the replication context is then placed in order by hash of > encoded region name into one of these N lists. Each of the N lists is then > sent all at once in one replication RPC. We do not test if the sum of data in > each N list will exceed RPC size limits. This code presumes each individual > edit is reasonably small. Not checking for aggregate size while assembling > the lists into RPCs is an oversight and can lead to replication failure when > that assumption is violated. > We can fix this by generating as many replication RPC calls as we need to > drain a list, keeping each RPC under limit, instead of assuming the whole > list will fit in one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18027) Replication should respect RPC size limits when batching edits
[
https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020497#comment-16020497
]
Hadoop QA commented on HBASE-18027:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 21s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
8s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 9s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
50s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
33m 0s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 20s
{color} | {color:red} hbase-server generated 1 new + 0 unchanged - 0 fixed = 1
total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 132m 45s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
19s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 180m 50s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hbase-server |
| | Inconsistent synchronization of
org.apache.hadoop.hbase.replication.regionserver.ReplicationSinkManager.sinks;
locked 72% of time Unsynchronized access at ReplicationSinkManager.java:72% of
time Unsynchronized access at ReplicationSinkManager.java:[line 175] |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12869338/HBASE-18027.patch |
| JIRA Issue | HBASE-18027 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 9fd7c2d008d5 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / f1a9990 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| findbugs |
https://builds.apache.org/job/PreCommit-HBASE-Build/6885/artifact/patchprocess/new-findbugs-hbase-server.html
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6885/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6885/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was au
[jira] [Updated] (HBASE-18060) Backport to branch-1 HBASE-9774 HBase native metrics and metric collection for coprocessors
[ https://issues.apache.org/jira/browse/HBASE-18060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vincent Poon updated HBASE-18060: - Attachment: HBASE-18060.branch-1.v5.patch HBASE-18060.branch-1.3.v5.patch fixed TestInterfaceAudienceAnnotations > Backport to branch-1 HBASE-9774 HBase native metrics and metric collection > for coprocessors > --- > > Key: HBASE-18060 > URL: https://issues.apache.org/jira/browse/HBASE-18060 > Project: HBase > Issue Type: New Feature >Affects Versions: 1.4.0, 1.3.2, 1.5.0 >Reporter: Vincent Poon >Assignee: Vincent Poon > Attachments: HBASE-18060.branch-1.3.v1.patch, > HBASE-18060.branch-1.3.v2.patch, HBASE-18060.branch-1.3.v3.patch, > HBASE-18060.branch-1.3.v4.patch, HBASE-18060.branch-1.3.v5.patch, > HBASE-18060.branch-1.v1.patch, HBASE-18060.branch-1.v2.patch, > HBASE-18060.branch-1.v3.patch, HBASE-18060.branch-1.v4.patch, > HBASE-18060.branch-1.v5.patch > > > I'd like to explore backporting HBASE-9774 to branch-1, as the ability for > coprocessors to report custom metrics through HBase is useful for us, and if > we have coprocessors use the native API, a re-write won't be necessary after > an upgrade to 2.0. > The main issues I see so far are: > - the usage of Java 8 language features. Seems we can work around this as > most of it is syntactic sugar. Will need to find a backport for LongAdder > - dropwizard 3.1.2 in Master. branch-1 is still on yammer metrics 2.2. Not > sure if these can coexist just for this feature -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Assigned] (HBASE-16392) Backup delete fault tolerance
[ https://issues.apache.org/jira/browse/HBASE-16392?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov reassigned HBASE-16392: - Assignee: Vladimir Rodionov > Backup delete fault tolerance > - > > Key: HBASE-16392 > URL: https://issues.apache.org/jira/browse/HBASE-16392 > Project: HBase > Issue Type: Sub-task >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > > Backup delete modified file system and backup system table. We have to make > sure that operation is atomic, durable and isolated. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17850) Backup system repair utility
[
https://issues.apache.org/jira/browse/HBASE-17850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020425#comment-16020425
]
Hadoop QA commented on HBASE-17850:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 25s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 20s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m 8s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
26s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
61m 2s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 205m 44s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
4s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 294m 2s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.master.balancer.TestStochasticLoadBalancer2
|
| | hadoop.hbase.client.TestAsyncBalancerAdminApi |
| Timed out junit tests |
org.apache.hadoop.hbase.master.procedure.TestServerCrashProcedure |
| | org.apache.hadoop.hbase.filter.TestFuzzyRowFilterEndToEnd |
| | org.apache.hadoop.hbase.master.procedure.TestAddColumnFamilyProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestRestoreSnapshotProcedure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12869314/HBASE-17850-v6.patch |
| JIRA Issue | HBASE-17850 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 2506d68e86bd 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 56cbc68 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/6880/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/6880/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6880/testReport/ |
| mo
[jira] [Updated] (HBASE-17850) Backup system repair utility
[
https://issues.apache.org/jira/browse/HBASE-17850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17850:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks for the patch, Vlad.
> Backup system repair utility
>
>
> Key: HBASE-17850
> URL: https://issues.apache.org/jira/browse/HBASE-17850
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-17850-v2.patch, HBASE-17850-v3.patch,
> HBASE-17850-v4.patch, HBASE-17850-v5.patch, HBASE-17850-v6.patch,
> HBASE-17850-v7.patch
>
>
> Backup repair tool restores integrity of a backup system table and removes
> artefacts of a failed backup session from a file system(s)
> This is command-line tool. To run the tool:
> {code}
> hbase backup repair
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17850) Backup system repair utility
[
https://issues.apache.org/jira/browse/HBASE-17850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-17850:
---
Summary: Backup system repair utility (was: Backup system restore /repair
utility)
> Backup system repair utility
>
>
> Key: HBASE-17850
> URL: https://issues.apache.org/jira/browse/HBASE-17850
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-17850-v2.patch, HBASE-17850-v3.patch,
> HBASE-17850-v4.patch, HBASE-17850-v5.patch, HBASE-17850-v6.patch,
> HBASE-17850-v7.patch
>
>
> Backup repair tool restores integrity of a backup system table and removes
> artefacts of a failed backup session from a file system(s)
> This is command-line tool. To run the tool:
> {code}
> hbase backup repair
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18093) Overloading the meaning of 'enabled' in Quota Manager to indicate either quota disabled or quota manager not ready is not good
[ https://issues.apache.org/jira/browse/HBASE-18093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020378#comment-16020378 ] Stephen Yuan Jiang commented on HBASE-18093: Rebase the latest change in master in V3 patch. > Overloading the meaning of 'enabled' in Quota Manager to indicate either > quota disabled or quota manager not ready is not good > -- > > Key: HBASE-18093 > URL: https://issues.apache.org/jira/browse/HBASE-18093 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 1.1.10 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang >Priority: Minor > Attachments: HBASE-18093.v1-master.patch, > HBASE-18093.v2-master.patch, HBASE-18093.v3-master.patch > > > In MasterQuotaManager, a member 'enabled' is used to indicate either quota > feature is disabled or quota manager is not fully initialized. This would > create confusion whether caller should wait for quota manager to be > initialized or change configuration to enable quota. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18093) Overloading the meaning of 'enabled' in Quota Manager to indicate either quota disabled or quota manager not ready is not good
[ https://issues.apache.org/jira/browse/HBASE-18093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-18093: --- Attachment: HBASE-18093.v3-master.patch > Overloading the meaning of 'enabled' in Quota Manager to indicate either > quota disabled or quota manager not ready is not good > -- > > Key: HBASE-18093 > URL: https://issues.apache.org/jira/browse/HBASE-18093 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 1.1.10 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang >Priority: Minor > Attachments: HBASE-18093.v1-master.patch, > HBASE-18093.v2-master.patch, HBASE-18093.v3-master.patch > > > In MasterQuotaManager, a member 'enabled' is used to indicate either quota > feature is disabled or quota manager is not fully initialized. This would > create confusion whether caller should wait for quota manager to be > initialized or change configuration to enable quota. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18041) Add pylintrc file to HBase
[ https://issues.apache.org/jira/browse/HBASE-18041?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020354#comment-16020354 ] Alex Leblang commented on HBASE-18041: -- [~dspivak] what do we need to do to move this forward? Do you want me to go through the checks I proposed not to use one by one? Should we try to get more opinions? > Add pylintrc file to HBase > -- > > Key: HBASE-18041 > URL: https://issues.apache.org/jira/browse/HBASE-18041 > Project: HBase > Issue Type: Improvement > Components: community >Reporter: Alex Leblang >Assignee: Alex Leblang > Attachments: HBASE-18041.branch-1.2.001.patch > > > Yetus runs all commits with python files through a linter. I think that the > HBase community should add a pylintrc file to actively choose the project's > python style instead of just relying on yetus defaults. > As an argument for this, the yetus project itself doesn't even use the > default python linter for its own commits. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17850) Backup system restore /repair utility
[
https://issues.apache.org/jira/browse/HBASE-17850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020338#comment-16020338
]
Hadoop QA commented on HBASE-17850:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 17s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 36s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 27s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 8s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 113m 0s
{color} | {color:green} hbase-server in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 152m 30s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12869324/HBASE-17850-v7.patch |
| JIRA Issue | HBASE-17850 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux ca7b1d0253b5 4.4.0-43-generic #63-Ubuntu SMP Wed Oct 12
13:48:03 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / f1a9990 |
| Default Java | 1.8.0_131 |
| findbugs | v3.0.0 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6882/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6882/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Backup system restore /repair utility
> -
>
> Key: HBASE-17850
> URL: https://issues.apache.org/jira/browse/HBASE-17850
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-17850-v2.patch, HBASE-17850-v3.patch,
> HBASE-17850
[jira] [Assigned] (HBASE-15602) Clean up using directives in cc files.
[ https://issues.apache.org/jira/browse/HBASE-15602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Hunt reassigned HBASE-15602: -- Assignee: Scott Hunt > Clean up using directives in cc files. > -- > > Key: HBASE-15602 > URL: https://issues.apache.org/jira/browse/HBASE-15602 > Project: HBase > Issue Type: Sub-task >Reporter: Elliott Clark >Assignee: Scott Hunt > Labels: beginner, easy, starter > Attachments: HBASE-15602.HBASE-14850.patch > > > There's a ton of files that just barf out all of folly, wangle, and hbase > into the global namespace. We should use the using directive better than that > when possible. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-15602) Clean up using directives in cc files.
[ https://issues.apache.org/jira/browse/HBASE-15602?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Scott Hunt updated HBASE-15602: --- Attachment: HBASE-15602.HBASE-14850.patch This is bigger than I would have liked. My first goal (though not in the description) was to remove usings from header files (because that can really do some bad/unexpected stuff.) Second goal was to remove or simplify most of the usings in source files, especially instances of using namespace. I tried to make sure all changes resulted in lines under 100 characters. I had to make a some guesses at what might be the desired style. Feel free to tell me where I need to make more changes. (i.e. I may have ended up under-using using statements for someone's taste.) Notes: I didn't touch any -test.cc files. I can do these if desired. While compiling, I encountered quite a few gcc warnings (primarily on constructor initializer order.) I have a further patch which attempts to clean all those up. > Clean up using directives in cc files. > -- > > Key: HBASE-15602 > URL: https://issues.apache.org/jira/browse/HBASE-15602 > Project: HBase > Issue Type: Sub-task >Reporter: Elliott Clark > Labels: beginner, easy, starter > Attachments: HBASE-15602.HBASE-14850.patch > > > There's a ton of files that just barf out all of folly, wangle, and hbase > into the global namespace. We should use the using directive better than that > when possible. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17955) Commit Reviewboard comments from Vlad
[ https://issues.apache.org/jira/browse/HBASE-17955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020311#comment-16020311 ] Hudson commented on HBASE-17955: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3059 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3059/]) HBASE-17955 Various reviewboard improvements to space quota work (elserj: rev d671a1dbc698160c4da6c8d9ca368832698c690f) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/FileSystemUtilizationChore.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/SpaceQuotaHelperForTests.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/RegionServerStatusProtos.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestSpaceQuotas.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/SpaceLimitingException.java * (edit) hbase-server/src/main/resources/hbase-webapps/master/table.jsp * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetricsMaster.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaStatusRPCs.java * (edit) hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerQuotaSource.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerRegionSpaceUseReport.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestActivePolicyEnforcement.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/RegionServerSpaceQuotaManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java * (edit) hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterQuotaSource.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/policies/NoWritesViolationPolicyEnforcement.java * (edit) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterQuotaSourceImpl.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/SpaceLimitSettings.java * (delete) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/policies/BulkLoadVerifyingViolationPolicyEnforcement.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/QuotaTableUtil.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaObserverChore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/SpaceViolationPolicyEnforcementFactory.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/SpaceQuotaSnapshot.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestMasterSpaceQuotaObserver.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/QuotaProtos.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestTableSpaceQuotaViolationNotifier.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/policies/NoInsertsViolationPolicyEnforcement.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/MasterQuotaManager.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/policies/TestBulkLoadCheckingViolationPolicyEnforcement.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/SpaceViolationPolicy.java * (edit) hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/QuotaProtos.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/QuotaSettingsFactory.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/TableQuotaSnapshotStore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/NamespaceQuotaSnapshotStore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/policies/AbstractViolationPolicyEnforcement.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestRegionServerSpaceQuotaManager.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/policies/MissingSnapshotViolationPolicyEnforcement.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/MasterSpaceQuotaObserver.java * (edit) hbase-protocol-shaded/src/main/protobuf/RegionServerStatus.proto * (edit) hbase-protocol-shaded/src/main/protobuf/Quota.proto * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterWrapperImpl.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/policies/DisableTableViol
[jira] [Commented] (HBASE-17981) Roll list_quota_violations into list_quota_snapshots
[ https://issues.apache.org/jira/browse/HBASE-17981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020313#comment-16020313 ] Hudson commented on HBASE-17981: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3059 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3059/]) HBASE-17981 Consolidate the space quota shell commands (elserj: rev ed618da906c56cfac7b7bcee862413d3ae851b57) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/RequestConverter.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/QuotaStatusCalls.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * (edit) hbase-protocol-shaded/src/main/protobuf/Admin.proto * (edit) hbase-shell/src/main/ruby/shell.rb * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/QuotaProtos.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaStatusRPCs.java * (edit) hbase-shell/src/main/ruby/hbase/quotas.rb * (edit) hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/QuotaProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/AdminProtos.java * (edit) hbase-protocol-shaded/src/main/protobuf/Quota.proto * (edit) hbase-shell/src/main/ruby/shell/commands/list_quota_table_sizes.rb * (delete) hbase-shell/src/main/ruby/shell/commands/list_quota_violations.rb * (edit) hbase-shell/src/test/ruby/hbase/quotas_test_no_cluster.rb * (edit) hbase-shell/src/main/ruby/hbase_constants.rb * (edit) hbase-shell/src/main/ruby/shell/commands/list_quota_snapshots.rb * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/QuotaTableUtil.java > Roll list_quota_violations into list_quota_snapshots > > > Key: HBASE-17981 > URL: https://issues.apache.org/jira/browse/HBASE-17981 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: HBASE-16961 > > Attachments: HBASE-17981.001.HBASE-16961.patch > > > [~apurtell] had the good suggestion on the parent issue to consolidate the > functionality of these two commands into one, with uniform output and some > filtering options. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17978) Investigate hbase superuser permissions in the face of quota violation
[ https://issues.apache.org/jira/browse/HBASE-17978?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020314#comment-16020314 ] Hudson commented on HBASE-17978: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3059 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3059/]) HBASE-17978 Ensure superusers can circumvent actions restricted by space (elserj: rev b971b449e8de700d7709df2032a5a1429d369394) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/SpaceQuotaHelperForTests.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestSuperUserQuotaPermissions.java > Investigate hbase superuser permissions in the face of quota violation > -- > > Key: HBASE-17978 > URL: https://issues.apache.org/jira/browse/HBASE-17978 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: HBASE-16961 > > Attachments: HBASE-17978.001.HBASE-16961.patch > > > Any hbase superuser should be able to manipulate the system, even in the face > of quota violation. > Add some tests to give us some confidence about what superusers can (and > cannot) do, fixing any glaring permissions issues about superusers facing > permission issues. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18093) Overloading the meaning of 'enabled' in Quota Manager to indicate either quota disabled or quota manager not ready is not good
[
https://issues.apache.org/jira/browse/HBASE-18093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020316#comment-16020316
]
Hadoop QA commented on HBASE-18093:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
29s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 26s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 27s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 27s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 11s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 22s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 35s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 4m 50s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.6.4. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 5s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.6.5. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m 15s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.7.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m 28s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.7.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 9m 42s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.7.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m 54s
{color} | {color:red} The patch causes 16 errors with Hadoop v3.0.0-alpha2.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 29s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 32s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 28s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
8s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 22m 35s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12869337/HBASE-18093.v2-master.patch
|
| JIRA Issue | HBASE-18093 |
| Optional Tests | asflicense javac javadoc
[jira] [Commented] (HBASE-17976) Remove stability annotation from public audience-marked classes
[ https://issues.apache.org/jira/browse/HBASE-17976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020312#comment-16020312 ] Hudson commented on HBASE-17976: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3059 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3059/]) HBASE-17976 Remove stability annotation from public audience class (elserj: rev 13b9a1ed01d8115863e09fd09d4628281d1dbb90) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/SpaceViolationPolicy.java > Remove stability annotation from public audience-marked classes > --- > > Key: HBASE-17976 > URL: https://issues.apache.org/jira/browse/HBASE-17976 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: HBASE-16961 > > Attachments: HBASE-17976.001.HBASE-16961.patch > > > Looks like I grabbed HBASE-17857 in the last rebase, but missed this test > failing. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17447) Automatically delete quota when table is deleted
[ https://issues.apache.org/jira/browse/HBASE-17447?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020310#comment-16020310 ] Hudson commented on HBASE-17447: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3059 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3059/]) HBASE-17447 Implement a MasterObserver for automatically deleting space (elserj: rev 98ace3d586d068db03cba14f3dc904628b917eb9) * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestMasterSpaceQuotaObserver.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/MasterSpaceQuotaObserver.java * (edit) src/main/asciidoc/_chapters/ops_mgt.adoc > Automatically delete quota when table is deleted > > > Key: HBASE-17447 > URL: https://issues.apache.org/jira/browse/HBASE-17447 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: HBASE-16961 > > Attachments: HBASE-17447.001.HBASE-16961.patch, > HBASE-17447.002.HBASE-16961.patch > > > If a table has a space quota defined on it, we can delete that quota when the > table is deleted. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17977) Enabled Master observer to delete quotas on table deletion by default
[ https://issues.apache.org/jira/browse/HBASE-17977?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020315#comment-16020315 ] Hudson commented on HBASE-17977: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3059 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3059/]) HBASE-17977 Enable the MasterSpaceQuotaObserver by default (elserj: rev f1a9990328a65c49cdd10af069f67ff5ba5147f9) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/MasterSpaceQuotaObserver.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestMasterSpaceQuotaObserverWithMocks.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestMasterSpaceQuotaObserver.java * (edit) src/main/asciidoc/_chapters/ops_mgt.adoc > Enabled Master observer to delete quotas on table deletion by default > - > > Key: HBASE-17977 > URL: https://issues.apache.org/jira/browse/HBASE-17977 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: HBASE-16961 > > Attachments: HBASE-17977.001.HBASE-16961.patch, > HBASE-17977.002.HBASE-16961.patch > > > Deleting a table's quota when we delete the table should be the common case, > not the exception. > Load the observer by default (when quota support is enabled), and switch the > flag to disable loading it. Make sure to update the docs too. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17557) HRegionServer#reportRegionSizesForQuotas() should respond to UnsupportedOperationException
[
https://issues.apache.org/jira/browse/HBASE-17557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020294#comment-16020294
]
Hudson commented on HBASE-17557:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3059 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/3059/])
HBASE-17557 HRegionServer#reportRegionSizesForQuotas() should respond to
(elserj: rev 7fb0ac26e3a85a01b285078c198578216e3e8524)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/FileSystemUtilizationChore.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
> HRegionServer#reportRegionSizesForQuotas() should respond to
> UnsupportedOperationException
> --
>
> Key: HBASE-17557
> URL: https://issues.apache.org/jira/browse/HBASE-17557
> Project: HBase
> Issue Type: Sub-task
>Reporter: Ted Yu
>Assignee: Ted Yu
> Labels: quota
> Fix For: HBASE-16961
>
> Attachments: 17557.HBASE-16961.v2.txt, 17557.HBASE-16961.v3.txt,
> 17557.HBASE-16961.v4.txt
>
>
> When master doesn't support quota, you would see the following repeatedly in
> region server log:
> {code}
> 2017-01-27 17:24:27,389 DEBUG
> [cn011.x.com,16020,1485468203653_ChoreService_1] regionserver.HRegionServer:
> Failed to report region sizes to Master. This will be retried.
> org.apache.hadoop.hbase.DoNotRetryIOException: /23:16000 is unable to read
> call parameter from client 21; java.lang.UnsupportedOperationException:
> ReportRegionSpaceUse
> at sun.reflect.GeneratedConstructorAccessor8.newInstance(Unknown
> Source)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
> at
> org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
> at
> org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:334)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportRegionSizesForQuotas(HRegionServer.java:1211)
> at
> org.apache.hadoop.hbase.quotas.FileSystemUtilizationChore.reportRegionSizesToMaster(FileSystemUtilizationChore.java:170)
> at
> org.apache.hadoop.hbase.quotas.FileSystemUtilizationChore.chore(FileSystemUtilizationChore.java:129)
> at org.apache.hadoop.hbase.ScheduledChore.run(ScheduledChore.java:185)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> Caused by:
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(org.apache.hadoop.hbase.DoNotRetryIOException):
> /23:16000 is unable to read call parameter from client 21; java.lang.
> UnsupportedOperationException: ReportRegionSpaceUse
> at
> org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1225)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:213)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:287)
> at
> org.apache.hadoop.hbase.protobuf.generated.RegionServerStatusProtos$RegionServerStatusService$BlockingStub.reportRegionSpaceUse(RegionServerStatusProtos.java:10919)
> at
> org.apache.hadoop.hbase.regionserver.HRegionServer.reportRegionSizesForQuotas(HRegionServer.java:1209)
> {code}
> HRegionServer.reportRegionSizesForQuotas() should respond to
> UnsupportedOperationException and stop retrying.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-17959) Canary timeout should be configurable on a per-table basis
[ https://issues.apache.org/jira/browse/HBASE-17959?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chinmay Kulkarni updated HBASE-17959: - Attachment: HBASE-17959.003.patch [~apurtell] I thought this would not be necessary since we have only one write-table, but I agree it would make logging more convenient and easier to parse. I have split the read and write latency counters to reflect this change. Please take a look at this amended patch. Thanks! > Canary timeout should be configurable on a per-table basis > -- > > Key: HBASE-17959 > URL: https://issues.apache.org/jira/browse/HBASE-17959 > Project: HBase > Issue Type: Improvement > Components: canary >Reporter: Andrew Purtell >Assignee: Chinmay Kulkarni >Priority: Minor > Attachments: HBASE-17959.002.patch, HBASE-17959.003.patch, > HBASE-17959.patch > > > The Canary read and write timeouts should be configurable on a per-table > basis, for cases where different tables have different latency SLAs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16998) [Master] Analyze table use reports and update quota violations
[ https://issues.apache.org/jira/browse/HBASE-16998?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020295#comment-16020295 ] Hudson commented on HBASE-16998: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3059 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3059/]) HBASE-16998 Implement Master-side analysis of region space reports (elserj: rev 533470f8c879174d023140a80d2b6d548685733e) * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaObserverChoreWithMiniCluster.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/SpaceQuotaViolationNotifier.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaTableUtil.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaObserverChore.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaObserverChore.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestTableQuotaViolationStore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/SpaceQuotaViolationNotifierForTest.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestNamespaceQuotaViolationStore.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/NamespaceQuotaViolationStore.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/TableQuotaViolationStore.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestTablesWithQuotas.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/MasterQuotaManager.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaViolationStore.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/QuotaRetriever.java > [Master] Analyze table use reports and update quota violations > -- > > Key: HBASE-16998 > URL: https://issues.apache.org/jira/browse/HBASE-16998 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: HBASE-16961 > > Attachments: HBASE-16998.001.patch, > HBASE-16998.002.HBASE-16961.patch, HBASE-16998.003.HBASE-16961.patch, > HBASE-16998.004.HBASE-16961.patch, HBASE-16998.005.HBASE-16961.patch > > > Given the collected table usage reports from RegionServers, the Master needs > to inspect all filesystem-use quotas and determine which need to move into > violation and which need to move out of violation. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17002) [Master] Report quotas and active violations on Master UI and JMX metrics
[ https://issues.apache.org/jira/browse/HBASE-17002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020307#comment-16020307 ] Hudson commented on HBASE-17002: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3059 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3059/]) HBASE-17002 JMX metrics and some UI additions for space quotas (elserj: rev 13af7f8ac6e393dc7895b6fb1d35fac2367c5a17) * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/QuotaProtos.java * (add) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterQuotaSourceImpl.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * (add) hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterQuotaSourceFactoryImpl.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ShortCircuitMasterConnection.java * (add) hbase-hadoop2-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.master.MetricsMasterQuotaSource * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterWrapperImpl.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestMasterMetricsWrapper.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaStatusRPCs.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/MasterProtos.java * (edit) hbase-protocol-shaded/src/main/protobuf/Quota.proto * (edit) hbase-server/src/main/resources/hbase-webapps/master/table.jsp * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java * (edit) hbase-protocol-shaded/src/main/protobuf/Master.proto * (add) hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterQuotaSource.java * (add) hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterQuotaSourceFactory.java * (add) hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/regionserver/MetricsRegionServerQuotaSource.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/QuotaTableUtil.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/QuotaStatusCalls.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MetricsMaster.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/RequestConverter.java * (add) hbase-hadoop2-compat/src/main/resources/META-INF/services/org.apache.hadoop.hbase.master.MetricsMasterQuotaSourceFactory * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaObserverChore.java * (edit) hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/master/MetricsMasterWrapper.java > [Master] Report quotas and active violations on Master UI and JMX metrics > - > > Key: HBASE-17002 > URL: https://issues.apache.org/jira/browse/HBASE-17002 > Project: HBase > Issue Type: Sub-task > Components: master, UI >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: HBASE-16961 > > Attachments: HBASE-17002.001.HBASE-16961.patch, > HBASE-17002.002.HBASE-16961.patch, HBASE-17002.003.HBASE-16961.patch, > TableQuotaDefined2.png, TableQuotaDefined.png > > > We should be able to view the Master UI and determine what quotas exist and > the quotas that are in violation to easily confirm/deny the state of > tables/namespaces. > The JMX metrics from the Master should also include the list of > tables/namespaces which have quotas whose violation policies are presently > being enforced. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17478) Avoid sending FSUtilization reports to master when quota support is not enabled
[ https://issues.apache.org/jira/browse/HBASE-17478?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020301#comment-16020301 ] Hudson commented on HBASE-17478: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3059 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3059/]) HBASE-17478 Avoid reporting FS use when quotas are disabled (elserj: rev 4ad49bc3ac86708144d126ea0a4cbfa68f226834) * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestMasterQuotaManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/MasterQuotaManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java > Avoid sending FSUtilization reports to master when quota support is not > enabled > --- > > Key: HBASE-17478 > URL: https://issues.apache.org/jira/browse/HBASE-17478 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Trivial > Fix For: HBASE-16961 > > Attachments: HBASE-17478.001.HBASE-16961.patch, > HBASE-17478.002.HBASE-16961.patch, HBASE-17478.003.HBASE-16961.patch > > > Trivial little change to make sure that the RS's do not send the filesystem > utilization reports to the master when hbase.quota.enabled=false and, > similarly, that the master gracefully handles these reports when the feature > is not enabled. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17003) Update book for filesystem use quotas
[ https://issues.apache.org/jira/browse/HBASE-17003?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020308#comment-16020308 ] Hudson commented on HBASE-17003: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3059 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3059/]) HBASE-17003 Documentation updates for space quotas (elserj: rev f7da41d0bf5812c074102beadda160b46b463158) * (edit) src/main/asciidoc/_chapters/ops_mgt.adoc > Update book for filesystem use quotas > - > > Key: HBASE-17003 > URL: https://issues.apache.org/jira/browse/HBASE-17003 > Project: HBase > Issue Type: Sub-task > Components: documentation >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: HBASE-16961 > > Attachments: book.html, HBASE-17003.001.HBASE-16961.patch, > HBASE-17003.002.HBASE-16961.patch > > > Need to make sure that the book's section on quotas reflect the final > implementation of the filesystem-use quota support. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17001) [RegionServer] Implement enforcement of quota violation policies
[ https://issues.apache.org/jira/browse/HBASE-17001?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020298#comment-16020298 ] Hudson commented on HBASE-17001: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3059 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3059/]) HBASE-17001 Enforce quota violation policies in the RegionServer (elserj: rev 34ba143fc8f484134e229eb3cfff0bd3a1e311d1) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/SpaceQuotaSnapshotNotifier.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/ActivePolicyEnforcement.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/QuotaExceededException.java * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/SpaceQuotaSnapshot.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestNamespaceQuotaViolationStore.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/SpaceLimitingException.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/SpaceViolationPolicyEnforcementFactory.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestActivePolicyEnforcement.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/QuotaTableUtil.java * (delete) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaViolationStore.java * (delete) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/NamespaceQuotaViolationStore.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/SpaceQuotaRefresherChore.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/policies/TestBulkLoadCheckingViolationPolicyEnforcement.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/policies/TestNoWritesViolationPolicyEnforcement.java * (delete) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/SpaceQuotaViolationNotifierFactory.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestTableSpaceQuotaViolationNotifier.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/SpaceQuotaSnapshotNotifierForTest.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestSpaceQuotas.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/policies/TestDisableTableViolationPolicyEnforcement.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/policies/DisableTableViolationPolicyEnforcement.java * (edit) hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/QuotaProtos.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/SpaceQuotaHelperForTests.java * (delete) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/SpaceQuotaViolationPolicyRefresherChore.java * (delete) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/TableQuotaViolationStore.java * (edit) hbase-protocol-shaded/src/main/protobuf/Quota.proto * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/policies/BaseViolationPolicyEnforcement.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/SpaceQuotaSnapshotNotifierFactory.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/SpaceViolationPolicyEnforcement.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaObserverChore.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaObserverChoreWithMiniCluster.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/TableQuotaSnapshotStore.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestSpaceQuotaViolationPolicyRefresherChore.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/NamespaceQuotaSnapshotStore.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/policies/NoWritesViolationPolicyEnforcement.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/policies/TestNoWritesCompactionsViolationPolicyEnforcement.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaObserverChore.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaTableUtil.java * (delete) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/SpaceQuotaViolationNotifierForTest.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/QuotaProtos.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/policies/TestNoInsertsViolationPolicyEnforcement.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quot
[jira] [Commented] (HBASE-17516) Table quota not taking precedence over namespace quota
[ https://issues.apache.org/jira/browse/HBASE-17516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020304#comment-16020304 ] Hudson commented on HBASE-17516: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3059 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3059/]) HBASE-17516 Correctly handle case where table and NS quotas both apply (elserj: rev f031b69969003b7b33111312060a73858bada319) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaObserverChoreWithMiniCluster.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestSpaceQuotas.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaObserverChore.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaStatusRPCs.java > Table quota not taking precedence over namespace quota > -- > > Key: HBASE-17516 > URL: https://issues.apache.org/jira/browse/HBASE-17516 > Project: HBase > Issue Type: Sub-task >Reporter: Romil Choksi >Assignee: Josh Elser > Fix For: HBASE-16961 > > Attachments: HBASE-17516.001.patch, HBASE-17516.002.HBASE-16961.patch > > > [~romil.choksi] found a bug in the current patch-set where a more restrictive > table quota did not take priority over a less-restrictive namespace quota. > Turns out some of the logic to handle this case was incorrect. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17602) Tweak chore delay/period defaults
[ https://issues.apache.org/jira/browse/HBASE-17602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020305#comment-16020305 ] Hudson commented on HBASE-17602: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3059 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3059/]) HBASE-17602 Reduce some quota chore periods/delays (elserj: rev 8159eae7810a6bf6e13596fed9f8cb7f1aff9dfc) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaObserverChore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/SpaceQuotaRefresherChore.java > Tweak chore delay/period defaults > - > > Key: HBASE-17602 > URL: https://issues.apache.org/jira/browse/HBASE-17602 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Trivial > Fix For: HBASE-16961 > > Attachments: HBASE-17602.001.HBASE-16961.patch > > > In testing, found that the default chore periods/delays were a little too > lax. This resulted in quotas not being updated as soon as they really should > have been. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17568) Expire region reports in the HMaster
[ https://issues.apache.org/jira/browse/HBASE-17568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020306#comment-16020306 ] Hudson commented on HBASE-17568: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3059 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3059/]) HBASE-17568 Better handle stale/missing region size reports (elserj: rev 91b4d2e827e61a4bedae232aae3f8f5a10015ae3) * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaObserverChoreRegionReports.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/MasterQuotaManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestMasterQuotaManager.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaObserverChore.java > Expire region reports in the HMaster > > > Key: HBASE-17568 > URL: https://issues.apache.org/jira/browse/HBASE-17568 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: HBASE-16961 > > Attachments: HBASE-17568.001.patch, > HBASE-17568.002.HBASE-16961.patch, HBASE-17568.003.HBASE-16961.patch > > > (From a TODO) > The RegionServers send reports of sizes from to the Master so the Master can > compute the "size" of each Table. > Presently, the Master has no way to expire these reports. Thus, reports for > tables that are deleted would be retained in memory. Retain the time at which > the report was received by the master, and then use that to determine when to > age it off. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17428) Expand on shell commands for detailed insight
[
https://issues.apache.org/jira/browse/HBASE-17428?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020302#comment-16020302
]
Hudson commented on HBASE-17428:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3059 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/3059/])
HBASE-17428 Implement informational RPCs for space quotas (elserj: rev
80a1f8fa2aec6687774ffeb03178efcdda2e136b)
* (edit) hbase-protocol-shaded/src/main/protobuf/Admin.proto
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/RequestConverter.java
* (edit) hbase-protocol-shaded/src/main/protobuf/Master.proto
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ConnectionImplementation.java
* (edit) hbase-shell/src/main/ruby/shell.rb
* (add) hbase-shell/src/main/ruby/shell/commands/list_quota_table_sizes.rb
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java
* (edit)
hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/QuotaProtos.java
* (edit) hbase-shell/src/main/ruby/hbase/quotas.rb
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ShortCircuitMasterConnection.java
* (add)
hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaStatusRPCs.java
* (add) hbase-shell/src/test/ruby/hbase/quotas_test_no_cluster.rb
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java
* (edit)
hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/AdminProtos.java
* (edit)
hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/QuotaProtos.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/QuotaTableUtil.java
* (edit)
hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/RegionServerStatusProtos.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/ActivePolicyEnforcement.java
* (add) hbase-shell/src/main/ruby/shell/commands/list_quota_snapshots.rb
* (edit)
hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/MasterProtos.java
* (edit) hbase-protocol-shaded/src/main/protobuf/Quota.proto
* (add) hbase-shell/src/main/ruby/shell/commands/list_quota_violations.rb
* (edit) hbase-shell/src/test/ruby/hbase/quotas_test.rb
* (add)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/QuotaStatusCalls.java
> Expand on shell commands for detailed insight
> -
>
> Key: HBASE-17428
> URL: https://issues.apache.org/jira/browse/HBASE-17428
> Project: HBase
> Issue Type: Sub-task
> Components: shell
>Reporter: Josh Elser
>Assignee: Josh Elser
> Fix For: HBASE-16961
>
> Attachments: HBASE-17428.001.patch, HBASE-17428.002.patch,
> HBASE-17428.003.HBASE-16961.patch, HBASE-17428.004.HBASE-16961.patch
>
>
> Was talking over how to build some tests with [~romil.choksi], and what kind
> of information we capture and expose via some more shell commands.
> * Current SpaceQuotaSnapshots that the Master sees
> * Current SpaceQuotaSnapshots that a RS sees (or all RSs)
> * Current SpaceViolationPolicyEnforcements that a RS has enabled
> We can build out on the {{status}} command, with some options for different
> levels of detail, perhaps. Shouldn't be too tricky -- we have the info, just
> need to wire up java API, RPC, and shell command.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-17025) [Shell] Support space quota get/set via the shell
[ https://issues.apache.org/jira/browse/HBASE-17025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020300#comment-16020300 ] Hudson commented on HBASE-17025: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3059 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3059/]) HBASE-17025 Add shell commands for space quotas (elserj: rev f1066cd7744e185044dcc33aae09623d10ed2631) * (edit) hbase-shell/src/main/ruby/hbase_constants.rb * (add) hbase-shell/src/test/ruby/hbase/quotas_test.rb * (edit) hbase-shell/src/main/ruby/hbase/quotas.rb * (edit) hbase-shell/src/test/ruby/tests_runner.rb * (edit) hbase-shell/src/test/java/org/apache/hadoop/hbase/client/AbstractTestShell.java * (edit) hbase-shell/src/main/ruby/shell/commands/set_quota.rb > [Shell] Support space quota get/set via the shell > - > > Key: HBASE-17025 > URL: https://issues.apache.org/jira/browse/HBASE-17025 > Project: HBase > Issue Type: Sub-task > Components: shell >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: HBASE-16961 > > Attachments: HBASE-17025.001.patch, HBASE-17025.002.patch, > HBASE-17025.003.HBASE-16961.patch, HBASE-17025.004.HBASE-16961.patch > > > Need to make sure that admins can use the shell to get/set the new space > quotas. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17000) [RegionServer] Compute region filesystem space use and report to Master
[ https://issues.apache.org/jira/browse/HBASE-17000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020293#comment-16020293 ] Hudson commented on HBASE-17000: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3059 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3059/]) HBASE-17000 Implement computation of online region sizes and report to (elserj: rev 6b334cd817e37a3727160cc754187e8f40893ced) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/MasterQuotaManager.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/RegionServerStatusProtos.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/FileSystemUtilizationChore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestRegionSizeUse.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestFileSystemUtilizationChore.java * (edit) hbase-protocol-shaded/src/main/protobuf/RegionServerStatus.proto * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionServerRegionSpaceUseReport.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java > [RegionServer] Compute region filesystem space use and report to Master > --- > > Key: HBASE-17000 > URL: https://issues.apache.org/jira/browse/HBASE-17000 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: HBASE-16961 > > Attachments: HBASE-17000.001.patch, > HBASE-17000.002.HBASE-16961.patch, HBASE-17000.004.HBASE-16961.patch, > HBASE-17000.005.HBASE-16961.patch > > > Each RegionServer needs to track how much space a Region takes up and roll > this up to the table level. > Aggregation of this information in the Master will be covered by HBASE-16997. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16999) [Master] Inform RegionServers on size quota violations
[ https://issues.apache.org/jira/browse/HBASE-16999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020297#comment-16020297 ] Hudson commented on HBASE-16999: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3059 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3059/]) HBASE-16999 Implement master and regionserver synchronization of quota (elserj: rev 98b4181f43a22c678ef66b6f568f6f19209720b5) * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/RegionServerRpcQuotaManager.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestTableSpaceQuotaViolationNotifier.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/TableSpaceQuotaViolationNotifier.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaObserverChoreWithMiniCluster.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestRegionServerSpaceQuotaManager.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/QuotaTableUtil.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/SpaceQuotaViolationPolicyRefresherChore.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaObserverChore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/RegionServerSpaceQuotaManager.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestSpaceQuotaViolationPolicyRefresherChore.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/SpaceQuotaViolationNotifier.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/MockRegionServerServices.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaThrottle.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/SpaceQuotaViolationNotifierFactory.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/SpaceQuotaViolationNotifierForTest.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaTableUtil.java * (delete) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/RegionServerQuotaManager.java > [Master] Inform RegionServers on size quota violations > -- > > Key: HBASE-16999 > URL: https://issues.apache.org/jira/browse/HBASE-16999 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: HBASE-16961 > > Attachments: HBASE-16999.002.HBASE-16961.patch, > HBASE-16999.003.HBASE-16961.patch, HBASE-16999-HBASE-16961.001.patch > > > The Master, after computing the total usage across all regionservers, needs > to inform the RegionServers about tables that need to change violation policy > enforcement (enable or disable enforcement). > RPC, ZK, or ProcV2's notification bus are some examples of ways this could be > implemented. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16996) Implement storage/retrieval of filesystem-use quotas into quota table
[
https://issues.apache.org/jira/browse/HBASE-16996?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020292#comment-16020292
]
Hudson commented on HBASE-16996:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3059 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/3059/])
HBASE-16996 Implement storage/retrieval of filesystem-use quotas into (elserj:
rev f74e051bce94200ae99c7d97c913c1bef658f53c)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/QuotaTableUtil.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaAdmin.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/MasterQuotaManager.java
> Implement storage/retrieval of filesystem-use quotas into quota table
> -
>
> Key: HBASE-16996
> URL: https://issues.apache.org/jira/browse/HBASE-16996
> Project: HBase
> Issue Type: Sub-task
>Reporter: Josh Elser
>Assignee: Josh Elser
> Fix For: HBASE-16961
>
> Attachments: HBASE-16996.001.patch, HBASE-16996-HBASE-16961.002.patch
>
>
> Provide read/write API for accessing the new filesystem-usage quotas in the
> existing {{hbase:quota}} table.
> Make sure that both the client can read quotas the quotas in the table as
> well as the Master can perform the necessary update/delete actions per the
> quota RPCs.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Comment Edited] (HBASE-17959) Canary timeout should be configurable on a per-table basis
[ https://issues.apache.org/jira/browse/HBASE-17959?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020296#comment-16020296 ] Chinmay Kulkarni edited comment on HBASE-17959 at 5/22/17 10:19 PM: [~apurtell] I thought this would not be necessary since we have only one write-table, but I agree it would make logging more convenient and easier to parse. I have split the read and write latency counters to reflect this change. Please take a look at the amended patch. Thanks! was (Author: ckulkarni): [~apurtell] I thought this would not be necessary since we have only one write-table, but I agree it would make logging more convenient and easier to parse. I have split the read and write latency counters to reflect this change. Please take a look at this amended patch. Thanks! > Canary timeout should be configurable on a per-table basis > -- > > Key: HBASE-17959 > URL: https://issues.apache.org/jira/browse/HBASE-17959 > Project: HBase > Issue Type: Improvement > Components: canary >Reporter: Andrew Purtell >Assignee: Chinmay Kulkarni >Priority: Minor > Attachments: HBASE-17959.002.patch, HBASE-17959.003.patch, > HBASE-17959.patch > > > The Canary read and write timeouts should be configurable on a per-table > basis, for cases where different tables have different latency SLAs. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17794) Remove lingering "violation" in favor of the accurate "snapshot"
[ https://issues.apache.org/jira/browse/HBASE-17794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020309#comment-16020309 ] Hudson commented on HBASE-17794: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3059 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3059/]) HBASE-17794 Swap "violation" for "snapshot" where appropriate (elserj: rev a8460b8bad746fc4815845bcaa9740a51bdd3c25) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/QuotaTableUtil.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/TableQuotaSnapshotStore.java * (edit) hbase-protocol-shaded/src/main/protobuf/Quota.proto * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/QuotaObserverChore.java > Remove lingering "violation" in favor of the accurate "snapshot" > > > Key: HBASE-17794 > URL: https://issues.apache.org/jira/browse/HBASE-17794 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Trivial > Fix For: HBASE-16961 > > Attachments: HBASE-17794.001.HBASE-16961.patch > > > Previously, quota violations used to be stored in the quota table. > Eventually, I realized that I needed to actually persist the utilization of > each table/NS, not just the violation state. > I know there are cases in the code where "violation" is incorrectly > describing things (it should be "snapshot"). Clean these up for clarity. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-17259) Missing functionality to remove space quota
[ https://issues.apache.org/jira/browse/HBASE-17259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020299#comment-16020299 ] Hudson commented on HBASE-17259: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3059 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3059/]) HBASE-17259 API to remove space quotas on a table/namespace (elserj: rev 6c9082fe162c2e940b556c9da5edc36e5501906e) * (edit) hbase-protocol-shaded/src/main/protobuf/Quota.proto * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaAdmin.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/QuotaSettingsFactory.java * (edit) hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/QuotaProtos.java * (edit) hbase-protocol/src/main/protobuf/Quota.proto * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/MasterQuotaManager.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/SpaceLimitSettings.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/QuotaTableUtil.java * (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaSettingsFactory.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/QuotaProtos.java > Missing functionality to remove space quota > --- > > Key: HBASE-17259 > URL: https://issues.apache.org/jira/browse/HBASE-17259 > Project: HBase > Issue Type: Sub-task >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: HBASE-16961 > > Attachments: HBASE-17259.001.patch, HBASE-17259.002.patch, > HBASE-17259.003.HBASE-16961.patch > > > I'm noticing that while I have create and update APIs for quotas, I missed > the remove functionality. > Need to add public API for that and some tests. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-16995) Build client Java API and client protobuf messages
[ https://issues.apache.org/jira/browse/HBASE-16995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020291#comment-16020291 ] Hudson commented on HBASE-16995: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3059 (See [https://builds.apache.org/job/HBase-Trunk_matrix/3059/]) HBASE-16995 Build client Java API and client protobuf messages (Josh (elserj: rev 4dfafd6e508e2e32c6a44e1957065abe2dd65fea) * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/SpaceLimitSettings.java * (add) hbase-client/src/test/java/org/apache/hadoop/hbase/quotas/TestSpaceLimitSettings.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/MasterProtos.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/QuotaSettingsFactory.java * (edit) hbase-protocol-shaded/src/main/protobuf/Quota.proto * (edit) hbase-protocol-shaded/src/main/protobuf/Master.proto * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/QuotaType.java * (edit) hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/QuotaProtos.java * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/QuotaProtos.java * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/SpaceViolationPolicy.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java * (add) hbase-client/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaSettingsFactory.java * (edit) hbase-protocol/src/main/protobuf/Quota.proto HBASE-16995 Build client Java API and client protobuf messages - (elserj: rev 140413c11b9b856f0dc688bf61bbaeae722a87c2) * (edit) hbase-protocol-shaded/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/generated/QuotaProtos.java * (edit) hbase-client/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaSettingsFactory.java * (edit) hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/QuotaProtos.java HBASE-16995 Build client Java API and client protobuf messages - (elserj: rev c5172169f23e20fb4df55f803dad115a47164be9) * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/shaded/protobuf/ProtobufUtil.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/QuotaSettingsFactory.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/quotas/SpaceLimitSettings.java > Build client Java API and client protobuf messages > -- > > Key: HBASE-16995 > URL: https://issues.apache.org/jira/browse/HBASE-16995 > Project: HBase > Issue Type: Sub-task > Components: Client >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: HBASE-16961 > > Attachments: HBASE-16995.001.patch, HBASE-16995.002.patch, > HBASE-16995.003.patch, HBASE-16995.004.patch, HBASE-16995.addendum.patch > > > Extend the existing Java API and protobuf messages to allow the client to set > filesystem-use quotas via the Master. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18084) Improve CleanerChore to clean from directory which consumes more disk space
[
https://issues.apache.org/jira/browse/HBASE-18084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020288#comment-16020288
]
huaxiang sun commented on HBASE-18084:
--
{code}
return (f1ConsumedSpace > f2ConsumedSpace) ? -1
: (f1ConsumedSpace < f2ConsumedSpace ? 1 : 0);
{code}
Replace it with Long.compare(long x, long y)?
> Improve CleanerChore to clean from directory which consumes more disk space
> ---
>
> Key: HBASE-18084
> URL: https://issues.apache.org/jira/browse/HBASE-18084
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
> Attachments: HBASE-18084.patch, HBASE-18084.v2.patch
>
>
> Currently CleanerChore cleans the directory in dictionary order, rather than
> from the directory with largest space usage. And when data abnormally
> accumulated to some huge volume in archive directory, the cleaning speed
> might not be enough.
> This proposal is another improvement working together with HBASE-18083 to
> resolve our online issue (archive dir consumed more than 1.8PB SSD space)
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18090) Improve TableSnapshotInputFormat to allow more multiple mappers per region
[
https://issues.apache.org/jira/browse/HBASE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020265#comment-16020265
]
Ted Yu commented on HBASE-18090:
For the new TableMapReduceUtil#initTableSnapshotMapJob method (in mapred
package), please add numSplitsPerRegion to @param
{code}
+} else if
(RegionSplitter.HexStringSplit.class.getSimpleName().equals(conf.get(SPLIT_ALGO)))
{
+ splitAlgo = new RegionSplitter.HexStringSplit();
+}
{code}
Add an else block for handling the case where split algorithm is not specified.
{code}
+if (splitAlgo == null && numSplitsPerRegion > 1) {
+ throw new IllegalArgumentException("Split algo can't be null, numSplits
must be >= 1!");
{code}
The condition seems to imply that numSplits can be 1 if splitAlgo is null.
Please modify the error message to be more precise.
> Improve TableSnapshotInputFormat to allow more multiple mappers per region
> --
>
> Key: HBASE-18090
> URL: https://issues.apache.org/jira/browse/HBASE-18090
> Project: HBase
> Issue Type: Improvement
> Components: mapreduce
>Affects Versions: 1.4.0
>Reporter: Mikhail Antonov
> Attachments: HBASE-18090-branch-1.3-v1.patch
>
>
> TableSnapshotInputFormat runs one map task per region in the table snapshot.
> This places unnecessary restriction that the region layout of the original
> table needs to take the processing resources available to MR job into
> consideration. Allowing to run multiple mappers per region (assuming
> reasonably even key distribution) would be useful.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18060) Backport to branch-1 HBASE-9774 HBase native metrics and metric collection for coprocessors
[ https://issues.apache.org/jira/browse/HBASE-18060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020264#comment-16020264 ] Andrew Purtell commented on HBASE-18060: [~vincentpoon] The TestInterfaceAudienceAnnotations failure is probably related to this change. Fix that and this is good from my perspective. > Backport to branch-1 HBASE-9774 HBase native metrics and metric collection > for coprocessors > --- > > Key: HBASE-18060 > URL: https://issues.apache.org/jira/browse/HBASE-18060 > Project: HBase > Issue Type: New Feature >Affects Versions: 1.4.0, 1.3.2, 1.5.0 >Reporter: Vincent Poon >Assignee: Vincent Poon > Attachments: HBASE-18060.branch-1.3.v1.patch, > HBASE-18060.branch-1.3.v2.patch, HBASE-18060.branch-1.3.v3.patch, > HBASE-18060.branch-1.3.v4.patch, HBASE-18060.branch-1.v1.patch, > HBASE-18060.branch-1.v2.patch, HBASE-18060.branch-1.v3.patch, > HBASE-18060.branch-1.v4.patch > > > I'd like to explore backporting HBASE-9774 to branch-1, as the ability for > coprocessors to report custom metrics through HBase is useful for us, and if > we have coprocessors use the native API, a re-write won't be necessary after > an upgrade to 2.0. > The main issues I see so far are: > - the usage of Java 8 language features. Seems we can work around this as > most of it is syntactic sugar. Will need to find a backport for LongAdder > - dropwizard 3.1.2 in Master. branch-1 is still on yammer metrics 2.2. Not > sure if these can coexist just for this feature -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Work started] (HBASE-18091) Add API for who currently holds a lock on namespace/ table/ region and log when state is LOCK_EVENT_WAIT
[ https://issues.apache.org/jira/browse/HBASE-18091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-18091 started by Umesh Agashe. > Add API for who currently holds a lock on namespace/ table/ region and log > when state is LOCK_EVENT_WAIT > > > Key: HBASE-18091 > URL: https://issues.apache.org/jira/browse/HBASE-18091 > Project: HBase > Issue Type: Sub-task > Components: proc-v2 >Reporter: Umesh Agashe >Assignee: Umesh Agashe > Fix For: 2.0.0 > > > Add API for who currently holds a lock on namespace/ table/ region and log > message when state is LOCK_EVENT_WAIT -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18027) Replication should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-18027: --- Attachment: HBASE-18027.patch Updated patch fixes findbugs warning. > Replication should respect RPC size limits when batching edits > -- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch, > HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry in the replication context is then placed in order by hash of > encoded region name into one of these N lists. Each of the N lists is then > sent all at once in one replication RPC. We do not test if the sum of data in > each N list will exceed RPC size limits. This code presumes each individual > edit is reasonably small. Not checking for aggregate size while assembling > the lists into RPCs is an oversight and can lead to replication failure when > that assumption is violated. > We can fix this by generating as many replication RPC calls as we need to > drain a list, keeping each RPC under limit, instead of assuming the whole > list will fit in one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18093) Overloading the meaning of 'enabled' in Quota Manager to indicate either quota disabled or quota manager not ready is not good
[ https://issues.apache.org/jira/browse/HBASE-18093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020259#comment-16020259 ] Stephen Yuan Jiang commented on HBASE-18093: The failure from V1 patch does not make sense. Things is good locally. Attach V2 patch addressing the typo issue found by [~te...@apache.org] > Overloading the meaning of 'enabled' in Quota Manager to indicate either > quota disabled or quota manager not ready is not good > -- > > Key: HBASE-18093 > URL: https://issues.apache.org/jira/browse/HBASE-18093 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 1.1.10 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang >Priority: Minor > Attachments: HBASE-18093.v1-master.patch, HBASE-18093.v2-master.patch > > > In MasterQuotaManager, a member 'enabled' is used to indicate either quota > feature is disabled or quota manager is not fully initialized. This would > create confusion whether caller should wait for quota manager to be > initialized or change configuration to enable quota. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18093) Overloading the meaning of 'enabled' in Quota Manager to indicate either quota disabled or quota manager not ready is not good
[ https://issues.apache.org/jira/browse/HBASE-18093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-18093: --- Attachment: HBASE-18093.v2-master.patch > Overloading the meaning of 'enabled' in Quota Manager to indicate either > quota disabled or quota manager not ready is not good > -- > > Key: HBASE-18093 > URL: https://issues.apache.org/jira/browse/HBASE-18093 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 1.1.10 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang >Priority: Minor > Attachments: HBASE-18093.v1-master.patch, HBASE-18093.v2-master.patch > > > In MasterQuotaManager, a member 'enabled' is used to indicate either quota > feature is disabled or quota manager is not fully initialized. This would > create confusion whether caller should wait for quota manager to be > initialized or change configuration to enable quota. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18027) Replication should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020255#comment-16020255 ] Andrew Purtell commented on HBASE-18027: Eh, spoke too soon about the findbugs warning actually. Next time I will look at the patch first rather than assume I can remember what I did last week. > Replication should respect RPC size limits when batching edits > -- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry in the replication context is then placed in order by hash of > encoded region name into one of these N lists. Each of the N lists is then > sent all at once in one replication RPC. We do not test if the sum of data in > each N list will exceed RPC size limits. This code presumes each individual > edit is reasonably small. Not checking for aggregate size while assembling > the lists into RPCs is an oversight and can lead to replication failure when > that assumption is violated. > We can fix this by generating as many replication RPC calls as we need to > drain a list, keeping each RPC under limit, instead of assuming the whole > list will fit in one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18027) Replication should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020254#comment-16020254 ] Andrew Purtell commented on HBASE-18027: Thanks [~ashu210890] Test failures and timeouts are not mine. The findbugs warning is not mine. I didn't add or remove any 'synchronize' keywords in ReplicationSinkManager. Waiting on feedback from [~lhofhansl] if the recent changes address his last review. > Replication should respect RPC size limits when batching edits > -- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry in the replication context is then placed in order by hash of > encoded region name into one of these N lists. Each of the N lists is then > sent all at once in one replication RPC. We do not test if the sum of data in > each N list will exceed RPC size limits. This code presumes each individual > edit is reasonably small. Not checking for aggregate size while assembling > the lists into RPCs is an oversight and can lead to replication failure when > that assumption is violated. > We can fix this by generating as many replication RPC calls as we need to > drain a list, keeping each RPC under limit, instead of assuming the whole > list will fit in one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18090) Improve TableSnapshotInputFormat to allow more multiple mappers per region
[ https://issues.apache.org/jira/browse/HBASE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-18090: Status: Patch Available (was: Open) > Improve TableSnapshotInputFormat to allow more multiple mappers per region > -- > > Key: HBASE-18090 > URL: https://issues.apache.org/jira/browse/HBASE-18090 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0 >Reporter: Mikhail Antonov > Attachments: HBASE-18090-branch-1.3-v1.patch > > > TableSnapshotInputFormat runs one map task per region in the table snapshot. > This places unnecessary restriction that the region layout of the original > table needs to take the processing resources available to MR job into > consideration. Allowing to run multiple mappers per region (assuming > reasonably even key distribution) would be useful. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18090) Improve TableSnapshotInputFormat to allow more multiple mappers per region
[ https://issues.apache.org/jira/browse/HBASE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-18090: Status: Open (was: Patch Available) > Improve TableSnapshotInputFormat to allow more multiple mappers per region > -- > > Key: HBASE-18090 > URL: https://issues.apache.org/jira/browse/HBASE-18090 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0 >Reporter: Mikhail Antonov > Attachments: HBASE-18090-branch-1.3-v1.patch > > > TableSnapshotInputFormat runs one map task per region in the table snapshot. > This places unnecessary restriction that the region layout of the original > table needs to take the processing resources available to MR job into > consideration. Allowing to run multiple mappers per region (assuming > reasonably even key distribution) would be useful. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18093) Overloading the meaning of 'enabled' in Quota Manager to indicate either quota disabled or quota manager not ready is not good
[
https://issues.apache.org/jira/browse/HBASE-18093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020227#comment-16020227
]
Hadoop QA commented on HBASE-18093:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 16s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
53s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 2s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} master passed {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 28s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 27s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 27s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 1m 20s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.6.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 2m 37s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.6.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 3m 53s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.6.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 5m 10s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.6.4. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 6m 26s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.6.5. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 7m 43s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.7.1. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 8m 59s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.7.2. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 10m 17s
{color} | {color:red} The patch causes 16 errors with Hadoop v2.7.3. {color} |
| {color:red}-1{color} | {color:red} hadoopcheck {color} | {color:red} 11m 33s
{color} | {color:red} The patch causes 16 errors with Hadoop v3.0.0-alpha2.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 25s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 31s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 28s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
9s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 24m 0s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12869327/HBASE-18093.v1-master.patch
|
| JIRA Issue | HBASE-18093 |
| Optional Tests | asflicense javac javadoc
[jira] [Commented] (HBASE-18090) Improve TableSnapshotInputFormat to allow more multiple mappers per region
[
https://issues.apache.org/jira/browse/HBASE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020202#comment-16020202
]
Hadoop QA commented on HBASE-18090:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 23s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 4m 54s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
22s {color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} branch-1.3 passed with JDK v1.8.0_131 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} branch-1.3 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
3s {color} | {color:green} branch-1.3 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
47s {color} | {color:green} branch-1.3 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 45s
{color} | {color:red} hbase-server in branch-1.3 has 1 extant Findbugs
warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} branch-1.3 passed with JDK v1.8.0_131 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} branch-1.3 passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
0s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed with JDK v1.8.0_131 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
15m 23s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed with JDK v1.8.0_131 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 43s
{color} | {color:green} the patch passed with JDK v1.7.0_80 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 32m 53s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 17s
{color} | {color:green} hbase-it in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
20s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 75m 1s {color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.ipc.TestSimpleRpcScheduler |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.3 Server=1.12.3 Image:yetus/hbase:e1e11ad |
| JIRA Patch URL |
https://issu
[jira] [Commented] (HBASE-18093) Overloading the meaning of 'enabled' in Quota Manager to indicate either quota disabled or quota manager not ready is not good
[
https://issues.apache.org/jira/browse/HBASE-18093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020190#comment-16020190
]
Ted Yu commented on HBASE-18093:
{code}
62private boolean initalized = false;
{code}
Typo in variable name.
Otherwise looks good.
> Overloading the meaning of 'enabled' in Quota Manager to indicate either
> quota disabled or quota manager not ready is not good
> --
>
> Key: HBASE-18093
> URL: https://issues.apache.org/jira/browse/HBASE-18093
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 1.1.10
>Reporter: Stephen Yuan Jiang
>Assignee: Stephen Yuan Jiang
>Priority: Minor
> Attachments: HBASE-18093.v1-master.patch
>
>
> In MasterQuotaManager, a member 'enabled' is used to indicate either quota
> feature is disabled or quota manager is not fully initialized. This would
> create confusion whether caller should wait for quota manager to be
> initialized or change configuration to enable quota.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-16488) Starting namespace and quota services in master startup asynchronizely
[ https://issues.apache.org/jira/browse/HBASE-16488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020184#comment-16020184 ] Stephen Yuan Jiang commented on HBASE-16488: hadoop.hbase.quotas.TestQuotaAdmin failure should be addressed in a generic fix in HBASE-18093 > Starting namespace and quota services in master startup asynchronizely > -- > > Key: HBASE-16488 > URL: https://issues.apache.org/jira/browse/HBASE-16488 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 2.0.0, 1.3.0, 1.0.3, 1.4.0, 1.1.5, 1.2.2 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > Attachments: HBASE-16488.v1-branch-1.patch, > HBASE-16488.v1-master.patch, HBASE-16488.v2-branch-1.patch, > HBASE-16488.v2-branch-1.patch, HBASE-16488.v3-branch-1.patch, > HBASE-16488.v3-branch-1.patch, HBASE-16488.v4-branch-1.patch, > HBASE-16488.v5-branch-1.patch, HBASE-16488.v6-branch-1.patch > > > From time to time, during internal IT test and from customer, we often see > master initialization failed due to namespace table region takes long time to > assign (eg. sometimes split log takes long time or hanging; or sometimes RS > is temporarily not available; sometimes due to some unknown assignment > issue). In the past, there was some proposal to improve this situation, eg. > HBASE-13556 / HBASE-14190 (Assign system tables ahead of user region > assignment) or HBASE-13557 (Special WAL handling for system tables) or > HBASE-14623 (Implement dedicated WAL for system tables). > This JIRA proposes another way to solve this master initialization fail > issue: namespace service is only used by a handful operations (eg. create > table / namespace DDL / get namespace API / some RS group DDL). Only quota > manager depends on it and quota management is off by default. Therefore, > namespace service is not really needed for master to be functional. So we > could start namespace service asynchronizely without blocking master startup. > -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18093) Overloading the meaning of 'enabled' in Quota Manager to indicate either quota disabled or quota manager not ready is not good
[ https://issues.apache.org/jira/browse/HBASE-18093?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020180#comment-16020180 ] Stephen Yuan Jiang commented on HBASE-18093: V1 patch to distinguish whether quota is disabled or quota manger is uninitialized. > Overloading the meaning of 'enabled' in Quota Manager to indicate either > quota disabled or quota manager not ready is not good > -- > > Key: HBASE-18093 > URL: https://issues.apache.org/jira/browse/HBASE-18093 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 1.1.10 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang >Priority: Minor > Attachments: HBASE-18093.v1-master.patch > > > In MasterQuotaManager, a member 'enabled' is used to indicate either quota > feature is disabled or quota manager is not fully initialized. This would > create confusion whether caller should wait for quota manager to be > initialized or change configuration to enable quota. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18093) Overloading 'enabled' in Quota Manager to indicate either quota disabled or quota manager not ready is not good
[ https://issues.apache.org/jira/browse/HBASE-18093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-18093: --- Summary: Overloading 'enabled' in Quota Manager to indicate either quota disabled or quota manager not ready is not good (was: Overload 'enabled' in Quota Manager) > Overloading 'enabled' in Quota Manager to indicate either quota disabled or > quota manager not ready is not good > --- > > Key: HBASE-18093 > URL: https://issues.apache.org/jira/browse/HBASE-18093 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 1.1.10 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang >Priority: Minor > Attachments: HBASE-18093.v1-master.patch > > > In MasterQuotaManager, a member 'enabled' is used to indicate either quota > feature is disabled or quota manager is not fully initialized. This would > create confusion whether caller should wait for quota manager to be > initialized or change configuration to enable quota. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18093) Overloading the meaning of 'enabled' in Quota Manager to indicate either quota disabled or quota manager not ready is not good
[ https://issues.apache.org/jira/browse/HBASE-18093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-18093: --- Status: Patch Available (was: Open) > Overloading the meaning of 'enabled' in Quota Manager to indicate either > quota disabled or quota manager not ready is not good > -- > > Key: HBASE-18093 > URL: https://issues.apache.org/jira/browse/HBASE-18093 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 1.1.10 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang >Priority: Minor > Attachments: HBASE-18093.v1-master.patch > > > In MasterQuotaManager, a member 'enabled' is used to indicate either quota > feature is disabled or quota manager is not fully initialized. This would > create confusion whether caller should wait for quota manager to be > initialized or change configuration to enable quota. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18093) Overloading the meaning of 'enabled' in Quota Manager to indicate either quota disabled or quota manager not ready is not good
[ https://issues.apache.org/jira/browse/HBASE-18093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-18093: --- Summary: Overloading the meaning of 'enabled' in Quota Manager to indicate either quota disabled or quota manager not ready is not good (was: Overloading 'enabled' in Quota Manager to indicate either quota disabled or quota manager not ready is not good) > Overloading the meaning of 'enabled' in Quota Manager to indicate either > quota disabled or quota manager not ready is not good > -- > > Key: HBASE-18093 > URL: https://issues.apache.org/jira/browse/HBASE-18093 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 1.1.10 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang >Priority: Minor > Attachments: HBASE-18093.v1-master.patch > > > In MasterQuotaManager, a member 'enabled' is used to indicate either quota > feature is disabled or quota manager is not fully initialized. This would > create confusion whether caller should wait for quota manager to be > initialized or change configuration to enable quota. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18093) Overload 'enabled' in Quota Manager
[ https://issues.apache.org/jira/browse/HBASE-18093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-18093: --- Attachment: HBASE-18093.v1-master.patch > Overload 'enabled' in Quota Manager > --- > > Key: HBASE-18093 > URL: https://issues.apache.org/jira/browse/HBASE-18093 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 1.1.10 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang >Priority: Minor > Attachments: HBASE-18093.v1-master.patch > > > In MasterQuotaManager, a member 'enabled' is used to indicate either quota > feature is disabled or quota manager is not fully initialized. This would > create confusion whether caller should wait for quota manager to be > initialized or change configuration to enable quota. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-18093) Overload 'enabled' in Quota Manager
Stephen Yuan Jiang created HBASE-18093: -- Summary: Overload 'enabled' in Quota Manager Key: HBASE-18093 URL: https://issues.apache.org/jira/browse/HBASE-18093 Project: HBase Issue Type: Bug Components: master Affects Versions: 1.1.10 Reporter: Stephen Yuan Jiang Assignee: Stephen Yuan Jiang Priority: Minor In MasterQuotaManager, a member 'enabled' is used to indicate either quota feature is disabled or quota manager is not fully initialized. This would create confusion whether caller should wait for quota manager to be initialized or change configuration to enable quota. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18068) Fix flaky test TestAsyncSnapshotAdminApi
[
https://issues.apache.org/jira/browse/HBASE-18068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020133#comment-16020133
]
Hudson commented on HBASE-18068:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3058 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/3058/])
HBASE-18068 Fix flaky test TestAsyncSnapshotAdminApi (appy: rev
23ea2c36f52283631ce665e04f2af433fc25ae55)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncSnapshotAdminApi.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/AsyncHBaseAdmin.java
> Fix flaky test TestAsyncSnapshotAdminApi
>
>
> Key: HBASE-18068
> URL: https://issues.apache.org/jira/browse/HBASE-18068
> Project: HBase
> Issue Type: Sub-task
> Components: Client
>Reporter: Appy
>Assignee: Appy
> Fix For: 2.0.0
>
> Attachments: HBASE-18068.master.001.patch
>
>
> Test failures
> {noformat}
> org.apache.hadoop.hbase.client.TestAsyncSnapshotAdminApi.testRestoreSnapshot
>
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotException:
> org.apache.hadoop.hbase.snapshot.RestoreSnapshotException: Restore already in
> progress on the table=testRestoreSnapshot
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.restoreSnapshot(SnapshotManager.java:854)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.restoreSnapshot(SnapshotManager.java:818)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.restoreOrCloneSnapshot(SnapshotManager.java:780)
> at org.apache.hadoop.hbase.master.HMaster$14.run(HMaster.java:2324)
> at
> org.apache.hadoop.hbase.master.procedure.MasterProcedureUtil.submitProcedure(MasterProcedureUtil.java:133)
> at org.apache.hadoop.hbase.master.HMaster.restoreSnapshot(HMaster.java:2320)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.restoreSnapshot(MasterRpcServices.java:1224)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:257)
>
> org.apache.hadoop.hbase.client.TestAsyncSnapshotAdminApi.testDeleteSnapshots
>
> org.apache.hadoop.hbase.snapshot.SnapshotCreationException:
> org.apache.hadoop.hbase.snapshot.SnapshotCreationException: Rejected taking {
> ss=snapshotName1 table=testDeleteSnapshots type=FLUSH } because we are
> already running another snapshot on the same table { ss=snapshotName1
> table=testDeleteSnapshots type=FLUSH }
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.prepareToTakeSnapshot(SnapshotManager.java:440)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.snapshotEnabledTable(SnapshotManager.java:497)
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.takeSnapshot(SnapshotManager.java:598)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.snapshot(MasterRpcServices.java:1299)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:257)
>
> org.apache.hadoop.hbase.client.TestAsyncSnapshotAdminApi.testListSnapshots
>
> org.apache.hadoop.hbase.snapshot.SnapshotDoesNotExistException: Snapshot
> 'snapshotName2' doesn't exist on the filesystem
> at
> org.apache.hadoop.hbase.master.snapshot.SnapshotManager.deleteSnapshot(SnapshotManager.java:289)
> at
> org.apache.hadoop.hbase.master.MasterRpcServices.deleteSnapshot(MasterRpcServices.java:461)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:277)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.
[jira] [Commented] (HBASE-18075) Support namespaces and tables with non-latin alphabetical characters
[
https://issues.apache.org/jira/browse/HBASE-18075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020135#comment-16020135
]
Hudson commented on HBASE-18075:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #3058 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/3058/])
HBASE-18075 (addendum) Correct javadoc for valid namespaces (elserj: rev
56cbc68b0da37db6a34e14e2bdc52245b848d173)
* (edit) hbase-common/src/main/java/org/apache/hadoop/hbase/TableName.java
> Support namespaces and tables with non-latin alphabetical characters
>
>
> Key: HBASE-18075
> URL: https://issues.apache.org/jira/browse/HBASE-18075
> Project: HBase
> Issue Type: Improvement
> Components: Client
>Reporter: Josh Elser
>Assignee: Josh Elser
> Fix For: 2.0.0
>
> Attachments: HBASE-18075.001.patch, HBASE-18075.002.patch,
> HBASE-18075.003.patch, HBASE-18075-addendum.patch
>
>
> On the heels of HBASE-18067, it would be nice to support namespaces and
> tables with names that fall outside of Latin alphabetical characters and
> numbers.
> Our current regex for allowable characters is approximately
> {{\[a-zA-Z0-9\]+}}.
> It would be nice to replace {{a-zA-Z}} with Java's {{\p\{IsAlphabetic\}}}
> which will naturally restrict the unicode character space down to just those
> that are part of the alphabet for each script (e.g. latin, cyrillic, greek).
> Technically, our possible scope of allowable characters is, best as I can
> tell, only limited by the limitations of ZooKeeper itself
> https://zookeeper.apache.org/doc/r3.4.10/zookeeperProgrammers.html#ch_zkDataModel
> (as both table and namespace are created as znodes).
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (HBASE-18060) Backport to branch-1 HBASE-9774 HBase native metrics and metric collection for coprocessors
[
https://issues.apache.org/jira/browse/HBASE-18060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020131#comment-16020131
]
Hadoop QA commented on HBASE-18060:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 20s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 14 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 20s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
3s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 38s
{color} | {color:green} branch-1 passed with JDK v1.8.0_131 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 43s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 4m
24s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
56s {color} | {color:green} branch-1 passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source:
hbase-assembly hbase-assembly . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 3s
{color} | {color:red} hbase-server in branch-1 has 1 extant Findbugs warnings.
{color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 5s
{color} | {color:red} hbase-server in branch-1 has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m 54s
{color} | {color:green} branch-1 passed with JDK v1.8.0_131 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m 28s
{color} | {color:green} branch-1 passed with JDK v1.7.0_80 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m 49s
{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 15s
{color} | {color:red} hbase-assembly in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m 13s
{color} | {color:red} root in the patch failed with JDK v1.8.0_131. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m 12s
{color} | {color:red} hbase-assembly in the patch failed with JDK v1.8.0_131.
{color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 17s {color}
| {color:red} hbase-common-jdk1.8.0_131 with JDK v1.8.0_131 generated 8 new +
10 unchanged - 10 fixed = 18 total (was 20) {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 9s
{color} | {color:green} hbase-metrics-api in the patch passed with JDK
v1.8.0_131. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 10s
{color} | {color:green} hbase-hadoop-compat in the patch passed with JDK
v1.8.0_131. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 1m 13s {color}
| {color:red} root in the patch failed with JDK v1.8.0_131. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 12s {color}
| {color:red} hbase-assembly in the patch failed with JDK v1.8.0_131. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 15s
{color} | {color:green} hbase-examples in the patch passed with JDK v1.8.0_131.
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 11s
{color} | {color:green} hbase-hadoop2-compat in the patch passed with JDK
v1.8.0_131. {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 10s
{color} | {color:green} hbase-metrics in the patch passed with JDK v1.8.0_131.
{color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s
{color} | {color:green} hbase-server-jdk1.8.0_131 with JDK v1.8.0_131 generated
0 new + 5 unchanged - 5 fixed = 5 total (was 10) {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 1m 9s
{color} | {color:red} root in the patch failed with JDK v1.7.0_80. {color} |
| {color:red}-1{color}
[jira] [Updated] (HBASE-17850) Backup system restore /repair utility
[
https://issues.apache.org/jira/browse/HBASE-17850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Rodionov updated HBASE-17850:
--
Attachment: HBASE-17850-v7.patch
Patch v7
> Backup system restore /repair utility
> -
>
> Key: HBASE-17850
> URL: https://issues.apache.org/jira/browse/HBASE-17850
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-17850-v2.patch, HBASE-17850-v3.patch,
> HBASE-17850-v4.patch, HBASE-17850-v5.patch, HBASE-17850-v6.patch,
> HBASE-17850-v7.patch
>
>
> Backup repair tool restores integrity of a backup system table and removes
> artefacts of a failed backup session from a file system(s)
> This is command-line tool. To run the tool:
> {code}
> hbase backup repair
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (HBASE-16961) FileSystem Quotas
[ https://issues.apache.org/jira/browse/HBASE-16961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Josh Elser resolved HBASE-16961. Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.0.0 Rebased and merged all sub-tasks onto master (2.0.0). RESULT of the vote for posterity: https://lists.apache.org/thread.html/6ae310a51232f19209d4af50c91805e01b5a6dc1852b60a40484fc2e@%3Cdev.hbase.apache.org%3E > FileSystem Quotas > - > > Key: HBASE-16961 > URL: https://issues.apache.org/jira/browse/HBASE-16961 > Project: HBase > Issue Type: New Feature >Reporter: Josh Elser >Assignee: Josh Elser > Fix For: 2.0.0 > > Attachments: hbase-quota-test.sh > > > Umbrella issue for tracking the filesystem utilization of HBase data, > defining quotas on that utilization, and enforcement when utilization exceeds > the limits of the quota. > At a high level: we can define quotas on tables and namespaces. Region size > is computed by RegionServers and sent to the Master. The Master inspects the > sizes of Regions, rolling up to table and namespace sizes. Defined quotas in > the quota table are evaluated given the computed sizes, and, for those > tables/namespaces violating the quota, RegionServers are informed to take > some action to limit any further filesystem growth by that table/namespace. > Discuss: > https://lists.apache.org/thread.html/66a4b0c3725b5cbdd61dd6111c43847adaeef7b7da5f4cd045df30ef@%3Cdev.hbase.apache.org%3E > Design Doc: > http://home.apache.org/~elserj/hbase/FileSystemQuotasforApacheHBase.pdf or > https://docs.google.com/document/d/1VtLWDkB2tpwc_zgCNPE1ulZOeecF-YA2FYSK3TSs_bw/edit?usp=sharing -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-18090) Improve TableSnapshotInputFormat to allow more multiple mappers per region
[ https://issues.apache.org/jira/browse/HBASE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-18090: Status: Patch Available (was: Open) > Improve TableSnapshotInputFormat to allow more multiple mappers per region > -- > > Key: HBASE-18090 > URL: https://issues.apache.org/jira/browse/HBASE-18090 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0 >Reporter: Mikhail Antonov > Attachments: HBASE-18090-branch-1.3-v1.patch > > > TableSnapshotInputFormat runs one map task per region in the table snapshot. > This places unnecessary restriction that the region layout of the original > table needs to take the processing resources available to MR job into > consideration. Allowing to run multiple mappers per region (assuming > reasonably even key distribution) would be useful. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18027) Replication should respect RPC size limits when batching edits
[ https://issues.apache.org/jira/browse/HBASE-18027?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16020024#comment-16020024 ] Ashu Pachauri commented on HBASE-18027: --- [~apurtell] bq. Perhaps we table this change and wait and see if the same problems occur with 1.3 I can confirm that the same occurs on 1.3. Just for the sake of completeness of discussion, there is probably an easier way to handle this if both source and peer are HBase versions 1.3+ by retrying entries serially if it receives RequestTooBigException. However, that would fail completely in a X --> 1.3+ replication scenario. > Replication should respect RPC size limits when batching edits > -- > > Key: HBASE-18027 > URL: https://issues.apache.org/jira/browse/HBASE-18027 > Project: HBase > Issue Type: Bug > Components: Replication >Affects Versions: 2.0.0, 1.4.0, 1.3.1 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 2.0.0, 1.4.0, 1.3.2 > > Attachments: HBASE-18027.patch, HBASE-18027.patch, HBASE-18027.patch > > > In HBaseInterClusterReplicationEndpoint#replicate we try to replicate in > batches. We create N lists. N is the minimum of configured replicator > threads, number of 100-waledit batches, or number of current sinks. Every > pending entry in the replication context is then placed in order by hash of > encoded region name into one of these N lists. Each of the N lists is then > sent all at once in one replication RPC. We do not test if the sum of data in > each N list will exceed RPC size limits. This code presumes each individual > edit is reasonably small. Not checking for aggregate size while assembling > the lists into RPCs is an oversight and can lead to replication failure when > that assumption is violated. > We can fix this by generating as many replication RPC calls as we need to > drain a list, keeping each RPC under limit, instead of assuming the whole > list will fit in one. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (HBASE-17850) Backup system restore /repair utility
[
https://issues.apache.org/jira/browse/HBASE-17850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladimir Rodionov updated HBASE-17850:
--
Attachment: HBASE-17850-v6.patch
patch v6. cc: [~te...@apache.org]
> Backup system restore /repair utility
> -
>
> Key: HBASE-17850
> URL: https://issues.apache.org/jira/browse/HBASE-17850
> Project: HBase
> Issue Type: Sub-task
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Labels: backup
> Fix For: 2.0.0
>
> Attachments: HBASE-17850-v2.patch, HBASE-17850-v3.patch,
> HBASE-17850-v4.patch, HBASE-17850-v5.patch, HBASE-17850-v6.patch
>
>
> Backup repair tool restores integrity of a backup system table and removes
> artefacts of a failed backup session from a file system(s)
> This is command-line tool. To run the tool:
> {code}
> hbase backup repair
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Created] (HBASE-18092) Removing a peer does not properly clean up the ReplicationSourceManager state and metrics
Ashu Pachauri created HBASE-18092: - Summary: Removing a peer does not properly clean up the ReplicationSourceManager state and metrics Key: HBASE-18092 URL: https://issues.apache.org/jira/browse/HBASE-18092 Project: HBase Issue Type: Bug Components: Replication Affects Versions: 1.3.1, 2.0.0 Reporter: Ashu Pachauri Removing a peer does not clean up the associated metrics and state from walsById map in the ReplicationSourceManager. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (HBASE-18091) Add API for who currently holds a lock on namespace/ table/ region and log when state is LOCK_EVENT_WAIT
Umesh Agashe created HBASE-18091: Summary: Add API for who currently holds a lock on namespace/ table/ region and log when state is LOCK_EVENT_WAIT Key: HBASE-18091 URL: https://issues.apache.org/jira/browse/HBASE-18091 Project: HBase Issue Type: Sub-task Components: proc-v2 Reporter: Umesh Agashe Assignee: Umesh Agashe Add API for who currently holds a lock on namespace/ table/ region and log message when state is LOCK_EVENT_WAIT -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-18087) Fix unit tests for TestTableFavoredNodes
[
https://issues.apache.org/jira/browse/HBASE-18087?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16019991#comment-16019991
]
Hadoop QA commented on HBASE-18087:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 10s {color}
| {color:red} HBASE-18087 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12869306/HBASE-18087.hbase-14614.001.patch
|
| JIRA Issue | HBASE-18087 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6879/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Fix unit tests for TestTableFavoredNodes
>
>
> Key: HBASE-18087
> URL: https://issues.apache.org/jira/browse/HBASE-18087
> Project: HBase
> Issue Type: Sub-task
> Components: master
>Reporter: Umesh Agashe
>Assignee: Umesh Agashe
> Fix For: 2.0.0
>
> Attachments: HBASE-18087.hbase-14614.001.patch
>
>
> Following tests are failing on hbase-14614 branch:
> org.apache.hadoop.hbase.client.TestTableFavoredNodes.testCreateTable
> org.apache.hadoop.hbase.client.TestTableFavoredNodes.testSystemTables
> org.apache.hadoop.hbase.client.TestTableFavoredNodes.testTruncateTable
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Updated] (HBASE-18087) Fix unit tests for TestTableFavoredNodes
[ https://issues.apache.org/jira/browse/HBASE-18087?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Umesh Agashe updated HBASE-18087: - Status: Patch Available (was: Open) > Fix unit tests for TestTableFavoredNodes > > > Key: HBASE-18087 > URL: https://issues.apache.org/jira/browse/HBASE-18087 > Project: HBase > Issue Type: Sub-task > Components: master >Reporter: Umesh Agashe >Assignee: Umesh Agashe > Fix For: 2.0.0 > > Attachments: HBASE-18087.hbase-14614.001.patch > > > Following tests are failing on hbase-14614 branch: > org.apache.hadoop.hbase.client.TestTableFavoredNodes.testCreateTable > org.apache.hadoop.hbase.client.TestTableFavoredNodes.testSystemTables > org.apache.hadoop.hbase.client.TestTableFavoredNodes.testTruncateTable -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (HBASE-15930) Make IntegrationTestReplication's waitForReplication() smarter
[
https://issues.apache.org/jira/browse/HBASE-15930?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16019989#comment-16019989
]
Hadoop QA commented on HBASE-15930:
---
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 31s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
6s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
35s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 0s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 17s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
38s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 30s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
67m 40s {color} | {color:green} Patch does not cause any errors with Hadoop
2.6.1 2.6.2 2.6.3 2.6.4 2.6.5 2.7.1 2.7.2 2.7.3 or 3.0.0-alpha2. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 27s
{color} | {color:green} hbase-it in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
10s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 81m 22s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.03.0-ce Server=17.03.0-ce Image:yetus/hbase:757bf37 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12869291/HBASE-15930.patch |
| JIRA Issue | HBASE-15930 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 63be2f7d4ced 4.8.3-std-1 #1 SMP Fri Oct 21 11:15:43 UTC 2016
x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 23ea2c3 |
| Default Java | 1.8.0_131 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/6876/testReport/ |
| modules | C: hbase-it U: hbase-it |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/6876/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Make IntegrationTestReplication's waitForReplication() smarter
> --
>
> Key: HBASE-15930
> URL: https://issues.apache.org/jira/browse/HBASE-15930
> Project: HBase
> Issue Type: Improvement
> Components: integration tests
>Reporter: Dima Spivak
>Assignee: Mike Drob
> Fix For: 2.0.0
>
> Attachments: HBASE-15930.patch
>
>
> {{IntegrationTestReplication}}
[jira] [Updated] (HBASE-18090) Improve TableSnapshotInputFormat to allow more multiple mappers per region
[ https://issues.apache.org/jira/browse/HBASE-18090?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-18090: Attachment: HBASE-18090-branch-1.3-v1.patch patch for review > Improve TableSnapshotInputFormat to allow more multiple mappers per region > -- > > Key: HBASE-18090 > URL: https://issues.apache.org/jira/browse/HBASE-18090 > Project: HBase > Issue Type: Improvement > Components: mapreduce >Affects Versions: 1.4.0 >Reporter: Mikhail Antonov > Attachments: HBASE-18090-branch-1.3-v1.patch > > > TableSnapshotInputFormat runs one map task per region in the table snapshot. > This places unnecessary restriction that the region layout of the original > table needs to take the processing resources available to MR job into > consideration. Allowing to run multiple mappers per region (assuming > reasonably even key distribution) would be useful. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
