[jira] [Updated] (HBASE-16423) Add re-compare option to VerifyReplication to avoid occasional inconsistent rows
[ https://issues.apache.org/jira/browse/HBASE-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianwei Cui updated HBASE-16423: Attachment: HBASE-16423-branch-1-v1.patch patch for branch-1. > Add re-compare option to VerifyReplication to avoid occasional inconsistent > rows > > > Key: HBASE-16423 > URL: https://issues.apache.org/jira/browse/HBASE-16423 > Project: HBase > Issue Type: Improvement > Components: Replication >Affects Versions: 2.0.0 >Reporter: Jianwei Cui >Assignee: Jianwei Cui >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16423-branch-1-v1.patch, HBASE-16423-v1.patch, > HBASE-16423-v2.patch, HBASE-16423-v3.patch > > > Because replication keeps eventually consistency, VerifyReplication may > report inconsistent rows if there are data being written to source or peer > clusters during scanning. These occasionally inconsistent rows will have the > same data if we do the comparison again after a short period. It is not easy > to find the really inconsistent rows if VerifyReplication report a large > number of such occasionally inconsistency. To avoid this case, we can add an > option to make VerifyReplication read out the inconsistent rows again after > sleeping a few seconds and re-compare the rows during scanning. This behavior > follows the eventually consistency of hbase's replication. Suggestions and > discussions are welcomed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[
https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-16643:
---
Attachment: HBASE-16643_6.patch
Updated patch correcting the test case failures. Every where we are now
supposed to do the seek before we do next() call. The hbase core does it that
way only but test cases were not doing it. So we may have to update such test
cases now.
> Reverse scanner heap creation may not allow MSLAB closure due to improper ref
> counting of segments
> --
>
> Key: HBASE-16643
> URL: https://issues.apache.org/jira/browse/HBASE-16643
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16643.patch, HBASE-16643_1.patch,
> HBASE-16643_2.patch, HBASE-16643_3.patch, HBASE-16643_4.patch,
> HBASE-16643_5.patch, HBASE-16643_6.patch
>
>
> In the reverse scanner case,
> While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the
> backward heap, we do a
> {code}
> if ((backwardHeap == null) && (forwardHeap != null)) {
> forwardHeap.close();
> forwardHeap = null;
> // before building the heap seek for the relevant key on the scanners,
> // for the heap to be built from the scanners correctly
> for (KeyValueScanner scan : scanners) {
> if (toLast) {
> res |= scan.seekToLastRow();
> } else {
> res |= scan.backwardSeek(cell);
> }
> }
> {code}
> forwardHeap.close(). This would internally decrement the MSLAB ref counter
> for the current active segment and snapshot segment.
> When the scan is actually closed again we do close() and that will again
> decrement the count. Here chances are there that the count would go negative
> and hence the actual MSLAB closure that checks for refCount==0 will fail.
> Apart from this, when the refCount becomes 0 after the firstClose if any
> other thread requests to close the segment, then we will end up in corrupted
> segment because the segment could be put back to the MSLAB pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[
https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-16643:
---
Status: Patch Available (was: Open)
> Reverse scanner heap creation may not allow MSLAB closure due to improper ref
> counting of segments
> --
>
> Key: HBASE-16643
> URL: https://issues.apache.org/jira/browse/HBASE-16643
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16643.patch, HBASE-16643_1.patch,
> HBASE-16643_2.patch, HBASE-16643_3.patch, HBASE-16643_4.patch,
> HBASE-16643_5.patch, HBASE-16643_6.patch
>
>
> In the reverse scanner case,
> While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the
> backward heap, we do a
> {code}
> if ((backwardHeap == null) && (forwardHeap != null)) {
> forwardHeap.close();
> forwardHeap = null;
> // before building the heap seek for the relevant key on the scanners,
> // for the heap to be built from the scanners correctly
> for (KeyValueScanner scan : scanners) {
> if (toLast) {
> res |= scan.seekToLastRow();
> } else {
> res |= scan.backwardSeek(cell);
> }
> }
> {code}
> forwardHeap.close(). This would internally decrement the MSLAB ref counter
> for the current active segment and snapshot segment.
> When the scan is actually closed again we do close() and that will again
> decrement the count. Here chances are there that the count would go negative
> and hence the actual MSLAB closure that checks for refCount==0 will fail.
> Apart from this, when the refCount becomes 0 after the firstClose if any
> other thread requests to close the segment, then we will end up in corrupted
> segment because the segment could be put back to the MSLAB pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16664) Timeout logic in AsyncProcess is broken
[ https://issues.apache.org/jira/browse/HBASE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515594#comment-15515594 ] Phil Yang edited comment on HBASE-16664 at 9/23/16 6:42 AM: And for some operations, like append, we use NoncedRegionServerCallable extends CancellableRegionServerCallable, but we use caller's callWithRetries, both of them have logic of operation timeout, they are conflicting . But the logic in CancellableRegionServerCallable is broken, so it just works... was (Author: yangzhe1991): And for some operations, like append, we use NoncedRegionServerCallable extends CancellableRegionServerCallable, but we use caller's callWithRetries, both of time have logic of operation timeout, they are conflicting . But the logic in CancellableRegionServerCallable is broken, so it just works... > Timeout logic in AsyncProcess is broken > --- > > Key: HBASE-16664 > URL: https://issues.apache.org/jira/browse/HBASE-16664 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang > Attachments: testhcm.patch > > > Have not checked the root cause, but I think timeout of all operations in > AsyncProcess is broken -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16664) Timeout logic in AsyncProcess is broken
[ https://issues.apache.org/jira/browse/HBASE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515594#comment-15515594 ] Phil Yang commented on HBASE-16664: --- And for some operations, like append, we use NoncedRegionServerCallable extends CancellableRegionServerCallable, but we use caller's callWithRetries, both of time have logic of operation timeout, they are conflicting . But the logic in CancellableRegionServerCallable is broken, so it just works... > Timeout logic in AsyncProcess is broken > --- > > Key: HBASE-16664 > URL: https://issues.apache.org/jira/browse/HBASE-16664 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang > Attachments: testhcm.patch > > > Have not checked the root cause, but I think timeout of all operations in > AsyncProcess is broken -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15871) Memstore flush doesn't finish because of backwardseek() in memstore scanner.
[ https://issues.apache.org/jira/browse/HBASE-15871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515582#comment-15515582 ] ramkrishna.s.vasudevan commented on HBASE-15871: bq.So we may have to have closed state in MemstoreScanner? Yes. We should add it now after HBAES-16643. https://issues.apache.org/jira/browse/HBASE-16643?focusedCommentId=15509011&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15509011. I need to check one more case in HBASE-16643 after that wil update this JIRA. > Memstore flush doesn't finish because of backwardseek() in memstore scanner. > > > Key: HBASE-15871 > URL: https://issues.apache.org/jira/browse/HBASE-15871 > Project: HBase > Issue Type: Bug > Components: Scanners >Affects Versions: 1.1.2 >Reporter: Jeongdae Kim >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15871-branch-1.patch, > HBASE-15871.branch-1.1.001.patch, HBASE-15871.branch-1.1.002.patch, > HBASE-15871.branch-1.1.003.patch, HBASE-15871.patch, HBASE-15871_1.patch, > HBASE-15871_1.patch, HBASE-15871_2.patch, HBASE-15871_3.patch, > HBASE-15871_4.patch, memstore_backwardSeek().PNG > > > Sometimes in our production hbase cluster, it takes a long time to finish > memstore flush.( for about more than 30 minutes) > the reason is that a memstore flusher thread calls > StoreScanner.updateReaders(), waits for acquiring a lock that store scanner > holds in StoreScanner.next() and backwardseek() in memstore scanner runs for > a long time. > I think that this condition could occur in reverse scan by the following > process. > 1) create a reversed store scanner by requesting a reverse scan. > 2) flush a memstore in the same HStore. > 3) puts a lot of cells in memstore and memstore is almost full. > 4) call the reverse scanner.next() and re-create all scanners in this store > because all scanners was already closed by 2)'s flush() and backwardseek() > with store's lastTop for all new scanners. > 5) in this status, memstore is almost full by 2) and all cells in memstore > have sequenceID greater than this scanner's readPoint because of 2)'s > flush(). this condition causes searching all cells in memstore, and > seekToPreviousRow() repeatly seach cells that are already searched if a row > has one column. (described this in more detail in a attached file.) > 6) flush a memstore again in the same HStore, and wait until 4-5) process > finished, to update store files in the same HStore after flusing. > I searched HBase jira. and found a similar issue. (HBASE-14497) but, > HBASE-14497's fix can't solve this issue because that fix just changed > recursive call to loop.(and already applied to our HBase version) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16664) Timeout logic in AsyncProcess is broken
[ https://issues.apache.org/jira/browse/HBASE-16664?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515576#comment-15515576 ] Phil Yang commented on HBASE-16664: --- The current CancellableRegionServerCallable's logic is wrong, I think. The tracker must be started from beginning, not each call. And in fact we will create new CancellableRegionServerCallable in each retrying, so the operation timeout is broken. My idea is pass a deadline (currentTime+operationTimeout) when we submit, we just check the remaining time and get min of remaining and rpcTimeout for each call. And we have RpcRetryingCaller, but in AP we only use callWithoutRetries. If we want to unify all operations into AP, the RpcRetryingCaller can be removed? > Timeout logic in AsyncProcess is broken > --- > > Key: HBASE-16664 > URL: https://issues.apache.org/jira/browse/HBASE-16664 > Project: HBase > Issue Type: Bug >Reporter: Phil Yang > Attachments: testhcm.patch > > > Have not checked the root cause, but I think timeout of all operations in > AsyncProcess is broken -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15871) Memstore flush doesn't finish because of backwardseek() in memstore scanner.
[ https://issues.apache.org/jira/browse/HBASE-15871?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515572#comment-15515572 ] Anoop Sam John commented on HBASE-15871: Now after fix of HBASE-16643, we can not do a peek and decide whether to include the memstore scanner or not. The heap within the MemstoreScanner is lazily created. So we may have to have closed state in MemstoreScanner? We will include the MemstoreScanner in store level heap but the subsequent calls to this MemstoreScanner checks the closed state and return null? The outer heap will remove any individual scanner within it when it see a null result from it no? > Memstore flush doesn't finish because of backwardseek() in memstore scanner. > > > Key: HBASE-15871 > URL: https://issues.apache.org/jira/browse/HBASE-15871 > Project: HBase > Issue Type: Bug > Components: Scanners >Affects Versions: 1.1.2 >Reporter: Jeongdae Kim >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-15871-branch-1.patch, > HBASE-15871.branch-1.1.001.patch, HBASE-15871.branch-1.1.002.patch, > HBASE-15871.branch-1.1.003.patch, HBASE-15871.patch, HBASE-15871_1.patch, > HBASE-15871_1.patch, HBASE-15871_2.patch, HBASE-15871_3.patch, > HBASE-15871_4.patch, memstore_backwardSeek().PNG > > > Sometimes in our production hbase cluster, it takes a long time to finish > memstore flush.( for about more than 30 minutes) > the reason is that a memstore flusher thread calls > StoreScanner.updateReaders(), waits for acquiring a lock that store scanner > holds in StoreScanner.next() and backwardseek() in memstore scanner runs for > a long time. > I think that this condition could occur in reverse scan by the following > process. > 1) create a reversed store scanner by requesting a reverse scan. > 2) flush a memstore in the same HStore. > 3) puts a lot of cells in memstore and memstore is almost full. > 4) call the reverse scanner.next() and re-create all scanners in this store > because all scanners was already closed by 2)'s flush() and backwardseek() > with store's lastTop for all new scanners. > 5) in this status, memstore is almost full by 2) and all cells in memstore > have sequenceID greater than this scanner's readPoint because of 2)'s > flush(). this condition causes searching all cells in memstore, and > seekToPreviousRow() repeatly seach cells that are already searched if a row > has one column. (described this in more detail in a attached file.) > 6) flush a memstore again in the same HStore, and wait until 4-5) process > finished, to update store files in the same HStore after flusing. > I searched HBase jira. and found a similar issue. (HBASE-14497) but, > HBASE-14497's fix can't solve this issue because that fix just changed > recursive call to loop.(and already applied to our HBase version) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16682) Fix Shell tests failure. NoClassDefFoundError for MiniKdc
[
https://issues.apache.org/jira/browse/HBASE-16682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515553#comment-15515553
]
Guanghao Zhang commented on HBASE-16682:
Agreed. I will update HBASE-1 after this be committed. v1 patch looks good
to me except the version info. The version will extend from parent pom and it
doesn't need to set. I am +1 for this. But it should need a +1 from committer?
> Fix Shell tests failure. NoClassDefFoundError for MiniKdc
> -
>
> Key: HBASE-16682
> URL: https://issues.apache.org/jira/browse/HBASE-16682
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-16682.master.001.patch,
> HBASE-16682.master.002.patch
>
>
> Stacktrace
> {noformat}
> java.lang.NoClassDefFoundError: org/apache/hadoop/minikdc/MiniKdc
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
> at java.lang.Class.getDeclaredMethods(Class.java:1975)
> at org.jruby.javasupport.JavaClass.getMethods(JavaClass.java:2110)
> at org.jruby.javasupport.JavaClass.setupClassMethods(JavaClass.java:955)
> at org.jruby.javasupport.JavaClass.access$700(JavaClass.java:99)
> at
> org.jruby.javasupport.JavaClass$ClassInitializer.initialize(JavaClass.java:650)
> at org.jruby.javasupport.JavaClass.setupProxy(JavaClass.java:689)
> at org.jruby.javasupport.Java.createProxyClass(Java.java:526)
> at org.jruby.javasupport.Java.getProxyClass(Java.java:455)
> at org.jruby.javasupport.Java.getInstance(Java.java:364)
> at
> org.jruby.javasupport.JavaUtil.convertJavaToUsableRubyObject(JavaUtil.java:166)
> at
> org.jruby.javasupport.JavaEmbedUtils.javaToRuby(JavaEmbedUtils.java:291)
> at
> org.jruby.embed.variable.AbstractVariable.updateByJavaObject(AbstractVariable.java:81)
> at
> org.jruby.embed.variable.GlobalVariable.(GlobalVariable.java:69)
> at
> org.jruby.embed.variable.GlobalVariable.getInstance(GlobalVariable.java:60)
> at
> org.jruby.embed.variable.VariableInterceptor.getVariableInstance(VariableInterceptor.java:97)
> at org.jruby.embed.internal.BiVariableMap.put(BiVariableMap.java:321)
> at org.jruby.embed.ScriptingContainer.put(ScriptingContainer.java:1123)
> at
> org.apache.hadoop.hbase.client.AbstractTestShell.setUpBeforeClass(AbstractTestShell.java:61)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:108)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:78)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:54)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:144)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.minikdc.M
[jira] [Commented] (HBASE-16604) Scanner retries on IOException can cause the scans to miss data
[
https://issues.apache.org/jira/browse/HBASE-16604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515547#comment-15515547
]
Hudson commented on HBASE-16604:
FAILURE: Integrated in Jenkins build HBase-1.4 #427 (See
[https://builds.apache.org/job/HBase-1.4/427/])
HBASE-16604 Scanner retries on IOException can cause the scans to miss (enis:
rev 8a797e81b83ded184f9ecaeecf26954a27348974)
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceBase.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
* (add)
hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/ScannerResetException.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/UnknownScannerException.java
* (edit)
hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSourceImpl.java
* (edit)
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSource.java
* (add)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/DelegatingKeyValueScanner.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatTestBase.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServer.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultithreadedTableMapper.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestTableSnapshotScanner.java
> Scanner retries on IOException can cause the scans to miss data
>
>
> Key: HBASE-16604
> URL: https://issues.apache.org/jira/browse/HBASE-16604
> Project: HBase
> Issue Type: Bug
> Components: regionserver, Scanners
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: hbase-16604_v1.patch, hbase-16604_v2.patch,
> hbase-16604_v3.branch-1.patch, hbase-16604_v3.patch
>
>
> Debugging an ITBLL failure, where the Verify did not "see" all the data in
> the cluster, I've noticed that if we end up getting a generic IOException
> from the HFileReader level, we may end up missing the rest of the data in the
> region. I was able to manually test this, and this stack trace helps to
> understand what is going on:
> {code}
> 2016-09-09 16:27:15,633 INFO [hconnection-0x71ad3d8a-shared--pool21-t9]
> client.ScannerCallable(376): Open scanner=1 for
> scan={"loadColumnFamiliesOnDemand":null,"startRow":"","stopRow":"","batch":-1,"cacheBlocks":true,"totalColumns":1,"maxResultSize":2097152,"families":{"testFamily":["testFamily"]},"caching":100,"maxVersions":1,"timeRange":[0,9223372036854775807]}
> on region
> region=testScanThrowsException,,1473463632707.b2adfb618e5d0fe225c1dc40c0eabfee.,
> hostname=hw10676,51833,1473463626529, seqNum=2
> 2016-09-09 16:27:15,634 INFO
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833]
> regionserver.RSRpcServices(2196): scan request:scanner_id: 1 number_of_rows:
> 100 close_scanner: false next_call_seq: 0 client_handles_partials: true
> client_handles_heartbeats: true renew: false
> 2016-09-09 16:27:15,635 INFO
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833]
> regionserver.RSRpcServices(2510): Rolling back next call seqId
> 2016-09-09 16:27:15,635 INFO
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833]
> regionserver.RSRpcServices(2565): Throwing new
> ServiceExceptionjava.io.IOException: Could not reseek
> StoreFileScanner[HFileScanner for reader
> reader=hdfs://localhost:51795/user/enis/test-data/d6fb1c70-93c1-4099-acb7-5723fc05a737/data/default/testScanThrowsException/b2adfb618e5d0fe225c1dc40c0eabfee/testFamily/5a213cc23b714e5e8e1a140ebbe72f2c,
> compression=none, cacheConf=blockCache=LruBlockCache{blockCount=0,
> currentSize=1567264, freeSize=1525578848, maxSize=1527146112,
> heapSize=1567264, minSize=1450788736, minFactor=0.95, multiSize=725394368,
> multiFactor=0.5, singleSize=362697184, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false, firstKey=aaa/testFamily:testFamily/1473463633859/Put,
> lastKey=zzz/testFamily:testFamily/1473463634271/Put, avgKeyLen=35,
> avgValueLen=3, entries=17576, length=866
[jira] [Updated] (HBASE-16423) Add re-compare option to VerifyReplication to avoid occasional inconsistent rows
[ https://issues.apache.org/jira/browse/HBASE-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jianwei Cui updated HBASE-16423: Attachment: HBASE-16423-v3.patch trunk patch v3 to fix whitespace. > Add re-compare option to VerifyReplication to avoid occasional inconsistent > rows > > > Key: HBASE-16423 > URL: https://issues.apache.org/jira/browse/HBASE-16423 > Project: HBase > Issue Type: Improvement > Components: Replication >Affects Versions: 2.0.0 >Reporter: Jianwei Cui >Assignee: Jianwei Cui >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16423-v1.patch, HBASE-16423-v2.patch, > HBASE-16423-v3.patch > > > Because replication keeps eventually consistency, VerifyReplication may > report inconsistent rows if there are data being written to source or peer > clusters during scanning. These occasionally inconsistent rows will have the > same data if we do the comparison again after a short period. It is not easy > to find the really inconsistent rows if VerifyReplication report a large > number of such occasionally inconsistency. To avoid this case, we can add an > option to make VerifyReplication read out the inconsistent rows again after > sleeping a few seconds and re-compare the rows during scanning. This behavior > follows the eventually consistency of hbase's replication. Suggestions and > discussions are welcomed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16604) Scanner retries on IOException can cause the scans to miss data
[
https://issues.apache.org/jira/browse/HBASE-16604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515532#comment-15515532
]
Hudson commented on HBASE-16604:
FAILURE: Integrated in Jenkins build HBase-1.2-JDK7 #31 (See
[https://builds.apache.org/job/HBase-1.2-JDK7/31/])
HBASE-16604 Scanner retries on IOException can cause the scans to miss (enis:
rev d307ad19fec359fab166f0a6271d6460324f1c16)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/UnknownScannerException.java
* (edit)
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSource.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java
* (add)
hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/ScannerResetException.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestTableSnapshotScanner.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceBase.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServer.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatTestBase.java
* (add)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/DelegatingKeyValueScanner.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* (edit)
hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSourceImpl.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultithreadedTableMapper.java
> Scanner retries on IOException can cause the scans to miss data
>
>
> Key: HBASE-16604
> URL: https://issues.apache.org/jira/browse/HBASE-16604
> Project: HBase
> Issue Type: Bug
> Components: regionserver, Scanners
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: hbase-16604_v1.patch, hbase-16604_v2.patch,
> hbase-16604_v3.branch-1.patch, hbase-16604_v3.patch
>
>
> Debugging an ITBLL failure, where the Verify did not "see" all the data in
> the cluster, I've noticed that if we end up getting a generic IOException
> from the HFileReader level, we may end up missing the rest of the data in the
> region. I was able to manually test this, and this stack trace helps to
> understand what is going on:
> {code}
> 2016-09-09 16:27:15,633 INFO [hconnection-0x71ad3d8a-shared--pool21-t9]
> client.ScannerCallable(376): Open scanner=1 for
> scan={"loadColumnFamiliesOnDemand":null,"startRow":"","stopRow":"","batch":-1,"cacheBlocks":true,"totalColumns":1,"maxResultSize":2097152,"families":{"testFamily":["testFamily"]},"caching":100,"maxVersions":1,"timeRange":[0,9223372036854775807]}
> on region
> region=testScanThrowsException,,1473463632707.b2adfb618e5d0fe225c1dc40c0eabfee.,
> hostname=hw10676,51833,1473463626529, seqNum=2
> 2016-09-09 16:27:15,634 INFO
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833]
> regionserver.RSRpcServices(2196): scan request:scanner_id: 1 number_of_rows:
> 100 close_scanner: false next_call_seq: 0 client_handles_partials: true
> client_handles_heartbeats: true renew: false
> 2016-09-09 16:27:15,635 INFO
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833]
> regionserver.RSRpcServices(2510): Rolling back next call seqId
> 2016-09-09 16:27:15,635 INFO
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833]
> regionserver.RSRpcServices(2565): Throwing new
> ServiceExceptionjava.io.IOException: Could not reseek
> StoreFileScanner[HFileScanner for reader
> reader=hdfs://localhost:51795/user/enis/test-data/d6fb1c70-93c1-4099-acb7-5723fc05a737/data/default/testScanThrowsException/b2adfb618e5d0fe225c1dc40c0eabfee/testFamily/5a213cc23b714e5e8e1a140ebbe72f2c,
> compression=none, cacheConf=blockCache=LruBlockCache{blockCount=0,
> currentSize=1567264, freeSize=1525578848, maxSize=1527146112,
> heapSize=1567264, minSize=1450788736, minFactor=0.95, multiSize=725394368,
> multiFactor=0.5, singleSize=362697184, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false, firstKey=aaa/testFamily:testFamily/1473463633859/Put,
> lastKey=zzz/testFamily:testFamily/1473463634271/Put, avgKeyLen=35,
> avgValueLen=3, entries=17576, le
[jira] [Commented] (HBASE-16604) Scanner retries on IOException can cause the scans to miss data
[
https://issues.apache.org/jira/browse/HBASE-16604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515515#comment-15515515
]
Hudson commented on HBASE-16604:
FAILURE: Integrated in Jenkins build HBase-1.2-JDK8 #28 (See
[https://builds.apache.org/job/HBase-1.2-JDK8/28/])
HBASE-16604 Scanner retries on IOException can cause the scans to miss (enis:
rev d307ad19fec359fab166f0a6271d6460324f1c16)
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java
* (edit)
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSource.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultithreadedTableMapper.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceBase.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatTestBase.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/UnknownScannerException.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
* (edit)
hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSourceImpl.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServer.java
* (add)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/DelegatingKeyValueScanner.java
* (add)
hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/ScannerResetException.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestTableSnapshotScanner.java
> Scanner retries on IOException can cause the scans to miss data
>
>
> Key: HBASE-16604
> URL: https://issues.apache.org/jira/browse/HBASE-16604
> Project: HBase
> Issue Type: Bug
> Components: regionserver, Scanners
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: hbase-16604_v1.patch, hbase-16604_v2.patch,
> hbase-16604_v3.branch-1.patch, hbase-16604_v3.patch
>
>
> Debugging an ITBLL failure, where the Verify did not "see" all the data in
> the cluster, I've noticed that if we end up getting a generic IOException
> from the HFileReader level, we may end up missing the rest of the data in the
> region. I was able to manually test this, and this stack trace helps to
> understand what is going on:
> {code}
> 2016-09-09 16:27:15,633 INFO [hconnection-0x71ad3d8a-shared--pool21-t9]
> client.ScannerCallable(376): Open scanner=1 for
> scan={"loadColumnFamiliesOnDemand":null,"startRow":"","stopRow":"","batch":-1,"cacheBlocks":true,"totalColumns":1,"maxResultSize":2097152,"families":{"testFamily":["testFamily"]},"caching":100,"maxVersions":1,"timeRange":[0,9223372036854775807]}
> on region
> region=testScanThrowsException,,1473463632707.b2adfb618e5d0fe225c1dc40c0eabfee.,
> hostname=hw10676,51833,1473463626529, seqNum=2
> 2016-09-09 16:27:15,634 INFO
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833]
> regionserver.RSRpcServices(2196): scan request:scanner_id: 1 number_of_rows:
> 100 close_scanner: false next_call_seq: 0 client_handles_partials: true
> client_handles_heartbeats: true renew: false
> 2016-09-09 16:27:15,635 INFO
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833]
> regionserver.RSRpcServices(2510): Rolling back next call seqId
> 2016-09-09 16:27:15,635 INFO
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833]
> regionserver.RSRpcServices(2565): Throwing new
> ServiceExceptionjava.io.IOException: Could not reseek
> StoreFileScanner[HFileScanner for reader
> reader=hdfs://localhost:51795/user/enis/test-data/d6fb1c70-93c1-4099-acb7-5723fc05a737/data/default/testScanThrowsException/b2adfb618e5d0fe225c1dc40c0eabfee/testFamily/5a213cc23b714e5e8e1a140ebbe72f2c,
> compression=none, cacheConf=blockCache=LruBlockCache{blockCount=0,
> currentSize=1567264, freeSize=1525578848, maxSize=1527146112,
> heapSize=1567264, minSize=1450788736, minFactor=0.95, multiSize=725394368,
> multiFactor=0.5, singleSize=362697184, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false, firstKey=aaa/testFamily:testFamily/1473463633859/Put,
> lastKey=zzz/testFamily:testFamily/1473463634271/Put, avgKeyLen=35,
> avgValueLen=3, entries=17576, le
[jira] [Commented] (HBASE-16689) Durability == ASYNC_WAL means no SYNC
[ https://issues.apache.org/jira/browse/HBASE-16689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515511#comment-15515511 ] Yu Li commented on HBASE-16689: --- +1, we also found this during benchmark testing weeks ago but didn't find a good way to fix it, and since there will always be some SYNC call online, we mark this as relatively low priority on our side. Already got some idea about how to fix this sir? [~stack] btw, we also found that because currently we are using one disruptor per WAL and the disruptor is consumed in sequential, we've seen more contention among different regions under high write pressure. I got a patch at hand and will open another JIRA to tell more details soon, JFYI. > Durability == ASYNC_WAL means no SYNC > - > > Key: HBASE-16689 > URL: https://issues.apache.org/jira/browse/HBASE-16689 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 1.0.3, 1.1.6, 1.2.3 >Reporter: stack >Assignee: stack >Priority: Critical > > Setting DURABILITY=ASYNC_WAL on a Table suspends all syncs for all table > Table appends. If all tables on a cluster have this setting, data is flushed > from the RS to the DN at some arbitrary time and a bunch may just hang out in > DFSClient buffers on the RS-side indefinitely if writes are sporadic, at > least until there is a WAL roll -- a log roll sends a sync through the write > pipeline to flush out any outstanding appends -- or a region close which does > similar or we crash and drop the data in buffers RS. > This is probably not what a user expects when they set ASYNC_WAL (We don't > doc anywhere that I could find clearly what ASYNC_WAL means). Worse, old-time > users probably associate ASYNC_WAL and DEFERRED_FLUSH, an old > HTableDescriptor config that was deprecated and replaced by ASYNC_WAL. > DEFERRED_FLUSH ran a background thread -- LogSyncer -- that on a configurable > interval, sent a sync down the write pipeline so any outstanding appends > since last last interval start get pushed out to the DN. ASYNC_WAL doesn't > do this (see below for history on how we let go of the LogSyncer feature). > Of note, we always sync meta edits. You can't turn this off. Also, given WALs > are per regionserver, if other regions on the RS are from tables that have > sync set, these writes will push out to the DN any appends done on tables > that have DEFERRED/ASYNC_WAL set. > To fix, we could do a few things: > * Simple and comprehensive would be always queuing a sync, even if ASYNC_WAL > is set but we let go of Handlers as soon as we write the memstore -- we don't > wait on the sync to complete as we do with the default setting of > Durability=SYNC_WAL. > * Be like a 'real' database and add in a sync after N bytes of data have > been appended (configurable) or after M milliseconds have passed, which ever > threshold happens first. The size check would be easy. The sync-ever-M-millis > would mean another thread. > * Doc what ASYNC_WAL means (and other durability options) > Let me take a look and report back. Will file a bit of history on how we got > here in next comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[
https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515508#comment-15515508
]
ramkrishna vasudevan commented on HBASE-16643:
--
Hi
Are you telling about some new issue @sunyu?
Regards
Ram
> Reverse scanner heap creation may not allow MSLAB closure due to improper ref
> counting of segments
> --
>
> Key: HBASE-16643
> URL: https://issues.apache.org/jira/browse/HBASE-16643
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16643.patch, HBASE-16643_1.patch,
> HBASE-16643_2.patch, HBASE-16643_3.patch, HBASE-16643_4.patch,
> HBASE-16643_5.patch
>
>
> In the reverse scanner case,
> While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the
> backward heap, we do a
> {code}
> if ((backwardHeap == null) && (forwardHeap != null)) {
> forwardHeap.close();
> forwardHeap = null;
> // before building the heap seek for the relevant key on the scanners,
> // for the heap to be built from the scanners correctly
> for (KeyValueScanner scan : scanners) {
> if (toLast) {
> res |= scan.seekToLastRow();
> } else {
> res |= scan.backwardSeek(cell);
> }
> }
> {code}
> forwardHeap.close(). This would internally decrement the MSLAB ref counter
> for the current active segment and snapshot segment.
> When the scan is actually closed again we do close() and that will again
> decrement the count. Here chances are there that the count would go negative
> and hence the actual MSLAB closure that checks for refCount==0 will fail.
> Apart from this, when the refCount becomes 0 after the firstClose if any
> other thread requests to close the segment, then we will end up in corrupted
> segment because the segment could be put back to the MSLAB pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16678) MapReduce jobs do not update counters from ScanMetrics
[
https://issues.apache.org/jira/browse/HBASE-16678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515463#comment-15515463
]
Hadoop QA commented on HBASE-16678:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 36s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
5s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 53s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
50s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
36m 7s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 127m 7s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
18s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 179m 40s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestRegionServerMetrics |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicasClient |
| | org.apache.hadoop.hbase.client.TestClientScannerRPCTimeout |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
| | org.apache.hadoop.hbase.client.TestBlockEvictionFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.0 Server=1.12.0 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829973/hbase-16678_v1.patch |
| JIRA Issue | HBASE-16678 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux eb8476a875b3 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 07ed155 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3677/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3677/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3677/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3677/
回复：[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
String msg = "Results sent from server=" + noOfResults + ". But
only got " + i+ " results completely at client. Resetting the
scanner to scan again."; LOG.error(msg); throw new
DoNotRetryIOException(msg);} } catch (IOException ioe) {
// We are getting IOE while retrieving the cells for Results.
// We have to scan for the same results again. Throwing DNRIOE as a client
retry on the// same scanner will result in
OutOfOrderScannerNextExceptionLOG.error("Exception while reading
cells from result." + "Resetting the scanner to scan again.",
ioe);throw new DoNotRetryIOException("Resetting the scanner.",
ioe); } cells.add(cellScanner.current());
System.out.println("ResponseConverter.getResults(1)--cells == "+ cells + "--j
== " + j + "---noOfCells == " + noOfCells);}results[i] =
Result.create(cells, null, response.getStale(), isPartial);
System.out.println("ResponseConverter.getResults(2)--cells == "+ cells +
"--results[i] == " + results[i] + "---noOfCells == " + noOfCells); } else
{// Result is pure pb.results[i] =
ProtobufUtil.toResult(response.getResults(i));
System.out.println("ResponseConverter.getResults(3)--results == " + results[i]
+ "--i == " + i); }}return results; thank you very much, anoop, do
you want to see this code, the exception should occur on the red mark line.
- 原始邮件 -
发件人："ramkrishna.s.vasudevan (JIRA)"
收件人：issues@hbase.apache.org
主题：[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow
MSLAB closure due to improper ref counting of segments
日期：2016年09月23日 13点25分
[
https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-16643:
---
Status: Open (was: Patch Available)
> Reverse scanner heap creation may not allow MSLAB closure due to improper ref
> counting of segments
> --
>
> Key: HBASE-16643
> URL: https://issues.apache.org/jira/browse/HBASE-16643
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16643.patch, HBASE-16643_1.patch,
> HBASE-16643_2.patch, HBASE-16643_3.patch, HBASE-16643_4.patch,
> HBASE-16643_5.patch
>
>
> In the reverse scanner case,
> While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the
> backward heap, we do a
> {code}
> if ((backwardHeap == null) && (forwardHeap != null)) {
> forwardHeap.close();
> forwardHeap = null;
> // before building the heap seek for the relevant key on the scanners,
> // for the heap to be built from the scanners correctly
> for (KeyValueScanner scan : scanners) {
> if (toLast) {
> res |= scan.seekToLastRow();
> } else {
> res |= scan.backwardSeek(cell);
> }
> }
> {code}
> forwardHeap.close(). This would internally decrement the MSLAB ref counter
> for the current active segment and snapshot segment.
> When the scan is actually closed again we do close() and that will again
> decrement the count. Here chances are there that the count would go negative
> and hence the actual MSLAB closure that checks for refCount==0 will fail.
> Apart from this, when the refCount becomes 0 after the firstClose if any
> other thread requests to close the segment, then we will end up in corrupted
> segment because the segment could be put back to the MSLAB pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[
https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-16643:
---
Status: Open (was: Patch Available)
> Reverse scanner heap creation may not allow MSLAB closure due to improper ref
> counting of segments
> --
>
> Key: HBASE-16643
> URL: https://issues.apache.org/jira/browse/HBASE-16643
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16643.patch, HBASE-16643_1.patch,
> HBASE-16643_2.patch, HBASE-16643_3.patch, HBASE-16643_4.patch,
> HBASE-16643_5.patch
>
>
> In the reverse scanner case,
> While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the
> backward heap, we do a
> {code}
> if ((backwardHeap == null) && (forwardHeap != null)) {
> forwardHeap.close();
> forwardHeap = null;
> // before building the heap seek for the relevant key on the scanners,
> // for the heap to be built from the scanners correctly
> for (KeyValueScanner scan : scanners) {
> if (toLast) {
> res |= scan.seekToLastRow();
> } else {
> res |= scan.backwardSeek(cell);
> }
> }
> {code}
> forwardHeap.close(). This would internally decrement the MSLAB ref counter
> for the current active segment and snapshot segment.
> When the scan is actually closed again we do close() and that will again
> decrement the count. Here chances are there that the count would go negative
> and hence the actual MSLAB closure that checks for refCount==0 will fail.
> Apart from this, when the refCount becomes 0 after the firstClose if any
> other thread requests to close the segment, then we will end up in corrupted
> segment because the segment could be put back to the MSLAB pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[
https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-16643:
---
Hadoop Flags: Reviewed
Status: Patch Available (was: Open)
> Reverse scanner heap creation may not allow MSLAB closure due to improper ref
> counting of segments
> --
>
> Key: HBASE-16643
> URL: https://issues.apache.org/jira/browse/HBASE-16643
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16643.patch, HBASE-16643_1.patch,
> HBASE-16643_2.patch, HBASE-16643_3.patch, HBASE-16643_4.patch,
> HBASE-16643_5.patch
>
>
> In the reverse scanner case,
> While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the
> backward heap, we do a
> {code}
> if ((backwardHeap == null) && (forwardHeap != null)) {
> forwardHeap.close();
> forwardHeap = null;
> // before building the heap seek for the relevant key on the scanners,
> // for the heap to be built from the scanners correctly
> for (KeyValueScanner scan : scanners) {
> if (toLast) {
> res |= scan.seekToLastRow();
> } else {
> res |= scan.backwardSeek(cell);
> }
> }
> {code}
> forwardHeap.close(). This would internally decrement the MSLAB ref counter
> for the current active segment and snapshot segment.
> When the scan is actually closed again we do close() and that will again
> decrement the count. Here chances are there that the count would go negative
> and hence the actual MSLAB closure that checks for refCount==0 will fail.
> Apart from this, when the refCount becomes 0 after the firstClose if any
> other thread requests to close the segment, then we will end up in corrupted
> segment because the segment could be put back to the MSLAB pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[
https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-16643:
---
Attachment: HBASE-16643_5.patch
Updated patch based on RB comments. I have left some replies in the RB for all
the comments and questions. The SegmentScanner change is needed is what I
believe. If that is not needed in this JIRA I can remove it and raise another
JIRA for discussion.
> Reverse scanner heap creation may not allow MSLAB closure due to improper ref
> counting of segments
> --
>
> Key: HBASE-16643
> URL: https://issues.apache.org/jira/browse/HBASE-16643
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16643.patch, HBASE-16643_1.patch,
> HBASE-16643_2.patch, HBASE-16643_3.patch, HBASE-16643_4.patch,
> HBASE-16643_5.patch
>
>
> In the reverse scanner case,
> While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the
> backward heap, we do a
> {code}
> if ((backwardHeap == null) && (forwardHeap != null)) {
> forwardHeap.close();
> forwardHeap = null;
> // before building the heap seek for the relevant key on the scanners,
> // for the heap to be built from the scanners correctly
> for (KeyValueScanner scan : scanners) {
> if (toLast) {
> res |= scan.seekToLastRow();
> } else {
> res |= scan.backwardSeek(cell);
> }
> }
> {code}
> forwardHeap.close(). This would internally decrement the MSLAB ref counter
> for the current active segment and snapshot segment.
> When the scan is actually closed again we do close() and that will again
> decrement the count. Here chances are there that the count would go negative
> and hence the actual MSLAB closure that checks for refCount==0 will fail.
> Apart from this, when the refCount becomes 0 after the firstClose if any
> other thread requests to close the segment, then we will end up in corrupted
> segment because the segment could be put back to the MSLAB pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[
https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-16643:
---
Status: Open (was: Patch Available)
> Reverse scanner heap creation may not allow MSLAB closure due to improper ref
> counting of segments
> --
>
> Key: HBASE-16643
> URL: https://issues.apache.org/jira/browse/HBASE-16643
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16643.patch, HBASE-16643_1.patch,
> HBASE-16643_2.patch, HBASE-16643_3.patch, HBASE-16643_4.patch
>
>
> In the reverse scanner case,
> While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the
> backward heap, we do a
> {code}
> if ((backwardHeap == null) && (forwardHeap != null)) {
> forwardHeap.close();
> forwardHeap = null;
> // before building the heap seek for the relevant key on the scanners,
> // for the heap to be built from the scanners correctly
> for (KeyValueScanner scan : scanners) {
> if (toLast) {
> res |= scan.seekToLastRow();
> } else {
> res |= scan.backwardSeek(cell);
> }
> }
> {code}
> forwardHeap.close(). This would internally decrement the MSLAB ref counter
> for the current active segment and snapshot segment.
> When the scan is actually closed again we do close() and that will again
> decrement the count. Here chances are there that the count would go negative
> and hence the actual MSLAB closure that checks for refCount==0 will fail.
> Apart from this, when the refCount becomes 0 after the firstClose if any
> other thread requests to close the segment, then we will end up in corrupted
> segment because the segment could be put back to the MSLAB pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16680) Reduce garbage in BufferChain
[ https://issues.apache.org/jira/browse/HBASE-16680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515426#comment-15515426 ] Ashish Singhi commented on HBASE-16680: --- My bad :( Thanks for noticing and correcting it. > Reduce garbage in BufferChain > - > > Key: HBASE-16680 > URL: https://issues.apache.org/jira/browse/HBASE-16680 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16680-master.patch, HBASE-16680-master_v2.patch > > > BufferChain accept a List and then convert it to ByteBuffer[], we > can directly produce ByteBuffer[] and handle it to BufferChain, so eliminate > the object List. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16689) Durability == ASYNC_WAL means no SYNC
[ https://issues.apache.org/jira/browse/HBASE-16689?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-16689: -- Description: Setting DURABILITY=ASYNC_WAL on a Table suspends all syncs for all table Table appends. If all tables on a cluster have this setting, data is flushed from the RS to the DN at some arbitrary time and a bunch may just hang out in DFSClient buffers on the RS-side indefinitely if writes are sporadic, at least until there is a WAL roll -- a log roll sends a sync through the write pipeline to flush out any outstanding appends -- or a region close which does similar or we crash and drop the data in buffers RS. This is probably not what a user expects when they set ASYNC_WAL (We don't doc anywhere that I could find clearly what ASYNC_WAL means). Worse, old-time users probably associate ASYNC_WAL and DEFERRED_FLUSH, an old HTableDescriptor config that was deprecated and replaced by ASYNC_WAL. DEFERRED_FLUSH ran a background thread -- LogSyncer -- that on a configurable interval, sent a sync down the write pipeline so any outstanding appends since last last interval start get pushed out to the DN. ASYNC_WAL doesn't do this (see below for history on how we let go of the LogSyncer feature). Of note, we always sync meta edits. You can't turn this off. Also, given WALs are per regionserver, if other regions on the RS are from tables that have sync set, these writes will push out to the DN any appends done on tables that have DEFERRED/ASYNC_WAL set. To fix, we could do a few things: * Simple and comprehensive would be always queuing a sync, even if ASYNC_WAL is set but we let go of Handlers as soon as we write the memstore -- we don't wait on the sync to complete as we do with the default setting of Durability=SYNC_WAL. * Be like a 'real' database and add in a sync after N bytes of data have been appended (configurable) or after M milliseconds have passed, which ever threshold happens first. The size check would be easy. The sync-ever-M-millis would mean another thread. * Doc what ASYNC_WAL means (and other durability options) Let me take a look and report back. Will file a bit of history on how we got here in next comment. was: Setting DURABILITY=ASYNC_WAL on a Table suspends all syncs for all table Table appends. If all tables on a cluster have this setting, data is flushed from the RS to the DN at some arbitrary time and a bunch may just hang out in DFSClient buffers on the RS-side indefinitely if writes are sporadic, at least until there is a WAL roll -- a log roll sends a sync through the write pipeline to flush out any outstanding appends -- or a region close which does similar or we crash and drop the data in buffers RS. This is probably not what a user expects when they set ASYNC_WAL (We don't doc anywhere that I could find clearly what ASYNC_WAL means). Worse, old-time users probably associate ASYNC_WAL and DEFERRED_FLUSH, an old HTableDescriptor config that was deprecated and replaced by ASYNC_WAL. DEFERRED_FLUSH ran a background thread -- LogSyncer -- that on a configurable interval, sent a sync down the write pipeline so any outstanding appends since last last interval start get pushed out to the DN. ASYNC_WAL doesn't do this (see below for history on how we let go of the LogSyncer feature). Of note, we always sync meta edits. You can't turn this off. Also, given WALs are per regionserver, if other regions on the RS are from tables that have sync set, these writes will push out to the DN any appends done on tables that have DEFERRED/ASYNC_WAL set. To fix, we could do a few things: * Simple and comprehensive would be always queuing a sync, even if ASYNC_WAL is set but we let go of Handlers as soon as we write the memstore -- we don't wait on the sync to complete as we do with the default setting of Durability=SYNC_WAL. * Be like a 'real' database and add in a sync after N bytes of data have been appended (configurable) or after M milliseconds have passed, which ever threshold happens first. The size check would be easy. The sync-ever-M-millis would mean another thread. Let me take a look and report back. Will file a bit of history on how we got here in next comment. > Durability == ASYNC_WAL means no SYNC > - > > Key: HBASE-16689 > URL: https://issues.apache.org/jira/browse/HBASE-16689 > Project: HBase > Issue Type: Bug > Components: wal >Affects Versions: 1.0.3, 1.1.6, 1.2.3 >Reporter: stack >Assignee: stack >Priority: Critical > > Setting DURABILITY=ASYNC_WAL on a Table suspends all syncs for all table > Table appends. If all tables on a cluster have this setting, data is flushed > from the RS to the DN at some arbitrary time and a bunch may just hang out in > DFSClient buffers on the RS-side indef
[jira] [Commented] (HBASE-16682) Fix Shell tests failure. NoClassDefFoundError for MiniKdc
[
https://issues.apache.org/jira/browse/HBASE-16682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515386#comment-15515386
]
Appy commented on HBASE-16682:
--
It bother me not knowing why such a thing is happening. I spent 45 min trying
to get it working by playing with deps in hbase-testing-util, but i still don't
understand why hbase-shell isn't getting it transitively.
So am fine with adding it to hbase-shell/pom for now to fix the broken tests.
This change is probably better here because it's linked with HBASE-14734, then
in HBASE-1 which has to do with replication.
Please +1 the v1 patch (QA failure was because of unmodified tests).
> Fix Shell tests failure. NoClassDefFoundError for MiniKdc
> -
>
> Key: HBASE-16682
> URL: https://issues.apache.org/jira/browse/HBASE-16682
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-16682.master.001.patch,
> HBASE-16682.master.002.patch
>
>
> Stacktrace
> {noformat}
> java.lang.NoClassDefFoundError: org/apache/hadoop/minikdc/MiniKdc
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
> at java.lang.Class.getDeclaredMethods(Class.java:1975)
> at org.jruby.javasupport.JavaClass.getMethods(JavaClass.java:2110)
> at org.jruby.javasupport.JavaClass.setupClassMethods(JavaClass.java:955)
> at org.jruby.javasupport.JavaClass.access$700(JavaClass.java:99)
> at
> org.jruby.javasupport.JavaClass$ClassInitializer.initialize(JavaClass.java:650)
> at org.jruby.javasupport.JavaClass.setupProxy(JavaClass.java:689)
> at org.jruby.javasupport.Java.createProxyClass(Java.java:526)
> at org.jruby.javasupport.Java.getProxyClass(Java.java:455)
> at org.jruby.javasupport.Java.getInstance(Java.java:364)
> at
> org.jruby.javasupport.JavaUtil.convertJavaToUsableRubyObject(JavaUtil.java:166)
> at
> org.jruby.javasupport.JavaEmbedUtils.javaToRuby(JavaEmbedUtils.java:291)
> at
> org.jruby.embed.variable.AbstractVariable.updateByJavaObject(AbstractVariable.java:81)
> at
> org.jruby.embed.variable.GlobalVariable.(GlobalVariable.java:69)
> at
> org.jruby.embed.variable.GlobalVariable.getInstance(GlobalVariable.java:60)
> at
> org.jruby.embed.variable.VariableInterceptor.getVariableInstance(VariableInterceptor.java:97)
> at org.jruby.embed.internal.BiVariableMap.put(BiVariableMap.java:321)
> at org.jruby.embed.ScriptingContainer.put(ScriptingContainer.java:1123)
> at
> org.apache.hadoop.hbase.client.AbstractTestShell.setUpBeforeClass(AbstractTestShell.java:61)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:108)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:78)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:54)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:144)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.mave
[jira] [Commented] (HBASE-16689) Durability == ASYNC_WAL means no SYNC
[
https://issues.apache.org/jira/browse/HBASE-16689?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515385#comment-15515385
]
stack commented on HBASE-16689:
---
h1. History
We committed the below in time for 0.98.0:
Author: Michael Stack
Date: Fri Dec 13 17:32:09 2013 +
HBASE-8755 A new write thread model for HLog to improve the overall HBase
write throughput
git-svn-id: https://svn.apache.org/repos/asf/hbase/trunk@1550778
13f79535-47bb-0310-9956-ffa450edef68
It removed the LogSyncer thread because it didn't fit the new model. From
comments in above issue:
+ * 6). No LogSyncer thread any more (since there is always
AsyncWriter/AsyncFlusher threads
+ * do the same job it does)
And from reviews:
{code}
[stack] How does deferred log flush still work when you remove stuff like
optionalFlushInterval? You say '...don't pend on HLog.syncer() waiting for its
txid to be sync-ed' but that is another behavior than what we had here
previously.
===> When say 'still support deferred log flush' I mean for 'deferred log
flush' it can still response write success to client without wait/pend on
syncer(txid),
in this sense, the AsyncWriter/AsyncSyncer do what the previous LogSyncer does
from the point view of the write handler threads: clients don't wait for the
write persist before get reponse success.
{code}
The above got further clarification over in HBASE-10324:
{code}
"By the new write thread model introduced by HBASE-8755, some
deferred-log-flush/Durability API/code/names should be change accordingly:
1. no timer-triggered deferred-log-flush since flush is always done by async
threads, so configuration 'hbase.regionserver.optionallogflushinterval' is no
longer needed
2. the async writer-syncer-notifier threads will always be triggered
implicitly, this semantic is that it always holds that
'hbase.regionserver.optionallogflushinterval' > 0, so deferredLogSyncDisabled
in HRegion.java which affects durability behavior should always be false
3. what HTableDescriptor.isDeferredLogFlush really means is the write can
return without waiting for the sync is done, so the interface name should be
changed to isAsyncLogFlush/setAsyncLogFlush to reflect their real meaning"
{code}
Reading the patch, we just always did sync. There was no support for deferred.
In 1.0.0, a new WAL refactor was brought in by HBASE-10156. It removed all
vestiges of deferred. They weren't working anyways. But it also changed the
model. It added support for durability with a variety of actions dependent on
how durability is set. ASYNC_WAL became a pass through for sync.
> Durability == ASYNC_WAL means no SYNC
> -
>
> Key: HBASE-16689
> URL: https://issues.apache.org/jira/browse/HBASE-16689
> Project: HBase
> Issue Type: Bug
> Components: wal
>Affects Versions: 1.0.3, 1.1.6, 1.2.3
>Reporter: stack
>Assignee: stack
>Priority: Critical
>
> Setting DURABILITY=ASYNC_WAL on a Table suspends all syncs for all table
> Table appends. If all tables on a cluster have this setting, data is flushed
> from the RS to the DN at some arbitrary time and a bunch may just hang out in
> DFSClient buffers on the RS-side indefinitely if writes are sporadic, at
> least until there is a WAL roll -- a log roll sends a sync through the write
> pipeline to flush out any outstanding appends -- or a region close which does
> similar or we crash and drop the data in buffers RS.
> This is probably not what a user expects when they set ASYNC_WAL (We don't
> doc anywhere that I could find clearly what ASYNC_WAL means). Worse, old-time
> users probably associate ASYNC_WAL and DEFERRED_FLUSH, an old
> HTableDescriptor config that was deprecated and replaced by ASYNC_WAL.
> DEFERRED_FLUSH ran a background thread -- LogSyncer -- that on a configurable
> interval, sent a sync down the write pipeline so any outstanding appends
> since last last interval start get pushed out to the DN. ASYNC_WAL doesn't
> do this (see below for history on how we let go of the LogSyncer feature).
> Of note, we always sync meta edits. You can't turn this off. Also, given WALs
> are per regionserver, if other regions on the RS are from tables that have
> sync set, these writes will push out to the DN any appends done on tables
> that have DEFERRED/ASYNC_WAL set.
> To fix, we could do a few things:
> * Simple and comprehensive would be always queuing a sync, even if ASYNC_WAL
> is set but we let go of Handlers as soon as we write the memstore -- we don't
> wait on the sync to complete as we do with the default setting of
> Durability=SYNC_WAL.
> * Be like a 'real' database and add in a sync after N bytes of data have
> been appended (configurable) or after M milliseconds have passed, which ever
> threshold happens f
[jira] [Commented] (HBASE-16671) Split TestExportSnapshot
[ https://issues.apache.org/jira/browse/HBASE-16671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515363#comment-15515363 ] Hudson commented on HBASE-16671: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1656 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1656/]) HBASE-16671 Split TestExportSnapshot (matteo.bertozzi: rev 07ed15598bcc618df74f51c3c3950c543daea788) * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotHelpers.java * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshotNoCluster.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestMobExportSnapshot.java > Split TestExportSnapshot > > > Key: HBASE-16671 > URL: https://issues.apache.org/jira/browse/HBASE-16671 > Project: HBase > Issue Type: Test > Components: snapshots, test >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16671-v0.patch > > > TestExportSnapshot contains 3 type of tests. > - MiniCluster creating a table, taking a snapshot and running export > - Mocked snapshot running export > - tool helpers tests > since now we have everything packed in a single test. 2 and 3 ended up having > a before and after that is creating a table and taking a snapshot which is > not used. Move those tests out and cut some time from the test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16662) Fix open POODLE vulnerabilities
[ https://issues.apache.org/jira/browse/HBASE-16662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515364#comment-15515364 ] Hudson commented on HBASE-16662: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1656 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1656/]) HBASE-16662 Fix open POODLE vulnerabilities (apurtell: rev 4b05f40984f06ccd094b5680177ff760c88f81ea) * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/SslRMIClientSocketFactorySecure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/JMXListener.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/jetty/SslSelectChannelConnectorSecure.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/SslRMIServerSocketFactorySecure.java * (edit) hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java * (edit) hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java > Fix open POODLE vulnerabilities > --- > > Key: HBASE-16662 > URL: https://issues.apache.org/jira/browse/HBASE-16662 > Project: HBase > Issue Type: Bug > Components: REST, Thrift >Reporter: Ben Lau >Assignee: Ben Lau > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 0.98.23, 1.2.4 > > Attachments: HBASE-16662-master.patch > > > We recently found a security issue in our HBase REST servers. The issue is a > variant of the POODLE vulnerability (https://en.wikipedia.org/wiki/POODLE) > and is present in the HBase Thrift server as well. It also appears to affect > the JMXListener coprocessor. The vulnerabilities probably affect all > versions of HBase that have the affected services. (If you don't use the > affected services with SSL then this ticket probably doesn't affect you). > Included is a patch to fix the known POODLE vulnerabilities in master. Let > us know if we missed any. From our end we only personally encountered the > HBase REST vulnerability. We do not use the Thrift server or JMXListener > coprocessor but discovered those problems after discussing the issue with > some of the HBase PMCs. > Coincidentally, Hadoop recently committed a SslSelectChannelConnectorSecure > which is more or less the same as one of the fixes in this patch. Hadoop > wasn't originally affected by the vulnerability in the > SslSelectChannelConnector, but about a month ago they committed HADOOP-12765 > which does use that class, so they added a SslSelectChannelConnectorSecure > class similar to this patch. Since this class is present in Hadoop 2.7.4+ > which hasn't been released yet, we will for now just include our own version > instead of depending on the Hadoop version. > After the patch is approved for master we can backport as necessary to older > versions of HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16684) The get() requests does not see locally buffered put() requests when autoflush is disabled
[ https://issues.apache.org/jira/browse/HBASE-16684?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515361#comment-15515361 ] ramkrishna.s.vasudevan commented on HBASE-16684: Is this a bug or is it a valid case? So in case of puts that gets buffered and the client dies even then those puts are not considered to be successful (as autoflush is false). In that case also any get() won't return those results correct? > The get() requests does not see locally buffered put() requests when > autoflush is disabled > -- > > Key: HBASE-16684 > URL: https://issues.apache.org/jira/browse/HBASE-16684 > Project: HBase > Issue Type: Improvement >Reporter: Haohui Mai >Priority: Minor > > When autoflush is disabled the put() requests are buffered locally. > Subsequent get() requests on the same host will always go to the network and > will not see the updates that are buffered locally. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16689) Durability == ASYNC_WAL means no SYNC
stack created HBASE-16689: - Summary: Durability == ASYNC_WAL means no SYNC Key: HBASE-16689 URL: https://issues.apache.org/jira/browse/HBASE-16689 Project: HBase Issue Type: Bug Components: wal Affects Versions: 1.2.3, 1.1.6, 1.0.3 Reporter: stack Assignee: stack Priority: Critical Setting DURABILITY=ASYNC_WAL on a Table suspends all syncs for all table Table appends. If all tables on a cluster have this setting, data is flushed from the RS to the DN at some arbitrary time and a bunch may just hang out in DFSClient buffers on the RS-side indefinitely if writes are sporadic, at least until there is a WAL roll -- a log roll sends a sync through the write pipeline to flush out any outstanding appends -- or a region close which does similar or we crash and drop the data in buffers RS. This is probably not what a user expects when they set ASYNC_WAL (We don't doc anywhere that I could find clearly what ASYNC_WAL means). Worse, old-time users probably associate ASYNC_WAL and DEFERRED_FLUSH, an old HTableDescriptor config that was deprecated and replaced by ASYNC_WAL. DEFERRED_FLUSH ran a background thread -- LogSyncer -- that on a configurable interval, sent a sync down the write pipeline so any outstanding appends since last last interval start get pushed out to the DN. ASYNC_WAL doesn't do this (see below for history on how we let go of the LogSyncer feature). Of note, we always sync meta edits. You can't turn this off. Also, given WALs are per regionserver, if other regions on the RS are from tables that have sync set, these writes will push out to the DN any appends done on tables that have DEFERRED/ASYNC_WAL set. To fix, we could do a few things: * Simple and comprehensive would be always queuing a sync, even if ASYNC_WAL is set but we let go of Handlers as soon as we write the memstore -- we don't wait on the sync to complete as we do with the default setting of Durability=SYNC_WAL. * Be like a 'real' database and add in a sync after N bytes of data have been appended (configurable) or after M milliseconds have passed, which ever threshold happens first. The size check would be easy. The sync-ever-M-millis would mean another thread. Let me take a look and report back. Will file a bit of history on how we got here in next comment. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16662) Fix open POODLE vulnerabilities
[ https://issues.apache.org/jira/browse/HBASE-16662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515327#comment-15515327 ] Hudson commented on HBASE-16662: FAILURE: Integrated in Jenkins build HBase-1.3-JDK7 #19 (See [https://builds.apache.org/job/HBase-1.3-JDK7/19/]) HBASE-16662 Fix open POODLE vulnerabilities (apurtell: rev 73d4edbfa122c3d6c592a43171bacb61a7c69ca8) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/JMXListener.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/SslRMIClientSocketFactorySecure.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/SslRMIServerSocketFactorySecure.java * (edit) hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java * (edit) hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/jetty/SslSelectChannelConnectorSecure.java > Fix open POODLE vulnerabilities > --- > > Key: HBASE-16662 > URL: https://issues.apache.org/jira/browse/HBASE-16662 > Project: HBase > Issue Type: Bug > Components: REST, Thrift >Reporter: Ben Lau >Assignee: Ben Lau > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 0.98.23, 1.2.4 > > Attachments: HBASE-16662-master.patch > > > We recently found a security issue in our HBase REST servers. The issue is a > variant of the POODLE vulnerability (https://en.wikipedia.org/wiki/POODLE) > and is present in the HBase Thrift server as well. It also appears to affect > the JMXListener coprocessor. The vulnerabilities probably affect all > versions of HBase that have the affected services. (If you don't use the > affected services with SSL then this ticket probably doesn't affect you). > Included is a patch to fix the known POODLE vulnerabilities in master. Let > us know if we missed any. From our end we only personally encountered the > HBase REST vulnerability. We do not use the Thrift server or JMXListener > coprocessor but discovered those problems after discussing the issue with > some of the HBase PMCs. > Coincidentally, Hadoop recently committed a SslSelectChannelConnectorSecure > which is more or less the same as one of the fixes in this patch. Hadoop > wasn't originally affected by the vulnerability in the > SslSelectChannelConnector, but about a month ago they committed HADOOP-12765 > which does use that class, so they added a SslSelectChannelConnectorSecure > class similar to this patch. Since this class is present in Hadoop 2.7.4+ > which hasn't been released yet, we will for now just include our own version > instead of depending on the Hadoop version. > After the patch is approved for master we can backport as necessary to older > versions of HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16423) Add re-compare option to VerifyReplication to avoid occasional inconsistent rows
[ https://issues.apache.org/jira/browse/HBASE-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16423: --- Assignee: Jianwei Cui Hadoop Flags: Reviewed Fix Version/s: 1.4.0 2.0.0 Test failures were not related to patch. > Add re-compare option to VerifyReplication to avoid occasional inconsistent > rows > > > Key: HBASE-16423 > URL: https://issues.apache.org/jira/browse/HBASE-16423 > Project: HBase > Issue Type: Improvement > Components: Replication >Affects Versions: 2.0.0 >Reporter: Jianwei Cui >Assignee: Jianwei Cui >Priority: Minor > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16423-v1.patch, HBASE-16423-v2.patch > > > Because replication keeps eventually consistency, VerifyReplication may > report inconsistent rows if there are data being written to source or peer > clusters during scanning. These occasionally inconsistent rows will have the > same data if we do the comparison again after a short period. It is not easy > to find the really inconsistent rows if VerifyReplication report a large > number of such occasionally inconsistency. To avoid this case, we can add an > option to make VerifyReplication read out the inconsistent rows again after > sleeping a few seconds and re-compare the rows during scanning. This behavior > follows the eventually consistency of hbase's replication. Suggestions and > discussions are welcomed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16604) Scanner retries on IOException can cause the scans to miss data
[
https://issues.apache.org/jira/browse/HBASE-16604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515306#comment-15515306
]
Hudson commented on HBASE-16604:
FAILURE: Integrated in Jenkins build HBase-1.3-JDK8 #21 (See
[https://builds.apache.org/job/HBase-1.3-JDK8/21/])
HBASE-16604 Scanner retries on IOException can cause the scans to miss (enis:
rev d600e8b70e10281ec19e3316ca0fd461d824a018)
* (edit)
hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSource.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java
* (add)
hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/ScannerResetException.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceBase.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServer.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduce.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TableSnapshotInputFormatTestBase.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestTableSnapshotScanner.java
* (edit)
hbase-hadoop2-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSourceImpl.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultithreadedTableMapper.java
* (edit)
hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java
* (edit)
hbase-client/src/main/java/org/apache/hadoop/hbase/UnknownScannerException.java
* (add)
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/DelegatingKeyValueScanner.java
> Scanner retries on IOException can cause the scans to miss data
>
>
> Key: HBASE-16604
> URL: https://issues.apache.org/jira/browse/HBASE-16604
> Project: HBase
> Issue Type: Bug
> Components: regionserver, Scanners
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: hbase-16604_v1.patch, hbase-16604_v2.patch,
> hbase-16604_v3.branch-1.patch, hbase-16604_v3.patch
>
>
> Debugging an ITBLL failure, where the Verify did not "see" all the data in
> the cluster, I've noticed that if we end up getting a generic IOException
> from the HFileReader level, we may end up missing the rest of the data in the
> region. I was able to manually test this, and this stack trace helps to
> understand what is going on:
> {code}
> 2016-09-09 16:27:15,633 INFO [hconnection-0x71ad3d8a-shared--pool21-t9]
> client.ScannerCallable(376): Open scanner=1 for
> scan={"loadColumnFamiliesOnDemand":null,"startRow":"","stopRow":"","batch":-1,"cacheBlocks":true,"totalColumns":1,"maxResultSize":2097152,"families":{"testFamily":["testFamily"]},"caching":100,"maxVersions":1,"timeRange":[0,9223372036854775807]}
> on region
> region=testScanThrowsException,,1473463632707.b2adfb618e5d0fe225c1dc40c0eabfee.,
> hostname=hw10676,51833,1473463626529, seqNum=2
> 2016-09-09 16:27:15,634 INFO
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833]
> regionserver.RSRpcServices(2196): scan request:scanner_id: 1 number_of_rows:
> 100 close_scanner: false next_call_seq: 0 client_handles_partials: true
> client_handles_heartbeats: true renew: false
> 2016-09-09 16:27:15,635 INFO
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833]
> regionserver.RSRpcServices(2510): Rolling back next call seqId
> 2016-09-09 16:27:15,635 INFO
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833]
> regionserver.RSRpcServices(2565): Throwing new
> ServiceExceptionjava.io.IOException: Could not reseek
> StoreFileScanner[HFileScanner for reader
> reader=hdfs://localhost:51795/user/enis/test-data/d6fb1c70-93c1-4099-acb7-5723fc05a737/data/default/testScanThrowsException/b2adfb618e5d0fe225c1dc40c0eabfee/testFamily/5a213cc23b714e5e8e1a140ebbe72f2c,
> compression=none, cacheConf=blockCache=LruBlockCache{blockCount=0,
> currentSize=1567264, freeSize=1525578848, maxSize=1527146112,
> heapSize=1567264, minSize=1450788736, minFactor=0.95, multiSize=725394368,
> multiFactor=0.5, singleSize=362697184, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false, firstKey=aaa/testFamily:testFamily/1473463633859/Put,
> lastKey=zzz/testFamily:testFamily/1473463634271/Put, avgKeyLen=35,
> avgValueLen=3, entries=17576, le
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515277#comment-15515277 ] Duo Zhang commented on HBASE-15921: --- And I think the refactoring work on zk could be done in another issue, and also we should fix the findbugs warnings. We should not write to a static field in a non-static method. > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Jurriaan Mous >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15921-v2.patch, HBASE-15921.demo.patch, > HBASE-15921.patch, HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15921) Add first AsyncTable impl and create TableImpl based on it
[ https://issues.apache.org/jira/browse/HBASE-15921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515258#comment-15515258 ] Duo Zhang commented on HBASE-15921: --- Let's talk about the retry and timeout logic. In general, I think we should only do retry at one place, which means, we always restart from the beginning. We can config different timeout values for different stages, but the actual timeout when executing the operation will also be limited by the whole operation timeout config. This could simplify our logic. For example, we do not need to implement retry logic in rpc(Yeah I have just removed the reconnect logic in NettyRpcConnection). And it is also much easier for us to implement the backoff logic. What do you think? [~stack] [~carp84] [~chenheng]. Thanks. > Add first AsyncTable impl and create TableImpl based on it > -- > > Key: HBASE-15921 > URL: https://issues.apache.org/jira/browse/HBASE-15921 > Project: HBase > Issue Type: Improvement >Affects Versions: 2.0.0 >Reporter: Jurriaan Mous >Assignee: Duo Zhang > Fix For: 2.0.0 > > Attachments: HBASE-15921-v2.patch, HBASE-15921.demo.patch, > HBASE-15921.patch, HBASE-15921.v1.patch > > > First we create an AsyncTable interface with implementation without the Scan > functionality. Those will land in a separate patch since they need a refactor > of existing scans. > Also added is a new TableImpl to replace HTable. It uses the AsyncTableImpl > internally and should be a bit faster because it does jump through less hoops > to do ProtoBuf transportation. This way we can run all existing tests on the > AsyncTableImpl to guarantee its quality. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16677) Add table size (total store file size) to table page
[
https://issues.apache.org/jira/browse/HBASE-16677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515249#comment-15515249
]
Hadoop QA commented on HBASE-16677:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 14s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
14s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 30s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
53s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
28m 49s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 29s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 98m 24s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
17s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 133m 43s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestBlockEvictionFromClient |
| Timed out junit tests |
org.apache.hadoop.hbase.master.procedure.TestCreateTableProcedure |
| | org.apache.hadoop.hbase.master.procedure.TestMasterFailoverWithProcedures
|
| | org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache |
| |
org.apache.hadoop.hbase.replication.regionserver.TestReplicationWALReaderManager
|
| | org.apache.hadoop.hbase.master.procedure.TestRestoreSnapshotProcedure |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829966/HBASE-16677_v3.patch |
| JIRA Issue | HBASE-16677 |
| Optional Tests | asflicense javac javadoc unit |
| uname | Linux 589aeb7283bf 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 07ed155 |
| Default Java | 1.8.0_101 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3676/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3676/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3676/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3676/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Add table size (total store file size) to table page
>
>
> Key: HBASE-16677
> URL: https://issues.apache.org/jira/browse/HBASE-16677
> Project: HBase
> Issue Type: New Feature
> Components: website
>Reporter: Guang Yang
>Priority: Minor
> Attachments: HBASE-16677_v0.patch, HBASE-16677_v1.patch,
> HBA
[jira] [Commented] (HBASE-16677) Add table size (total store file size) to table page
[
https://issues.apache.org/jira/browse/HBASE-16677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515236#comment-15515236
]
Heng Chen commented on HBASE-16677:
---
+1 for patch v3, Not see regions hidden in your upload pic?
{code}
+ This table has <%= numRegions %> regions in total, in order to
improve the page load time,
+ only <%= numRegionsRendered %> regions are displayed here, click
+ here to see all regions.
+<% } %>
{code}
I mean this hit in your pic.
> Add table size (total store file size) to table page
>
>
> Key: HBASE-16677
> URL: https://issues.apache.org/jira/browse/HBASE-16677
> Project: HBase
> Issue Type: New Feature
> Components: website
>Reporter: Guang Yang
>Priority: Minor
> Attachments: HBASE-16677_v0.patch, HBASE-16677_v1.patch,
> HBASE-16677_v2.patch, HBASE-16677_v3.patch, mini_cluster_master.png,
> prod_cluster_partial.png
>
>
> Currently there is not an easy way to get the table size from the web UI,
> though we have the region size on the page, it is still convenient to have a
> table for the table size stat.
> Another pain point is that when the table grow large with tens of thousands
> of regions, it took extremely long time to load the page, however, sometimes
> we don't want to check all the regions. An optimization could be to accept a
> query parameter to specify the number of regions to render.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16680) Reduce garbage in BufferChain
[
https://issues.apache.org/jira/browse/HBASE-16680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515224#comment-15515224
]
Yu Li commented on HBASE-16680:
---
[~ashish singhi] I happen to find the typo of JIRA number in commit message
(should be 16680 rather than 16880) , so I just revert and resubmitted it, FYI
:-)
{noformat}
commit ce493642c0e295a08701cdcfe3ddc6755cdd7718
Author: Ashish Singhi
Date: Thu Sep 22 13:59:18 2016 +0530
HBASE-16880 Reduce garbage in BufferChain (binlijin)
{noformat}
> Reduce garbage in BufferChain
> -
>
> Key: HBASE-16680
> URL: https://issues.apache.org/jira/browse/HBASE-16680
> Project: HBase
> Issue Type: Improvement
>Reporter: binlijin
>Assignee: binlijin
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16680-master.patch, HBASE-16680-master_v2.patch
>
>
> BufferChain accept a List and then convert it to ByteBuffer[], we
> can directly produce ByteBuffer[] and handle it to BufferChain, so eliminate
> the object List.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16682) Fix Shell tests failure. NoClassDefFoundError for MiniKdc
[
https://issues.apache.org/jira/browse/HBASE-16682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515178#comment-15515178
]
Guanghao Zhang commented on HBASE-16682:
After HBASE-14734, HBaseTestingUtility use MiniKdc class and it is used in
hbase-rsgroup and hbase-thrift package, too. But the unit tests in
hbase-rsgroup and hbase-thrift didn't have this problem. So it is not a common
problem and maybe we can only fix it in hbase-shell pom?
> Fix Shell tests failure. NoClassDefFoundError for MiniKdc
> -
>
> Key: HBASE-16682
> URL: https://issues.apache.org/jira/browse/HBASE-16682
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-16682.master.001.patch,
> HBASE-16682.master.002.patch
>
>
> Stacktrace
> {noformat}
> java.lang.NoClassDefFoundError: org/apache/hadoop/minikdc/MiniKdc
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
> at java.lang.Class.getDeclaredMethods(Class.java:1975)
> at org.jruby.javasupport.JavaClass.getMethods(JavaClass.java:2110)
> at org.jruby.javasupport.JavaClass.setupClassMethods(JavaClass.java:955)
> at org.jruby.javasupport.JavaClass.access$700(JavaClass.java:99)
> at
> org.jruby.javasupport.JavaClass$ClassInitializer.initialize(JavaClass.java:650)
> at org.jruby.javasupport.JavaClass.setupProxy(JavaClass.java:689)
> at org.jruby.javasupport.Java.createProxyClass(Java.java:526)
> at org.jruby.javasupport.Java.getProxyClass(Java.java:455)
> at org.jruby.javasupport.Java.getInstance(Java.java:364)
> at
> org.jruby.javasupport.JavaUtil.convertJavaToUsableRubyObject(JavaUtil.java:166)
> at
> org.jruby.javasupport.JavaEmbedUtils.javaToRuby(JavaEmbedUtils.java:291)
> at
> org.jruby.embed.variable.AbstractVariable.updateByJavaObject(AbstractVariable.java:81)
> at
> org.jruby.embed.variable.GlobalVariable.(GlobalVariable.java:69)
> at
> org.jruby.embed.variable.GlobalVariable.getInstance(GlobalVariable.java:60)
> at
> org.jruby.embed.variable.VariableInterceptor.getVariableInstance(VariableInterceptor.java:97)
> at org.jruby.embed.internal.BiVariableMap.put(BiVariableMap.java:321)
> at org.jruby.embed.ScriptingContainer.put(ScriptingContainer.java:1123)
> at
> org.apache.hadoop.hbase.client.AbstractTestShell.setUpBeforeClass(AbstractTestShell.java:61)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:108)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:78)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:54)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:144)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Caused by: java.lang.ClassNot
[jira] [Commented] (HBASE-16662) Fix open POODLE vulnerabilities
[ https://issues.apache.org/jira/browse/HBASE-16662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515164#comment-15515164 ] Hudson commented on HBASE-16662: FAILURE: Integrated in Jenkins build HBase-1.4 #426 (See [https://builds.apache.org/job/HBase-1.4/426/]) HBASE-16662 Fix open POODLE vulnerabilities (apurtell: rev 69733040263d48a195d179c2390325a69416200f) * (edit) hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/jetty/SslSelectChannelConnectorSecure.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/SslRMIClientSocketFactorySecure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/JMXListener.java * (edit) hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/SslRMIServerSocketFactorySecure.java > Fix open POODLE vulnerabilities > --- > > Key: HBASE-16662 > URL: https://issues.apache.org/jira/browse/HBASE-16662 > Project: HBase > Issue Type: Bug > Components: REST, Thrift >Reporter: Ben Lau >Assignee: Ben Lau > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 0.98.23, 1.2.4 > > Attachments: HBASE-16662-master.patch > > > We recently found a security issue in our HBase REST servers. The issue is a > variant of the POODLE vulnerability (https://en.wikipedia.org/wiki/POODLE) > and is present in the HBase Thrift server as well. It also appears to affect > the JMXListener coprocessor. The vulnerabilities probably affect all > versions of HBase that have the affected services. (If you don't use the > affected services with SSL then this ticket probably doesn't affect you). > Included is a patch to fix the known POODLE vulnerabilities in master. Let > us know if we missed any. From our end we only personally encountered the > HBase REST vulnerability. We do not use the Thrift server or JMXListener > coprocessor but discovered those problems after discussing the issue with > some of the HBase PMCs. > Coincidentally, Hadoop recently committed a SslSelectChannelConnectorSecure > which is more or less the same as one of the fixes in this patch. Hadoop > wasn't originally affected by the vulnerability in the > SslSelectChannelConnector, but about a month ago they committed HADOOP-12765 > which does use that class, so they added a SslSelectChannelConnectorSecure > class similar to this patch. Since this class is present in Hadoop 2.7.4+ > which hasn't been released yet, we will for now just include our own version > instead of depending on the Hadoop version. > After the patch is approved for master we can backport as necessary to older > versions of HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15315) Remove always set super user call as high priority
[ https://issues.apache.org/jira/browse/HBASE-15315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515149#comment-15515149 ] Hudson commented on HBASE-15315: FAILURE: Integrated in Jenkins build HBase-1.2-JDK7 #30 (See [https://builds.apache.org/job/HBase-1.2-JDK7/30/]) HBASE-15315 Remove always set super user call as high priority (Yong (apurtell: rev 32a7f2c4b80b4d5f05889f1669aa1538450b6d7a) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AnnotationReadingPriorityFunction.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestPriorityRpc.java > Remove always set super user call as high priority > -- > > Key: HBASE-15315 > URL: https://issues.apache.org/jira/browse/HBASE-15315 > Project: HBase > Issue Type: Improvement >Reporter: Yong Zhang >Assignee: Yong Zhang > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4 > > Attachments: HBASE-15315.001.patch > > > Current implementation set superuser call as ADMIN_QOS, but we have many > customers use superuser to do normal table operation such as put/get data and > so on. If client put much data during region assignment, RPC from HMaster may > timeout because of no handle. so it is better to remove always set super user > call as high priority. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16662) Fix open POODLE vulnerabilities
[ https://issues.apache.org/jira/browse/HBASE-16662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515150#comment-15515150 ] Hudson commented on HBASE-16662: FAILURE: Integrated in Jenkins build HBase-1.2-JDK7 #30 (See [https://builds.apache.org/job/HBase-1.2-JDK7/30/]) HBASE-16662 Fix open POODLE vulnerabilities (apurtell: rev e382b2c9f48cd896d525025c3965fa252f344e08) * (edit) hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/SslRMIServerSocketFactorySecure.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/jetty/SslSelectChannelConnectorSecure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/JMXListener.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/SslRMIClientSocketFactorySecure.java * (edit) hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java > Fix open POODLE vulnerabilities > --- > > Key: HBASE-16662 > URL: https://issues.apache.org/jira/browse/HBASE-16662 > Project: HBase > Issue Type: Bug > Components: REST, Thrift >Reporter: Ben Lau >Assignee: Ben Lau > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 0.98.23, 1.2.4 > > Attachments: HBASE-16662-master.patch > > > We recently found a security issue in our HBase REST servers. The issue is a > variant of the POODLE vulnerability (https://en.wikipedia.org/wiki/POODLE) > and is present in the HBase Thrift server as well. It also appears to affect > the JMXListener coprocessor. The vulnerabilities probably affect all > versions of HBase that have the affected services. (If you don't use the > affected services with SSL then this ticket probably doesn't affect you). > Included is a patch to fix the known POODLE vulnerabilities in master. Let > us know if we missed any. From our end we only personally encountered the > HBase REST vulnerability. We do not use the Thrift server or JMXListener > coprocessor but discovered those problems after discussing the issue with > some of the HBase PMCs. > Coincidentally, Hadoop recently committed a SslSelectChannelConnectorSecure > which is more or less the same as one of the fixes in this patch. Hadoop > wasn't originally affected by the vulnerability in the > SslSelectChannelConnector, but about a month ago they committed HADOOP-12765 > which does use that class, so they added a SslSelectChannelConnectorSecure > class similar to this patch. Since this class is present in Hadoop 2.7.4+ > which hasn't been released yet, we will for now just include our own version > instead of depending on the Hadoop version. > After the patch is approved for master we can backport as necessary to older > versions of HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14123) HBase Backup/Restore Phase 2
[ https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14123: --- Attachment: 14123-master.v25.txt Patch v25 syncs up to commit 9952b134fc26b3e4745b2a93afaddb12565975a7 > HBase Backup/Restore Phase 2 > > > Key: HBASE-14123 > URL: https://issues.apache.org/jira/browse/HBASE-14123 > Project: HBase > Issue Type: Umbrella >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: 14123-master.v14.txt, 14123-master.v15.txt, > 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt, > 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt, > 14123-master.v21.txt, 14123-master.v24.txt, 14123-master.v25.txt, > 14123-master.v3.txt, 14123-master.v5.txt, 14123-master.v6.txt, > 14123-master.v7.txt, 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt, > HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch, > HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch, > HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch, > HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch, > HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch, > HBASE-14123-v7.patch, HBASE-14123-v9.patch > > > Phase 2 umbrella JIRA. See HBASE-7912 for design document and description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[
https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515119#comment-15515119
]
Ted Yu commented on HBASE-14123:
Ran the following tests with patch v24 locally:
{code}
Running org.apache.hadoop.hbase.regionserver.TestCompactionState
Tests run: 5, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 89.159 sec - in
org.apache.hadoop.hbase.regionserver.TestCompactionState
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m;
support was removed in 8.0
Running
org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoadWithOldClient
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 102.288 sec -
in org.apache.hadoop.hbase.regionserver.TestHRegionServerBulkLoadWithOldClient
Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m;
support was removed in 8.0
Running org.apache.hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush
Tests run: 104, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 79.693 sec -
in org.apache.hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush
Running org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover
Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 88.745 sec - in
org.apache.hadoop.hbase.regionserver.TestRegionReplicaFailover
{code}
> HBase Backup/Restore Phase 2
>
>
> Key: HBASE-14123
> URL: https://issues.apache.org/jira/browse/HBASE-14123
> Project: HBase
> Issue Type: Umbrella
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: 14123-master.v14.txt, 14123-master.v15.txt,
> 14123-master.v16.txt, 14123-master.v17.txt, 14123-master.v18.txt,
> 14123-master.v19.txt, 14123-master.v2.txt, 14123-master.v20.txt,
> 14123-master.v21.txt, 14123-master.v24.txt, 14123-master.v3.txt,
> 14123-master.v5.txt, 14123-master.v6.txt, 14123-master.v7.txt,
> 14123-master.v8.txt, 14123-master.v9.txt, 14123-v14.txt,
> HBASE-14123-for-7912-v1.patch, HBASE-14123-for-7912-v6.patch,
> HBASE-14123-v1.patch, HBASE-14123-v10.patch, HBASE-14123-v11.patch,
> HBASE-14123-v12.patch, HBASE-14123-v13.patch, HBASE-14123-v15.patch,
> HBASE-14123-v16.patch, HBASE-14123-v2.patch, HBASE-14123-v3.patch,
> HBASE-14123-v4.patch, HBASE-14123-v5.patch, HBASE-14123-v6.patch,
> HBASE-14123-v7.patch, HBASE-14123-v9.patch
>
>
> Phase 2 umbrella JIRA. See HBASE-7912 for design document and description.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16678) MapReduce jobs do not update counters from ScanMetrics
[
https://issues.apache.org/jira/browse/HBASE-16678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-16678:
--
Status: Patch Available (was: Open)
> MapReduce jobs do not update counters from ScanMetrics
> --
>
> Key: HBASE-16678
> URL: https://issues.apache.org/jira/browse/HBASE-16678
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: hbase-16678_v1.patch
>
>
> Was inspecting a perf issue, where we needed the scanner metrics as counters
> for a MR job. Turns out that the HBase scan counters are no longer working in
> 1.0+. I think it got broken via HBASE-13030.
> These are the counters:
> {code}
> HBase Counters
> BYTES_IN_REMOTE_RESULTS=0
> BYTES_IN_RESULTS=280
> MILLIS_BETWEEN_NEXTS=11
> NOT_SERVING_REGION_EXCEPTION=0
> NUM_SCANNER_RESTARTS=0
> NUM_SCAN_RESULTS_STALE=0
> REGIONS_SCANNED=1
> REMOTE_RPC_CALLS=0
> REMOTE_RPC_RETRIES=0
> RPC_CALLS=3
> RPC_RETRIES=0
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16678) MapReduce jobs do not update counters from ScanMetrics
[
https://issues.apache.org/jira/browse/HBASE-16678?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-16678:
--
Attachment: hbase-16678_v1.patch
v1 patch.
> MapReduce jobs do not update counters from ScanMetrics
> --
>
> Key: HBASE-16678
> URL: https://issues.apache.org/jira/browse/HBASE-16678
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: hbase-16678_v1.patch
>
>
> Was inspecting a perf issue, where we needed the scanner metrics as counters
> for a MR job. Turns out that the HBase scan counters are no longer working in
> 1.0+. I think it got broken via HBASE-13030.
> These are the counters:
> {code}
> HBase Counters
> BYTES_IN_REMOTE_RESULTS=0
> BYTES_IN_RESULTS=280
> MILLIS_BETWEEN_NEXTS=11
> NOT_SERVING_REGION_EXCEPTION=0
> NUM_SCANNER_RESTARTS=0
> NUM_SCAN_RESULTS_STALE=0
> REGIONS_SCANNED=1
> REMOTE_RPC_CALLS=0
> REMOTE_RPC_RETRIES=0
> RPC_CALLS=3
> RPC_RETRIES=0
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16604) Scanner retries on IOException can cause the scans to miss data
[
https://issues.apache.org/jira/browse/HBASE-16604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Enis Soztutar updated HBASE-16604:
--
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Pushed to rest of 1.1+. Test failures are flaky.
> Scanner retries on IOException can cause the scans to miss data
>
>
> Key: HBASE-16604
> URL: https://issues.apache.org/jira/browse/HBASE-16604
> Project: HBase
> Issue Type: Bug
> Components: regionserver, Scanners
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: hbase-16604_v1.patch, hbase-16604_v2.patch,
> hbase-16604_v3.branch-1.patch, hbase-16604_v3.patch
>
>
> Debugging an ITBLL failure, where the Verify did not "see" all the data in
> the cluster, I've noticed that if we end up getting a generic IOException
> from the HFileReader level, we may end up missing the rest of the data in the
> region. I was able to manually test this, and this stack trace helps to
> understand what is going on:
> {code}
> 2016-09-09 16:27:15,633 INFO [hconnection-0x71ad3d8a-shared--pool21-t9]
> client.ScannerCallable(376): Open scanner=1 for
> scan={"loadColumnFamiliesOnDemand":null,"startRow":"","stopRow":"","batch":-1,"cacheBlocks":true,"totalColumns":1,"maxResultSize":2097152,"families":{"testFamily":["testFamily"]},"caching":100,"maxVersions":1,"timeRange":[0,9223372036854775807]}
> on region
> region=testScanThrowsException,,1473463632707.b2adfb618e5d0fe225c1dc40c0eabfee.,
> hostname=hw10676,51833,1473463626529, seqNum=2
> 2016-09-09 16:27:15,634 INFO
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833]
> regionserver.RSRpcServices(2196): scan request:scanner_id: 1 number_of_rows:
> 100 close_scanner: false next_call_seq: 0 client_handles_partials: true
> client_handles_heartbeats: true renew: false
> 2016-09-09 16:27:15,635 INFO
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833]
> regionserver.RSRpcServices(2510): Rolling back next call seqId
> 2016-09-09 16:27:15,635 INFO
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833]
> regionserver.RSRpcServices(2565): Throwing new
> ServiceExceptionjava.io.IOException: Could not reseek
> StoreFileScanner[HFileScanner for reader
> reader=hdfs://localhost:51795/user/enis/test-data/d6fb1c70-93c1-4099-acb7-5723fc05a737/data/default/testScanThrowsException/b2adfb618e5d0fe225c1dc40c0eabfee/testFamily/5a213cc23b714e5e8e1a140ebbe72f2c,
> compression=none, cacheConf=blockCache=LruBlockCache{blockCount=0,
> currentSize=1567264, freeSize=1525578848, maxSize=1527146112,
> heapSize=1567264, minSize=1450788736, minFactor=0.95, multiSize=725394368,
> multiFactor=0.5, singleSize=362697184, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false, firstKey=aaa/testFamily:testFamily/1473463633859/Put,
> lastKey=zzz/testFamily:testFamily/1473463634271/Put, avgKeyLen=35,
> avgValueLen=3, entries=17576, length=866998,
> cur=/testFamily:/OLDEST_TIMESTAMP/Minimum/vlen=0/seqid=0] to key
> /testFamily:testFamily/LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0
> 2016-09-09 16:27:15,635 DEBUG
> [B.fifo.QRpcServer.handler=5,queue=0,port=51833] ipc.CallRunner(110):
> B.fifo.QRpcServer.handler=5,queue=0,port=51833: callId: 26 service:
> ClientService methodName: Scan size: 26 connection: 192.168.42.75:51903
> java.io.IOException: Could not reseek StoreFileScanner[HFileScanner for
> reader
> reader=hdfs://localhost:51795/user/enis/test-data/d6fb1c70-93c1-4099-acb7-5723fc05a737/data/default/testScanThrowsException/b2adfb618e5d0fe225c1dc40c0eabfee/testFamily/5a213cc23b714e5e8e1a140ebbe72f2c,
> compression=none, cacheConf=blockCache=LruBlockCache{blockCount=0,
> currentSize=1567264, freeSize=1525578848, maxSize=1527146112,
> heapSize=1567264, minSize=1450788736, minFactor=0.95, multiSize=725394368,
> multiFactor=0.5, singleSize=362697184, singleFactor=0.25},
> cacheDataOnRead=true, cacheDataOnWrite=false, cacheIndexesOnWrite=false,
> cacheBloomsOnWrite=false, cacheEvictOnClose=false, cacheDataCompressed=false,
> prefetchOnOpen=false, firstKey=aaa/testFamily:testFamily/1473463633859/Put,
> lastKey=zzz/testFamily:testFamily/1473463634271/Put, avgKeyLen=35,
> avgValueLen=3, entries=17576, length=866998,
> cur=/testFamily:/OLDEST_TIMESTAMP/Minimum/vlen=0/seqid=0] to key
> /testFamily:testFamily/LATEST_TIMESTAMP/Maximum/vlen=0/seqid=0
> at
> org.apache.hadoop.hbase.regionserver.StoreFileScanner.reseek(StoreFileScanner.java:224)
> at
> org.apache.hadoop.hbase.regionserver.NonLazyKeyValueScanner.doRealSeek(NonLazyKeyValueScanner.java:55)
>
[jira] [Commented] (HBASE-16662) Fix open POODLE vulnerabilities
[ https://issues.apache.org/jira/browse/HBASE-16662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515046#comment-15515046 ] Hudson commented on HBASE-16662: FAILURE: Integrated in Jenkins build HBase-1.3-JDK8 #20 (See [https://builds.apache.org/job/HBase-1.3-JDK8/20/]) HBASE-16662 Fix open POODLE vulnerabilities (apurtell: rev 73d4edbfa122c3d6c592a43171bacb61a7c69ca8) * (edit) hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java * (edit) hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/SslRMIServerSocketFactorySecure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/JMXListener.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/SslRMIClientSocketFactorySecure.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/jetty/SslSelectChannelConnectorSecure.java > Fix open POODLE vulnerabilities > --- > > Key: HBASE-16662 > URL: https://issues.apache.org/jira/browse/HBASE-16662 > Project: HBase > Issue Type: Bug > Components: REST, Thrift >Reporter: Ben Lau >Assignee: Ben Lau > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 0.98.23, 1.2.4 > > Attachments: HBASE-16662-master.patch > > > We recently found a security issue in our HBase REST servers. The issue is a > variant of the POODLE vulnerability (https://en.wikipedia.org/wiki/POODLE) > and is present in the HBase Thrift server as well. It also appears to affect > the JMXListener coprocessor. The vulnerabilities probably affect all > versions of HBase that have the affected services. (If you don't use the > affected services with SSL then this ticket probably doesn't affect you). > Included is a patch to fix the known POODLE vulnerabilities in master. Let > us know if we missed any. From our end we only personally encountered the > HBase REST vulnerability. We do not use the Thrift server or JMXListener > coprocessor but discovered those problems after discussing the issue with > some of the HBase PMCs. > Coincidentally, Hadoop recently committed a SslSelectChannelConnectorSecure > which is more or less the same as one of the fixes in this patch. Hadoop > wasn't originally affected by the vulnerability in the > SslSelectChannelConnector, but about a month ago they committed HADOOP-12765 > which does use that class, so they added a SslSelectChannelConnectorSecure > class similar to this patch. Since this class is present in Hadoop 2.7.4+ > which hasn't been released yet, we will for now just include our own version > instead of depending on the Hadoop version. > After the patch is approved for master we can backport as necessary to older > versions of HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16662) Fix open POODLE vulnerabilities
[ https://issues.apache.org/jira/browse/HBASE-16662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515037#comment-15515037 ] Hudson commented on HBASE-16662: FAILURE: Integrated in Jenkins build HBase-1.2-JDK8 #27 (See [https://builds.apache.org/job/HBase-1.2-JDK8/27/]) HBASE-16662 Fix open POODLE vulnerabilities (apurtell: rev e382b2c9f48cd896d525025c3965fa252f344e08) * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/JMXListener.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/SslRMIServerSocketFactorySecure.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/SslRMIClientSocketFactorySecure.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/jetty/SslSelectChannelConnectorSecure.java * (edit) hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java * (edit) hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java > Fix open POODLE vulnerabilities > --- > > Key: HBASE-16662 > URL: https://issues.apache.org/jira/browse/HBASE-16662 > Project: HBase > Issue Type: Bug > Components: REST, Thrift >Reporter: Ben Lau >Assignee: Ben Lau > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 0.98.23, 1.2.4 > > Attachments: HBASE-16662-master.patch > > > We recently found a security issue in our HBase REST servers. The issue is a > variant of the POODLE vulnerability (https://en.wikipedia.org/wiki/POODLE) > and is present in the HBase Thrift server as well. It also appears to affect > the JMXListener coprocessor. The vulnerabilities probably affect all > versions of HBase that have the affected services. (If you don't use the > affected services with SSL then this ticket probably doesn't affect you). > Included is a patch to fix the known POODLE vulnerabilities in master. Let > us know if we missed any. From our end we only personally encountered the > HBase REST vulnerability. We do not use the Thrift server or JMXListener > coprocessor but discovered those problems after discussing the issue with > some of the HBase PMCs. > Coincidentally, Hadoop recently committed a SslSelectChannelConnectorSecure > which is more or less the same as one of the fixes in this patch. Hadoop > wasn't originally affected by the vulnerability in the > SslSelectChannelConnector, but about a month ago they committed HADOOP-12765 > which does use that class, so they added a SslSelectChannelConnectorSecure > class similar to this patch. Since this class is present in Hadoop 2.7.4+ > which hasn't been released yet, we will for now just include our own version > instead of depending on the Hadoop version. > After the patch is approved for master we can backport as necessary to older > versions of HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-15315) Remove always set super user call as high priority
[ https://issues.apache.org/jira/browse/HBASE-15315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15515036#comment-15515036 ] Hudson commented on HBASE-15315: FAILURE: Integrated in Jenkins build HBase-1.2-JDK8 #27 (See [https://builds.apache.org/job/HBase-1.2-JDK8/27/]) HBASE-15315 Remove always set super user call as high priority (Yong (apurtell: rev 32a7f2c4b80b4d5f05889f1669aa1538450b6d7a) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestPriorityRpc.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/AnnotationReadingPriorityFunction.java > Remove always set super user call as high priority > -- > > Key: HBASE-15315 > URL: https://issues.apache.org/jira/browse/HBASE-15315 > Project: HBase > Issue Type: Improvement >Reporter: Yong Zhang >Assignee: Yong Zhang > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4 > > Attachments: HBASE-15315.001.patch > > > Current implementation set superuser call as ADMIN_QOS, but we have many > customers use superuser to do normal table operation such as put/get data and > so on. If client put much data during region assignment, RPC from HMaster may > timeout because of no handle. so it is better to remove always set super user > call as high priority. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16630) Fragmentation in long running Bucket Cache
[ https://issues.apache.org/jira/browse/HBASE-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16630: --- Attachment: 16630-v3-suggest.patch Patch with DEBUG log at the end of freeEntireBuckets(). > Fragmentation in long running Bucket Cache > -- > > Key: HBASE-16630 > URL: https://issues.apache.org/jira/browse/HBASE-16630 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.1.6, 1.3.1, 1.2.3 >Reporter: deepankar >Assignee: deepankar > Attachments: 16630-v2-suggest.patch, 16630-v3-suggest.patch, > HBASE-16630-v2.patch, HBASE-16630-v3.patch, HBASE-16630.patch > > > As we are running bucket cache for a long time in our system, we are > observing cases where some nodes after some time does not fully utilize the > bucket cache, in some cases it is even worse in the sense they get stuck at a > value < 0.25 % of the bucket cache (DEFAULT_MEMORY_FACTOR as all our tables > are configured in-memory for simplicity sake). > We took a heap dump and analyzed what is happening and saw that is classic > case of fragmentation, current implementation of BucketCache (mainly > BucketAllocator) relies on the logic that fullyFreeBuckets are available for > switching/adjusting cache usage between different bucketSizes . But once a > compaction / bulkload happens and the blocks are evicted from a bucket size , > these are usually evicted from random places of the buckets of a bucketSize > and thus locking the number of buckets associated with a bucketSize and in > the worst case of the fragmentation we have seen some bucketSizes with > occupancy ratio of < 10 % But they dont have any completelyFreeBuckets to > share with the other bucketSize. > Currently the existing eviction logic helps in the cases where cache used is > more the MEMORY_FACTOR or MULTI_FACTOR and once those evictions are also > done, the eviction (freeSpace function) will not evict anything and the cache > utilization will be stuck at that value without any allocations for other > required sizes. > The fix for this we came up with is simple that we do deFragmentation ( > compaction) of the bucketSize and thus increasing the occupancy ratio and > also freeing up the buckets to be fullyFree, this logic itself is not > complicated as the bucketAllocator takes care of packing the blocks in the > buckets, we need evict and re-allocate the blocks for all the BucketSizes > that dont fit the criteria. > I am attaching an initial patch just to give an idea of what we are thinking > and I'll improve it based on the comments from the community. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16677) Add table size (total store file size) to table page
[ https://issues.apache.org/jira/browse/HBASE-16677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-16677: -- Status: Patch Available (was: Open) > Add table size (total store file size) to table page > > > Key: HBASE-16677 > URL: https://issues.apache.org/jira/browse/HBASE-16677 > Project: HBase > Issue Type: New Feature > Components: website >Reporter: Guang Yang >Priority: Minor > Attachments: HBASE-16677_v0.patch, HBASE-16677_v1.patch, > HBASE-16677_v2.patch, HBASE-16677_v3.patch, mini_cluster_master.png, > prod_cluster_partial.png > > > Currently there is not an easy way to get the table size from the web UI, > though we have the region size on the page, it is still convenient to have a > table for the table size stat. > Another pain point is that when the table grow large with tens of thousands > of regions, it took extremely long time to load the page, however, sometimes > we don't want to check all the regions. An optimization could be to accept a > query parameter to specify the number of regions to render. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16677) Add table size (total store file size) to table page
[ https://issues.apache.org/jira/browse/HBASE-16677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514989#comment-15514989 ] Guang Yang commented on HBASE-16677: [~enis], sure updated changing to use the utility. Thanks. > Add table size (total store file size) to table page > > > Key: HBASE-16677 > URL: https://issues.apache.org/jira/browse/HBASE-16677 > Project: HBase > Issue Type: New Feature > Components: website >Reporter: Guang Yang >Priority: Minor > Attachments: HBASE-16677_v0.patch, HBASE-16677_v1.patch, > HBASE-16677_v2.patch, HBASE-16677_v3.patch, mini_cluster_master.png, > prod_cluster_partial.png > > > Currently there is not an easy way to get the table size from the web UI, > though we have the region size on the page, it is still convenient to have a > table for the table size stat. > Another pain point is that when the table grow large with tens of thousands > of regions, it took extremely long time to load the page, however, sometimes > we don't want to check all the regions. An optimization could be to accept a > query parameter to specify the number of regions to render. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16677) Add table size (total store file size) to table page
[ https://issues.apache.org/jira/browse/HBASE-16677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guang Yang updated HBASE-16677: --- Attachment: HBASE-16677_v3.patch > Add table size (total store file size) to table page > > > Key: HBASE-16677 > URL: https://issues.apache.org/jira/browse/HBASE-16677 > Project: HBase > Issue Type: New Feature > Components: website >Reporter: Guang Yang >Priority: Minor > Attachments: HBASE-16677_v0.patch, HBASE-16677_v1.patch, > HBASE-16677_v2.patch, HBASE-16677_v3.patch, mini_cluster_master.png, > prod_cluster_partial.png > > > Currently there is not an easy way to get the table size from the web UI, > though we have the region size on the page, it is still convenient to have a > table for the table size stat. > Another pain point is that when the table grow large with tens of thousands > of regions, it took extremely long time to load the page, however, sometimes > we don't want to check all the regions. An optimization could be to accept a > query parameter to specify the number of regions to render. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16656) BackupID must include backup set name
[
https://issues.apache.org/jira/browse/HBASE-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514967#comment-15514967
]
Hadoop QA commented on HBASE-16656:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color}
| {color:red} HBASE-16656 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829964/HBASE-16656-v2.patch |
| JIRA Issue | HBASE-16656 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3675/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> BackupID must include backup set name
> -
>
> Key: HBASE-16656
> URL: https://issues.apache.org/jira/browse/HBASE-16656
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-16656-v1.patch, HBASE-16656-v2.patch
>
>
> Default backup set name is "backup". If we do backup for a backup set
> "SomeSetName", by default, backup id will be generated in a form:
> *SomeSetName_timestamp*.
> The goal is to separate backup images between different sets.
> The history command will be updated and the new command - merge will use this
> backup names (prefixes)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16656) BackupID must include backup set name
[ https://issues.apache.org/jira/browse/HBASE-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16656: -- Attachment: HBASE-16656-v2.patch Patch v2. > BackupID must include backup set name > - > > Key: HBASE-16656 > URL: https://issues.apache.org/jira/browse/HBASE-16656 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16656-v1.patch, HBASE-16656-v2.patch > > > Default backup set name is "backup". If we do backup for a backup set > "SomeSetName", by default, backup id will be generated in a form: > *SomeSetName_timestamp*. > The goal is to separate backup images between different sets. > The history command will be updated and the new command - merge will use this > backup names (prefixes) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16688) Split TestMasterFailoverWithProcedures
[
https://issues.apache.org/jira/browse/HBASE-16688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514954#comment-15514954
]
Hadoop QA commented on HBASE-16688:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
49s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 39s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
42s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 25s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
46s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
26m 31s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
13s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
59s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 84m 43s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
13s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 123m 34s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestBlockEvictionFromClient |
| Timed out junit tests | org.apache.hadoop.hbase.master.TestMasterFailover |
| | org.apache.hadoop.hbase.master.TestMasterFailoverBalancerPersistence |
| | org.apache.hadoop.hbase.master.TestTableLockManager |
| | org.apache.hadoop.hbase.master.snapshot.TestSnapshotFileCache |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829940/HBASE-16688-v1.patch |
| JIRA Issue | HBASE-16688 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 8a9a9d237c3d 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 4b05f40 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3673/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3673/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3673/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/PreCommit-HBA
[jira] [Commented] (HBASE-12894) Upgrade Jetty to 9.2.6
[ https://issues.apache.org/jira/browse/HBASE-12894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514927#comment-15514927 ] Sean Busbey commented on HBASE-12894: - the master branch (what will be HBase 2.0) is already jdk 8 only. So let's just remove jetty 6 entirely from that branch and move to jetty 9 throughout. > Upgrade Jetty to 9.2.6 > -- > > Key: HBASE-12894 > URL: https://issues.apache.org/jira/browse/HBASE-12894 > Project: HBase > Issue Type: Improvement > Components: REST >Affects Versions: 0.98.0 >Reporter: Rick Hallihan >Assignee: Ted Yu > Labels: MicrosoftSupport > Fix For: 2.0.0 > > > The Jetty component that is used for the HBase Stargate REST endpoint is > version 6.1.26 and is fairly outdated. We recently had a customer inquire > about enabling cross-origin resource sharing (CORS) for the REST endpoint and > found that this older version does not include the necessary filter or > configuration options, highlighted at: > http://wiki.eclipse.org/Jetty/Feature/Cross_Origin_Filter > The Jetty project has had significant updates through versions 7, 8 and 9, > including a transition to be an Eclipse subproject, so updating to the latest > version may be non-trivial. The last update to the Jetty component in > https://issues.apache.org/jira/browse/HBASE-3377 was a minor version update > and did not require significant work. This update will include a package > namespace update so there will likely be a larger number of required changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16146) Counters are expensive...
[ https://issues.apache.org/jira/browse/HBASE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514911#comment-15514911 ] Enis Soztutar commented on HBASE-16146: --- Maybe we can make it so that Counter just delegates to LongAdder if it is available. > Counters are expensive... > - > > Key: HBASE-16146 > URL: https://issues.apache.org/jira/browse/HBASE-16146 > Project: HBase > Issue Type: Sub-task >Reporter: stack > Attachments: HBASE-16146.branch-1.3.001.patch, counters.patch, > less_and_less_counters.png > > > Doing workloadc, perf shows 10%+ of CPU being spent on counter#add. If I > disable some of the hot ones -- see patch -- I can get 10% more throughput > (390k to 440k). Figure something better. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16688) Split TestMasterFailoverWithProcedures
[
https://issues.apache.org/jira/browse/HBASE-16688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514905#comment-15514905
]
Hadoop QA commented on HBASE-16688:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 4s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 5s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 40s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
5s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
37m 58s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
27s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 111m 55s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 171m 4s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hbase.regionserver.throttle.TestFlushWithThroughputController |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestMetaWithReplicas |
| | org.apache.hadoop.hbase.client.TestClientOperationInterrupt |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
| | org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.0 Server=1.12.0 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829931/HBASE-16688-v0.patch |
| JIRA Issue | HBASE-16688 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 944314fd42af 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 83cf44c |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3672/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3672/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3672/testReport/ |
| modules | C: hbase-server U: hbase-server |
| Console output |
https://builds.apache.org/job/P
[jira] [Commented] (HBASE-16604) Scanner retries on IOException can cause the scans to miss data
[ https://issues.apache.org/jira/browse/HBASE-16604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514893#comment-15514893 ] Hudson commented on HBASE-16604: SUCCESS: Integrated in Jenkins build HBase-Trunk_matrix #1655 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1655/]) HBASE-16604 Scanner retries on IOException can cause the scans to miss (enis: rev 83cf44cd3f19c841ac53889d09454ed5247ce591) * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift2/ThriftServer.java" * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift/IncrementCoalescer.java" * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift/generated/TRowResult.java" * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift2/generated/TDelete.java" * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift2/generated/THRegionInfo.java" * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestTableMapReduceBase.java * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift2/generated/TMutation.java" * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestMultithreadedTableMapper.java * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift2/generated/TCellVisibility.java" * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift2/generated/TServerName.java" * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift/generated/TCell.java" * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift/generated/BatchMutation.java" * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift2/ThriftUtilities.java" * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift2/generated/TAuthorization.java" * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift/ThriftHttpServlet.java" * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift2/generated/THRegionLocation.java" * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift2/generated/TTimeRange.java" * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift/generated/Hbase.java" * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift2/generated/TAppend.java" * (add) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/DelegatingKeyValueScanner.java * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift/TBoundedThreadPoolServer.java" * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift2/ThriftHBaseServiceHandler.java" * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift/IncrementCoalescerMBean.java" * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java" * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift2/generated/TCompareOp.java" * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java * (edit) hbase-client/src/main/java/org/apache/hadoop/hbase/UnknownScannerException.java * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift/generated/Mutation.java" * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift/ThriftMetrics.java" * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift/ThriftServer.java" * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift2/generated/TColumnIncrement.java" * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift2/generated/TIOError.java" * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift2/generated/TIllegalArgument.java" * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift/generated/TScan.java" * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift2/generated/TDeleteType.java" * (add) hbase-client/src/main/java/org/apache/hadoop/hbase/exceptions/ScannerResetException.java * (edit) hbase-hadoop-compat/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServerSource.java * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift2/generated/TScan.java" * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift2/generated/TGet.java" * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift/generated/AlreadyExists.java" * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift2/generated/TResult.java" * (add) "hbase-thrift\036src/main/java/org/apache/hadoop/hbase/thrift2/generated/TPut.java" * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/ipc/MetricsHBaseServer.java * (add) "hbase-thrift\036src/main/java/org/a
[jira] [Commented] (HBASE-16677) Add table size (total store file size) to table page
[ https://issues.apache.org/jira/browse/HBASE-16677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514888#comment-15514888 ] Enis Soztutar commented on HBASE-16677: --- bq. It does not seem fit since here the store file is already MB, and in order to leverage it, we will need to convert it back to B and then call that method? Yes. It will be consistent with all human-readable sizes that we use in the UI. > Add table size (total store file size) to table page > > > Key: HBASE-16677 > URL: https://issues.apache.org/jira/browse/HBASE-16677 > Project: HBase > Issue Type: New Feature > Components: website >Reporter: Guang Yang >Priority: Minor > Attachments: HBASE-16677_v0.patch, HBASE-16677_v1.patch, > HBASE-16677_v2.patch, mini_cluster_master.png, prod_cluster_partial.png > > > Currently there is not an easy way to get the table size from the web UI, > though we have the region size on the page, it is still convenient to have a > table for the table size stat. > Another pain point is that when the table grow large with tens of thousands > of regions, it took extremely long time to load the page, however, sometimes > we don't want to check all the regions. An optimization could be to accept a > query parameter to specify the number of regions to render. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16662) Fix open POODLE vulnerabilities
[ https://issues.apache.org/jira/browse/HBASE-16662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514868#comment-15514868 ] Hudson commented on HBASE-16662: SUCCESS: Integrated in Jenkins build HBase-1.1-JDK7 #1785 (See [https://builds.apache.org/job/HBase-1.1-JDK7/1785/]) HBASE-16662 Fix open POODLE vulnerabilities (apurtell: rev 97ce640f5d71cc10828a7895298e2bbb482b1068) * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/SslRMIServerSocketFactorySecure.java * (edit) hbase-rest/src/main/java/org/apache/hadoop/hbase/rest/RESTServer.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/SslRMIClientSocketFactorySecure.java * (add) hbase-server/src/main/java/org/apache/hadoop/hbase/jetty/SslSelectChannelConnectorSecure.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/JMXListener.java * (edit) hbase-thrift/src/main/java/org/apache/hadoop/hbase/thrift/ThriftServerRunner.java > Fix open POODLE vulnerabilities > --- > > Key: HBASE-16662 > URL: https://issues.apache.org/jira/browse/HBASE-16662 > Project: HBase > Issue Type: Bug > Components: REST, Thrift >Reporter: Ben Lau >Assignee: Ben Lau > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 0.98.23, 1.2.4 > > Attachments: HBASE-16662-master.patch > > > We recently found a security issue in our HBase REST servers. The issue is a > variant of the POODLE vulnerability (https://en.wikipedia.org/wiki/POODLE) > and is present in the HBase Thrift server as well. It also appears to affect > the JMXListener coprocessor. The vulnerabilities probably affect all > versions of HBase that have the affected services. (If you don't use the > affected services with SSL then this ticket probably doesn't affect you). > Included is a patch to fix the known POODLE vulnerabilities in master. Let > us know if we missed any. From our end we only personally encountered the > HBase REST vulnerability. We do not use the Thrift server or JMXListener > coprocessor but discovered those problems after discussing the issue with > some of the HBase PMCs. > Coincidentally, Hadoop recently committed a SslSelectChannelConnectorSecure > which is more or less the same as one of the fixes in this patch. Hadoop > wasn't originally affected by the vulnerability in the > SslSelectChannelConnector, but about a month ago they committed HADOOP-12765 > which does use that class, so they added a SslSelectChannelConnectorSecure > class similar to this patch. Since this class is present in Hadoop 2.7.4+ > which hasn't been released yet, we will for now just include our own version > instead of depending on the Hadoop version. > After the patch is approved for master we can backport as necessary to older > versions of HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16423) Add re-compare option to VerifyReplication to avoid occasional inconsistent rows
[
https://issues.apache.org/jira/browse/HBASE-16423?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514866#comment-15514866
]
Hadoop QA commented on HBASE-16423:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 5m
33s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
25s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
57s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 50s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 1 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
45m 53s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
25s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 107m 1s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 175m 31s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests |
org.apache.hadoop.hbase.regionserver.wal.TestSecureWALReplay |
| | org.apache.hadoop.hbase.regionserver.wal.TestAsyncLogRolling |
| | org.apache.hadoop.hbase.regionserver.wal.TestLogRolling |
| | org.apache.hadoop.hbase.mapreduce.TestTableMapReduce |
| | org.apache.hadoop.hbase.regionserver.TestCompoundBloomFilter |
| | org.apache.hadoop.hbase.mapreduce.TestLoadIncrementalHFiles |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829858/HBASE-16423-v2.patch |
| JIRA Issue | HBASE-16423 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux df7e1575e1fb 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build@2/component/dev-support/hbase-personality.sh
|
| git revision | master / 83cf44c |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| whitespace |
https://builds.apache.org/job/PreCommit-HBASE-Build/3670/artifact/patchprocess/whitespace-eol.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3670/artifact/patchprocess/patch-unit-hbase-s
[jira] [Commented] (HBASE-16604) Scanner retries on IOException can cause the scans to miss data
[
https://issues.apache.org/jira/browse/HBASE-16604?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514849#comment-15514849
]
Hadoop QA commented on HBASE-16604:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 15m 55s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 8 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m 14s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
47s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 36s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 34s
{color} | {color:green} branch-1 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
26s {color} | {color:green} branch-1 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
15s {color} | {color:green} branch-1 passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 57s
{color} | {color:red} hbase-server in branch-1 has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 52s
{color} | {color:green} branch-1 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 10s
{color} | {color:green} branch-1 passed with JDK v1.7.0_111 {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 16s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 15s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 49s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 49s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 2m
33s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
24m 39s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 1m
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
57s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 52s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 18s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 23s
{color} | {color:green} hbase-hadoop-compat in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 32s
{color} | {color:green} hbase-hadoop2-compat in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 147m 2s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 1m
55s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} |
[jira] [Commented] (HBASE-16146) Counters are expensive...
[ https://issues.apache.org/jira/browse/HBASE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514767#comment-15514767 ] Gary Helmling commented on HBASE-16146: --- [~stack], I'm curious if this patch provides any improvement for your YCSB workload. > Counters are expensive... > - > > Key: HBASE-16146 > URL: https://issues.apache.org/jira/browse/HBASE-16146 > Project: HBase > Issue Type: Sub-task >Reporter: stack > Attachments: HBASE-16146.branch-1.3.001.patch, counters.patch, > less_and_less_counters.png > > > Doing workloadc, perf shows 10%+ of CPU being spent on counter#add. If I > disable some of the hot ones -- see patch -- I can get 10% more throughput > (390k to 440k). Figure something better. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16146) Counters are expensive...
[ https://issues.apache.org/jira/browse/HBASE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gary Helmling updated HBASE-16146: -- Attachment: HBASE-16146.branch-1.3.001.patch The attached patch removes the use of an instance-level ThreadLocal in Counter to store the per-thread last used cell index. Instead, we just recompute the hash for each access. This comes at a bit of a cost for writes, but provides big memory savings when many counters are in use, as well as avoiding spinning in ThreadLocalMap.getEntryAfterMiss(). > Counters are expensive... > - > > Key: HBASE-16146 > URL: https://issues.apache.org/jira/browse/HBASE-16146 > Project: HBase > Issue Type: Sub-task >Reporter: stack > Attachments: HBASE-16146.branch-1.3.001.patch, counters.patch, > less_and_less_counters.png > > > Doing workloadc, perf shows 10%+ of CPU being spent on counter#add. If I > disable some of the hot ones -- see patch -- I can get 10% more throughput > (390k to 440k). Figure something better. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12894) Upgrade Jetty to 9.2.6
[ https://issues.apache.org/jira/browse/HBASE-12894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514751#comment-15514751 ] Guang Yang commented on HBASE-12894: Hi [~ted_yu], are you working on this? We recently moved to Jetty 9 for REST and see good performance (will share more later), if you haven't started on this, I can work on a patch against master branch for review. As for this effort, there are two things needs some discussion though: 1) Jetty 9 needs JDK 8, which means we will need to move to JDK 8 at the same time. 2) The infoserver from HDFS is still on Jetty 6, so that we run both Jetty 9 and Jetty 6 together, that seems not a problem though per our testing as they are in different namespaces. Thoughts? > Upgrade Jetty to 9.2.6 > -- > > Key: HBASE-12894 > URL: https://issues.apache.org/jira/browse/HBASE-12894 > Project: HBase > Issue Type: Improvement > Components: REST >Affects Versions: 0.98.0 >Reporter: Rick Hallihan >Assignee: Ted Yu > Labels: MicrosoftSupport > Fix For: 2.0.0 > > > The Jetty component that is used for the HBase Stargate REST endpoint is > version 6.1.26 and is fairly outdated. We recently had a customer inquire > about enabling cross-origin resource sharing (CORS) for the REST endpoint and > found that this older version does not include the necessary filter or > configuration options, highlighted at: > http://wiki.eclipse.org/Jetty/Feature/Cross_Origin_Filter > The Jetty project has had significant updates through versions 7, 8 and 9, > including a transition to be an Eclipse subproject, so updating to the latest > version may be non-trivial. The last update to the Jetty component in > https://issues.apache.org/jira/browse/HBASE-3377 was a minor version update > and did not require significant work. This update will include a package > namespace update so there will likely be a larger number of required changes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16685) Revisit execution of SnapshotCopy in MapReduceBackupCopyService
[
https://issues.apache.org/jira/browse/HBASE-16685?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514740#comment-15514740
]
Vladimir Rodionov commented on HBASE-16685:
---
One handler blocked for hours is OK as long as we have 100 of them. I do not
see any issue here. I do not mind changing code if it is not a big one though.
Master does not have direct dependencies on M/R I think - only on backup
package.
> Revisit execution of SnapshotCopy in MapReduceBackupCopyService
> ---
>
> Key: HBASE-16685
> URL: https://issues.apache.org/jira/browse/HBASE-16685
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
> Labels: backup
>
> During code review of backup / restore, Matteo raised comment on the
> following code:
> {code}
> res = snapshotCp.run(options);
> {code}
> I think this will be a first time where HBase server has a direct dependency
> on a MR job.
> also we need to revisit this code to avoid having one handler blocked for N
> hours while we are doing a copy.
> This issue is to address the above comment.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16146) Counters are expensive...
[ https://issues.apache.org/jira/browse/HBASE-16146?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514741#comment-15514741 ] Gary Helmling commented on HBASE-16146: --- We've seen Counter come up as a source of high CPU utilization in 1.3, especially since the switch of metrics to use FastLongHistogram (each instance of which uses 260 Counter instances internally) from HBASE-15222. I think this is due to the use of the instance-level ThreadLocal in Counter to track the per-thread cell index, as perf output on hot nodes shows a huge amount of time in ThreadLocalMap.getEntryAfterMiss(). As the number of Counter instances (and ThreadLocal instances) we're retaining in memory grows, performance seems to degrade. This is all moot for master, since we've already deprecated Counter and replaced its usage with LongAdder. But we still need a solution for Counter in branch-1. I'm testing a patch which removes the ThreadLocal usage, which I'll attach here. Benchmarking shows a small reduction in Counter performance, but a big improvement in FastLongHistogram performance when many histograms are retained in memory. > Counters are expensive... > - > > Key: HBASE-16146 > URL: https://issues.apache.org/jira/browse/HBASE-16146 > Project: HBase > Issue Type: Sub-task >Reporter: stack > Attachments: counters.patch, less_and_less_counters.png > > > Doing workloadc, perf shows 10%+ of CPU being spent on counter#add. If I > disable some of the hot ones -- see patch -- I can get 10% more throughput > (390k to 440k). Figure something better. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16677) Add table size (total store file size) to table page
[
https://issues.apache.org/jira/browse/HBASE-16677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514732#comment-15514732
]
Guang Yang commented on HBASE-16677:
Thanks [~enis] for looking at the patch.
{quote}
Do we actually stop rendering here? I did not see it. Have you tested with more
than 10K regions?
{quote}
The rendering part is in the *if* block, thus if it reaches to the threshold,
it would stop rendering more regions. I didn't test it with 10K but I tested
with query parameter like "numRegions=10" and it truncate the number of
regions to 10 in this case.
{quote}
Please use TraditionalBinaryPrefix.long2String() instead of this:
{quote}
It does not seem fit since here the store file is already MB, and in order to
leverage it, we will need to convert it back to B and then call that method?
{quote}
Can we please change the param name from regions to numRegions, and also have a
link in the warning message to the same page with the parameter set to render
all regions.
{quote}
Fixed, thanks.
> Add table size (total store file size) to table page
>
>
> Key: HBASE-16677
> URL: https://issues.apache.org/jira/browse/HBASE-16677
> Project: HBase
> Issue Type: New Feature
> Components: website
>Reporter: Guang Yang
>Priority: Minor
> Attachments: HBASE-16677_v0.patch, HBASE-16677_v1.patch,
> HBASE-16677_v2.patch, mini_cluster_master.png, prod_cluster_partial.png
>
>
> Currently there is not an easy way to get the table size from the web UI,
> though we have the region size on the page, it is still convenient to have a
> table for the table size stat.
> Another pain point is that when the table grow large with tens of thousands
> of regions, it took extremely long time to load the page, however, sometimes
> we don't want to check all the regions. An optimization could be to accept a
> query parameter to specify the number of regions to render.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16677) Add table size (total store file size) to table page
[ https://issues.apache.org/jira/browse/HBASE-16677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guang Yang updated HBASE-16677: --- Attachment: HBASE-16677_v2.patch > Add table size (total store file size) to table page > > > Key: HBASE-16677 > URL: https://issues.apache.org/jira/browse/HBASE-16677 > Project: HBase > Issue Type: New Feature > Components: website >Reporter: Guang Yang >Priority: Minor > Attachments: HBASE-16677_v0.patch, HBASE-16677_v1.patch, > HBASE-16677_v2.patch, mini_cluster_master.png, prod_cluster_partial.png > > > Currently there is not an easy way to get the table size from the web UI, > though we have the region size on the page, it is still convenient to have a > table for the table size stat. > Another pain point is that when the table grow large with tens of thousands > of regions, it took extremely long time to load the page, however, sometimes > we don't want to check all the regions. An optimization could be to accept a > query parameter to specify the number of regions to render. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16671) Split TestExportSnapshot
[ https://issues.apache.org/jira/browse/HBASE-16671?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16671: Resolution: Fixed Status: Resolved (was: Patch Available) > Split TestExportSnapshot > > > Key: HBASE-16671 > URL: https://issues.apache.org/jira/browse/HBASE-16671 > Project: HBase > Issue Type: Test > Components: snapshots, test >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16671-v0.patch > > > TestExportSnapshot contains 3 type of tests. > - MiniCluster creating a table, taking a snapshot and running export > - Mocked snapshot running export > - tool helpers tests > since now we have everything packed in a single test. 2 and 3 ended up having > a before and after that is creating a table and taking a snapshot which is > not used. Move those tests out and cut some time from the test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16657) Expose per-region last major compaction timestamp in RegionServer UI
[ https://issues.apache.org/jira/browse/HBASE-16657?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514698#comment-15514698 ] Dustin Pho commented on HBASE-16657: Regarding manual steps taken to verify -- 1. use major_compact in bin/hbase shell 2. Go to the corresponding :/rs-status#regionCompactStats. The last major compaction should be updated. I don't think my changes affect the failed junit tests. > Expose per-region last major compaction timestamp in RegionServer UI > > > Key: HBASE-16657 > URL: https://issues.apache.org/jira/browse/HBASE-16657 > Project: HBase > Issue Type: Improvement > Components: regionserver, UI >Reporter: Gary Helmling >Assignee: Dustin Pho >Priority: Minor > Attachments: HBASE-16657.001.patch, with-patch-changes.png, > without-patch-changes.png > > > HBASE-12859 added some tracking for the last major compaction completed for > each region. However, this is currently only exposed through the cluster > status reporting and the Admin API. Since the regionserver is already > reporting this information, it would be nice to fold it in somewhere to the > region listing in the regionserver UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16657) Expose per-region last major compaction timestamp in RegionServer UI
[ https://issues.apache.org/jira/browse/HBASE-16657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dustin Pho updated HBASE-16657: --- Attachment: without-patch-changes.png > Expose per-region last major compaction timestamp in RegionServer UI > > > Key: HBASE-16657 > URL: https://issues.apache.org/jira/browse/HBASE-16657 > Project: HBase > Issue Type: Improvement > Components: regionserver, UI >Reporter: Gary Helmling >Assignee: Dustin Pho >Priority: Minor > Attachments: HBASE-16657.001.patch, with-patch-changes.png, > without-patch-changes.png > > > HBASE-12859 added some tracking for the last major compaction completed for > each region. However, this is currently only exposed through the cluster > status reporting and the Admin API. Since the regionserver is already > reporting this information, it would be nice to fold it in somewhere to the > region listing in the regionserver UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16688) Split TestMasterFailoverWithProcedures
[ https://issues.apache.org/jira/browse/HBASE-16688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514682#comment-15514682 ] Matteo Bertozzi commented on HBASE-16688: - TestRegionServerMetrics was other stuff I was working on, removed in v1. and renamed TestMasterProcedureWAL in TestMasterProcedureWalLease > Split TestMasterFailoverWithProcedures > -- > > Key: HBASE-16688 > URL: https://issues.apache.org/jira/browse/HBASE-16688 > Project: HBase > Issue Type: Bug > Components: proc-v2, test >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-16688-v0.patch, HBASE-16688-v1.patch > > > extract the WAL lease tests from the TestMasterFailoverWithProcedures. > leaving TestMasterFailoverWithProcedures with only the proc test on master > failover -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16657) Expose per-region last major compaction timestamp in RegionServer UI
[ https://issues.apache.org/jira/browse/HBASE-16657?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dustin Pho updated HBASE-16657: --- Attachment: with-patch-changes.png > Expose per-region last major compaction timestamp in RegionServer UI > > > Key: HBASE-16657 > URL: https://issues.apache.org/jira/browse/HBASE-16657 > Project: HBase > Issue Type: Improvement > Components: regionserver, UI >Reporter: Gary Helmling >Assignee: Dustin Pho >Priority: Minor > Attachments: HBASE-16657.001.patch, with-patch-changes.png > > > HBASE-12859 added some tracking for the last major compaction completed for > each region. However, this is currently only exposed through the cluster > status reporting and the Admin API. Since the regionserver is already > reporting this information, it would be nice to fold it in somewhere to the > region listing in the regionserver UI. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-15315) Remove always set super user call as high priority
[ https://issues.apache.org/jira/browse/HBASE-15315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-15315: --- Fix Version/s: 1.2.4 > Remove always set super user call as high priority > -- > > Key: HBASE-15315 > URL: https://issues.apache.org/jira/browse/HBASE-15315 > Project: HBase > Issue Type: Improvement >Reporter: Yong Zhang >Assignee: Yong Zhang > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.2.4 > > Attachments: HBASE-15315.001.patch > > > Current implementation set superuser call as ADMIN_QOS, but we have many > customers use superuser to do normal table operation such as put/get data and > so on. If client put much data during region assignment, RPC from HMaster may > timeout because of no handle. so it is better to remove always set super user > call as high priority. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16630) Fragmentation in long running Bucket Cache
[ https://issues.apache.org/jira/browse/HBASE-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514645#comment-15514645 ] Vladimir Rodionov commented on HBASE-16630: --- Yes, my bad. Somehow missed ! > Fragmentation in long running Bucket Cache > -- > > Key: HBASE-16630 > URL: https://issues.apache.org/jira/browse/HBASE-16630 > Project: HBase > Issue Type: Bug > Components: BucketCache >Affects Versions: 2.0.0, 1.1.6, 1.3.1, 1.2.3 >Reporter: deepankar >Assignee: deepankar > Attachments: 16630-v2-suggest.patch, HBASE-16630-v2.patch, > HBASE-16630-v3.patch, HBASE-16630.patch > > > As we are running bucket cache for a long time in our system, we are > observing cases where some nodes after some time does not fully utilize the > bucket cache, in some cases it is even worse in the sense they get stuck at a > value < 0.25 % of the bucket cache (DEFAULT_MEMORY_FACTOR as all our tables > are configured in-memory for simplicity sake). > We took a heap dump and analyzed what is happening and saw that is classic > case of fragmentation, current implementation of BucketCache (mainly > BucketAllocator) relies on the logic that fullyFreeBuckets are available for > switching/adjusting cache usage between different bucketSizes . But once a > compaction / bulkload happens and the blocks are evicted from a bucket size , > these are usually evicted from random places of the buckets of a bucketSize > and thus locking the number of buckets associated with a bucketSize and in > the worst case of the fragmentation we have seen some bucketSizes with > occupancy ratio of < 10 % But they dont have any completelyFreeBuckets to > share with the other bucketSize. > Currently the existing eviction logic helps in the cases where cache used is > more the MEMORY_FACTOR or MULTI_FACTOR and once those evictions are also > done, the eviction (freeSpace function) will not evict anything and the cache > utilization will be stuck at that value without any allocations for other > required sizes. > The fix for this we came up with is simple that we do deFragmentation ( > compaction) of the bucketSize and thus increasing the occupancy ratio and > also freeing up the buckets to be fullyFree, this logic itself is not > complicated as the bucketAllocator takes care of packing the blocks in the > buckets, we need evict and re-allocate the blocks for all the BucketSizes > that dont fit the criteria. > I am attaching an initial patch just to give an idea of what we are thinking > and I'll improve it based on the comments from the community. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14123) HBase Backup/Restore Phase 2
[
https://issues.apache.org/jira/browse/HBASE-14123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514637#comment-15514637
]
Hadoop QA commented on HBASE-14123:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:blue}0{color} | {color:blue} shelldocs {color} | {color:blue} 0m 5s
{color} | {color:blue} Shelldocs was not available. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 47 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 8s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 10s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 14m
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 1m
59s {color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 33s
{color} | {color:red} hbase-common in master has 1 extant Findbugs warnings.
{color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 49s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 9s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 4m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 12s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 15m
14s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 2m
4s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
4s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 1s
{color} | {color:red} The patch has 654 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 20s
{color} | {color:red} The patch 1 line(s) with tabs. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 2s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 24s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m 0s
{color} | {color:blue} Skipped patched modules with no Java source: . {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 1s
{color} | {color:red} hbase-server generated 2 new + 0 unchanged - 0 fixed = 2
total (was 0) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 0m 17s
{color} | {color:red} hbase-client generated 4 new + 14 unchanged - 0 fixed =
18 total (was 14) {color} |
| {color:red}-1{color} | {color:red} javadoc {color} | {color:red} 1m 42s
{color} | {color:red} root generated 4 new + 20 unchanged - 0 fixed = 24 total
(was 20) {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 25s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{col
[jira] [Updated] (HBASE-16688) Split TestMasterFailoverWithProcedures
[ https://issues.apache.org/jira/browse/HBASE-16688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16688: Attachment: HBASE-16688-v1.patch > Split TestMasterFailoverWithProcedures > -- > > Key: HBASE-16688 > URL: https://issues.apache.org/jira/browse/HBASE-16688 > Project: HBase > Issue Type: Bug > Components: proc-v2, test >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-16688-v0.patch, HBASE-16688-v1.patch > > > extract the WAL lease tests from the TestMasterFailoverWithProcedures. > leaving TestMasterFailoverWithProcedures with only the proc test on master > failover -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514628#comment-15514628 ] Ted Yu commented on HBASE-14417: Currently there is one BulkLoadHandler inside each region server which writes BulkLoadDescriptor periodically to hbase:backup table. Ideally only BulkLoadDescriptor's for tables which have gone through full backup should be written. Looking for a way to pass Set of such tables to region servers so that each server doesn't have to poll hbase:backup table periodically. > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > Attachments: 14417.v1.txt, 14417.v2.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-16676: --- Resolution: Duplicate Assignee: (was: Andrew Purtell) Fix Version/s: (was: 1.2.4) Status: Resolved (was: Patch Available) > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3 >Reporter: Andrew Purtell > Attachments: HBASE-16676-branch-1.2.patch > > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514613#comment-15514613 ] Andrew Purtell commented on HBASE-16676: 👍 > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.2.4 > > Attachments: HBASE-16676-branch-1.2.patch > > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16682) Fix Shell tests failure. NoClassDefFoundError for MiniKdc
[
https://issues.apache.org/jira/browse/HBASE-16682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514604#comment-15514604
]
Hadoop QA commented on HBASE-16682:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 19s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
15s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
9s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 16s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
27m 39s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
10s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 15s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 49s
{color} | {color:green} hbase-common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
6s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 35m 8s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829929/HBASE-16682.master.002.patch
|
| JIRA Issue | HBASE-16682 |
| Optional Tests | asflicense javac javadoc unit xml compile |
| uname | Linux 03b544b7526a 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 83cf44c |
| Default Java | 1.8.0_101 |
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3671/testReport/ |
| modules | C: hbase-common U: hbase-common |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3671/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Fix Shell tests failure. NoClassDefFoundError for MiniKdc
> -
>
> Key: HBASE-16682
> URL: https://issues.apache.org/jira/browse/HBASE-16682
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-16682.master.001.patch,
> HBASE-16682.master.002.patch
>
>
> Stacktrace
> {noformat}
> java.lang.NoClassDefFoundError: org/apache/hadoop/minikdc/MiniKdc
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2701
[jira] [Commented] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514596#comment-15514596 ] Dima Spivak commented on HBASE-16676: - Wow, awesome find, [~apurtell]! As [~stack] mentioned, I saw reproducible failures in the generator phase of ITBLL while testing 1.2.3, but just assumed something was wrong with my machine. Time for me to start trusting tests more :). > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.2.4 > > Attachments: HBASE-16676-branch-1.2.patch > > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16662) Fix open POODLE vulnerabilities
[ https://issues.apache.org/jira/browse/HBASE-16662?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-16662: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.2.4 0.98.23 1.1.7 1.4.0 1.3.0 2.0.0 Status: Resolved (was: Patch Available) > Fix open POODLE vulnerabilities > --- > > Key: HBASE-16662 > URL: https://issues.apache.org/jira/browse/HBASE-16662 > Project: HBase > Issue Type: Bug > Components: REST, Thrift >Reporter: Ben Lau >Assignee: Ben Lau > Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 0.98.23, 1.2.4 > > Attachments: HBASE-16662-master.patch > > > We recently found a security issue in our HBase REST servers. The issue is a > variant of the POODLE vulnerability (https://en.wikipedia.org/wiki/POODLE) > and is present in the HBase Thrift server as well. It also appears to affect > the JMXListener coprocessor. The vulnerabilities probably affect all > versions of HBase that have the affected services. (If you don't use the > affected services with SSL then this ticket probably doesn't affect you). > Included is a patch to fix the known POODLE vulnerabilities in master. Let > us know if we missed any. From our end we only personally encountered the > HBase REST vulnerability. We do not use the Thrift server or JMXListener > coprocessor but discovered those problems after discussing the issue with > some of the HBase PMCs. > Coincidentally, Hadoop recently committed a SslSelectChannelConnectorSecure > which is more or less the same as one of the fixes in this patch. Hadoop > wasn't originally affected by the vulnerability in the > SslSelectChannelConnector, but about a month ago they committed HADOOP-12765 > which does use that class, so they added a SslSelectChannelConnectorSecure > class similar to this patch. Since this class is present in Hadoop 2.7.4+ > which hasn't been released yet, we will for now just include our own version > instead of depending on the Hadoop version. > After the patch is approved for master we can backport as necessary to older > versions of HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16662) Fix open POODLE vulnerabilities
[ https://issues.apache.org/jira/browse/HBASE-16662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514560#comment-15514560 ] Andrew Purtell edited comment on HBASE-16662 at 9/22/16 9:41 PM: - Going to commit to all branches. For 0.98, in order to avoid POODLE vulnerabilities in the embedded servlets (via InfoServer) it is also necessary to build HBase with a version of Hadoop that has HADOOP-11260, which I think is 2.5.2 or later. The default 0.98 build specifies 2.2.0. I'm sure nobody runs that in production at this point, but we still do that for sake of build compatibility. was (Author: apurtell): Going to commit to all branches. For 0.98, in order to avoid POODLE vulnerabilities it is also necessary to build HBase with a version of Hadoop that has HADOOP-11260, which I think is 2.5.2 or later. The default 0.98 build specifies 2.2.0. I'm sure nobody runs that in production at this point, but we still do that for sake of build compatibility. > Fix open POODLE vulnerabilities > --- > > Key: HBASE-16662 > URL: https://issues.apache.org/jira/browse/HBASE-16662 > Project: HBase > Issue Type: Bug > Components: REST, Thrift >Reporter: Ben Lau >Assignee: Ben Lau > Attachments: HBASE-16662-master.patch > > > We recently found a security issue in our HBase REST servers. The issue is a > variant of the POODLE vulnerability (https://en.wikipedia.org/wiki/POODLE) > and is present in the HBase Thrift server as well. It also appears to affect > the JMXListener coprocessor. The vulnerabilities probably affect all > versions of HBase that have the affected services. (If you don't use the > affected services with SSL then this ticket probably doesn't affect you). > Included is a patch to fix the known POODLE vulnerabilities in master. Let > us know if we missed any. From our end we only personally encountered the > HBase REST vulnerability. We do not use the Thrift server or JMXListener > coprocessor but discovered those problems after discussing the issue with > some of the HBase PMCs. > Coincidentally, Hadoop recently committed a SslSelectChannelConnectorSecure > which is more or less the same as one of the fixes in this patch. Hadoop > wasn't originally affected by the vulnerability in the > SslSelectChannelConnector, but about a month ago they committed HADOOP-12765 > which does use that class, so they added a SslSelectChannelConnectorSecure > class similar to this patch. Since this class is present in Hadoop 2.7.4+ > which hasn't been released yet, we will for now just include our own version > instead of depending on the Hadoop version. > After the patch is approved for master we can backport as necessary to older > versions of HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16662) Fix open POODLE vulnerabilities
[ https://issues.apache.org/jira/browse/HBASE-16662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514560#comment-15514560 ] Andrew Purtell commented on HBASE-16662: Going to commit to all branches. For 0.98, in order to avoid POODLE vulnerabilities it is also necessary to build HBase with a version of Hadoop that has HADOOP-11260, which I think is 2.5.2 or later. The default 0.98 build specifies 2.2.0. I'm sure nobody runs that in production at this point, but we still do that for sake of build compatibility. > Fix open POODLE vulnerabilities > --- > > Key: HBASE-16662 > URL: https://issues.apache.org/jira/browse/HBASE-16662 > Project: HBase > Issue Type: Bug > Components: REST, Thrift >Reporter: Ben Lau >Assignee: Ben Lau > Attachments: HBASE-16662-master.patch > > > We recently found a security issue in our HBase REST servers. The issue is a > variant of the POODLE vulnerability (https://en.wikipedia.org/wiki/POODLE) > and is present in the HBase Thrift server as well. It also appears to affect > the JMXListener coprocessor. The vulnerabilities probably affect all > versions of HBase that have the affected services. (If you don't use the > affected services with SSL then this ticket probably doesn't affect you). > Included is a patch to fix the known POODLE vulnerabilities in master. Let > us know if we missed any. From our end we only personally encountered the > HBase REST vulnerability. We do not use the Thrift server or JMXListener > coprocessor but discovered those problems after discussing the issue with > some of the HBase PMCs. > Coincidentally, Hadoop recently committed a SslSelectChannelConnectorSecure > which is more or less the same as one of the fixes in this patch. Hadoop > wasn't originally affected by the vulnerability in the > SslSelectChannelConnector, but about a month ago they committed HADOOP-12765 > which does use that class, so they added a SslSelectChannelConnectorSecure > class similar to this patch. Since this class is present in Hadoop 2.7.4+ > which hasn't been released yet, we will for now just include our own version > instead of depending on the Hadoop version. > After the patch is approved for master we can backport as necessary to older > versions of HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16671) Split TestExportSnapshot
[ https://issues.apache.org/jira/browse/HBASE-16671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514524#comment-15514524 ] Jonathan Hsieh commented on HBASE-16671: +1 lgtm. > Split TestExportSnapshot > > > Key: HBASE-16671 > URL: https://issues.apache.org/jira/browse/HBASE-16671 > Project: HBase > Issue Type: Test > Components: snapshots, test >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16671-v0.patch > > > TestExportSnapshot contains 3 type of tests. > - MiniCluster creating a table, taking a snapshot and running export > - Mocked snapshot running export > - tool helpers tests > since now we have everything packed in a single test. 2 and 3 ended up having > a before and after that is creating a table and taking a snapshot which is > not used. Move those tests out and cut some time from the test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16688) Split TestMasterFailoverWithProcedures
[ https://issues.apache.org/jira/browse/HBASE-16688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514516#comment-15514516 ] Jonathan Hsieh commented on HBASE-16688: MasterProcedureTestingUtility, TestMasterFailoverWithProcedures, and TestMasterProcedureWAL content wise look fine. Maybe rename TestMasterPrcedureWAL to something to mention lease recovery or fencing? Are the changes to TestRegionServerMetrics intentionally included? > Split TestMasterFailoverWithProcedures > -- > > Key: HBASE-16688 > URL: https://issues.apache.org/jira/browse/HBASE-16688 > Project: HBase > Issue Type: Bug > Components: proc-v2, test >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-16688-v0.patch > > > extract the WAL lease tests from the TestMasterFailoverWithProcedures. > leaving TestMasterFailoverWithProcedures with only the proc test on master > failover -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16682) Fix Shell tests failure. NoClassDefFoundError for MiniKdc
[
https://issues.apache.org/jira/browse/HBASE-16682?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514408#comment-15514408
]
Appy edited comment on HBASE-16682 at 9/22/16 9:11 PM:
---
(edited)
[~zghaobac], i thought about it and realized later that the right fix would
actually be adding the deps in --hbase-common--.
I have been playing with pom for last 30 min. But can't seem to figure it out.
The right fix is adding minikdc deps in hbase-testing-util, but that's not
fixing it.
was (Author: appy):
[~zghaobac], i thought about it and realized later that the right fix would
actually be adding the deps in hbase-common.
Putting up the patch.
> Fix Shell tests failure. NoClassDefFoundError for MiniKdc
> -
>
> Key: HBASE-16682
> URL: https://issues.apache.org/jira/browse/HBASE-16682
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-16682.master.001.patch,
> HBASE-16682.master.002.patch
>
>
> Stacktrace
> {noformat}
> java.lang.NoClassDefFoundError: org/apache/hadoop/minikdc/MiniKdc
> at java.lang.Class.getDeclaredMethods0(Native Method)
> at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
> at java.lang.Class.getDeclaredMethods(Class.java:1975)
> at org.jruby.javasupport.JavaClass.getMethods(JavaClass.java:2110)
> at org.jruby.javasupport.JavaClass.setupClassMethods(JavaClass.java:955)
> at org.jruby.javasupport.JavaClass.access$700(JavaClass.java:99)
> at
> org.jruby.javasupport.JavaClass$ClassInitializer.initialize(JavaClass.java:650)
> at org.jruby.javasupport.JavaClass.setupProxy(JavaClass.java:689)
> at org.jruby.javasupport.Java.createProxyClass(Java.java:526)
> at org.jruby.javasupport.Java.getProxyClass(Java.java:455)
> at org.jruby.javasupport.Java.getInstance(Java.java:364)
> at
> org.jruby.javasupport.JavaUtil.convertJavaToUsableRubyObject(JavaUtil.java:166)
> at
> org.jruby.javasupport.JavaEmbedUtils.javaToRuby(JavaEmbedUtils.java:291)
> at
> org.jruby.embed.variable.AbstractVariable.updateByJavaObject(AbstractVariable.java:81)
> at
> org.jruby.embed.variable.GlobalVariable.(GlobalVariable.java:69)
> at
> org.jruby.embed.variable.GlobalVariable.getInstance(GlobalVariable.java:60)
> at
> org.jruby.embed.variable.VariableInterceptor.getVariableInstance(VariableInterceptor.java:97)
> at org.jruby.embed.internal.BiVariableMap.put(BiVariableMap.java:321)
> at org.jruby.embed.ScriptingContainer.put(ScriptingContainer.java:1123)
> at
> org.apache.hadoop.hbase.client.AbstractTestShell.setUpBeforeClass(AbstractTestShell.java:61)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:24)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runners.Suite.runChild(Suite.java:128)
> at org.junit.runners.Suite.runChild(Suite.java:27)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:137)
> at org.junit.runner.JUnitCore.run(JUnitCore.java:115)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.createRequestAndRun(JUnitCoreWrapper.java:108)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.executeEager(JUnitCoreWrapper.java:78)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreWrapper.execute(JUnitCoreWrapper.java:54)
> at
> org.apache.maven.surefire.junitcore.JUnitCoreProvider.invoke(JUnitCoreProvider.java:144)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
>
[jira] [Comment Edited] (HBASE-16683) Address review comments for mega patch of backup / restore feature
[ https://issues.apache.org/jira/browse/HBASE-16683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514158#comment-15514158 ] Ted Yu edited comment on HBASE-16683 at 9/22/16 9:10 PM: - Checking in after running backup tests. Will keep the JIRA open for further comments. was (Author: yuzhih...@gmail.com): Clarification: waiting for more review comments before checking in. > Address review comments for mega patch of backup / restore feature > -- > > Key: HBASE-16683 > URL: https://issues.apache.org/jira/browse/HBASE-16683 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu > Labels: backup > Attachments: 16683.v1.txt > > > There're review comments for the mega patch posted over HBASE-14123. > See https://reviews.apache.org/r/51823/ > This issue is for addressing review comments in HBASE-7912 branch. > Mega patch for master branch would be posted / tested on HBASE-14123. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-16687) Remove MaxPermSize from surefire/failsave command line
[
https://issues.apache.org/jira/browse/HBASE-16687?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Purtell resolved HBASE-16687.
Resolution: Duplicate
Assignee: (was: Andrew Purtell)
Fix Version/s: (was: 2.0.0)
> Remove MaxPermSize from surefire/failsave command line
> --
>
> Key: HBASE-16687
> URL: https://issues.apache.org/jira/browse/HBASE-16687
> Project: HBase
> Issue Type: Task
>Reporter: Andrew Purtell
>Priority: Trivial
> Attachments: HBASE-16687.patch
>
>
> Master branch requires Java 8, so we can eliminate the reason for these noisy
> messages:
> {noformat}
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16687) Remove MaxPermSize from surefire/failsave command line
[
https://issues.apache.org/jira/browse/HBASE-16687?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514459#comment-15514459
]
Andrew Purtell commented on HBASE-16687:
Funny how that works with different people coming to the same conclusion around
the same time. Fine to close this as dup
> Remove MaxPermSize from surefire/failsave command line
> --
>
> Key: HBASE-16687
> URL: https://issues.apache.org/jira/browse/HBASE-16687
> Project: HBase
> Issue Type: Task
>Reporter: Andrew Purtell
>Assignee: Andrew Purtell
>Priority: Trivial
> Fix For: 2.0.0
>
> Attachments: HBASE-16687.patch
>
>
> Master branch requires Java 8, so we can eliminate the reason for these noisy
> messages:
> {noformat}
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=256m;
> support was removed in 8.0
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16688) Split TestMasterFailoverWithProcedures
[ https://issues.apache.org/jira/browse/HBASE-16688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16688: Status: Patch Available (was: Open) > Split TestMasterFailoverWithProcedures > -- > > Key: HBASE-16688 > URL: https://issues.apache.org/jira/browse/HBASE-16688 > Project: HBase > Issue Type: Bug > Components: proc-v2, test >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-16688-v0.patch > > > extract the WAL lease tests from the TestMasterFailoverWithProcedures. > leaving TestMasterFailoverWithProcedures with only the proc test on master > failover -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16681) Fix flaky TestReplicationSourceManagerZkImpl
[
https://issues.apache.org/jira/browse/HBASE-16681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514426#comment-15514426
]
Appy commented on HBASE-16681:
--
[~ashu210890] confirmed that it's connected to jira mentioned above.
https://issues.apache.org/jira/browse/HBASE-16096?focusedCommentId=15514351&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15514351
> Fix flaky TestReplicationSourceManagerZkImpl
>
>
> Key: HBASE-16681
> URL: https://issues.apache.org/jira/browse/HBASE-16681
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Ashu Pachauri
>
> Stack Trace
> {noformat}
> java.io.IOException: java.lang.reflect.InvocationTargetException
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:233)
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:118)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.init(HBaseInterClusterReplicationEndpoint.java:119)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.getReplicationSource(ReplicationSourceManager.java:502)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.addSource(ReplicationSourceManager.java:273)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.init(ReplicationSourceManager.java:246)
> at
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager.testLogRoll(TestReplicationSourceManager.java:228)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Caused by: java.lang.reflect.InvocationTargetException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:231)
> at
> org.apac
[jira] [Commented] (HBASE-16630) Fragmentation in long running Bucket Cache
[
https://issues.apache.org/jira/browse/HBASE-16630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15514424#comment-15514424
]
Ted Yu commented on HBASE-16630:
{code}
+ Set candidateBuckets = bucketAllocator.getLeastFilledBuckets(
+ inUseBuckets, completelyFreeBucketsNeeded);
...
+ * @param excludedBuckets the buckets that need to be excluded due to
+ *currently being in used
{code}
In use buckets are excluded from freeing.
> Fragmentation in long running Bucket Cache
> --
>
> Key: HBASE-16630
> URL: https://issues.apache.org/jira/browse/HBASE-16630
> Project: HBase
> Issue Type: Bug
> Components: BucketCache
>Affects Versions: 2.0.0, 1.1.6, 1.3.1, 1.2.3
>Reporter: deepankar
>Assignee: deepankar
> Attachments: 16630-v2-suggest.patch, HBASE-16630-v2.patch,
> HBASE-16630-v3.patch, HBASE-16630.patch
>
>
> As we are running bucket cache for a long time in our system, we are
> observing cases where some nodes after some time does not fully utilize the
> bucket cache, in some cases it is even worse in the sense they get stuck at a
> value < 0.25 % of the bucket cache (DEFAULT_MEMORY_FACTOR as all our tables
> are configured in-memory for simplicity sake).
> We took a heap dump and analyzed what is happening and saw that is classic
> case of fragmentation, current implementation of BucketCache (mainly
> BucketAllocator) relies on the logic that fullyFreeBuckets are available for
> switching/adjusting cache usage between different bucketSizes . But once a
> compaction / bulkload happens and the blocks are evicted from a bucket size ,
> these are usually evicted from random places of the buckets of a bucketSize
> and thus locking the number of buckets associated with a bucketSize and in
> the worst case of the fragmentation we have seen some bucketSizes with
> occupancy ratio of < 10 % But they dont have any completelyFreeBuckets to
> share with the other bucketSize.
> Currently the existing eviction logic helps in the cases where cache used is
> more the MEMORY_FACTOR or MULTI_FACTOR and once those evictions are also
> done, the eviction (freeSpace function) will not evict anything and the cache
> utilization will be stuck at that value without any allocations for other
> required sizes.
> The fix for this we came up with is simple that we do deFragmentation (
> compaction) of the bucketSize and thus increasing the occupancy ratio and
> also freeing up the buckets to be fullyFree, this logic itself is not
> complicated as the bucketAllocator takes care of packing the blocks in the
> buckets, we need evict and re-allocate the blocks for all the BucketSizes
> that dont fit the criteria.
> I am attaching an initial patch just to give an idea of what we are thinking
> and I'll improve it based on the comments from the community.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16688) Split TestMasterFailoverWithProcedures
[ https://issues.apache.org/jira/browse/HBASE-16688?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-16688: Attachment: HBASE-16688-v0.patch > Split TestMasterFailoverWithProcedures > -- > > Key: HBASE-16688 > URL: https://issues.apache.org/jira/browse/HBASE-16688 > Project: HBase > Issue Type: Bug > Components: proc-v2, test >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 2.0.0 > > Attachments: HBASE-16688-v0.patch > > > extract the WAL lease tests from the TestMasterFailoverWithProcedures. > leaving TestMasterFailoverWithProcedures with only the proc test on master > failover -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16681) Fix flaky TestReplicationSourceManagerZkImpl
[
https://issues.apache.org/jira/browse/HBASE-16681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Appy updated HBASE-16681:
-
Assignee: Ashu Pachauri
> Fix flaky TestReplicationSourceManagerZkImpl
>
>
> Key: HBASE-16681
> URL: https://issues.apache.org/jira/browse/HBASE-16681
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Ashu Pachauri
>
> Stack Trace
> {noformat}
> java.io.IOException: java.lang.reflect.InvocationTargetException
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:233)
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:118)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.init(HBaseInterClusterReplicationEndpoint.java:119)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.getReplicationSource(ReplicationSourceManager.java:502)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.addSource(ReplicationSourceManager.java:273)
> at
> org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager.init(ReplicationSourceManager.java:246)
> at
> org.apache.hadoop.hbase.replication.regionserver.TestReplicationSourceManager.testLogRoll(TestReplicationSourceManager.java:228)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:50)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:55)
> at org.junit.rules.RunRules.evaluate(RunRules.java:20)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:325)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:78)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:57)
> at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
> at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
> at
> org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
> at
> org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
> at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:283)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:173)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:153)
> at
> org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:128)
> at
> org.apache.maven.surefire.booter.ForkedBooter.invokeProviderInSameClassLoader(ForkedBooter.java:203)
> at
> org.apache.maven.surefire.booter.ForkedBooter.runSuitesInProcess(ForkedBooter.java:155)
> at
> org.apache.maven.surefire.booter.ForkedBooter.main(ForkedBooter.java:103)
> Caused by: java.lang.reflect.InvocationTargetException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:231)
> at
> org.apache.hadoop.hbase.client.ConnectionFactory.createConnection(ConnectionFactory.java:118)
> at
> org.apache.hadoop.hbase.replication.regionserver.HBaseInterClusterReplicationEndpoint.init(HBaseInterClusterReplicationEndpoint.java:119)
> at
> org.apache
[jira] [Created] (HBASE-16688) Split TestMasterFailoverWithProcedures
Matteo Bertozzi created HBASE-16688: --- Summary: Split TestMasterFailoverWithProcedures Key: HBASE-16688 URL: https://issues.apache.org/jira/browse/HBASE-16688 Project: HBase Issue Type: Bug Components: proc-v2, test Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Fix For: 2.0.0 extract the WAL lease tests from the TestMasterFailoverWithProcedures. leaving TestMasterFailoverWithProcedures with only the proc test on master failover -- This message was sent by Atlassian JIRA (v6.3.4#6332)
