[jira] [Comment Edited] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[ https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512359#comment-15512359 ] Ted Yu edited comment on HBASE-16672 at 9/22/16 6:57 AM: - Restoring to another cluster is one scenario. On the cluster where restore is performed, multiple destination tables should be supported. Local restore (target table on the same cluster) should also be supported. was (Author: yuzhih...@gmail.com): Restoring to another cluster is one scenario. Local restore (target table on the same cluster) should also be supported. > Add option for bulk load to copy hfile(s) instead of renaming > - > > Key: HBASE-16672 > URL: https://issues.apache.org/jira/browse/HBASE-16672 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16672.v1.txt, 16672.v2.txt, 16672.v3.txt, 16672.v4.txt, > 16672.v5.txt > > > This is related to HBASE-14417, to support incrementally restoring to > multiple destinations, this issue adds option which would always copy > hfile(s) during bulk load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12088) Remove un-used profiles in non-root poms
[ https://issues.apache.org/jira/browse/HBASE-12088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512363#comment-15512363 ] Hudson commented on HBASE-12088: FAILURE: Integrated in Jenkins build HBase-1.4 #422 (See [https://builds.apache.org/job/HBase-1.4/422/]) HBASE-12088 Remove unused hadoop-1.0, hadoop-1.1 profiles from non-root (jmhsieh: rev 13d6acbc7f719bad6e0b618455c034577deae528) * (edit) hbase-external-blockcache/pom.xml * (edit) hbase-procedure/pom.xml * (edit) hbase-server/pom.xml * (edit) hbase-thrift/pom.xml * (edit) hbase-common/pom.xml * (edit) hbase-client/pom.xml * (edit) hbase-examples/pom.xml * (edit) hbase-it/pom.xml * (edit) hbase-prefix-tree/pom.xml * (edit) hbase-testing-util/pom.xml * (edit) hbase-shell/pom.xml HBASE-12088 Addendum - fix spacing (jmhsieh: rev ecc1c294f526f02d03e877af872bc6fc284f0a28) * (edit) hbase-common/pom.xml * (edit) hbase-external-blockcache/pom.xml * (edit) hbase-client/pom.xml > Remove un-used profiles in non-root poms > > > Key: HBASE-12088 > URL: https://issues.apache.org/jira/browse/HBASE-12088 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Jonathan Hsieh > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-12088.v0.patch > > > Some of the poms still have hadoop 1 and hadoop 1.1 profiles even though the > root pom has them removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16665) Check whether KeyValueUtil.createXXX could be replaced by CellUtil without copy
[ https://issues.apache.org/jira/browse/HBASE-16665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512356#comment-15512356 ] Anoop Sam John commented on HBASE-16665: +1. createXXX method should go away from KVUtil.. We can keep KVUtil though. Wherever we do ops with Cell in a KV format way, we can use this util. Like the Oswrite method.. Here we write the Cell in a KV format. Similar things. All others we better put in CellUtil alone? You up for doing all those cleanup as part of this work? :-) Am ok if u want that to be part of another also. > Check whether KeyValueUtil.createXXX could be replaced by CellUtil without > copy > --- > > Key: HBASE-16665 > URL: https://issues.apache.org/jira/browse/HBASE-16665 > Project: HBase > Issue Type: Bug >Reporter: Heng Chen > Attachments: HBASE-16665.patch, HBASE-16665.v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[ https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512359#comment-15512359 ] Ted Yu commented on HBASE-16672: Restoring to another cluster is one scenario. Local restore (target table on the same cluster) should also be supported. > Add option for bulk load to copy hfile(s) instead of renaming > - > > Key: HBASE-16672 > URL: https://issues.apache.org/jira/browse/HBASE-16672 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16672.v1.txt, 16672.v2.txt, 16672.v3.txt, 16672.v4.txt, > 16672.v5.txt > > > This is related to HBASE-14417, to support incrementally restoring to > multiple destinations, this issue adds option which would always copy > hfile(s) during bulk load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16670) Make RpcServer#processRequest logic more robust
[
https://issues.apache.org/jira/browse/HBASE-16670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512346#comment-15512346
]
Anoop Sam John commented on HBASE-16670:
+1. Its ok to fix. DNRIOE looks much better Exception reported back, rather
than NPE. Go for the commit pls.
> Make RpcServer#processRequest logic more robust
> ---
>
> Key: HBASE-16670
> URL: https://issues.apache.org/jira/browse/HBASE-16670
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Minor
> Attachments: HBASE-16670.patch
>
>
> Currently in {{RpcServer#processRequest}}, we will check whether the request
> header has parameters but missed handling the abnormal case, so if there's no
> param in the header, it will throw NPE like below:
> {noformat}
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):
> java.io.IOException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2269)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:189)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:169)
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2211)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16643) Reverse scanner heap creation may not allow MSLAB closure due to improper ref counting of segments
[
https://issues.apache.org/jira/browse/HBASE-16643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512343#comment-15512343
]
Anoop Sam John commented on HBASE-16643:
Not sure for the bug fix we need this big change.
We can just remove below line and that will fix the bug
forwardHeap.close();
It is ok also.. As we dont really closing the scanners here.. We just replace
the container KVHeap with a ReverseKVHeap with same all SegmentScanners. Pls
consider fix that way?
The reverse scan info was not passed from StoreScanner level onwards. If u
strongly feel we need some refactor and pass the reverse type throughout,
suggest fix in another IA issue. There can be some other improvements also
possible there.
BTW just noticed that methods in MemstoreScanner is made synchronized. I dont
think we really need it. That also can be avoided in that IA issue.
Pls add some comments why we dont close forwardHeap but just nullify it.
> Reverse scanner heap creation may not allow MSLAB closure due to improper ref
> counting of segments
> --
>
> Key: HBASE-16643
> URL: https://issues.apache.org/jira/browse/HBASE-16643
> Project: HBase
> Issue Type: Bug
>Reporter: ramkrishna.s.vasudevan
>Assignee: ramkrishna.s.vasudevan
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HBASE-16643.patch, HBASE-16643_1.patch,
> HBASE-16643_2.patch
>
>
> In the reverse scanner case,
> While doing 'initBackwardHeapIfNeeded' in MemstoreScanner for setting the
> backward heap, we do a
> {code}
> if ((backwardHeap == null) && (forwardHeap != null)) {
> forwardHeap.close();
> forwardHeap = null;
> // before building the heap seek for the relevant key on the scanners,
> // for the heap to be built from the scanners correctly
> for (KeyValueScanner scan : scanners) {
> if (toLast) {
> res |= scan.seekToLastRow();
> } else {
> res |= scan.backwardSeek(cell);
> }
> }
> {code}
> forwardHeap.close(). This would internally decrement the MSLAB ref counter
> for the current active segment and snapshot segment.
> When the scan is actually closed again we do close() and that will again
> decrement the count. Here chances are there that the count would go negative
> and hence the actual MSLAB closure that checks for refCount==0 will fail.
> Apart from this, when the refCount becomes 0 after the firstClose if any
> other thread requests to close the segment, then we will end up in corrupted
> segment because the segment could be put back to the MSLAB pool.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HBASE-16665) Check whether KeyValueUtil.createXXX could be replaced by CellUtil without copy
[ https://issues.apache.org/jira/browse/HBASE-16665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512330#comment-15512330 ] Heng Chen edited comment on HBASE-16665 at 9/22/16 6:22 AM: Another thoughts about KeyValueUtil and CellUtil, we'd better to tell our developers in which scenario should use CellUtil to avoid copy instead of KeyValueUtil, maybe the methods "createXXX" in KeyValueUtil could be removed directly (most seems to be used only in test case)? was (Author: chenheng): Another thoughts about KeyValueUtil and CellUtil, we'd better to tell our developers in which scenario should use CellUtil to avoid copy instead of KeyValueUtil, maybe the methods of createXXX could be removed directly (most seems to be used only in test case)? > Check whether KeyValueUtil.createXXX could be replaced by CellUtil without > copy > --- > > Key: HBASE-16665 > URL: https://issues.apache.org/jira/browse/HBASE-16665 > Project: HBase > Issue Type: Bug >Reporter: Heng Chen > Attachments: HBASE-16665.patch, HBASE-16665.v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16665) Check whether KeyValueUtil.createXXX could be replaced by CellUtil without copy
[ https://issues.apache.org/jira/browse/HBASE-16665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512330#comment-15512330 ] Heng Chen commented on HBASE-16665: --- Another thoughts about KeyValueUtil and CellUtil, we'd better to tell our developers in which scenario should use CellUtil to avoid copy instead of KeyValueUtil, maybe the methods of createXXX could be removed directly (most seems to be used only in test case)? > Check whether KeyValueUtil.createXXX could be replaced by CellUtil without > copy > --- > > Key: HBASE-16665 > URL: https://issues.apache.org/jira/browse/HBASE-16665 > Project: HBase > Issue Type: Bug >Reporter: Heng Chen > Attachments: HBASE-16665.patch, HBASE-16665.v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16670) Make RpcServer#processRequest logic more robust
[
https://issues.apache.org/jira/browse/HBASE-16670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512318#comment-15512318
]
Heng Chen commented on HBASE-16670:
---
sounds reasonable. Will be more friendly to our users. +1
> Make RpcServer#processRequest logic more robust
> ---
>
> Key: HBASE-16670
> URL: https://issues.apache.org/jira/browse/HBASE-16670
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Minor
> Attachments: HBASE-16670.patch
>
>
> Currently in {{RpcServer#processRequest}}, we will check whether the request
> header has parameters but missed handling the abnormal case, so if there's no
> param in the header, it will throw NPE like below:
> {noformat}
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):
> java.io.IOException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2269)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:189)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:169)
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2211)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16665) Check whether KeyValueUtil.createXXX could be replaced by CellUtil without copy
[ https://issues.apache.org/jira/browse/HBASE-16665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512319#comment-15512319 ] binlijin commented on HBASE-16665: -- +1 > Check whether KeyValueUtil.createXXX could be replaced by CellUtil without > copy > --- > > Key: HBASE-16665 > URL: https://issues.apache.org/jira/browse/HBASE-16665 > Project: HBase > Issue Type: Bug >Reporter: Heng Chen > Attachments: HBASE-16665.patch, HBASE-16665.v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[ https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512313#comment-15512313 ] Ashish Singhi commented on HBASE-16672: --- One doubt here, pardon me I haven't went through backup design yet. What I understand is backup of data will be stored on another cluster. On restore, the data will be transferred from backup cluster to the main cluster so there will be two different FS involved, so data will finally be copied to source cluster, it will not be renamed. Is my understanding correct ? > Add option for bulk load to copy hfile(s) instead of renaming > - > > Key: HBASE-16672 > URL: https://issues.apache.org/jira/browse/HBASE-16672 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16672.v1.txt, 16672.v2.txt, 16672.v3.txt, 16672.v4.txt, > 16672.v5.txt > > > This is related to HBASE-14417, to support incrementally restoring to > multiple destinations, this issue adds option which would always copy > hfile(s) during bulk load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16665) Check whether KeyValueUtil.createXXX could be replaced by CellUtil without copy
[ https://issues.apache.org/jira/browse/HBASE-16665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-16665: -- Attachment: HBASE-16665.v1.patch patch v1 deal with createLastOnRow and address [~anoop.hbase] suggestion. > Check whether KeyValueUtil.createXXX could be replaced by CellUtil without > copy > --- > > Key: HBASE-16665 > URL: https://issues.apache.org/jira/browse/HBASE-16665 > Project: HBase > Issue Type: Bug >Reporter: Heng Chen > Attachments: HBASE-16665.patch, HBASE-16665.v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16665) Check whether KeyValueUtil.createXXX could be replaced by CellUtil without copy
[ https://issues.apache.org/jira/browse/HBASE-16665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Heng Chen updated HBASE-16665: -- Status: Patch Available (was: Open) > Check whether KeyValueUtil.createXXX could be replaced by CellUtil without > copy > --- > > Key: HBASE-16665 > URL: https://issues.apache.org/jira/browse/HBASE-16665 > Project: HBase > Issue Type: Bug >Reporter: Heng Chen > Attachments: HBASE-16665.patch, HBASE-16665.v1.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16670) Make RpcServer#processRequest logic more robust
[
https://issues.apache.org/jira/browse/HBASE-16670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512286#comment-15512286
]
Yu Li commented on HBASE-16670:
---
I found this incomplete logic when developing async table codes, some mistake
lead me to some logic like calling {{getStub().mutate(controller, null);}}. So
in our current codes there won't be such a kind of request, this is why I set
the priority of this JIRA to Minor, but I think this is still something worth
improving. Agree?:-)
> Make RpcServer#processRequest logic more robust
> ---
>
> Key: HBASE-16670
> URL: https://issues.apache.org/jira/browse/HBASE-16670
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Minor
> Attachments: HBASE-16670.patch
>
>
> Currently in {{RpcServer#processRequest}}, we will check whether the request
> header has parameters but missed handling the abnormal case, so if there's no
> param in the header, it will throw NPE like below:
> {noformat}
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):
> java.io.IOException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2269)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:189)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:169)
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2211)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14734) BindException when setting up MiniKdc
[ https://issues.apache.org/jira/browse/HBASE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512263#comment-15512263 ] Hudson commented on HBASE-14734: FAILURE: Integrated in Jenkins build HBase-1.2-JDK8 #26 (See [https://builds.apache.org/job/HBase-1.2-JDK8/26/]) HBASE-14734 Prevent BindException when setting up MiniKdc. Port for kdc (appy: rev 79531fd95f1b7b72ad1f764d7cd57e37467a4b2a) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestUsersOperationsWithSecureHadoop.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestSecureRPC.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/token/TestGenerateDelegationToken.java > BindException when setting up MiniKdc > - > > Key: HBASE-14734 > URL: https://issues.apache.org/jira/browse/HBASE-14734 > Project: HBase > Issue Type: Sub-task > Components: flakey, test >Reporter: stack >Assignee: Appy > Fix For: 1.3.0, 1.4.0, 1.2.4, 2.0.. > > Attachments: HBASE-14734.master.001.patch > > > Root cause : Port for kdc service gets selected in the constructor, but we > bind to it later in MiniKdc.start()-->MiniKdc.initKDCServer() --> > KdcServer.start(). In meantime, some other service can capture the port which > results in BindException. The solution here is to catch the exception and > retry. > From > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/330/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.security.token/TestGenerateDelegationToken/org_apache_hadoop_hbase_security_token_TestGenerateDelegationToken/ > Error Message > Address already in use > Stacktrace > java.net.BindException: Address already in use > at sun.nio.ch.Net.bind0(Native Method) > at sun.nio.ch.Net.bind(Net.java:444) > at sun.nio.ch.Net.bind(Net.java:436) > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422) > at > org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Can this utility be made to not fail if address taken? Try another? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16651) LRUBlockCache#returnBlock should try return block to Victim Handler L2 cache.
[ https://issues.apache.org/jira/browse/HBASE-16651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-16651: --- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the review Ram. > LRUBlockCache#returnBlock should try return block to Victim Handler L2 cache. > - > > Key: HBASE-16651 > URL: https://issues.apache.org/jira/browse/HBASE-16651 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 2.0.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16651.patch > > > In case of L1 and L2 cache usage with combinedMode = false, L2 is used as a > victim handler cache for L1 cache. When a getBlock() request comes, L1 will > see if block is in it and if not it will try to provide the block from L2 > cache. In such a case, the return block must return the block to L2 cache and > count down the ref count for the block. But right now we just ignore the > returnBlock call in LRUCache -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16651) LRUBlockCache#returnBlock should try return block to Victim Handler L2 cache.
[ https://issues.apache.org/jira/browse/HBASE-16651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512256#comment-15512256 ] ramkrishna.s.vasudevan commented on HBASE-16651: +1 > LRUBlockCache#returnBlock should try return block to Victim Handler L2 cache. > - > > Key: HBASE-16651 > URL: https://issues.apache.org/jira/browse/HBASE-16651 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 2.0.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16651.patch > > > In case of L1 and L2 cache usage with combinedMode = false, L2 is used as a > victim handler cache for L1 cache. When a getBlock() request comes, L1 will > see if block is in it and if not it will try to provide the block from L2 > cache. In such a case, the return block must return the block to L2 cache and > count down the ref count for the block. But right now we just ignore the > returnBlock call in LRUCache -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16650) Wrong usage of BlockCache eviction stat for heap memory tuning
[ https://issues.apache.org/jira/browse/HBASE-16650?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-16650: --- Attachment: HBASE-16650_V2.patch > Wrong usage of BlockCache eviction stat for heap memory tuning > -- > > Key: HBASE-16650 > URL: https://issues.apache.org/jira/browse/HBASE-16650 > Project: HBase > Issue Type: Sub-task >Affects Versions: 1.0.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 1.4.0 > > Attachments: HBASE-16650.patch, HBASE-16650_V2.patch > > > 1. We use the stat evictedBlocksCount - A block can get evicted because of > eviction thread due to lack of space or because of removal of an HFile itself > (After a compaction). We should not consider the latter in the tune decision > at all. These are actually invalidation of blocks. Should the stat counter > itself not use this count of evicted blocks? I think yes. This will give > wrong message to users that there are lot of real eviction happening. > 2. In case L1+ L2 combined block cache, what we use is the sum of evictions > from both. But we will be tuning L1 size alone. Eviction count from L2 should > not affect the tuning of L1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16680) Reduce garbage in BufferChain
[ https://issues.apache.org/jira/browse/HBASE-16680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512231#comment-15512231 ] Ashish Singhi commented on HBASE-16680: --- +1 > Reduce garbage in BufferChain > - > > Key: HBASE-16680 > URL: https://issues.apache.org/jira/browse/HBASE-16680 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16680-master.patch, HBASE-16680-master_v2.patch > > > BufferChain accept a List and then convert it to ByteBuffer[], we > can directly produce ByteBuffer[] and handle it to BufferChain, so eliminate > the object List. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16680) Reduce garbage in BufferChain
[ https://issues.apache.org/jira/browse/HBASE-16680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512227#comment-15512227 ] binlijin commented on HBASE-16680: -- HBASE-16680-master_v2.patch fix it. > Reduce garbage in BufferChain > - > > Key: HBASE-16680 > URL: https://issues.apache.org/jira/browse/HBASE-16680 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16680-master.patch, HBASE-16680-master_v2.patch > > > BufferChain accept a List and then convert it to ByteBuffer[], we > can directly produce ByteBuffer[] and handle it to BufferChain, so eliminate > the object List. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512226#comment-15512226 ] Ashish Singhi commented on HBASE-16676: --- I am +1 to commit this in branch-1.2 > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.2.4 > > Attachments: HBASE-16676-branch-1.2.patch > > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16680) Reduce garbage in BufferChain
[ https://issues.apache.org/jira/browse/HBASE-16680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-16680: - Attachment: HBASE-16680-master_v2.patch > Reduce garbage in BufferChain > - > > Key: HBASE-16680 > URL: https://issues.apache.org/jira/browse/HBASE-16680 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16680-master.patch, HBASE-16680-master_v2.patch > > > BufferChain accept a List and then convert it to ByteBuffer[], we > can directly produce ByteBuffer[] and handle it to BufferChain, so eliminate > the object List. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=1551#comment-1551 ] Sean Busbey commented on HBASE-16676: - it seems silly to not include the fix in branch-1.2 if many of the deployments our committers are involved with using 1.2.z are going patch for the issue. > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.2.4 > > Attachments: HBASE-16676-branch-1.2.patch > > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[ https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512213#comment-15512213 ] Ted Yu commented on HBASE-16672: SecureBulkLoadListener has logic for the copy through staging dir. I think we should reuse this part of code. > Add option for bulk load to copy hfile(s) instead of renaming > - > > Key: HBASE-16672 > URL: https://issues.apache.org/jira/browse/HBASE-16672 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16672.v1.txt, 16672.v2.txt, 16672.v3.txt, 16672.v4.txt, > 16672.v5.txt > > > This is related to HBASE-14417, to support incrementally restoring to > multiple destinations, this issue adds option which would always copy > hfile(s) during bulk load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16650) Wrong usage of BlockCache eviction stat for heap memory tuning
[
https://issues.apache.org/jira/browse/HBASE-16650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512212#comment-15512212
]
Anoop Sam John commented on HBASE-16650:
bq.We are setting some flags in this block that are used later. Do we need
those flags when an eviction because an hfile has been removed?
No there are no flags being set inside the if and used outside..
Code before patch
{code}
stats.evicted(block.getCachedTime(), block.getCacheKey().isPrimary());
if (evictedByEvictionProcess && victimHandler != null) {
if (victimHandler instanceof BucketCache) {
boolean wait = getCurrentSize() < acceptableSize();
boolean inMemory = block.getPriority() == BlockPriority.MEMORY;
((BucketCache)victimHandler).cacheBlockWithWait(block.getCacheKey(),
block.getBuffer(),
inMemory, wait);
} else {
victimHandler.cacheBlock(block.getCacheKey(), block.getBuffer());
}
}
{code}
Code after this patch
{code}
if (evictedByEvictionProcess ) {
stats.evicted(block.getCachedTime(), block.getCacheKey().isPrimary());
if(victimHandler != null){
if (victimHandler instanceof BucketCache) {
boolean wait = getCurrentSize() < acceptableSize();
boolean inMemory = block.getPriority() == BlockPriority.MEMORY;
((BucketCache)victimHandler).cacheBlockWithWait(block.getCacheKey(),
block.getBuffer(),
inMemory, wait);
} else {
victimHandler.cacheBlock(block.getCacheKey(), block.getBuffer());
}
}
}
{code}
Will commit with addressing ur 1st comment
> Wrong usage of BlockCache eviction stat for heap memory tuning
> --
>
> Key: HBASE-16650
> URL: https://issues.apache.org/jira/browse/HBASE-16650
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 1.0.0
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-16650.patch
>
>
> 1. We use the stat evictedBlocksCount - A block can get evicted because of
> eviction thread due to lack of space or because of removal of an HFile itself
> (After a compaction). We should not consider the latter in the tune decision
> at all. These are actually invalidation of blocks. Should the stat counter
> itself not use this count of evicted blocks? I think yes. This will give
> wrong message to users that there are lot of real eviction happening.
> 2. In case L1+ L2 combined block cache, what we use is the sum of evictions
> from both. But we will be tuning L1 size alone. Eviction count from L2 should
> not affect the tuning of L1
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512207#comment-15512207 ] Andrew Purtell commented on HBASE-16676: Sorry I missed the discussion on the other issue. We will definitely patch for this if moving onto 1.2 based code given how it will impact us. Chasing this down after seeing ITBLL failures with 1.2 burned a lot of time for me. Related, I can report a decent probability this will fail ITBLL if you use clusterdock as it is now to test. > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.2.4 > > Attachments: HBASE-16676-branch-1.2.patch > > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16651) LRUBlockCache#returnBlock should try return block to Victim Handler L2 cache.
[ https://issues.apache.org/jira/browse/HBASE-16651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512199#comment-15512199 ] Anoop Sam John commented on HBASE-16651: [~ram_krish] does ur concerns being answered above? Can I get a +1 pls? > LRUBlockCache#returnBlock should try return block to Victim Handler L2 cache. > - > > Key: HBASE-16651 > URL: https://issues.apache.org/jira/browse/HBASE-16651 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 2.0.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0 > > Attachments: HBASE-16651.patch > > > In case of L1 and L2 cache usage with combinedMode = false, L2 is used as a > victim handler cache for L1 cache. When a getBlock() request comes, L1 will > see if block is in it and if not it will try to provide the block from L2 > cache. In such a case, the return block must return the block to L2 cache and > count down the ref count for the block. But right now we just ignore the > returnBlock call in LRUCache -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512187#comment-15512187 ] Ashish Singhi commented on HBASE-16676: --- I think we should bring this change in branch-1.2 also. Even our customers reported this and unfortunately we are maintaining this as a private change in our version. As at that time it was decided that it will not be committed in branch-1.2. Committing it in branch-1.2 will help many of us and IMO this seems more like a bug fix! > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.2.4 > > Attachments: HBASE-16676-branch-1.2.patch > > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16666) Add append and remove peer namespaces cmds for replication
[ https://issues.apache.org/jira/browse/HBASE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-1: --- Attachment: HBASE-1-v1.patch Attach v1. Failed ut Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.minikdc.MiniKdc. Update hbase-shell pom for minikdc. > Add append and remove peer namespaces cmds for replication > -- > > Key: HBASE-1 > URL: https://issues.apache.org/jira/browse/HBASE-1 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Minor > Attachments: HBASE-1-v1.patch, HBASE-1.patch > > > After HBASE-16447, we support replication by namespaces config in peer. Like > append_peer_tableCFs and remove_peer_tableCFs, I thought we need two new > shell cmd: append_peer_namespaces and remove_peer_namespaces. Then we can > easily change the namespaces config. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[ https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512172#comment-15512172 ] Anoop Sam John commented on HBASE-16672: The copy of the backup files should be done by the restore op and then pass the copied file path to the bulk load rather than giving option for the bulk load itself to rename/copy? > Add option for bulk load to copy hfile(s) instead of renaming > - > > Key: HBASE-16672 > URL: https://issues.apache.org/jira/browse/HBASE-16672 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16672.v1.txt, 16672.v2.txt, 16672.v3.txt, 16672.v4.txt, > 16672.v5.txt > > > This is related to HBASE-14417, to support incrementally restoring to > multiple destinations, this issue adds option which would always copy > hfile(s) during bulk load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16679) Flush throughput controller: Minor perf change and fix flaky TestFlushWithThroughputController
[
https://issues.apache.org/jira/browse/HBASE-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512174#comment-15512174
]
Duo Zhang commented on HBASE-16679:
---
The {{maxThroughputPerOperation}} should be volatile.
And the {{HBaseTestingUtility}} does not need to be static as you recreate it
in setUp?
Thanks.
> Flush throughput controller: Minor perf change and fix flaky
> TestFlushWithThroughputController
> --
>
> Key: HBASE-16679
> URL: https://issues.apache.org/jira/browse/HBASE-16679
> Project: HBase
> Issue Type: Bug
>Reporter: Appy
>Assignee: Appy
> Attachments: HBASE-16679.master.001.patch,
> HBASE-16679.master.002.patch
>
>
> Minor perf change:
> Calculate maxThroughputPerOperation outside of control() since start()&end()
> are called only once per operation, but control can be called
> hundreds/thousands of time.
> Flaky test:
> Problems in current test:
> - writes only 2.5MB each iteration but control triggers sleep only every 1Mb
> write (decided by HBASE_HSTORE_FLUSH_THROUGHPUT_CONTROL_CHECK_INTERVAL).
> Either increase data written in each batch or decreasing this threshold for
> better throughput control.
> - We shouldn't be timing table disable/delete/create and populating data in
> throughput calculations.
> See the differences below.
> With patch (total data written 30M)
> run 1:
> Throughput is: 1.0113841089709052 MB/s
> Throughput w/o limit is: 14.665069580078125 MB/s
> With 1M/s limit, flush use 29683ms; without limit, flush use 2130ms
> run 2:
> Throughput is: 1.0113841089709052 MB/s
> Throughput w/o limit is: 14.665069580078125 MB/s
> With 1M/s limit, flush use 29674ms; without limit, flush use 2027ms
> Without patch (total data written 25M)
> run 1:
> Throughput is: 0.921681903523776 MB/s
> Throughput w/o limit is: 4.06833346870301 MB/s
> With 1M/s limit, flush use 27189ms; without limit, flush use 6159ms
> run 2:
> Throughput is: 0.9422982728478803 MB/s
> Throughput w/o limit is: 4.047858424942981 MB/s
> With 1M/s limit, flush use 26594ms; without limit, flush use 6190ms
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512170#comment-15512170 ] Sean Busbey commented on HBASE-16676: - no worries [~ashish singhi]. do you think the impact doesn't warrant the change in behavior? > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.2.4 > > Attachments: HBASE-16676-branch-1.2.patch > > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512161#comment-15512161 ] Ashish Singhi commented on HBASE-16676: --- Sorry, I did not refresh the page and missed Sean's comment. > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.2.4 > > Attachments: HBASE-16676-branch-1.2.patch > > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512151#comment-15512151 ] Ashish Singhi commented on HBASE-16676: --- The discussion regarding this was happened on HBASE-15315 and was decided to not commit in branch-1.2 as it will introduce a behavior change. > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.2.4 > > Attachments: HBASE-16676-branch-1.2.patch > > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16670) Make RpcServer#processRequest logic more robust
[
https://issues.apache.org/jira/browse/HBASE-16670?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512149#comment-15512149
]
Anoop Sam John commented on HBASE-16670:
So how u r getting a requesting of this type?
> Make RpcServer#processRequest logic more robust
> ---
>
> Key: HBASE-16670
> URL: https://issues.apache.org/jira/browse/HBASE-16670
> Project: HBase
> Issue Type: Bug
>Reporter: Yu Li
>Assignee: Yu Li
>Priority: Minor
> Attachments: HBASE-16670.patch
>
>
> Currently in {{RpcServer#processRequest}}, we will check whether the request
> header has parameters but missed handling the abnormal case, so if there's no
> param in the header, it will throw NPE like below:
> {noformat}
> org.apache.hadoop.hbase.ipc.RemoteWithExtrasException(java.io.IOException):
> java.io.IOException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2269)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:123)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:189)
> at
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:169)
> Caused by: java.lang.NullPointerException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2211)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512141#comment-15512141 ] Sean Busbey commented on HBASE-16676: - This is effectively a backport of HBASE-15315, right? The original reasoning for that not ending up in branch-1.2 was that it would change operational behavior too much for a maintenance release. It sounds like another ~6 months of time on the 1.2 branch has led to a belief that the downside of leaving this in place is severe enough to change that decision. Is that right? If so, that sounds reasonable to me. ping [~eclark] since he was the other person in favor of leaving HBASE-15315 out of branch-1.2. > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.2.4 > > Attachments: HBASE-16676-branch-1.2.patch > > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[
https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512135#comment-15512135
]
Hadoop QA commented on HBASE-16676:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 28m 48s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
19s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} branch-1.2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} branch-1.2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
16s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s
{color} | {color:green} branch-1.2 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 47s
{color} | {color:green} branch-1.2 passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 45s
{color} | {color:green} branch-1.2 passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
9s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 48s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
18m 43s {color} | {color:green} The patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 28s
{color} | {color:green} the patch passed with JDK v1.8.0_101 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed with JDK v1.7.0_111 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 118m 18s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 192m 56s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestRegionReplicaFailover |
| | hadoop.hbase.regionserver.TestWALLockup |
| | hadoop.hbase.regionserver.TestHRegion |
| | hadoop.hbase.coprocessor.TestRowProcessorEndpoint |
| | hadoop.hbase.mapreduce.TestMultiTableSnapshotInputFormat |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:date2016-09-22 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829735/HBASE-16676-branch-1.2.patch
|
| JIRA Issue | HBASE-16676 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 5b5
[jira] [Commented] (HBASE-16680) Reduce garbage in BufferChain
[
https://issues.apache.org/jira/browse/HBASE-16680?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512129#comment-15512129
]
Ashish Singhi commented on HBASE-16680:
---
bq. for (int i = 0; i < cellBlock.size(); i++) {
Nit: extract cellBlock.size() to a variable and use it.
> Reduce garbage in BufferChain
> -
>
> Key: HBASE-16680
> URL: https://issues.apache.org/jira/browse/HBASE-16680
> Project: HBase
> Issue Type: Improvement
>Reporter: binlijin
>Assignee: binlijin
>Priority: Minor
> Fix For: 2.0.0
>
> Attachments: HBASE-16680-master.patch
>
>
> BufferChain accept a List and then convert it to ByteBuffer[], we
> can directly produce ByteBuffer[] and handle it to BufferChain, so eliminate
> the object List.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14734) BindException when setting up MiniKdc
[ https://issues.apache.org/jira/browse/HBASE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512123#comment-15512123 ] Hudson commented on HBASE-14734: FAILURE: Integrated in Jenkins build HBase-1.3-JDK7 #18 (See [https://builds.apache.org/job/HBase-1.3-JDK7/18/]) HBASE-14734 Prevent BindException when setting up MiniKdc. Port for kdc (appy: rev 1c42948d5ec691b1f27b5c61f5589fdf59e16b6c) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestUsersOperationsWithSecureHadoop.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/AbstractTestSecureIPC.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/token/SecureTestCluster.java > BindException when setting up MiniKdc > - > > Key: HBASE-14734 > URL: https://issues.apache.org/jira/browse/HBASE-14734 > Project: HBase > Issue Type: Sub-task > Components: flakey, test >Reporter: stack >Assignee: Appy > Fix For: 1.3.0, 1.4.0, 1.2.4, 2.0.. > > Attachments: HBASE-14734.master.001.patch > > > Root cause : Port for kdc service gets selected in the constructor, but we > bind to it later in MiniKdc.start()-->MiniKdc.initKDCServer() --> > KdcServer.start(). In meantime, some other service can capture the port which > results in BindException. The solution here is to catch the exception and > retry. > From > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/330/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.security.token/TestGenerateDelegationToken/org_apache_hadoop_hbase_security_token_TestGenerateDelegationToken/ > Error Message > Address already in use > Stacktrace > java.net.BindException: Address already in use > at sun.nio.ch.Net.bind0(Native Method) > at sun.nio.ch.Net.bind(Net.java:444) > at sun.nio.ch.Net.bind(Net.java:436) > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422) > at > org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Can this utility be made to not fail if address taken? Try another? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16680) Reduce garbage in BufferChain
[ https://issues.apache.org/jira/browse/HBASE-16680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-16680: --- Assignee: binlijin > Reduce garbage in BufferChain > - > > Key: HBASE-16680 > URL: https://issues.apache.org/jira/browse/HBASE-16680 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Assignee: binlijin >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16680-master.patch > > > BufferChain accept a List and then convert it to ByteBuffer[], we > can directly produce ByteBuffer[] and handle it to BufferChain, so eliminate > the object List. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16294) hbck reporting "No HDFS region dir found" for replicas
[
https://issues.apache.org/jira/browse/HBASE-16294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512122#comment-15512122
]
Hudson commented on HBASE-16294:
FAILURE: Integrated in Jenkins build HBase-1.3-JDK7 #18 (See
[https://builds.apache.org/job/HBase-1.3-JDK7/18/])
HBASE-16294 hbck reporting "No HDFS region dir found" for replicas
(matteo.bertozzi: rev 9aaff6045b151518a0cdaf694ebd477bff30b2c0)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
> hbck reporting "No HDFS region dir found" for replicas
> --
>
> Key: HBASE-16294
> URL: https://issues.apache.org/jira/browse/HBASE-16294
> Project: HBase
> Issue Type: Bug
> Components: hbck
>Affects Versions: 2.0.0, 1.3.0, 1.0.3, 1.4.0, 1.1.5, 1.2.2
>Reporter: Matteo Bertozzi
>Assignee: Umesh Agashe
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: HBASE-16294.v1.patch
>
>
> simple test, create a table with replicas and then run hbck.
> we don't filter out the replicas for the loadHdfsRegioninfo()
> {noformat}
> $ hbase shell
> hbase(main):001:0> create 'myTable', 'myCF', {REGION_REPLICATION => '3'}
> $ hbase hbck
> 2016-07-27 13:47:38,090 WARN [hbasefsck-pool1-t2] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.,
> replicaId => 2 } meta={ENCODED => 9dea3506e09e00910158dc91fa21e550, NAME =>
> 'myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 2}
> 2016-07-27 13:47:38,092 WARN [hbasefsck-pool1-t1] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.,
> replicaId => 1 } meta={ENCODED => a03250bca30781ff7002a91c281b4e92, NAME =>
> 'myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 1}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16680) Reduce garbage in BufferChain
[ https://issues.apache.org/jira/browse/HBASE-16680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-16680: --- Hadoop Flags: Reviewed Status: Patch Available (was: Open) +1 > Reduce garbage in BufferChain > - > > Key: HBASE-16680 > URL: https://issues.apache.org/jira/browse/HBASE-16680 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16680-master.patch > > > BufferChain accept a List and then convert it to ByteBuffer[], we > can directly produce ByteBuffer[] and handle it to BufferChain, so eliminate > the object List. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16680) Reduce garbage in BufferChain
[ https://issues.apache.org/jira/browse/HBASE-16680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-16680: --- Fix Version/s: 2.0.0 > Reduce garbage in BufferChain > - > > Key: HBASE-16680 > URL: https://issues.apache.org/jira/browse/HBASE-16680 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Priority: Minor > Fix For: 2.0.0 > > Attachments: HBASE-16680-master.patch > > > BufferChain accept a List and then convert it to ByteBuffer[], we > can directly produce ByteBuffer[] and handle it to BufferChain, so eliminate > the object List. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16679) Flush throughput controller: Minor perf change and fix flaky TestFlushWithThroughputController
[ https://issues.apache.org/jira/browse/HBASE-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16679: - Status: Patch Available (was: Open) > Flush throughput controller: Minor perf change and fix flaky > TestFlushWithThroughputController > -- > > Key: HBASE-16679 > URL: https://issues.apache.org/jira/browse/HBASE-16679 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: HBASE-16679.master.001.patch, > HBASE-16679.master.002.patch > > > Minor perf change: > Calculate maxThroughputPerOperation outside of control() since start()&end() > are called only once per operation, but control can be called > hundreds/thousands of time. > Flaky test: > Problems in current test: > - writes only 2.5MB each iteration but control triggers sleep only every 1Mb > write (decided by HBASE_HSTORE_FLUSH_THROUGHPUT_CONTROL_CHECK_INTERVAL). > Either increase data written in each batch or decreasing this threshold for > better throughput control. > - We shouldn't be timing table disable/delete/create and populating data in > throughput calculations. > See the differences below. > With patch (total data written 30M) > run 1: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29683ms; without limit, flush use 2130ms > run 2: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29674ms; without limit, flush use 2027ms > Without patch (total data written 25M) > run 1: > Throughput is: 0.921681903523776 MB/s > Throughput w/o limit is: 4.06833346870301 MB/s > With 1M/s limit, flush use 27189ms; without limit, flush use 6159ms > run 2: > Throughput is: 0.9422982728478803 MB/s > Throughput w/o limit is: 4.047858424942981 MB/s > With 1M/s limit, flush use 26594ms; without limit, flush use 6190ms -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16679) Flush throughput controller: Minor perf change and fix flaky TestFlushWithThroughputController
[ https://issues.apache.org/jira/browse/HBASE-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512109#comment-15512109 ] Appy commented on HBASE-16679: -- [~stack] ready for review. > Flush throughput controller: Minor perf change and fix flaky > TestFlushWithThroughputController > -- > > Key: HBASE-16679 > URL: https://issues.apache.org/jira/browse/HBASE-16679 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: HBASE-16679.master.001.patch, > HBASE-16679.master.002.patch > > > Minor perf change: > Calculate maxThroughputPerOperation outside of control() since start()&end() > are called only once per operation, but control can be called > hundreds/thousands of time. > Flaky test: > Problems in current test: > - writes only 2.5MB each iteration but control triggers sleep only every 1Mb > write (decided by HBASE_HSTORE_FLUSH_THROUGHPUT_CONTROL_CHECK_INTERVAL). > Either increase data written in each batch or decreasing this threshold for > better throughput control. > - We shouldn't be timing table disable/delete/create and populating data in > throughput calculations. > See the differences below. > With patch (total data written 30M) > run 1: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29683ms; without limit, flush use 2130ms > run 2: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29674ms; without limit, flush use 2027ms > Without patch (total data written 25M) > run 1: > Throughput is: 0.921681903523776 MB/s > Throughput w/o limit is: 4.06833346870301 MB/s > With 1M/s limit, flush use 27189ms; without limit, flush use 6159ms > run 2: > Throughput is: 0.9422982728478803 MB/s > Throughput w/o limit is: 4.047858424942981 MB/s > With 1M/s limit, flush use 26594ms; without limit, flush use 6190ms -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12088) Remove un-used profiles in non-root poms
[ https://issues.apache.org/jira/browse/HBASE-12088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512104#comment-15512104 ] Sean Busbey commented on HBASE-12088: - +1 for 1.2, presuming it makes it to 1.3. > Remove un-used profiles in non-root poms > > > Key: HBASE-12088 > URL: https://issues.apache.org/jira/browse/HBASE-12088 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Jonathan Hsieh > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-12088.v0.patch > > > Some of the poms still have hadoop 1 and hadoop 1.1 profiles even though the > root pom has them removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16679) Flush throughput controller: Minor perf change and fix flaky TestFlushWithThroughputController
[ https://issues.apache.org/jira/browse/HBASE-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16679: - Attachment: HBASE-16679.master.002.patch > Flush throughput controller: Minor perf change and fix flaky > TestFlushWithThroughputController > -- > > Key: HBASE-16679 > URL: https://issues.apache.org/jira/browse/HBASE-16679 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: HBASE-16679.master.001.patch, > HBASE-16679.master.002.patch > > > Minor perf change: > Calculate maxThroughputPerOperation outside of control() since start()&end() > are called only once per operation, but control can be called > hundreds/thousands of time. > Flaky test: > Problems in current test: > - writes only 2.5MB each iteration but control triggers sleep only every 1Mb > write (decided by HBASE_HSTORE_FLUSH_THROUGHPUT_CONTROL_CHECK_INTERVAL). > Either increase data written in each batch or decreasing this threshold for > better throughput control. > - We shouldn't be timing table disable/delete/create and populating data in > throughput calculations. > See the differences below. > With patch (total data written 30M) > run 1: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29683ms; without limit, flush use 2130ms > run 2: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29674ms; without limit, flush use 2027ms > Without patch (total data written 25M) > run 1: > Throughput is: 0.921681903523776 MB/s > Throughput w/o limit is: 4.06833346870301 MB/s > With 1M/s limit, flush use 27189ms; without limit, flush use 6159ms > run 2: > Throughput is: 0.9422982728478803 MB/s > Throughput w/o limit is: 4.047858424942981 MB/s > With 1M/s limit, flush use 26594ms; without limit, flush use 6190ms -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16294) hbck reporting "No HDFS region dir found" for replicas
[
https://issues.apache.org/jira/browse/HBASE-16294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512087#comment-15512087
]
Hudson commented on HBASE-16294:
FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1650 (See
[https://builds.apache.org/job/HBase-Trunk_matrix/1650/])
HBASE-16294 hbck reporting "No HDFS region dir found" for replicas
(matteo.bertozzi: rev 6a4c292a901386fd0b8c800f00a05572f0cb5cb5)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
> hbck reporting "No HDFS region dir found" for replicas
> --
>
> Key: HBASE-16294
> URL: https://issues.apache.org/jira/browse/HBASE-16294
> Project: HBase
> Issue Type: Bug
> Components: hbck
>Affects Versions: 2.0.0, 1.3.0, 1.0.3, 1.4.0, 1.1.5, 1.2.2
>Reporter: Matteo Bertozzi
>Assignee: Umesh Agashe
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: HBASE-16294.v1.patch
>
>
> simple test, create a table with replicas and then run hbck.
> we don't filter out the replicas for the loadHdfsRegioninfo()
> {noformat}
> $ hbase shell
> hbase(main):001:0> create 'myTable', 'myCF', {REGION_REPLICATION => '3'}
> $ hbase hbck
> 2016-07-27 13:47:38,090 WARN [hbasefsck-pool1-t2] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.,
> replicaId => 2 } meta={ENCODED => 9dea3506e09e00910158dc91fa21e550, NAME =>
> 'myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 2}
> 2016-07-27 13:47:38,092 WARN [hbasefsck-pool1-t1] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.,
> replicaId => 1 } meta={ENCODED => a03250bca30781ff7002a91c281b4e92, NAME =>
> 'myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 1}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14734) BindException when setting up MiniKdc
[ https://issues.apache.org/jira/browse/HBASE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512088#comment-15512088 ] Hudson commented on HBASE-14734: FAILURE: Integrated in Jenkins build HBase-Trunk_matrix #1650 (See [https://builds.apache.org/job/HBase-Trunk_matrix/1650/]) HBASE-14734 Prevent BindException when setting up MiniKdc. Port for kdc (appy: rev 593fb750835ebf5e74cdd42945d65fcf71b32c1e) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/asyncfs/TestSaslFanOutOneBlockAsyncDFSOutput.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/token/SecureTestCluster.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestSecureIPC.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestUsersOperationsWithSecureHadoop.java > BindException when setting up MiniKdc > - > > Key: HBASE-14734 > URL: https://issues.apache.org/jira/browse/HBASE-14734 > Project: HBase > Issue Type: Sub-task > Components: flakey, test >Reporter: stack >Assignee: Appy > Fix For: 1.3.0, 1.4.0, 1.2.4, 2.0.. > > Attachments: HBASE-14734.master.001.patch > > > Root cause : Port for kdc service gets selected in the constructor, but we > bind to it later in MiniKdc.start()-->MiniKdc.initKDCServer() --> > KdcServer.start(). In meantime, some other service can capture the port which > results in BindException. The solution here is to catch the exception and > retry. > From > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/330/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.security.token/TestGenerateDelegationToken/org_apache_hadoop_hbase_security_token_TestGenerateDelegationToken/ > Error Message > Address already in use > Stacktrace > java.net.BindException: Address already in use > at sun.nio.ch.Net.bind0(Native Method) > at sun.nio.ch.Net.bind(Net.java:444) > at sun.nio.ch.Net.bind(Net.java:436) > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422) > at > org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Can this utility be made to not fail if address taken? Try another? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16680) Reduce garbage in BufferChain
[ https://issues.apache.org/jira/browse/HBASE-16680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] binlijin updated HBASE-16680: - Attachment: HBASE-16680-master.patch > Reduce garbage in BufferChain > - > > Key: HBASE-16680 > URL: https://issues.apache.org/jira/browse/HBASE-16680 > Project: HBase > Issue Type: Improvement >Reporter: binlijin >Priority: Minor > Attachments: HBASE-16680-master.patch > > > BufferChain accept a List and then convert it to ByteBuffer[], we > can directly produce ByteBuffer[] and handle it to BufferChain, so eliminate > the object List. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16680) Reduce garbage in BufferChain
binlijin created HBASE-16680: Summary: Reduce garbage in BufferChain Key: HBASE-16680 URL: https://issues.apache.org/jira/browse/HBASE-16680 Project: HBase Issue Type: Improvement Reporter: binlijin Priority: Minor BufferChain accept a List and then convert it to ByteBuffer[], we can directly produce ByteBuffer[] and handle it to BufferChain, so eliminate the object List. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12088) Remove un-used profiles in non-root poms
[ https://issues.apache.org/jira/browse/HBASE-12088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-12088: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 1.4.0 2.0.0 Status: Resolved (was: Patch Available) Thanks for the review enis. Committed to trunk and branch-1. [~mantonov], [~busbey], [~ndimiduk] -- let me know if you want in 1.3, 1.2, or 1.1. > Remove un-used profiles in non-root poms > > > Key: HBASE-12088 > URL: https://issues.apache.org/jira/browse/HBASE-12088 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Jonathan Hsieh > Fix For: 2.0.0, 1.4.0 > > Attachments: hbase-12088.v0.patch > > > Some of the poms still have hadoop 1 and hadoop 1.1 profiles even though the > root pom has them removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[ https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-16672: --- Attachment: 16672.v5.txt > Add option for bulk load to copy hfile(s) instead of renaming > - > > Key: HBASE-16672 > URL: https://issues.apache.org/jira/browse/HBASE-16672 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16672.v1.txt, 16672.v2.txt, 16672.v3.txt, 16672.v4.txt, > 16672.v5.txt > > > This is related to HBASE-14417, to support incrementally restoring to > multiple destinations, this issue adds option which would always copy > hfile(s) during bulk load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16679) Flush throughput controller: Minor perf change and fix flaky TestFlushWithThroughputController
[ https://issues.apache.org/jira/browse/HBASE-16679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-16679: - Attachment: HBASE-16679.master.001.patch > Flush throughput controller: Minor perf change and fix flaky > TestFlushWithThroughputController > -- > > Key: HBASE-16679 > URL: https://issues.apache.org/jira/browse/HBASE-16679 > Project: HBase > Issue Type: Bug >Reporter: Appy >Assignee: Appy > Attachments: HBASE-16679.master.001.patch > > > Minor perf change: > Calculate maxThroughputPerOperation outside of control() since start()&end() > are called only once per operation, but control can be called > hundreds/thousands of time. > Flaky test: > Problems in current test: > - writes only 2.5MB each iteration but control triggers sleep only every 1Mb > write (decided by HBASE_HSTORE_FLUSH_THROUGHPUT_CONTROL_CHECK_INTERVAL). > Either increase data written in each batch or decreasing this threshold for > better throughput control. > - We shouldn't be timing table disable/delete/create and populating data in > throughput calculations. > See the differences below. > With patch (total data written 30M) > run 1: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29683ms; without limit, flush use 2130ms > run 2: > Throughput is: 1.0113841089709052 MB/s > Throughput w/o limit is: 14.665069580078125 MB/s > With 1M/s limit, flush use 29674ms; without limit, flush use 2027ms > Without patch (total data written 25M) > run 1: > Throughput is: 0.921681903523776 MB/s > Throughput w/o limit is: 4.06833346870301 MB/s > With 1M/s limit, flush use 27189ms; without limit, flush use 6159ms > run 2: > Throughput is: 0.9422982728478803 MB/s > Throughput w/o limit is: 4.047858424942981 MB/s > With 1M/s limit, flush use 26594ms; without limit, flush use 6190ms -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16679) Flush throughput controller: Minor perf change and fix flaky TestFlushWithThroughputController
Appy created HBASE-16679: Summary: Flush throughput controller: Minor perf change and fix flaky TestFlushWithThroughputController Key: HBASE-16679 URL: https://issues.apache.org/jira/browse/HBASE-16679 Project: HBase Issue Type: Bug Reporter: Appy Assignee: Appy Minor perf change: Calculate maxThroughputPerOperation outside of control() since start()&end() are called only once per operation, but control can be called hundreds/thousands of time. Flaky test: Problems in current test: - writes only 2.5MB each iteration but control triggers sleep only every 1Mb write (decided by HBASE_HSTORE_FLUSH_THROUGHPUT_CONTROL_CHECK_INTERVAL). Either increase data written in each batch or decreasing this threshold for better throughput control. - We shouldn't be timing table disable/delete/create and populating data in throughput calculations. See the differences below. With patch (total data written 30M) run 1: Throughput is: 1.0113841089709052 MB/s Throughput w/o limit is: 14.665069580078125 MB/s With 1M/s limit, flush use 29683ms; without limit, flush use 2130ms run 2: Throughput is: 1.0113841089709052 MB/s Throughput w/o limit is: 14.665069580078125 MB/s With 1M/s limit, flush use 29674ms; without limit, flush use 2027ms Without patch (total data written 25M) run 1: Throughput is: 0.921681903523776 MB/s Throughput w/o limit is: 4.06833346870301 MB/s With 1M/s limit, flush use 27189ms; without limit, flush use 6159ms run 2: Throughput is: 0.9422982728478803 MB/s Throughput w/o limit is: 4.047858424942981 MB/s With 1M/s limit, flush use 26594ms; without limit, flush use 6190ms -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16675) Average region size may be incorrect when there is region whose RegionLoad cannot be retrieved
[ https://issues.apache.org/jira/browse/HBASE-16675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512022#comment-15512022 ] Heng Chen commented on HBASE-16675: --- +1 > Average region size may be incorrect when there is region whose RegionLoad > cannot be retrieved > -- > > Key: HBASE-16675 > URL: https://issues.apache.org/jira/browse/HBASE-16675 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16675.v1.txt > > > HBASE-15933 fixed the NullPointerException bug. > When there is one or more region whose RegionLoad cannot be retrieved, the > average region size may be incorrect. > We should not use tableRegions.size() as denominator - the number of regions > whose RegionLoad can be retrieved should be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16669) fix flaky TestAdmin#testmergeRegions
[
https://issues.apache.org/jira/browse/HBASE-16669?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jonathan Hsieh updated HBASE-16669:
---
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks for the review Appy. Test only change and other unittests that failed
are unrelated.
> fix flaky TestAdmin#testmergeRegions
>
>
> Key: HBASE-16669
> URL: https://issues.apache.org/jira/browse/HBASE-16669
> Project: HBase
> Issue Type: Bug
> Components: test
>Affects Versions: 2.0.0
>Reporter: Jonathan Hsieh
>Assignee: Jonathan Hsieh
> Fix For: 2.0.0
>
> Attachments: hbase-16669.v1.patch
>
>
> Recent test runs show that this test is failing with these error messages:
> {code}
> java.lang.AssertionError: expected:<2> but was:<3>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.hadoop.hbase.client.TestAdmin1.testMergeRegions(TestAdmin1.java:1385)
> {code}
> or
> {code}
> java.lang.AssertionError: expected:<1> but was:<2>
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.failNotEquals(Assert.java:834)
> at org.junit.Assert.assertEquals(Assert.java:645)
> at org.junit.Assert.assertEquals(Assert.java:631)
> at
> org.apache.hadoop.hbase.client.TestAdmin1.testMergeRegions(TestAdmin1.java:1394)
> {code}
> looking at the code this indicates that merge operation did not complete or
> didn't work.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[
https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15512001#comment-15512001
]
Hadoop QA commented on HBASE-16672:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 18s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 25s
{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 15s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 7m
19s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 3s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s
{color} | {color:green} master passed {color} |
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m 10s
{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 32s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 8m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
36s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
33m 4s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m 30s
{color} | {color:green} hbase-protocol in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 10s
{color} | {color:green} hbase-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 100m 10s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 175m 11s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestBlockEvictionFromClient |
| | hadoop.hbase.regionserver.TestHRegion |
| Timed out junit tests |
org.apache.hadoop.hbase.snapshot.TestMobSecureExportSnapshot |
| | org.apache.hadoop.hbase.snapshot.TestRestoreFlushSnapshotFromClient |
| | org.apache.hadoop.hbase.client.TestHCM |
| | org.apache.hadoop.hbase.snapshot.TestMobFlushSnapshotFromClient |
| | org.apache.hadoop.hbase.snapshot.TestFlushSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829723/16672.v4.txt |
| JI
[jira] [Commented] (HBASE-16666) Add append and remove peer namespaces cmds for replication
[
https://issues.apache.org/jira/browse/HBASE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511978#comment-15511978
]
Hadoop QA commented on HBASE-1:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 9m 40s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} rubocop {color} | {color:blue} 0m 11s
{color} | {color:blue} rubocop was not available. {color} |
| {color:blue}0{color} | {color:blue} ruby-lint {color} | {color:blue} 0m 11s
{color} | {color:blue} Ruby-lint was not available. {color} |
| {color:green}+1{color} | {color:green} hbaseanti {color} | {color:green} 0m
0s {color} | {color:green} Patch does not have any anti-patterns. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
26s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
21s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
48m 10s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 10s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 25s {color}
| {color:red} hbase-shell in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
21s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 71m 9s {color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.client.TestReplicationShell |
| | hadoop.hbase.client.rsgroup.TestShellRSGroups |
| | hadoop.hbase.client.TestShell |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829544/HBASE-1.patch |
| JIRA Issue | HBASE-1 |
| Optional Tests | asflicense javac javadoc unit rubocop ruby_lint |
| uname | Linux b0560da77c1e 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 593fb75 |
| Default Java | 1.8.0_101 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3646/artifact/patchprocess/patch-unit-hbase-shell.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3646/artifact/patchprocess/patch-unit-hbase-shell.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3646/testReport/ |
| modules | C: hbase-shell U: hbase-shell |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3646/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Add append and remove peer namespaces cmds for replication
> --
>
> Key: HBASE-1
> URL: https://issues.apache.org/jira/browse/HBASE-1
> Project: HBase
> Issue Type: Improvement
> Components: Replication
>Reporter: Guanghao Zhang
>Assignee: Guanghao Zhang
>Priority: Minor
> Attachments: HBASE-1.patch
>
>
> After HBASE-16447, we support replication by namespaces config in peer. Like
> append_peer_tableCFs and remove_peer_tableCFs, I thought we need two n
[jira] [Updated] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-14417: --- Attachment: 14417.v2.txt Patch v2 adds second full back up in the test. Not working yet. > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > Attachments: 14417.v1.txt, 14417.v2.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16294) hbck reporting "No HDFS region dir found" for replicas
[
https://issues.apache.org/jira/browse/HBASE-16294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511930#comment-15511930
]
Hudson commented on HBASE-16294:
FAILURE: Integrated in Jenkins build HBase-1.4 #421 (See
[https://builds.apache.org/job/HBase-1.4/421/])
HBASE-16294 hbck reporting "No HDFS region dir found" for replicas
(matteo.bertozzi: rev 23c5ea39bd237c2012c0c5a13951b8338dc54a8d)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
> hbck reporting "No HDFS region dir found" for replicas
> --
>
> Key: HBASE-16294
> URL: https://issues.apache.org/jira/browse/HBASE-16294
> Project: HBase
> Issue Type: Bug
> Components: hbck
>Affects Versions: 2.0.0, 1.3.0, 1.0.3, 1.4.0, 1.1.5, 1.2.2
>Reporter: Matteo Bertozzi
>Assignee: Umesh Agashe
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: HBASE-16294.v1.patch
>
>
> simple test, create a table with replicas and then run hbck.
> we don't filter out the replicas for the loadHdfsRegioninfo()
> {noformat}
> $ hbase shell
> hbase(main):001:0> create 'myTable', 'myCF', {REGION_REPLICATION => '3'}
> $ hbase hbck
> 2016-07-27 13:47:38,090 WARN [hbasefsck-pool1-t2] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.,
> replicaId => 2 } meta={ENCODED => 9dea3506e09e00910158dc91fa21e550, NAME =>
> 'myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 2}
> 2016-07-27 13:47:38,092 WARN [hbasefsck-pool1-t1] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.,
> replicaId => 1 } meta={ENCODED => a03250bca30781ff7002a91c281b4e92, NAME =>
> 'myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 1}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14734) BindException when setting up MiniKdc
[ https://issues.apache.org/jira/browse/HBASE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511931#comment-15511931 ] Hudson commented on HBASE-14734: FAILURE: Integrated in Jenkins build HBase-1.4 #421 (See [https://builds.apache.org/job/HBase-1.4/421/]) HBASE-14734 Prevent BindException when setting up MiniKdc. Port for kdc (appy: rev e7e660d5b2b6ed915ec7614e8d7070f9a51d091f) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/AbstractTestSecureIPC.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestUsersOperationsWithSecureHadoop.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/token/SecureTestCluster.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java > BindException when setting up MiniKdc > - > > Key: HBASE-14734 > URL: https://issues.apache.org/jira/browse/HBASE-14734 > Project: HBase > Issue Type: Sub-task > Components: flakey, test >Reporter: stack >Assignee: Appy > Fix For: 1.3.0, 1.4.0, 1.2.4, 2.0.. > > Attachments: HBASE-14734.master.001.patch > > > Root cause : Port for kdc service gets selected in the constructor, but we > bind to it later in MiniKdc.start()-->MiniKdc.initKDCServer() --> > KdcServer.start(). In meantime, some other service can capture the port which > results in BindException. The solution here is to catch the exception and > retry. > From > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/330/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.security.token/TestGenerateDelegationToken/org_apache_hadoop_hbase_security_token_TestGenerateDelegationToken/ > Error Message > Address already in use > Stacktrace > java.net.BindException: Address already in use > at sun.nio.ch.Net.bind0(Native Method) > at sun.nio.ch.Net.bind(Net.java:444) > at sun.nio.ch.Net.bind(Net.java:436) > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422) > at > org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Can this utility be made to not fail if address taken? Try another? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12949) Scanner can be stuck in infinite loop if the HFile is corrupted
[
https://issues.apache.org/jira/browse/HBASE-12949?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511932#comment-15511932
]
Hudson commented on HBASE-12949:
FAILURE: Integrated in Jenkins build HBase-1.4 #421 (See
[https://builds.apache.org/job/HBase-1.4/421/])
HBASE-12949 Scanner can be stuck in infinite loop if the HFile is (jerryjch:
rev c80d671a062737c806d4e20a992443e9b6b86b02)
* (edit)
hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java
> Scanner can be stuck in infinite loop if the HFile is corrupted
> ---
>
> Key: HBASE-12949
> URL: https://issues.apache.org/jira/browse/HBASE-12949
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.3, 0.98.10
>Reporter: Jerry He
>Assignee: Jerry He
> Fix For: 2.0.0, 1.4.0
>
> Attachments: HBASE-12949-branch-1-v3.patch, HBASE-12949-master-v2
> (1).patch, HBASE-12949-master-v2.patch, HBASE-12949-master-v2.patch,
> HBASE-12949-master-v2.patch, HBASE-12949-master-v3.patch,
> HBASE-12949-master.patch
>
>
> We've encountered problem where compaction hangs and never completes.
> After looking into it further, we found that the compaction scanner was stuck
> in a infinite loop. See stack below.
> {noformat}
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:296)
> org.apache.hadoop.hbase.regionserver.KeyValueHeap.reseek(KeyValueHeap.java:257)
> org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:697)
> org.apache.hadoop.hbase.regionserver.StoreScanner.seekToNextRow(StoreScanner.java:672)
> org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:529)
> org.apache.hadoop.hbase.regionserver.compactions.Compactor.performCompaction(Compactor.java:223)
> {noformat}
> We identified the hfile that seems to be corrupted. Using HFile tool shows
> the following:
> {noformat}
> [biadmin@hdtest009 bin]$ hbase org.apache.hadoop.hbase.io.hfile.HFile -v -k
> -m -f
> /user/biadmin/CUMMINS_INSITE_V1/7106432d294dd844be15996ccbf2ba84/attributes/f1a7e3113c2c4047ac1fc8fbcb41d8b7
> 15/01/23 11:53:17 INFO Configuration.deprecation: hadoop.native.lib is
> deprecated. Instead, use io.native.lib.available
> 15/01/23 11:53:18 INFO util.ChecksumType: Checksum using
> org.apache.hadoop.util.PureJavaCrc32
> 15/01/23 11:53:18 INFO util.ChecksumType: Checksum can use
> org.apache.hadoop.util.PureJavaCrc32C
> 15/01/23 11:53:18 INFO Configuration.deprecation: fs.default.name is
> deprecated. Instead, use fs.defaultFS
> Scanning ->
> /user/biadmin/CUMMINS_INSITE_V1/7106432d294dd844be15996ccbf2ba84/attributes/f1a7e3113c2c4047ac1fc8fbcb41d8b7
> WARNING, previous row is greater then current row
> filename ->
> /user/biadmin/CUMMINS_INSITE_V1/7106432d294dd844be15996ccbf2ba84/attributes/f1a7e3113c2c4047ac1fc8fbcb41d8b7
> previous ->
> \x00/20110203-094231205-79442793-1410161293068203000\x0Aattributes16794406\x00\x00\x01\x00\x00\x00\x00\x00\x00
> current ->
> Exception in thread "main" java.nio.BufferUnderflowException
> at java.nio.Buffer.nextGetIndex(Buffer.java:489)
> at java.nio.HeapByteBuffer.getInt(HeapByteBuffer.java:347)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.readKeyValueLen(HFileReaderV2.java:856)
> at
> org.apache.hadoop.hbase.io.hfile.HFileReaderV2$ScannerV2.next(HFileReaderV2.java:768)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.scanKeysValues(HFilePrettyPrinter.java:362)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:262)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:220)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at
> org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.main(HFilePrettyPrinter.java:539)
> at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:802)
> {noformat}
> Turning on Java Assert shows the following:
> {noformat}
> Exception in thread "main" java.lang.AssertionError: Key
> 20110203-094231205-79442793-1410161293068203000/attributes:16794406/1099511627776/Minimum/vlen=15/mvcc=0
> followed by a smaller key //0/Minimum/vlen=0/mvcc=0 in cf attributes
> at
> org.apache.hadoop.hbase.regionserver.StoreScanner.checkScanOrder(StoreScanner.java:672)
> {noformat}
> It shows that the hfile seems to be corrupted -- the keys don't seem to be
> right.
> But Scanner is not able to give a meaningful error, but stuck in an infinite
> loop in here:
> {code}
> KeyValueHeap.generalizedSeek()
> while ((scanner = heap.poll()) != null) {
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-14734) BindException when setting up MiniKdc
[ https://issues.apache.org/jira/browse/HBASE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511919#comment-15511919 ] Hudson commented on HBASE-14734: FAILURE: Integrated in Jenkins build HBase-1.2-JDK7 #29 (See [https://builds.apache.org/job/HBase-1.2-JDK7/29/]) HBASE-14734 Prevent BindException when setting up MiniKdc. Port for kdc (appy: rev 79531fd95f1b7b72ad1f764d7cd57e37467a4b2a) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/token/TestGenerateDelegationToken.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestUsersOperationsWithSecureHadoop.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/security/TestSecureRPC.java > BindException when setting up MiniKdc > - > > Key: HBASE-14734 > URL: https://issues.apache.org/jira/browse/HBASE-14734 > Project: HBase > Issue Type: Sub-task > Components: flakey, test >Reporter: stack >Assignee: Appy > Fix For: 1.3.0, 1.4.0, 1.2.4, 2.0.. > > Attachments: HBASE-14734.master.001.patch > > > Root cause : Port for kdc service gets selected in the constructor, but we > bind to it later in MiniKdc.start()-->MiniKdc.initKDCServer() --> > KdcServer.start(). In meantime, some other service can capture the port which > results in BindException. The solution here is to catch the exception and > retry. > From > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/330/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.security.token/TestGenerateDelegationToken/org_apache_hadoop_hbase_security_token_TestGenerateDelegationToken/ > Error Message > Address already in use > Stacktrace > java.net.BindException: Address already in use > at sun.nio.ch.Net.bind0(Native Method) > at sun.nio.ch.Net.bind(Net.java:444) > at sun.nio.ch.Net.bind(Net.java:436) > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422) > at > org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Can this utility be made to not fail if address taken? Try another? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16294) hbck reporting "No HDFS region dir found" for replicas
[
https://issues.apache.org/jira/browse/HBASE-16294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511918#comment-15511918
]
Hudson commented on HBASE-16294:
FAILURE: Integrated in Jenkins build HBase-1.2-JDK7 #29 (See
[https://builds.apache.org/job/HBase-1.2-JDK7/29/])
HBASE-16294 hbck reporting "No HDFS region dir found" for replicas
(matteo.bertozzi: rev c665424540afd0c771ac01fda8beac0a36949c1f)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
> hbck reporting "No HDFS region dir found" for replicas
> --
>
> Key: HBASE-16294
> URL: https://issues.apache.org/jira/browse/HBASE-16294
> Project: HBase
> Issue Type: Bug
> Components: hbck
>Affects Versions: 2.0.0, 1.3.0, 1.0.3, 1.4.0, 1.1.5, 1.2.2
>Reporter: Matteo Bertozzi
>Assignee: Umesh Agashe
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: HBASE-16294.v1.patch
>
>
> simple test, create a table with replicas and then run hbck.
> we don't filter out the replicas for the loadHdfsRegioninfo()
> {noformat}
> $ hbase shell
> hbase(main):001:0> create 'myTable', 'myCF', {REGION_REPLICATION => '3'}
> $ hbase hbck
> 2016-07-27 13:47:38,090 WARN [hbasefsck-pool1-t2] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.,
> replicaId => 2 } meta={ENCODED => 9dea3506e09e00910158dc91fa21e550, NAME =>
> 'myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 2}
> 2016-07-27 13:47:38,092 WARN [hbasefsck-pool1-t1] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.,
> replicaId => 1 } meta={ENCODED => a03250bca30781ff7002a91c281b4e92, NAME =>
> 'myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 1}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16675) Average region size may be incorrect when there is region whose RegionLoad cannot be retrieved
[ https://issues.apache.org/jira/browse/HBASE-16675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511912#comment-15511912 ] Ted Yu commented on HBASE-16675: Test failures were not related to the patch - normalizer is not used in those tests. > Average region size may be incorrect when there is region whose RegionLoad > cannot be retrieved > -- > > Key: HBASE-16675 > URL: https://issues.apache.org/jira/browse/HBASE-16675 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16675.v1.txt > > > HBASE-15933 fixed the NullPointerException bug. > When there is one or more region whose RegionLoad cannot be retrieved, the > average region size may be incorrect. > We should not use tableRegions.size() as denominator - the number of regions > whose RegionLoad can be retrieved should be used. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[ https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511904#comment-15511904 ] Jerry He commented on HBASE-16672: -- If SecureBulkLoadListener is not modified, the files will be moved/renamed from the source to staging dir, then based on the flag will be copied to region store directory. You don't want the files to be kept in the staging dir. You want them to be kept at the source. > Add option for bulk load to copy hfile(s) instead of renaming > - > > Key: HBASE-16672 > URL: https://issues.apache.org/jira/browse/HBASE-16672 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16672.v1.txt, 16672.v2.txt, 16672.v3.txt, 16672.v4.txt > > > This is related to HBASE-14417, to support incrementally restoring to > multiple destinations, this issue adds option which would always copy > hfile(s) during bulk load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16294) hbck reporting "No HDFS region dir found" for replicas
[
https://issues.apache.org/jira/browse/HBASE-16294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511892#comment-15511892
]
Hudson commented on HBASE-16294:
SUCCESS: Integrated in Jenkins build HBase-1.2-JDK8 #25 (See
[https://builds.apache.org/job/HBase-1.2-JDK8/25/])
HBASE-16294 hbck reporting "No HDFS region dir found" for replicas
(matteo.bertozzi: rev c665424540afd0c771ac01fda8beac0a36949c1f)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
> hbck reporting "No HDFS region dir found" for replicas
> --
>
> Key: HBASE-16294
> URL: https://issues.apache.org/jira/browse/HBASE-16294
> Project: HBase
> Issue Type: Bug
> Components: hbck
>Affects Versions: 2.0.0, 1.3.0, 1.0.3, 1.4.0, 1.1.5, 1.2.2
>Reporter: Matteo Bertozzi
>Assignee: Umesh Agashe
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: HBASE-16294.v1.patch
>
>
> simple test, create a table with replicas and then run hbck.
> we don't filter out the replicas for the loadHdfsRegioninfo()
> {noformat}
> $ hbase shell
> hbase(main):001:0> create 'myTable', 'myCF', {REGION_REPLICATION => '3'}
> $ hbase hbck
> 2016-07-27 13:47:38,090 WARN [hbasefsck-pool1-t2] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.,
> replicaId => 2 } meta={ENCODED => 9dea3506e09e00910158dc91fa21e550, NAME =>
> 'myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 2}
> 2016-07-27 13:47:38,092 WARN [hbasefsck-pool1-t1] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.,
> replicaId => 1 } meta={ENCODED => a03250bca30781ff7002a91c281b4e92, NAME =>
> 'myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 1}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16675) Average region size may be incorrect when there is region whose RegionLoad cannot be retrieved
[
https://issues.apache.org/jira/browse/HBASE-16675?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511882#comment-15511882
]
Hadoop QA commented on HBASE-16675:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 13s
{color} | {color:blue} Docker mode activated. {color} |
| {color:blue}0{color} | {color:blue} patch {color} | {color:blue} 0m 2s
{color} | {color:blue} The patch file was not named according to hbase's naming
conventions. Please see
https://yetus.apache.org/documentation/0.3.0/precommit-patchnames for
instructions. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 3m
47s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
55s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
23s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 38s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 34s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
12s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
25m 40s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
11s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
58s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 26s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 82m 51s {color}
| {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
12s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 123m 17s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Timed out junit tests | org.apache.hadoop.hbase.client.TestReplicasClient |
| | org.apache.hadoop.hbase.io.asyncfs.TestSaslFanOutOneBlockAsyncDFSOutput |
| | org.apache.hadoop.hbase.client.TestHCM |
| |
org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClientWithRegionReplicas |
| | org.apache.hadoop.hbase.client.TestTableSnapshotScanner |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.11.2 Server=1.11.2 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829720/16675.v1.txt |
| JIRA Issue | HBASE-16675 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux 2432c0ce18ac 3.13.0-93-generic #140-Ubuntu SMP Mon Jul 18
21:21:05 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 593fb75 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.
[jira] [Updated] (HBASE-16666) Add append and remove peer namespaces cmds for replication
[ https://issues.apache.org/jira/browse/HBASE-1?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang updated HBASE-1: --- Status: Patch Available (was: Open) > Add append and remove peer namespaces cmds for replication > -- > > Key: HBASE-1 > URL: https://issues.apache.org/jira/browse/HBASE-1 > Project: HBase > Issue Type: Improvement > Components: Replication >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Minor > Attachments: HBASE-1.patch > > > After HBASE-16447, we support replication by namespaces config in peer. Like > append_peer_tableCFs and remove_peer_tableCFs, I thought we need two new > shell cmd: append_peer_namespaces and remove_peer_namespaces. Then we can > easily change the namespaces config. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16656) BackupID must include backup set name
[
https://issues.apache.org/jira/browse/HBASE-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511804#comment-15511804
]
Hadoop QA commented on HBASE-16656:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 5s {color}
| {color:red} HBASE-16656 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829707/HBASE-16656-v1.patch |
| JIRA Issue | HBASE-16656 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3645/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> BackupID must include backup set name
> -
>
> Key: HBASE-16656
> URL: https://issues.apache.org/jira/browse/HBASE-16656
> Project: HBase
> Issue Type: Improvement
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-16656-v1.patch
>
>
> Default backup set name is "backup". If we do backup for a backup set
> "SomeSetName", by default, backup id will be generated in a form:
> *SomeSetName_timestamp*.
> The goal is to separate backup images between different sets.
> The history command will be updated and the new command - merge will use this
> backup names (prefixes)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-12088) Remove un-used profiles in non-root poms
[ https://issues.apache.org/jira/browse/HBASE-12088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511790#comment-15511790 ] Enis Soztutar commented on HBASE-12088: --- +1. > Remove un-used profiles in non-root poms > > > Key: HBASE-12088 > URL: https://issues.apache.org/jira/browse/HBASE-12088 > Project: HBase > Issue Type: Bug >Reporter: Elliott Clark >Assignee: Jonathan Hsieh > Attachments: hbase-12088.v0.patch > > > Some of the poms still have hadoop 1 and hadoop 1.1 profiles even though the > root pom has them removed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16673) Enhance history command: history per backup set
[
https://issues.apache.org/jira/browse/HBASE-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511791#comment-15511791
]
Hadoop QA commented on HBASE-16673:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 10s {color}
| {color:red} HBASE-16673 does not apply to master. Rebase required? Wrong
Branch? See https://yetus.apache.org/documentation/0.3.0/precommit-patchnames
for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829734/HBASE-16673-v1.patch |
| JIRA Issue | HBASE-16673 |
| Console output |
https://builds.apache.org/job/PreCommit-HBASE-Build/3643/console |
| Powered by | Apache Yetus 0.3.0 http://yetus.apache.org |
This message was automatically generated.
> Enhance history command: history per backup set
> ---
>
> Key: HBASE-16673
> URL: https://issues.apache.org/jira/browse/HBASE-16673
> Project: HBase
> Issue Type: New Feature
>Reporter: Vladimir Rodionov
>Assignee: Vladimir Rodionov
> Attachments: HBASE-16673-v1.patch
>
>
> New command-line option: -set setName. Will retrieve backup history for a
> particular backup set
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16294) hbck reporting "No HDFS region dir found" for replicas
[
https://issues.apache.org/jira/browse/HBASE-16294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511785#comment-15511785
]
Hudson commented on HBASE-16294:
SUCCESS: Integrated in Jenkins build HBase-1.3-JDK8 #18 (See
[https://builds.apache.org/job/HBase-1.3-JDK8/18/])
HBASE-16294 hbck reporting "No HDFS region dir found" for replicas
(matteo.bertozzi: rev 9aaff6045b151518a0cdaf694ebd477bff30b2c0)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
> hbck reporting "No HDFS region dir found" for replicas
> --
>
> Key: HBASE-16294
> URL: https://issues.apache.org/jira/browse/HBASE-16294
> Project: HBase
> Issue Type: Bug
> Components: hbck
>Affects Versions: 2.0.0, 1.3.0, 1.0.3, 1.4.0, 1.1.5, 1.2.2
>Reporter: Matteo Bertozzi
>Assignee: Umesh Agashe
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: HBASE-16294.v1.patch
>
>
> simple test, create a table with replicas and then run hbck.
> we don't filter out the replicas for the loadHdfsRegioninfo()
> {noformat}
> $ hbase shell
> hbase(main):001:0> create 'myTable', 'myCF', {REGION_REPLICATION => '3'}
> $ hbase hbck
> 2016-07-27 13:47:38,090 WARN [hbasefsck-pool1-t2] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.,
> replicaId => 2 } meta={ENCODED => 9dea3506e09e00910158dc91fa21e550, NAME =>
> 'myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 2}
> 2016-07-27 13:47:38,092 WARN [hbasefsck-pool1-t1] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.,
> replicaId => 1 } meta={ENCODED => a03250bca30781ff7002a91c281b4e92, NAME =>
> 'myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 1}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16678) MapReduce jobs do not update counters from ScanMetrics
[
https://issues.apache.org/jira/browse/HBASE-16678?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511782#comment-15511782
]
Enis Soztutar commented on HBASE-16678:
---
Patch is pretty simple:
{code}
diff --git
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
index 4349537..748d2be 100644
---
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
+++
hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/TableRecordReaderImpl.java
@@ -268,8 +268,9 @@ public class TableRecordReaderImpl {
* @throws IOException
*/
private void updateCounters() throws IOException {
-ScanMetrics scanMetrics = this.scan.getScanMetrics();
+ScanMetrics scanMetrics = currentScan.getScanMetrics();
{code}
Let me see whether we need a unit test.
> MapReduce jobs do not update counters from ScanMetrics
> --
>
> Key: HBASE-16678
> URL: https://issues.apache.org/jira/browse/HBASE-16678
> Project: HBase
> Issue Type: Bug
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
>
> Was inspecting a perf issue, where we needed the scanner metrics as counters
> for a MR job. Turns out that the HBase scan counters are no longer working in
> 1.0+. I think it got broken via HBASE-13030.
> These are the counters:
> {code}
> HBase Counters
> BYTES_IN_REMOTE_RESULTS=0
> BYTES_IN_RESULTS=280
> MILLIS_BETWEEN_NEXTS=11
> NOT_SERVING_REGION_EXCEPTION=0
> NUM_SCANNER_RESTARTS=0
> NUM_SCAN_RESULTS_STALE=0
> REGIONS_SCANNED=1
> REMOTE_RPC_CALLS=0
> REMOTE_RPC_RETRIES=0
> RPC_CALLS=3
> RPC_RETRIES=0
> {code}
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16678) MapReduce jobs do not update counters from ScanMetrics
Enis Soztutar created HBASE-16678:
-
Summary: MapReduce jobs do not update counters from ScanMetrics
Key: HBASE-16678
URL: https://issues.apache.org/jira/browse/HBASE-16678
Project: HBase
Issue Type: Bug
Reporter: Enis Soztutar
Assignee: Enis Soztutar
Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
Was inspecting a perf issue, where we needed the scanner metrics as counters
for a MR job. Turns out that the HBase scan counters are no longer working in
1.0+. I think it got broken via HBASE-13030.
These are the counters:
{code}
HBase Counters
BYTES_IN_REMOTE_RESULTS=0
BYTES_IN_RESULTS=280
MILLIS_BETWEEN_NEXTS=11
NOT_SERVING_REGION_EXCEPTION=0
NUM_SCANNER_RESTARTS=0
NUM_SCAN_RESULTS_STALE=0
REGIONS_SCANNED=1
REMOTE_RPC_CALLS=0
REMOTE_RPC_RETRIES=0
RPC_CALLS=3
RPC_RETRIES=0
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16662) Fix open POODLE vulnerabilities
[ https://issues.apache.org/jira/browse/HBASE-16662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511738#comment-15511738 ] Andrew Purtell commented on HBASE-16662: Got sidetracked. I can commit tomorrow unless someone else wants to get to it first. > Fix open POODLE vulnerabilities > --- > > Key: HBASE-16662 > URL: https://issues.apache.org/jira/browse/HBASE-16662 > Project: HBase > Issue Type: Bug > Components: REST, Thrift >Reporter: Ben Lau >Assignee: Ben Lau > Attachments: HBASE-16662-master.patch > > > We recently found a security issue in our HBase REST servers. The issue is a > variant of the POODLE vulnerability (https://en.wikipedia.org/wiki/POODLE) > and is present in the HBase Thrift server as well. It also appears to affect > the JMXListener coprocessor. The vulnerabilities probably affect all > versions of HBase that have the affected services. (If you don't use the > affected services with SSL then this ticket probably doesn't affect you). > Included is a patch to fix the known POODLE vulnerabilities in master. Let > us know if we missed any. From our end we only personally encountered the > HBase REST vulnerability. We do not use the Thrift server or JMXListener > coprocessor but discovered those problems after discussing the issue with > some of the HBase PMCs. > Coincidentally, Hadoop recently committed a SslSelectChannelConnectorSecure > which is more or less the same as one of the fixes in this patch. Hadoop > wasn't originally affected by the vulnerability in the > SslSelectChannelConnector, but about a month ago they committed HADOOP-12765 > which does use that class, so they added a SslSelectChannelConnectorSecure > class similar to this patch. Since this class is present in Hadoop 2.7.4+ > which hasn't been released yet, we will for now just include our own version > instead of depending on the Hadoop version. > After the patch is approved for master we can backport as necessary to older > versions of HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511733#comment-15511733 ] Andrew Purtell commented on HBASE-16676: Hey [~busbey] I have something for you to look at if you have a sec > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.2.4 > > Attachments: HBASE-16676-branch-1.2.patch > > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-16676: --- Attachment: HBASE-16676-branch-1.2.patch Here's a patch for branch-1.2 that reverts only the behavioral change leaving the LimitedPrivate API changes in place for coprocessor compatibility. > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3 >Reporter: Andrew Purtell > Fix For: 1.2.4 > > Attachments: HBASE-16676-branch-1.2.patch > > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-16676: --- Assignee: Andrew Purtell Fix Version/s: 1.2.4 Status: Patch Available (was: Open) > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.3, 1.2.2, 1.2.1, 1.2.0 >Reporter: Andrew Purtell >Assignee: Andrew Purtell > Fix For: 1.2.4 > > Attachments: HBASE-16676-branch-1.2.patch > > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[
https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511718#comment-15511718
]
Ted Yu commented on HBASE-16672:
w.r.t. the SecureBulkLoadListener, we have this code:
{code}
if (bulkLoadListener != null) {
finalPath = bulkLoadListener.prepareBulkLoad(familyName, path);
}
Path commitedStoreFile = store.bulkLoadHFile(finalPath, seqId,
copyFile);
{code}
The new flag would be effective based on the return value from
bulkLoadListener.prepareBulkLoad().
I feel there is no need to modify BulkLoadListener.
Let me know if I missed something.
> Add option for bulk load to copy hfile(s) instead of renaming
> -
>
> Key: HBASE-16672
> URL: https://issues.apache.org/jira/browse/HBASE-16672
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16672.v1.txt, 16672.v2.txt, 16672.v3.txt, 16672.v4.txt
>
>
> This is related to HBASE-14417, to support incrementally restoring to
> multiple destinations, this issue adds option which would always copy
> hfile(s) during bulk load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16656) BackupID must include backup set name
[ https://issues.apache.org/jira/browse/HBASE-16656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16656: -- Status: Patch Available (was: Open) > BackupID must include backup set name > - > > Key: HBASE-16656 > URL: https://issues.apache.org/jira/browse/HBASE-16656 > Project: HBase > Issue Type: Improvement >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16656-v1.patch > > > Default backup set name is "backup". If we do backup for a backup set > "SomeSetName", by default, backup id will be generated in a form: > *SomeSetName_timestamp*. > The goal is to separate backup images between different sets. > The history command will be updated and the new command - merge will use this > backup names (prefixes) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16673) Enhance history command: history per backup set
[ https://issues.apache.org/jira/browse/HBASE-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16673: -- Attachment: HBASE-16673-v1.patch patch v1. cc; [~tedyu] > Enhance history command: history per backup set > --- > > Key: HBASE-16673 > URL: https://issues.apache.org/jira/browse/HBASE-16673 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16673-v1.patch > > > New command-line option: -set setName. Will retrieve backup history for a > particular backup set -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16673) Enhance history command: history per backup set
[ https://issues.apache.org/jira/browse/HBASE-16673?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vladimir Rodionov updated HBASE-16673: -- Status: Patch Available (was: Open) > Enhance history command: history per backup set > --- > > Key: HBASE-16673 > URL: https://issues.apache.org/jira/browse/HBASE-16673 > Project: HBase > Issue Type: New Feature >Reporter: Vladimir Rodionov >Assignee: Vladimir Rodionov > Attachments: HBASE-16673-v1.patch > > > New command-line option: -set setName. Will retrieve backup history for a > particular backup set -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-14417) Incremental backup and bulk loading
[ https://issues.apache.org/jira/browse/HBASE-14417?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511698#comment-15511698 ] Ted Yu commented on HBASE-14417: I was adding new test which does one more full table backup after the incremental restore and verifies that BulkLoadDescriptor's are cleaned up from hbase:backup. It turns out the bulk loaded hfiles were renamed during the incremental restore which resulted in FileNotFoundException. Filed HBASE-16672 so that the bulk loaded hfiles can be used for multiple restore destinations. > Incremental backup and bulk loading > --- > > Key: HBASE-14417 > URL: https://issues.apache.org/jira/browse/HBASE-14417 > Project: HBase > Issue Type: New Feature >Affects Versions: 2.0.0 >Reporter: Vladimir Rodionov >Assignee: Ted Yu >Priority: Critical > Labels: backup > Fix For: 2.0.0 > > Attachments: 14417.v1.txt > > > Currently, incremental backup is based on WAL files. Bulk data loading > bypasses WALs for obvious reasons, breaking incremental backups. The only way > to continue backups after bulk loading is to create new full backup of a > table. This may not be feasible for customers who do bulk loading regularly > (say, every day). > Google doc for design: > https://docs.google.com/document/d/1ACCLsecHDvzVSasORgqqRNrloGx4mNYIbvAU7lq5lJE -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511700#comment-15511700 ] Andrew Purtell commented on HBASE-16676: That functionality of HBASE-13375 only exists in branch-1.2 . > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3 >Reporter: Andrew Purtell > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-16676: --- Affects Version/s: (was: 1.4.0) (was: 1.3.0) > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.2.1, 1.2.2, 1.2.3 >Reporter: Andrew Purtell > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
[ https://issues.apache.org/jira/browse/HBASE-16676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-16676: --- Affects Version/s: (was: 2.0.0) > All RPC requests serviced by PriorityRpcServer in some deploys after > HBASE-13375 > > > Key: HBASE-16676 > URL: https://issues.apache.org/jira/browse/HBASE-16676 > Project: HBase > Issue Type: Bug >Affects Versions: 1.2.0, 1.3.0, 1.2.1, 1.4.0, 1.2.2, 1.2.3 >Reporter: Andrew Purtell > > I have been trying to track down why 1.2.x won't sometimes pass a 1 billion > row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC > prioritization could explain it. We get stuck during the loading phase and > the loader job eventually fails. > All testing is done in an insecure environment under the same UNIX user > (clusterdock) so effectively all ops are issued by the superuser. > Doing unrelated work - or so I thought! - I was looking at object allocations > by YCSB workload by thread and when looking at the RegionServer RPC threads > noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of > allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 > and up, instead the vast majority are from threads named > "PriorityRpcServer.handler*" with very little from threads named > "B.defaultRpcServer.handler*". A git bisect to find the change that causes > this leads to HBASE-13375, and so of course this makes sense out of what I am > seeing, but is this really what we want? What about production environments > (insecure and degenerate secure) where all ops are effectively issued by the > superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[ https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511681#comment-15511681 ] Ted Yu commented on HBASE-16672: I agree that there is no need to dive into the details of the aforementioned stack trace. I simplified the description accordingly. Discussion on the stack trace can be carried out over in HBASE-14417. > Add option for bulk load to copy hfile(s) instead of renaming > - > > Key: HBASE-16672 > URL: https://issues.apache.org/jira/browse/HBASE-16672 > Project: HBase > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 16672.v1.txt, 16672.v2.txt, 16672.v3.txt, 16672.v4.txt > > > This is related to HBASE-14417, to support incrementally restoring to > multiple destinations, this issue adds option which would always copy > hfile(s) during bulk load. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[
https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16672:
---
Description: This is related to HBASE-14417, to support incrementally
restoring to multiple destinations, this issue adds option which would always
copy hfile(s) during bulk load. (was: While working on HBASE-14417, I found
that taking full backup of table which received bulk loaded hfiles and later
gone through incremental restore failed with the following exception:
{code}
2016-09-21 11:33:06,340 WARN [member: '10.22.9.171,53915,1474482617015'
subprocedure-pool7-thread-1]
snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(347): Got
Exception in SnapshotSubprocedurePool
java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File
does not exist:
hdfs://localhost:53901/user/tyu/test-data/48928d44-9757-4923-af29-60288fb8a553/data/ns1/test-1474482646700/
d1a93d05a75c596533ba4331a378fb3a/f/3dfb3a37cf6b4b519769b3116d8ab35a_SeqId_205_
at java.util.concurrent.FutureTask.report(FutureTask.java:122)
at java.util.concurrent.FutureTask.get(FutureTask.java:192)
at
org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool.waitForOutstandingTasks(RegionServerSnapshotManager.java:323)
at
org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.flushSnapshot(FlushSnapshotSubprocedure.java:139)
at
org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.insideBarrier(FlushSnapshotSubprocedure.java:159)
{code}
The cause was that the bulk loaded hfiles in source table were renamed during
bulk load step of incremental restore.
To support incrementally restoring to multiple destinations, this issue adds
option which would always copy hfile(s) during bulk load.)
> Add option for bulk load to copy hfile(s) instead of renaming
> -
>
> Key: HBASE-16672
> URL: https://issues.apache.org/jira/browse/HBASE-16672
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16672.v1.txt, 16672.v2.txt, 16672.v3.txt, 16672.v4.txt
>
>
> This is related to HBASE-14417, to support incrementally restoring to
> multiple destinations, this issue adds option which would always copy
> hfile(s) during bulk load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16671) Split TestExportSnapshot
[
https://issues.apache.org/jira/browse/HBASE-16671?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511663#comment-15511663
]
Hadoop QA commented on HBASE-16671:
---
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 1m 3s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 6m
4s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
52s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
54s {color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 37s
{color} | {color:green} master passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} hadoopcheck {color} | {color:green}
37m 21s {color} | {color:green} Patch does not cause any errors with Hadoop
2.4.0 2.4.1 2.5.0 2.5.1 2.5.2 2.6.1 2.6.2 2.6.3 2.7.1. {color} |
| {color:green}+1{color} | {color:green} hbaseprotoc {color} | {color:green} 0m
17s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 35s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 110m 57s
{color} | {color:red} hbase-server in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
23s {color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 166m 49s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hbase.regionserver.TestHRegionWithInMemoryFlush |
| | hadoop.hbase.regionserver.throttle.TestFlushWithThroughputController |
| | hadoop.hbase.client.TestAdmin1 |
| | hadoop.hbase.regionserver.TestRegionServerMetrics |
| Timed out junit tests | org.apache.hadoop.hbase.client.TestMetaWithReplicas |
| | org.apache.hadoop.hbase.client.TestMobCloneSnapshotFromClient |
| | org.apache.hadoop.hbase.client.TestRestoreSnapshotFromClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=1.12.0 Server=1.12.0 Image:yetus/hbase:7bda515 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12829697/HBASE-16671-v0.patch |
| JIRA Issue | HBASE-16671 |
| Optional Tests | asflicense javac javadoc unit findbugs hadoopcheck
hbaseanti checkstyle compile |
| uname | Linux d340db98720d 3.13.0-92-generic #139-Ubuntu SMP Tue Jun 28
20:42:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
/home/jenkins/jenkins-slave/workspace/PreCommit-HBASE-Build/component/dev-support/hbase-personality.sh
|
| git revision | master / 6a4c292 |
| Default Java | 1.8.0_101 |
| findbugs | v3.0.0 |
| unit |
https://builds.apache.org/job/PreCommit-HBASE-Build/3639/artifact/patchprocess/patch-unit-hbase-server.txt
|
| unit test logs |
https://builds.apache.org/job/PreCommit-HBASE-Build/3639/artifact/patchprocess/patch-unit-hbase-server.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HBASE-Build/3639/testReport/ |
| modu
[jira] [Updated] (HBASE-16677) Add table size (total store file size) to table page
[ https://issues.apache.org/jira/browse/HBASE-16677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guang Yang updated HBASE-16677: --- Attachment: HBASE-16677_v0.patch > Add table size (total store file size) to table page > > > Key: HBASE-16677 > URL: https://issues.apache.org/jira/browse/HBASE-16677 > Project: HBase > Issue Type: New Feature > Components: website >Reporter: Guang Yang >Priority: Minor > Attachments: HBASE-16677_v0.patch > > > Currently there is not an easy way to get the table size from the web UI, > though we have the region size on the page, it is still convenient to have a > table for the table size stat. > Another pain point is that when the table grow large with tens of thousands > of regions, it took extremely long time to load the page, however, sometimes > we don't want to check all the regions. An optimization could be to accept a > query parameter to specify the number of regions to render. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[
https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511653#comment-15511653
]
Jerry He commented on HBASE-16672:
--
Also there is another place that needs to be taken care of, in the
SecureBulkLoadListener where we move or copy the files to staging dir. In the
secure bulk load case, that is the place to decide copy or rename.
> Add option for bulk load to copy hfile(s) instead of renaming
> -
>
> Key: HBASE-16672
> URL: https://issues.apache.org/jira/browse/HBASE-16672
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16672.v1.txt, 16672.v2.txt, 16672.v3.txt, 16672.v4.txt
>
>
> While working on HBASE-14417, I found that taking full backup of table which
> received bulk loaded hfiles and later gone through incremental restore failed
> with the following exception:
> {code}
> 2016-09-21 11:33:06,340 WARN [member: '10.22.9.171,53915,1474482617015'
> subprocedure-pool7-thread-1]
> snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(347): Got
> Exception in SnapshotSubprocedurePool
> java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File
> does not exist:
> hdfs://localhost:53901/user/tyu/test-data/48928d44-9757-4923-af29-60288fb8a553/data/ns1/test-1474482646700/
>
> d1a93d05a75c596533ba4331a378fb3a/f/3dfb3a37cf6b4b519769b3116d8ab35a_SeqId_205_
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool.waitForOutstandingTasks(RegionServerSnapshotManager.java:323)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.flushSnapshot(FlushSnapshotSubprocedure.java:139)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.insideBarrier(FlushSnapshotSubprocedure.java:159)
> {code}
> The cause was that the bulk loaded hfiles in source table were renamed during
> bulk load step of incremental restore.
> To support incrementally restoring to multiple destinations, this issue adds
> option which would always copy hfile(s) during bulk load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16294) hbck reporting "No HDFS region dir found" for replicas
[
https://issues.apache.org/jira/browse/HBASE-16294?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511652#comment-15511652
]
Hudson commented on HBASE-16294:
FAILURE: Integrated in Jenkins build HBase-1.1-JDK7 #1784 (See
[https://builds.apache.org/job/HBase-1.1-JDK7/1784/])
HBASE-16294 hbck reporting "No HDFS region dir found" for replicas
(matteo.bertozzi: rev f7961ea70a040d450c7b1393792e43eba076f752)
* (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
> hbck reporting "No HDFS region dir found" for replicas
> --
>
> Key: HBASE-16294
> URL: https://issues.apache.org/jira/browse/HBASE-16294
> Project: HBase
> Issue Type: Bug
> Components: hbck
>Affects Versions: 2.0.0, 1.3.0, 1.0.3, 1.4.0, 1.1.5, 1.2.2
>Reporter: Matteo Bertozzi
>Assignee: Umesh Agashe
>Priority: Minor
> Fix For: 2.0.0, 1.3.0, 1.4.0, 1.1.7, 1.2.4
>
> Attachments: HBASE-16294.v1.patch
>
>
> simple test, create a table with replicas and then run hbck.
> we don't filter out the replicas for the loadHdfsRegioninfo()
> {noformat}
> $ hbase shell
> hbase(main):001:0> create 'myTable', 'myCF', {REGION_REPLICATION => '3'}
> $ hbase hbck
> 2016-07-27 13:47:38,090 WARN [hbasefsck-pool1-t2] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.,
> replicaId => 2 } meta={ENCODED => 9dea3506e09e00910158dc91fa21e550, NAME =>
> 'myTable,,1469652448440_0002.9dea3506e09e00910158dc91fa21e550.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 2}
> 2016-07-27 13:47:38,092 WARN [hbasefsck-pool1-t1] util.HBaseFsck: No HDFS
> region dir found: { meta =>
> myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92., hdfs => null,
> deployed =>
> u1604srv,42895,1469652420413;myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.,
> replicaId => 1 } meta={ENCODED => a03250bca30781ff7002a91c281b4e92, NAME =>
> 'myTable,,1469652448440_0001.a03250bca30781ff7002a91c281b4e92.', STARTKEY =>
> '', ENDKEY => '', REPLICA_ID => 1}
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HBASE-16677) Add table size (total store file size) to table page
Guang Yang created HBASE-16677: -- Summary: Add table size (total store file size) to table page Key: HBASE-16677 URL: https://issues.apache.org/jira/browse/HBASE-16677 Project: HBase Issue Type: New Feature Components: website Reporter: Guang Yang Priority: Minor Currently there is not an easy way to get the table size from the web UI, though we have the region size on the page, it is still convenient to have a table for the table size stat. Another pain point is that when the table grow large with tens of thousands of regions, it took extremely long time to load the page, however, sometimes we don't want to check all the regions. An optimization could be to accept a query parameter to specify the number of regions to render. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[
https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511645#comment-15511645
]
Jerry He commented on HBASE-16672:
--
Hi, Ted
The stack trace looks confusing.
1. Why incremental restore was bulk loading the file paths that are in an
existing table? The incremental restore should be restoring files from the
backup image. The files in the backup image may be coming from the existing
table originally. But the files should be physically in the backup image. This
is different from the snapshot restore, where it is a in-place restore.
2. What the snapshot failed with the missing file?
An bulk load option to keep the original files (not move/rename them to target)
seems a fine option. We can simplify the JIRA description to describe just
that, without the confusing stack trace.
> Add option for bulk load to copy hfile(s) instead of renaming
> -
>
> Key: HBASE-16672
> URL: https://issues.apache.org/jira/browse/HBASE-16672
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16672.v1.txt, 16672.v2.txt, 16672.v3.txt, 16672.v4.txt
>
>
> While working on HBASE-14417, I found that taking full backup of table which
> received bulk loaded hfiles and later gone through incremental restore failed
> with the following exception:
> {code}
> 2016-09-21 11:33:06,340 WARN [member: '10.22.9.171,53915,1474482617015'
> subprocedure-pool7-thread-1]
> snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(347): Got
> Exception in SnapshotSubprocedurePool
> java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File
> does not exist:
> hdfs://localhost:53901/user/tyu/test-data/48928d44-9757-4923-af29-60288fb8a553/data/ns1/test-1474482646700/
>
> d1a93d05a75c596533ba4331a378fb3a/f/3dfb3a37cf6b4b519769b3116d8ab35a_SeqId_205_
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool.waitForOutstandingTasks(RegionServerSnapshotManager.java:323)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.flushSnapshot(FlushSnapshotSubprocedure.java:139)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.insideBarrier(FlushSnapshotSubprocedure.java:159)
> {code}
> The cause was that the bulk loaded hfiles in source table were renamed during
> bulk load step of incremental restore.
> To support incrementally restoring to multiple destinations, this issue adds
> option which would always copy hfile(s) during bulk load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HBASE-16672) Add option for bulk load to copy hfile(s) instead of renaming
[
https://issues.apache.org/jira/browse/HBASE-16672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated HBASE-16672:
---
Attachment: 16672.v4.txt
Patch v4 addresses TestHRegionServerBulkLoadWithOldClient.
> Add option for bulk load to copy hfile(s) instead of renaming
> -
>
> Key: HBASE-16672
> URL: https://issues.apache.org/jira/browse/HBASE-16672
> Project: HBase
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Ted Yu
> Attachments: 16672.v1.txt, 16672.v2.txt, 16672.v3.txt, 16672.v4.txt
>
>
> While working on HBASE-14417, I found that taking full backup of table which
> received bulk loaded hfiles and later gone through incremental restore failed
> with the following exception:
> {code}
> 2016-09-21 11:33:06,340 WARN [member: '10.22.9.171,53915,1474482617015'
> subprocedure-pool7-thread-1]
> snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool(347): Got
> Exception in SnapshotSubprocedurePool
> java.util.concurrent.ExecutionException: java.io.FileNotFoundException: File
> does not exist:
> hdfs://localhost:53901/user/tyu/test-data/48928d44-9757-4923-af29-60288fb8a553/data/ns1/test-1474482646700/
>
> d1a93d05a75c596533ba4331a378fb3a/f/3dfb3a37cf6b4b519769b3116d8ab35a_SeqId_205_
> at java.util.concurrent.FutureTask.report(FutureTask.java:122)
> at java.util.concurrent.FutureTask.get(FutureTask.java:192)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.RegionServerSnapshotManager$SnapshotSubprocedurePool.waitForOutstandingTasks(RegionServerSnapshotManager.java:323)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.flushSnapshot(FlushSnapshotSubprocedure.java:139)
> at
> org.apache.hadoop.hbase.regionserver.snapshot.FlushSnapshotSubprocedure.insideBarrier(FlushSnapshotSubprocedure.java:159)
> {code}
> The cause was that the bulk loaded hfiles in source table were renamed during
> bulk load step of incremental restore.
> To support incrementally restoring to multiple destinations, this issue adds
> option which would always copy hfile(s) during bulk load.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HBASE-16662) Fix open POODLE vulnerabilities
[ https://issues.apache.org/jira/browse/HBASE-16662?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15511605#comment-15511605 ] Andrew Purtell commented on HBASE-16662: +1 Those test failures do not look related. Let me poke around with this a bit, and if it looks good I will commit with backport to all active branches. > Fix open POODLE vulnerabilities > --- > > Key: HBASE-16662 > URL: https://issues.apache.org/jira/browse/HBASE-16662 > Project: HBase > Issue Type: Bug > Components: REST, Thrift >Reporter: Ben Lau >Assignee: Ben Lau > Attachments: HBASE-16662-master.patch > > > We recently found a security issue in our HBase REST servers. The issue is a > variant of the POODLE vulnerability (https://en.wikipedia.org/wiki/POODLE) > and is present in the HBase Thrift server as well. It also appears to affect > the JMXListener coprocessor. The vulnerabilities probably affect all > versions of HBase that have the affected services. (If you don't use the > affected services with SSL then this ticket probably doesn't affect you). > Included is a patch to fix the known POODLE vulnerabilities in master. Let > us know if we missed any. From our end we only personally encountered the > HBase REST vulnerability. We do not use the Thrift server or JMXListener > coprocessor but discovered those problems after discussing the issue with > some of the HBase PMCs. > Coincidentally, Hadoop recently committed a SslSelectChannelConnectorSecure > which is more or less the same as one of the fixes in this patch. Hadoop > wasn't originally affected by the vulnerability in the > SslSelectChannelConnector, but about a month ago they committed HADOOP-12765 > which does use that class, so they added a SslSelectChannelConnectorSecure > class similar to this patch. Since this class is present in Hadoop 2.7.4+ > which hasn't been released yet, we will for now just include our own version > instead of depending on the Hadoop version. > After the patch is approved for master we can backport as necessary to older > versions of HBase. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-14734) BindException when setting up MiniKdc
[ https://issues.apache.org/jira/browse/HBASE-14734?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Appy updated HBASE-14734: - Resolution: Fixed Status: Resolved (was: Patch Available) > BindException when setting up MiniKdc > - > > Key: HBASE-14734 > URL: https://issues.apache.org/jira/browse/HBASE-14734 > Project: HBase > Issue Type: Sub-task > Components: flakey, test >Reporter: stack >Assignee: Appy > Fix For: 1.3.0, 1.4.0, 1.2.4, 2.0.. > > Attachments: HBASE-14734.master.001.patch > > > Root cause : Port for kdc service gets selected in the constructor, but we > bind to it later in MiniKdc.start()-->MiniKdc.initKDCServer() --> > KdcServer.start(). In meantime, some other service can capture the port which > results in BindException. The solution here is to catch the exception and > retry. > From > https://builds.apache.org/view/H-L/view/HBase/job/HBase-1.2/330/jdk=latest1.7,label=Hadoop/testReport/junit/org.apache.hadoop.hbase.security.token/TestGenerateDelegationToken/org_apache_hadoop_hbase_security_token_TestGenerateDelegationToken/ > Error Message > Address already in use > Stacktrace > java.net.BindException: Address already in use > at sun.nio.ch.Net.bind0(Native Method) > at sun.nio.ch.Net.bind(Net.java:444) > at sun.nio.ch.Net.bind(Net.java:436) > at > sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:214) > at sun.nio.ch.ServerSocketAdaptor.bind(ServerSocketAdaptor.java:74) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:198) > at > org.apache.mina.transport.socket.nio.NioSocketAcceptor.open(NioSocketAcceptor.java:51) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.registerHandles(AbstractPollingIoAcceptor.java:547) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor.access$400(AbstractPollingIoAcceptor.java:68) > at > org.apache.mina.core.polling.AbstractPollingIoAcceptor$Acceptor.run(AbstractPollingIoAcceptor.java:422) > at > org.apache.mina.util.NamePreservingRunnable.run(NamePreservingRunnable.java:64) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Can this utility be made to not fail if address taken? Try another? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-16676) All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375
Andrew Purtell created HBASE-16676: -- Summary: All RPC requests serviced by PriorityRpcServer in some deploys after HBASE-13375 Key: HBASE-16676 URL: https://issues.apache.org/jira/browse/HBASE-16676 Project: HBase Issue Type: Bug Affects Versions: 1.2.3, 1.2.2, 1.2.1, 1.2.0, 2.0.0, 1.3.0, 1.4.0 Reporter: Andrew Purtell I have been trying to track down why 1.2.x won't sometimes pass a 1 billion row ITBLL run while 0.98.22 and 1.1.6 will always, and a defeat of RPC prioritization could explain it. We get stuck during the loading phase and the loader job eventually fails. All testing is done in an insecure environment under the same UNIX user (clusterdock) so effectively all ops are issued by the superuser. Doing unrelated work - or so I thought! - I was looking at object allocations by YCSB workload by thread and when looking at the RegionServer RPC threads noticed that for 0.98.22 and 1.1.6, as expected, the vast majority of allocations are from threads named "B.defaultRpcServer.handler*". In 1.2.0 and up, instead the vast majority are from threads named "PriorityRpcServer.handler*" with very little from threads named "B.defaultRpcServer.handler*". A git bisect to find the change that causes this leads to HBASE-13375, and so of course this makes sense out of what I am seeing, but is this really what we want? What about production environments (insecure and degenerate secure) where all ops are effectively issued by the superuser? We run one of these at Salesforce. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
