[jira] [Commented] (HBASE-12336) RegionServer failed to shutdown for NodeFailoverWorker thread
[ https://issues.apache.org/jira/browse/HBASE-12336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188087#comment-14188087 ] Liu Shaohui commented on HBASE-12336: - [~tianq] Sorry, the log has been deleted automatically. We encountered this problem when we upgrade the zk cluster from 3 nodes to 5 fives and the zk cluster was restarted. > RegionServer failed to shutdown for NodeFailoverWorker thread > - > > Key: HBASE-12336 > URL: https://issues.apache.org/jira/browse/HBASE-12336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.11 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Attachments: stack > > > After enabling hbase.zookeeper.useMulti in hbase cluster, we found that > regionserver failed to shutdown. Other threads have exited except a > NodeFailoverWorker thread. > {code} > "ReplicationExecutor-0" prio=10 tid=0x7f0d40195ad0 nid=0x73a in > Object.wait() [0x7f0dc8fe6000] >java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:485) > at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309) > - locked <0x0005a16df080> (a > org.apache.zookeeper.ClientCnxn$Packet) > at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:930) > at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:912) > at > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.multi(RecoverableZooKeeper.java:531) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.multiOrSequential(ZKUtil.java:1518) > at > org.apache.hadoop.hbase.replication.ReplicationZookeeper.copyQueuesFromRSUsingMulti(ReplicationZookeeper.java:804) > at > org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:612) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > {code} > It's sure that the shutdown method of the executor is called in > ReplicationSourceManager#join. > > I am looking for the root cause and suggestions are welcomed. Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12075) Preemptive Fast Fail
[ https://issues.apache.org/jira/browse/HBASE-12075?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188088#comment-14188088 ] Manukranth Kolloju commented on HBASE-12075: Thanks [~stack], [~tedyu], [~eclark] for patiently going through my patch. > Preemptive Fast Fail > > > Key: HBASE-12075 > URL: https://issues.apache.org/jira/browse/HBASE-12075 > Project: HBase > Issue Type: Sub-task > Components: Client >Affects Versions: 0.99.0, 2.0.0, 0.98.6.1 >Reporter: Manukranth Kolloju >Assignee: Manukranth Kolloju > Fix For: 2.0.0, 0.99.2 > > Attachments: 0001-Add-a-test-case-for-Preemptive-Fast-Fail.patch, > 0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch, > 0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch, > 0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch, > 0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch, > 0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch, > 0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch, > 0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch, > 0001-HBASE-12075-Implement-Preemptive-Fast-Fail.patch, > 0001-Implement-Preemptive-Fast-Fail.patch, > 0001-Implement-Preemptive-Fast-Fail.patch, > 0001-Implement-Preemptive-Fast-Fail.patch, > 0001-Implement-Preemptive-Fast-Fail.patch, > 0001-Implement-Preemptive-Fast-Fail.patch, > HBASE-12075-Preemptive-Fast-Fail-V15.patch > > > In multi threaded clients, we use a feature developed on 0.89-fb branch > called Preemptive Fast Fail. This allows the client threads which would > potentially fail, fail fast. The idea behind this feature is that we allow, > among the hundreds of client threads, one thread to try and establish > connection with the regionserver and if that succeeds, we mark it as a live > node again. Meanwhile, other threads which are trying to establish connection > to the same server would ideally go into the timeouts which is effectively > unfruitful. We can in those cases return appropriate exceptions to those > clients instead of letting them retry. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-10856) Prep for 1.0
[ https://issues.apache.org/jira/browse/HBASE-10856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-10856. --- Resolution: Fixed Resolving. Only outstanding subtasks are the update of jars in time for release 1.0 in mvn and our dependencies. That is on going. No other dependent issues (those not being worked on were moved out). I believe I covered all the outstanding doc issues in here and pushed the doc out. > Prep for 1.0 > > > Key: HBASE-10856 > URL: https://issues.apache.org/jira/browse/HBASE-10856 > Project: HBase > Issue Type: Umbrella >Reporter: stack > Fix For: 0.99.2 > > > Tasks for 1.0 copied here from our '1.0.0' mailing list discussion. Idea is > to file subtasks off this one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12336) RegionServer failed to shutdown for NodeFailoverWorker thread
[ https://issues.apache.org/jira/browse/HBASE-12336?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188091#comment-14188091 ] Liu Shaohui commented on HBASE-12336: - [~stack] Yes, call setDaemon on ThreadFactoryBuilder will fix this problem. But i am wondering why this thread did not exist even if we called shutdown on shutdown. > RegionServer failed to shutdown for NodeFailoverWorker thread > - > > Key: HBASE-12336 > URL: https://issues.apache.org/jira/browse/HBASE-12336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.11 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Attachments: stack > > > After enabling hbase.zookeeper.useMulti in hbase cluster, we found that > regionserver failed to shutdown. Other threads have exited except a > NodeFailoverWorker thread. > {code} > "ReplicationExecutor-0" prio=10 tid=0x7f0d40195ad0 nid=0x73a in > Object.wait() [0x7f0dc8fe6000] >java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:485) > at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309) > - locked <0x0005a16df080> (a > org.apache.zookeeper.ClientCnxn$Packet) > at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:930) > at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:912) > at > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.multi(RecoverableZooKeeper.java:531) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.multiOrSequential(ZKUtil.java:1518) > at > org.apache.hadoop.hbase.replication.ReplicationZookeeper.copyQueuesFromRSUsingMulti(ReplicationZookeeper.java:804) > at > org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:612) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > {code} > It's sure that the shutdown method of the executor is called in > ReplicationSourceManager#join. > > I am looking for the root cause and suggestions are welcomed. Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12336) RegionServer failed to shutdown for NodeFailoverWorker thread
[ https://issues.apache.org/jira/browse/HBASE-12336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liu Shaohui updated HBASE-12336: Attachment: HBASE-12336-trunk-v1.diff Call setDaemon on ThreadFactoryBuilder to fix this problem. > RegionServer failed to shutdown for NodeFailoverWorker thread > - > > Key: HBASE-12336 > URL: https://issues.apache.org/jira/browse/HBASE-12336 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.11 >Reporter: Liu Shaohui >Assignee: Liu Shaohui >Priority: Minor > Attachments: HBASE-12336-trunk-v1.diff, stack > > > After enabling hbase.zookeeper.useMulti in hbase cluster, we found that > regionserver failed to shutdown. Other threads have exited except a > NodeFailoverWorker thread. > {code} > "ReplicationExecutor-0" prio=10 tid=0x7f0d40195ad0 nid=0x73a in > Object.wait() [0x7f0dc8fe6000] >java.lang.Thread.State: WAITING (on object monitor) > at java.lang.Object.wait(Native Method) > at java.lang.Object.wait(Object.java:485) > at org.apache.zookeeper.ClientCnxn.submitRequest(ClientCnxn.java:1309) > - locked <0x0005a16df080> (a > org.apache.zookeeper.ClientCnxn$Packet) > at org.apache.zookeeper.ZooKeeper.multiInternal(ZooKeeper.java:930) > at org.apache.zookeeper.ZooKeeper.multi(ZooKeeper.java:912) > at > org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper.multi(RecoverableZooKeeper.java:531) > at > org.apache.hadoop.hbase.zookeeper.ZKUtil.multiOrSequential(ZKUtil.java:1518) > at > org.apache.hadoop.hbase.replication.ReplicationZookeeper.copyQueuesFromRSUsingMulti(ReplicationZookeeper.java:804) > at > org.apache.hadoop.hbase.replication.regionserver.ReplicationSourceManager$NodeFailoverWorker.run(ReplicationSourceManager.java:612) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > {code} > It's sure that the shutdown method of the executor is called in > ReplicationSourceManager#join. > > I am looking for the root cause and suggestions are welcomed. Thanks -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10201) Port 'Make flush decisions per column family' to trunk
[ https://issues.apache.org/jira/browse/HBASE-10201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188093#comment-14188093 ] zhangduo commented on HBASE-10201: -- {quote} Out of interest, are you using the hbase formatter? {quote} No, I just use the default formatter with indent and max length changed. Only new code is formatted, old code is "format" manually to keep the patch clean... I will try the hbase formatter later. I found it when looking for test-patch.sh, thanks. {quote} You going to try hbase-it? {quote} Yes I have run it with 'mvn verify' under hbase-it. There are some fails and errors, I need to see the source code to identify the reason. Thanks. > Port 'Make flush decisions per column family' to trunk > -- > > Key: HBASE-10201 > URL: https://issues.apache.org/jira/browse/HBASE-10201 > Project: HBase > Issue Type: Improvement > Components: wal >Reporter: Ted Yu >Assignee: zhangduo >Priority: Critical > Fix For: 2.0.0, 0.99.2 > > Attachments: 3149-trunk-v1.txt, HBASE-10201-0.98.patch, > HBASE-10201-0.98_1.patch, HBASE-10201-0.98_2.patch, HBASE-10201-0.99.patch, > HBASE-10201.patch, HBASE-10201_1.patch, HBASE-10201_2.patch, > HBASE-10201_3.patch > > > Currently the flush decision is made using the aggregate size of all column > families. When large and small column families co-exist, this causes many > small flushes of the smaller CF. We need to make per-CF flush decisions. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12354) Update dependencies in time for 1.0 release
[ https://issues.apache.org/jira/browse/HBASE-12354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188095#comment-14188095 ] Hadoop QA commented on HBASE-12354: --- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12677827/12354v2.txt against trunk revision . ATTACHMENT ID: 12677827 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified tests. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 checkstyle{color}. The applied patch does not increase the total number of checkstyle errors {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/11502//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11502//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11502//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11502//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11502//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11502//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11502//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11502//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11502//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11502//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11502//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11502//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/11502//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11502//console This message is automatically generated. > Update dependencies in time for 1.0 release > --- > > Key: HBASE-12354 > URL: https://issues.apache.org/jira/browse/HBASE-12354 > Project: HBase > Issue Type: Sub-task > Components: dependencies >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 0.99.2 > > Attachments: 12354.txt, 12354v2.txt > > > Going through and updating egregiously old dependencies for 1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12355) Update maven plugins
[ https://issues.apache.org/jira/browse/HBASE-12355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188096#comment-14188096 ] Hadoop QA commented on HBASE-12355: --- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12677828/12355v6.txt against trunk revision . ATTACHMENT ID: 12677828 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 2 new or modified tests. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 checkstyle{color}. The applied patch does not increase the total number of checkstyle errors {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/11503//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11503//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11503//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11503//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11503//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11503//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11503//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11503//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11503//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11503//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11503//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11503//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/11503//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11503//console This message is automatically generated. > Update maven plugins > > > Key: HBASE-12355 > URL: https://issues.apache.org/jira/browse/HBASE-12355 > Project: HBase > Issue Type: Sub-task > Components: build >Reporter: stack >Assignee: stack > Fix For: 0.99.2 > > Attachments: 12355.txt, 12355v2.txt, 12355v3.txt, 12355v5.txt, > 12355v6.txt, 12355v6.txt > > > Update maven plugins. Some are way old. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12355) Update maven plugins
[ https://issues.apache.org/jira/browse/HBASE-12355?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188097#comment-14188097 ] Elliott Clark commented on HBASE-12355: --- +1 > Update maven plugins > > > Key: HBASE-12355 > URL: https://issues.apache.org/jira/browse/HBASE-12355 > Project: HBase > Issue Type: Sub-task > Components: build >Reporter: stack >Assignee: stack > Fix For: 0.99.2 > > Attachments: 12355.txt, 12355v2.txt, 12355v3.txt, 12355v5.txt, > 12355v6.txt, 12355v6.txt > > > Update maven plugins. Some are way old. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12312) Another couple of createTable race conditions
[ https://issues.apache.org/jira/browse/HBASE-12312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188099#comment-14188099 ] Hudson commented on HBASE-12312: SUCCESS: Integrated in HBase-TRUNK #5713 (See [https://builds.apache.org/job/HBase-TRUNK/5713/]) HBASE-12312 Another couple of createTable race conditions (stack: rev 95282f2ea53bdb55af7fe5a0a749c1fcde824b6c) * hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessController.java * hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestScanEarlyTermination.java * hbase-server/src/test/java/org/apache/hadoop/hbase/replication/TestReplicationBase.java > Another couple of createTable race conditions > - > > Key: HBASE-12312 > URL: https://issues.apache.org/jira/browse/HBASE-12312 > Project: HBase > Issue Type: Bug >Reporter: Dima Spivak >Assignee: Dima Spivak > Fix For: 2.0.0, 0.99.2 > > Attachments: HBASE-12312_master_v1.patch, > HBASE-12312_master_v2.patch, HBASE-12312_master_v3 (1).patch, > HBASE-12312_master_v3.patch, HBASE-12312_master_v3.patch, > HBASE-12312_master_v3.patch, HBASE-12312_master_v4.patch > > > Found a couple more failing tests in TestAccessController and > TestScanEarlyTermination caused by my favorite race condition. :) Will post a > patch in a second. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12313) Redo the hfile index length optimization so cell-based rather than serialized KV key
[ https://issues.apache.org/jira/browse/HBASE-12313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188111#comment-14188111 ] Hadoop QA commented on HBASE-12313: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12677833/12313v10.txt against trunk revision . ATTACHMENT ID: 12677833 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 36 new or modified tests. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 checkstyle{color}. The applied patch does not increase the total number of checkstyle errors {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 lineLengths{color}. The patch introduces the following lines longer than 100: +FSReaderImpl(FSDataInputStream istream, long fileSize, HFileContext fileContext) throws IOException { +HFileBlock.FSReaderImpl fsBlockReaderV2 = new HFileBlock.FSReaderImpl(fsdis, fileSize, hfs, path, + HFileBlock.FSReaderImpl hbr = new HFileBlock.FSReaderImpl(new FSDataInputStreamWrapper(is), {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:green}+1 core tests{color}. The patch passed unit tests in . {color:red}-1 core zombie tests{color}. There are 1 zombie test(s): at org.apache.hadoop.hdfs.server.namenode.TestMetaSave.testMetasaveAfterDelete(TestMetaSave.java:126) Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/11504//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11504//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11504//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11504//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11504//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11504//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11504//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11504//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11504//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11504//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11504//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11504//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/11504//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11504//console This message is automatically generated. > Redo the hfile index length optimization so cell-based rather than serialized > KV key > > > Key: HBASE-12313 > URL: https://issues.apache.org/jira/browse/HBASE-12313 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: stack >Assignee: stack > Attachments: > 0001-HBASE-12313-Redo-the-hfile-index-length-optimization.patch, > 0001-HBASE-12313-Redo-the-hfile-index-length-optimization.patch, > 0001-HBASE-12313-Redo-the-hfile-index-length-optimization.patch, > 0001-HBASE-12313-Redo-the-hfile-index-length-optimization.patch, > 0001-HBASE-12313-Redo-the-hfile-index-length-optimization.patch, > 12313v10.txt, 12313v5.txt, 12313v6.txt, 12313v8.txt > > > Trying to remove API that returns the 'key' of a KV serialized into a byte > array is thorny.
[jira] [Updated] (HBASE-12375) LoadIncrementalHFiles fails to load data in table when CF name starts with '_'
[ https://issues.apache.org/jira/browse/HBASE-12375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-12375: -- Status: Patch Available (was: Open) > LoadIncrementalHFiles fails to load data in table when CF name starts with '_' > -- > > Key: HBASE-12375 > URL: https://issues.apache.org/jira/browse/HBASE-12375 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.5 >Reporter: Ashish Singhi >Assignee: Ashish Singhi >Priority: Minor > Attachments: HBASE-12375.patch > > > We do not restrict user from creating a table having column family starting > with '_'. > So when user creates a table in such a way then LoadIncrementalHFiles will > skip those family data to load into the table. > {code} > // Skip _logs, etc > if (familyDir.getName().startsWith("_")) continue; > {code} > I think we should remove that check as I do not see any _logs directory being > created by the bulkload tool in the output directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12375) LoadIncrementalHFiles fails to load data in table when CF name starts with '_'
[ https://issues.apache.org/jira/browse/HBASE-12375?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-12375: -- Attachment: HBASE-12375.patch Patch for master branch. Can some one please review. > LoadIncrementalHFiles fails to load data in table when CF name starts with '_' > -- > > Key: HBASE-12375 > URL: https://issues.apache.org/jira/browse/HBASE-12375 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.5 >Reporter: Ashish Singhi >Assignee: Ashish Singhi >Priority: Minor > Attachments: HBASE-12375.patch > > > We do not restrict user from creating a table having column family starting > with '_'. > So when user creates a table in such a way then LoadIncrementalHFiles will > skip those family data to load into the table. > {code} > // Skip _logs, etc > if (familyDir.getName().startsWith("_")) continue; > {code} > I think we should remove that check as I do not see any _logs directory being > created by the bulkload tool in the output directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12068) [Branch-1] Avoid need to always do KeyValueUtil#ensureKeyValue for Filter transformCell
[ https://issues.apache.org/jira/browse/HBASE-12068?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188134#comment-14188134 ] Hudson commented on HBASE-12068: SUCCESS: Integrated in HBase-TRUNK #5714 (See [https://builds.apache.org/job/HBase-TRUNK/5714/]) Add note to upgrade section on HBASE-12068; i.e. things to do if you have custom filters (stack: rev 3a9cf5b2cdc3c3d24a085b3a4d4c289dcce09766) * src/main/docbkx/upgrading.xml > [Branch-1] Avoid need to always do KeyValueUtil#ensureKeyValue for Filter > transformCell > --- > > Key: HBASE-12068 > URL: https://issues.apache.org/jira/browse/HBASE-12068 > Project: HBase > Issue Type: Sub-task > Components: Filters >Affects Versions: 0.99.0 >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.99.1 > > Attachments: HBASE-12068.patch > > > During read with Filters added to Scan/Get, the core code calls > transformCell(Cell) on the Filter. Most of the filters do not implement > transform API so the method from FilterBase will get executed > {code} > @Override > public Cell transformCell(Cell v) throws IOException { > // Old filters based off of this class will override KeyValue > transform(KeyValue). > // Thus to maintain compatibility we need to call the old version. > return transform(KeyValueUtil.ensureKeyValue(v)); > } > {code} > Here always it do KeyValueUtil.ensureKeyValue. When a non KV cell comes in, > we need recreate KV and do deep copy of key and value! > We have to stick with this model in branch-1 for BC. > So as a workaround to avoid possible KV convert, we can implement > transformCell(Cell) method in all of our individual Filter classes which just > return the incoming cell (So that method from FilterBase wont get executed) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12297) Support DBB usage in Bloom and HFileIndex area
[ https://issues.apache.org/jira/browse/HBASE-12297?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-12297: --- Resolution: Fixed Fix Version/s: 0.99.2 2.0.0 Hadoop Flags: Incompatible change Status: Resolved (was: Patch Available) Pushed to 0.99 and trunk. Thanks for the reviews Stack and Ted. > Support DBB usage in Bloom and HFileIndex area > -- > > Key: HBASE-12297 > URL: https://issues.apache.org/jira/browse/HBASE-12297 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 0.99.2 > > Attachments: HBASE-12297.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12282) Ensure Cells and its implementations work with Buffers also
[ https://issues.apache.org/jira/browse/HBASE-12282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188157#comment-14188157 ] Anoop Sam John commented on HBASE-12282: I think Stack suggestion is to add hasArray() in Cell We have new BBBackedCell which extending Cell. In impl of BB backed let hasArray() return false. In comparators, based on hasArray() use getxxxArray() or getxxxBuffer() Correct Stack? > Ensure Cells and its implementations work with Buffers also > --- > > Key: HBASE-12282 > URL: https://issues.apache.org/jira/browse/HBASE-12282 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 0.99.1 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 0.99.2 > > Attachments: HBASE-12224_2.patch > > > This issue can be used to brainstorm and then do the necessary changes for > the offheap work. All impl of cells deal with byte[] but when we change the > Hfileblocks/Readers to work purely with Buffers then the byte[] usage would > mean that always the data is copied to the onheap. Cell may need some > interface change to implement this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12346) Scan's default auths behavior under Visibility labels
[ https://issues.apache.org/jira/browse/HBASE-12346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188161#comment-14188161 ] Anoop Sam John commented on HBASE-12346: bq.Does this mean we have EnforcingScanLabelGenerator and DefaultScanLabelGenerator both doing the same work? No. EnforcingScanLabelGenerator dont care what Auths passed in Scan at all. (if some is there or none) It always enforces *all* labels the user is auth for. DefaultScanLabelGenerator even after with this patch honor the Authroizations in Scan (if any). It can be a subset of the user auths. If none is passed then we default back to all. (Like Accumulo behaves) > Scan's default auths behavior under Visibility labels > - > > Key: HBASE-12346 > URL: https://issues.apache.org/jira/browse/HBASE-12346 > Project: HBase > Issue Type: Bug > Components: API, security >Affects Versions: 0.98.7, 0.99.1 >Reporter: Jerry He > Fix For: 0.98.8, 0.99.2 > > Attachments: HBASE-12346-master-v2.patch, HBASE-12346-master.patch > > > In Visibility Labels security, a set of labels (auths) are administered and > associated with a user. > A user can normally only see cell data during scan that are part of the > user's label set (auths). > Scan uses setAuthorizations to indicates its wants to use the auths to access > the cells. > Similarly in the shell: > {code} > scan 'table1', AUTHORIZATIONS => ['private'] > {code} > But it is a surprise to find that setAuthorizations seems to be 'mandatory' > in the default visibility label security setting. Every scan needs to > setAuthorizations before the scan can get any cells even the cells are under > the labels the request user is part of. > The following steps will illustrate the issue: > Run as superuser. > {code} > 1. create a visibility label called 'private' > 2. create 'table1' > 3. put into 'table1' data and label the data as 'private' > 4. set_auths 'user1', 'private' > 5. grant 'user1', 'RW', 'table1' > {code} > Run as 'user1': > {code} > 1. scan 'table1' > This show no cells. > 2. scan 'table1', scan 'table1', AUTHORIZATIONS => ['private'] > This will show all the data. > {code} > I am not sure if this is expected by design or a bug. > But a more reasonable, more client application backward compatible, and less > surprising default behavior should probably look like this: > A scan's default auths, if its Authorizations attributes is not set > explicitly, should be all the auths the request user is administered and > allowed on the server. > If scan.setAuthorizations is used, then the server further filter the auths > during scan: use the input auths minus what is not in user's label set on the > server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12358) Create ByteBuffer backed Cell
[ https://issues.apache.org/jira/browse/HBASE-12358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188168#comment-14188168 ] Anoop Sam John commented on HBASE-12358: bq.CP would be exposed with both type of cells. Can we have annotations in the API that are exposed to cells and filters that tells which one to use and which one to use? That may be confusing but if we can have a cleaner way of showing which APIs are in the read and which one in write then may be it may make sense to extend Cell only for the new BB based cell. If we continue to pass Cell in read path.. every where we need buffers, we will end up in casting. That is ugly. Can we change all places in read path to new interface BBBackedCell or some other better name. Like what we pass to Filter, to cps, what StoreScanner, InternalScanner, RegionScanner returns.. etc... It can land in 2.0 only I believe. One option as Stack suggested have a hasArray() in Cell and based on decide which API to call in places like Comparators. (where we deal with Cells only). Here the impls should throw Exception out of getxxxArray() API if hasArray() is false. (like BB impls) Or else not make BBBackedCell to extend Cell at all. So we will see only getxxxBuffer() in read path. One adv is there is no diff in Public exposed Cell. It might not make much sense for hasArray() at client side because there we deal with Cells backed by Array only. Only in read path it make sense. > Create ByteBuffer backed Cell > - > > Key: HBASE-12358 > URL: https://issues.apache.org/jira/browse/HBASE-12358 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 0.99.2 > > Attachments: HBASE-12358.patch > > > As part of HBASE-12224 and HBASE-12282 we wanted a Cell that is backed by BB. > Changing the core Cell impl would not be needed as it is used in server > only. So we will create a BB backed Cell and use it in the Server side read > path. This JIRA just creates an interface that extends Cell and adds the > needed API. > The getTimeStamp and getTypebyte() can still refer to the original Cell API > only. The getXXxOffset() and getXXXLength() can also refer to the original > Cell only. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12375) LoadIncrementalHFiles fails to load data in table when CF name starts with '_'
[ https://issues.apache.org/jira/browse/HBASE-12375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188167#comment-14188167 ] Matteo Bertozzi commented on HBASE-12375: - I think that by removing the check you'll fail in case of splits (but I haven't checked) the problem is that the LoadIncrementalHFiles will create a "_tmp" directory. In theory is enough adding to this patch the rename of "_tmp" to something like .tmp did you tried to run this patch with a set of files that requires splitting? > LoadIncrementalHFiles fails to load data in table when CF name starts with '_' > -- > > Key: HBASE-12375 > URL: https://issues.apache.org/jira/browse/HBASE-12375 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.5 >Reporter: Ashish Singhi >Assignee: Ashish Singhi >Priority: Minor > Attachments: HBASE-12375.patch > > > We do not restrict user from creating a table having column family starting > with '_'. > So when user creates a table in such a way then LoadIncrementalHFiles will > skip those family data to load into the table. > {code} > // Skip _logs, etc > if (familyDir.getName().startsWith("_")) continue; > {code} > I think we should remove that check as I do not see any _logs directory being > created by the bulkload tool in the output directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12374) Change DBEs to work with new BB based cell
[ https://issues.apache.org/jira/browse/HBASE-12374?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188169#comment-14188169 ] Anoop Sam John commented on HBASE-12374: The entire read path APIs to have this new Interface as param/return type rather than Cell. Till the RPC response. > Change DBEs to work with new BB based cell > -- > > Key: HBASE-12374 > URL: https://issues.apache.org/jira/browse/HBASE-12374 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > > Once we are changing the read path to use BB based cell then the DBEs should > also return BB based cells. Currently they are byte[] array backed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12373) Provide a command to list visibility labels
[ https://issues.apache.org/jira/browse/HBASE-12373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188173#comment-14188173 ] Anoop Sam John commented on HBASE-12373: It should be allowed only for a user who is having system label auth. A client side API and a shell command (?) You there for a patch Jerry? Thanks. > Provide a command to list visibility labels > --- > > Key: HBASE-12373 > URL: https://issues.apache.org/jira/browse/HBASE-12373 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.98.7, 0.99.1 >Reporter: Jerry He >Priority: Minor > > A command to list visibility labels that are in place would be handy. > This is also in line with many of the other hbase list commands. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12282) Ensure Cells and its implementations work with Buffers also
[ https://issues.apache.org/jira/browse/HBASE-12282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188180#comment-14188180 ] ramkrishna.s.vasudevan commented on HBASE-12282: bq.In comparators, based on hasArray() use getxxxArray() or getxxxBuffer() Yes true. This I agree. So how should the cell be created? So based on the buffer from the HFileBlock we wil decide if it is a direct BB or on heap BB and create KV based on that ? That KV is either a normal KV that we use now (create cell based on buffer.array) or it would be a new KV which will have buffer in it? > Ensure Cells and its implementations work with Buffers also > --- > > Key: HBASE-12282 > URL: https://issues.apache.org/jira/browse/HBASE-12282 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 0.99.1 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 0.99.2 > > Attachments: HBASE-12224_2.patch > > > This issue can be used to brainstorm and then do the necessary changes for > the offheap work. All impl of cells deal with byte[] but when we change the > Hfileblocks/Readers to work purely with Buffers then the byte[] usage would > mean that always the data is copied to the onheap. Cell may need some > interface change to implement this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12282) Ensure Cells and its implementations work with Buffers also
[ https://issues.apache.org/jira/browse/HBASE-12282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188181#comment-14188181 ] ramkrishna.s.vasudevan commented on HBASE-12282: Another thing is if that is the case then all the new APIs that we add in ByteBufferUtils could just handle only the DBB cases (including the Unsafe cases for DBB). Not the cases of normal BB because in this case we could still work with byte[] inside the BB. > Ensure Cells and its implementations work with Buffers also > --- > > Key: HBASE-12282 > URL: https://issues.apache.org/jira/browse/HBASE-12282 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 0.99.1 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 0.99.2 > > Attachments: HBASE-12224_2.patch > > > This issue can be used to brainstorm and then do the necessary changes for > the offheap work. All impl of cells deal with byte[] but when we change the > Hfileblocks/Readers to work purely with Buffers then the byte[] usage would > mean that always the data is copied to the onheap. Cell may need some > interface change to implement this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12282) Ensure Cells and its implementations work with Buffers also
[ https://issues.apache.org/jira/browse/HBASE-12282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188184#comment-14188184 ] Anoop Sam John commented on HBASE-12282: bq.So how should the cell be created? So based on the buffer from the HFileBlock we wil decide if it is a direct BB or on heap BB and create KV based on that ? That KV is either a normal KV that we use now (create cell based on buffer.array) or it would be a new KV which will have buffer in it? IMHO just not worry abt DBB/HBB here. Always create a new KV type which implements new BBBackedBuffer. We have optimization in Comparators etc to check based on hasArry() and do. > Ensure Cells and its implementations work with Buffers also > --- > > Key: HBASE-12282 > URL: https://issues.apache.org/jira/browse/HBASE-12282 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 0.99.1 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 0.99.2 > > Attachments: HBASE-12224_2.patch > > > This issue can be used to brainstorm and then do the necessary changes for > the offheap work. All impl of cells deal with byte[] but when we change the > Hfileblocks/Readers to work purely with Buffers then the byte[] usage would > mean that always the data is copied to the onheap. Cell may need some > interface change to implement this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12282) Ensure Cells and its implementations work with Buffers also
[ https://issues.apache.org/jira/browse/HBASE-12282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188186#comment-14188186 ] Anoop Sam John commented on HBASE-12282: Even the KeyOnlyKeyValue also. Let it be a BB backed one. It will be better we have this BufferBackedCell only in read path. The Compartors no need to worry one type is buffer backed and other array backed. Let both be Buffer backed. Even we can have new Comparators to avoid checks in all method. > Ensure Cells and its implementations work with Buffers also > --- > > Key: HBASE-12282 > URL: https://issues.apache.org/jira/browse/HBASE-12282 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 0.99.1 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 0.99.2 > > Attachments: HBASE-12224_2.patch > > > This issue can be used to brainstorm and then do the necessary changes for > the offheap work. All impl of cells deal with byte[] but when we change the > Hfileblocks/Readers to work purely with Buffers then the byte[] usage would > mean that always the data is copied to the onheap. Cell may need some > interface change to implement this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12219) Cache more efficiently getAll() and get() in FSTableDescriptors
[ https://issues.apache.org/jira/browse/HBASE-12219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-12219: Attachment: HBASE-12219-v1.patch > Cache more efficiently getAll() and get() in FSTableDescriptors > --- > > Key: HBASE-12219 > URL: https://issues.apache.org/jira/browse/HBASE-12219 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.94.24, 0.99.1, 0.98.6.1 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez > Labels: scalability > Attachments: HBASE-12219-v1.patch, HBASE-12219.v0.txt, list.png > > > Currently table descriptors and tables are cached once they are accessed for > the first time. Next calls to the master only require a trip to HDFS to > lookup the modified time in order to reload the table descriptors if > modified. However in clusters with a large number of tables or concurrent > clients and this can be too aggressive to HDFS and the master causing > contention to process other requests. A simple solution is to have a TTL > based cached for FSTableDescriptors#getAll() and > FSTableDescriptors#TableDescriptorAndModtime() that can allow the master to > process those calls faster without causing contention without having to > perform a trip to HDFS for every call. to listtables() or getTableDescriptor() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12282) Ensure Cells and its implementations work with Buffers also
[ https://issues.apache.org/jira/browse/HBASE-12282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188193#comment-14188193 ] ramkrishna.s.vasudevan commented on HBASE-12282: bq. Let it be a BB backed one. Yes. That is why in the attached patch for ease of use created buffer for even the fake keys that we create that includes KeyOnlyKV. In the read path both should be BB based only. > Ensure Cells and its implementations work with Buffers also > --- > > Key: HBASE-12282 > URL: https://issues.apache.org/jira/browse/HBASE-12282 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 0.99.1 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 0.99.2 > > Attachments: HBASE-12224_2.patch > > > This issue can be used to brainstorm and then do the necessary changes for > the offheap work. All impl of cells deal with byte[] but when we change the > Hfileblocks/Readers to work purely with Buffers then the byte[] usage would > mean that always the data is copied to the onheap. Cell may need some > interface change to implement this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-12219) Cache more efficiently getAll() and get() in FSTableDescriptors
[ https://issues.apache.org/jira/browse/HBASE-12219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi reassigned HBASE-12219: --- Assignee: Matteo Bertozzi (was: Esteban Gutierrez) > Cache more efficiently getAll() and get() in FSTableDescriptors > --- > > Key: HBASE-12219 > URL: https://issues.apache.org/jira/browse/HBASE-12219 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.94.24, 0.99.1, 0.98.6.1 >Reporter: Esteban Gutierrez >Assignee: Matteo Bertozzi > Labels: scalability > Attachments: HBASE-12219-v1.patch, HBASE-12219.v0.txt, list.png > > > Currently table descriptors and tables are cached once they are accessed for > the first time. Next calls to the master only require a trip to HDFS to > lookup the modified time in order to reload the table descriptors if > modified. However in clusters with a large number of tables or concurrent > clients and this can be too aggressive to HDFS and the master causing > contention to process other requests. A simple solution is to have a TTL > based cached for FSTableDescriptors#getAll() and > FSTableDescriptors#TableDescriptorAndModtime() that can allow the master to > process those calls faster without causing contention without having to > perform a trip to HDFS for every call. to listtables() or getTableDescriptor() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12219) Cache more efficiently getAll() and get() in FSTableDescriptors
[ https://issues.apache.org/jira/browse/HBASE-12219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-12219: Status: Patch Available (was: In Progress) > Cache more efficiently getAll() and get() in FSTableDescriptors > --- > > Key: HBASE-12219 > URL: https://issues.apache.org/jira/browse/HBASE-12219 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.98.6.1, 0.99.1, 0.94.24 >Reporter: Esteban Gutierrez >Assignee: Matteo Bertozzi > Labels: scalability > Attachments: HBASE-12219-v1.patch, HBASE-12219.v0.txt, list.png > > > Currently table descriptors and tables are cached once they are accessed for > the first time. Next calls to the master only require a trip to HDFS to > lookup the modified time in order to reload the table descriptors if > modified. However in clusters with a large number of tables or concurrent > clients and this can be too aggressive to HDFS and the master causing > contention to process other requests. A simple solution is to have a TTL > based cached for FSTableDescriptors#getAll() and > FSTableDescriptors#TableDescriptorAndModtime() that can allow the master to > process those calls faster without causing contention without having to > perform a trip to HDFS for every call. to listtables() or getTableDescriptor() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HBASE-12219) Cache more efficiently getAll() and get() in FSTableDescriptors
[ https://issues.apache.org/jira/browse/HBASE-12219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi reassigned HBASE-12219: --- Assignee: Esteban Gutierrez (was: Matteo Bertozzi) > Cache more efficiently getAll() and get() in FSTableDescriptors > --- > > Key: HBASE-12219 > URL: https://issues.apache.org/jira/browse/HBASE-12219 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.94.24, 0.99.1, 0.98.6.1 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez > Labels: scalability > Attachments: HBASE-12219-v1.patch, HBASE-12219.v0.txt, list.png > > > Currently table descriptors and tables are cached once they are accessed for > the first time. Next calls to the master only require a trip to HDFS to > lookup the modified time in order to reload the table descriptors if > modified. However in clusters with a large number of tables or concurrent > clients and this can be too aggressive to HDFS and the master causing > contention to process other requests. A simple solution is to have a TTL > based cached for FSTableDescriptors#getAll() and > FSTableDescriptors#TableDescriptorAndModtime() that can allow the master to > process those calls faster without causing contention without having to > perform a trip to HDFS for every call. to listtables() or getTableDescriptor() -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11419) After increasing TTL value of a hbase table having pre-split regions and decreasing TTL value, table becomes inaccessible.
[ https://issues.apache.org/jira/browse/HBASE-11419?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188197#comment-14188197 ] Prabhu Joseph commented on HBASE-11419: --- HFILE: on viewing metadata of the hfile , it throws error as dataBlockIndexReader is empty hbase hfile -m -f /hbase/AccountHistoryMA1/5640608f0ab19ee100ca71974acd5677/d/e8be48b383e1428698736f71f85b0049 Block index size as per heapsize: 336 Exception in thread "main" java.lang.NullPointerException at org.apache.hadoop.hbase.KeyValue.keyToString(KeyValue.java:716) at org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toStringFirstKey(AbstractHFileReader.java:138) at org.apache.hadoop.hbase.io.hfile.AbstractHFileReader.toString(AbstractHFileReader.java:149) at org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.printMeta(HFilePrettyPrinter.java:318) at org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.processFile(HFilePrettyPrinter.java:234) at org.apache.hadoop.hbase.io.hfile.HFilePrettyPrinter.run(HFilePrettyPrinter.java:189) at org.apache.hadoop.hbase.io.hfile.HFile.main(HFile.java:750) > After increasing TTL value of a hbase table having pre-split regions and > decreasing TTL value, table becomes inaccessible. > -- > > Key: HBASE-11419 > URL: https://issues.apache.org/jira/browse/HBASE-11419 > Project: HBase > Issue Type: Bug > Components: HFile >Affects Versions: 0.94.6 > Environment: Linux x86_64 >Reporter: Prabhu Joseph >Priority: Blocker > Attachments: HBaseExporter.java, account.csv, hbase-site.xml > > Original Estimate: 96h > Remaining Estimate: 96h > > After increasing and decreasing the TTL value of a Hbase Table , table gets > inaccessible. Scan table not working. > Scan in hbase shell throws > java.lang.IllegalStateException: Block index not loaded > at com.google.common.base.Preconditions.checkState(Preconditions.java:145) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV1.blockContainingKey(HFileReaderV1.java:181) > at > org.apache.hadoop.hbase.io.hfile.HFileReaderV1$AbstractScannerV1.seekTo(HFileReaderV1.java:426) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:226) > at > org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:145) > at > org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:131) > at org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:2015) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.(HRegion.java:3706) > at > org.apache.hadoop.hbase.regionserver.HRegion.instantiateRegionScanner(HRegion.java:1761) > at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1753) > at org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1730) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.openScanner(HRegionServer.java:2409) > at sun.reflect.GeneratedMethodAccessor56.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:320) > at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1426) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12358) Create ByteBuffer backed Cell
[ https://issues.apache.org/jira/browse/HBASE-12358?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188198#comment-14188198 ] ramkrishna.s.vasudevan commented on HBASE-12358: May be hasArray() would be the best option so that the exposed APIs are not changed. So users using the CP and filters should use the hasArray() to determine which one to use getXXxArray or getXXXBuffer. > Create ByteBuffer backed Cell > - > > Key: HBASE-12358 > URL: https://issues.apache.org/jira/browse/HBASE-12358 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 0.99.2 > > Attachments: HBASE-12358.patch > > > As part of HBASE-12224 and HBASE-12282 we wanted a Cell that is backed by BB. > Changing the core Cell impl would not be needed as it is used in server > only. So we will create a BB backed Cell and use it in the Server side read > path. This JIRA just creates an interface that extends Cell and adds the > needed API. > The getTimeStamp and getTypebyte() can still refer to the original Cell API > only. The getXXxOffset() and getXXXLength() can also refer to the original > Cell only. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12375) LoadIncrementalHFiles fails to load data in table when CF name starts with '_'
[ https://issues.apache.org/jira/browse/HBASE-12375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188200#comment-14188200 ] Hadoop QA commented on HBASE-12375: --- {color:green}+1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12677847/HBASE-12375.patch against trunk revision . ATTACHMENT ID: 12677847 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified tests. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 checkstyle{color}. The applied patch does not increase the total number of checkstyle errors {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/11505//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11505//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11505//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11505//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11505//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11505//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11505//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11505//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11505//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11505//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11505//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11505//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/11505//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11505//console This message is automatically generated. > LoadIncrementalHFiles fails to load data in table when CF name starts with '_' > -- > > Key: HBASE-12375 > URL: https://issues.apache.org/jira/browse/HBASE-12375 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.5 >Reporter: Ashish Singhi >Assignee: Ashish Singhi >Priority: Minor > Attachments: HBASE-12375.patch > > > We do not restrict user from creating a table having column family starting > with '_'. > So when user creates a table in such a way then LoadIncrementalHFiles will > skip those family data to load into the table. > {code} > // Skip _logs, etc > if (familyDir.getName().startsWith("_")) continue; > {code} > I think we should remove that check as I do not see any _logs directory being > created by the bulkload tool in the output directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11683) Metrics for MOB
[ https://issues.apache.org/jira/browse/HBASE-11683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jingcheng Du updated HBASE-11683: - Attachment: HBASE-11683-V7.diff > Metrics for MOB > --- > > Key: HBASE-11683 > URL: https://issues.apache.org/jira/browse/HBASE-11683 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jingcheng Du > Attachments: HBASE-11683-V2.diff, HBASE-11683-V3.diff, > HBASE-11683-V4.diff, HBASE-11683-V5.diff, HBASE-11683-V6.diff, > HBASE-11683-V7.diff, HBASE-11683.diff > > > We need to make sure to capture metrics about mobs. > Some basic ones include: > # of mob writes > # of mob reads > # avg size of mob (?) > # mob files > # of mob compactions / sweeps -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11683) Metrics for MOB
[ https://issues.apache.org/jira/browse/HBASE-11683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188210#comment-14188210 ] Jingcheng Du commented on HBASE-11683: -- Upload the latest patch V7. > Metrics for MOB > --- > > Key: HBASE-11683 > URL: https://issues.apache.org/jira/browse/HBASE-11683 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh >Assignee: Jingcheng Du > Attachments: HBASE-11683-V2.diff, HBASE-11683-V3.diff, > HBASE-11683-V4.diff, HBASE-11683-V5.diff, HBASE-11683-V6.diff, > HBASE-11683-V7.diff, HBASE-11683.diff > > > We need to make sure to capture metrics about mobs. > Some basic ones include: > # of mob writes > # of mob reads > # avg size of mob (?) > # mob files > # of mob compactions / sweeps -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11819) Unit test for CoprocessorHConnection
[ https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Talat UYARER updated HBASE-11819: - Attachment: HBASE-11819.patch Hi [~apurtell], I create a test with your instruction. Could you review my patch ? > Unit test for CoprocessorHConnection > - > > Key: HBASE-11819 > URL: https://issues.apache.org/jira/browse/HBASE-11819 > Project: HBase > Issue Type: Test >Reporter: Andrew Purtell >Assignee: Talat UYARER >Priority: Minor > Labels: newbie++ > Fix For: 2.0.0, 0.98.8, 0.99.2 > > Attachments: HBASE-11819.patch > > > Add a unit test to hbase-server that exercises CoprocessorHConnection . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12297) Support DBB usage in Bloom and HFileIndex area
[ https://issues.apache.org/jira/browse/HBASE-12297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188224#comment-14188224 ] Hudson commented on HBASE-12297: FAILURE: Integrated in HBase-1.0 #379 (See [https://builds.apache.org/job/HBase-1.0/379/]) HBASE-12297 Support DBB usage in Bloom and HFileIndex area. (anoop.s.john: rev e1d1ba564bf808278c38bcbbdc527b233b4cbfca) * hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/ChecksumUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/ByteBloomFilter.java * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDefaultDecodingContext.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/CompoundBloomFilter.java > Support DBB usage in Bloom and HFileIndex area > -- > > Key: HBASE-12297 > URL: https://issues.apache.org/jira/browse/HBASE-12297 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 0.99.2 > > Attachments: HBASE-12297.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12297) Support DBB usage in Bloom and HFileIndex area
[ https://issues.apache.org/jira/browse/HBASE-12297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188252#comment-14188252 ] Hudson commented on HBASE-12297: SUCCESS: Integrated in HBase-TRUNK #5715 (See [https://builds.apache.org/job/HBase-TRUNK/5715/]) HBASE-12297 Support DBB usage in Bloom and HFileIndex area. (anoop.s.john: rev cbb334035d87542ed06693bb9c8534f64360672b) * hbase-common/src/main/java/org/apache/hadoop/hbase/io/encoding/HFileBlockDefaultDecodingContext.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/CompoundBloomFilter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/ChecksumUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/util/ByteBloomFilter.java * hbase-common/src/main/java/org/apache/hadoop/hbase/util/ByteBufferUtils.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java > Support DBB usage in Bloom and HFileIndex area > -- > > Key: HBASE-12297 > URL: https://issues.apache.org/jira/browse/HBASE-12297 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 2.0.0, 0.99.2 > > Attachments: HBASE-12297.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12219) Cache more efficiently getAll() and get() in FSTableDescriptors
[ https://issues.apache.org/jira/browse/HBASE-12219?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188259#comment-14188259 ] Hadoop QA commented on HBASE-12219: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12677859/HBASE-12219-v1.patch against trunk revision . ATTACHMENT ID: 12677859 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 8 new or modified tests. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:red}-1 checkstyle{color}. The applied patch generated 3792 checkstyle errors (more than the trunk's current 3790 errors). {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 lineLengths{color}. The patch introduces the following lines longer than 100: + tds.put(this.metaTableDescritor.getNameAsString(), new TableDescriptor(metaTableDescritor, TableState.State.ENABLED)); + public static TableDescriptor getTableDescriptorFromFs(FileSystem fs, Path tableDir, boolean rewritePb) +FSTableDescriptors htds = new FSTableDescriptorsTest(UTIL.getConfiguration(), fs, rootdir, false, false); +FSTableDescriptors nonchtds = new FSTableDescriptorsTest(UTIL.getConfiguration(), fs, rootdir, false, false); {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestRegionServerNoMaster Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/11506//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11506//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11506//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11506//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11506//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11506//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11506//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11506//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11506//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11506//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11506//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11506//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/11506//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11506//console This message is automatically generated. > Cache more efficiently getAll() and get() in FSTableDescriptors > --- > > Key: HBASE-12219 > URL: https://issues.apache.org/jira/browse/HBASE-12219 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.94.24, 0.99.1, 0.98.6.1 >Reporter: Esteban Gutierrez >Assignee: Esteban Gutierrez > Labels: scalability > Attachments: HBASE-12219-v1.patch, HBASE-12219.v0.txt, list.png > > > Currently table descriptors and tables are cached once they are accessed for > the first time. Next calls to the master only require a trip to HDFS to > lookup the modified time in order to reload the table descriptors if > modified. However in clusters with a large number of tables or conc
[jira] [Commented] (HBASE-12282) Ensure Cells and its implementations work with Buffers also
[ https://issues.apache.org/jira/browse/HBASE-12282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188279#comment-14188279 ] ramkrishna.s.vasudevan commented on HBASE-12282: Forgot to add one more point here. When we want to compare a KV in Memstore and a fake key backed by BB or with a key from a store file (that is backed by BB), we need to compare using both types of KV. I had raised a subtask for changing the memstore to BB. Otherwise we have to have both types of comparison as done in the patch attached here. > Ensure Cells and its implementations work with Buffers also > --- > > Key: HBASE-12282 > URL: https://issues.apache.org/jira/browse/HBASE-12282 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 0.99.1 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 0.99.2 > > Attachments: HBASE-12224_2.patch > > > This issue can be used to brainstorm and then do the necessary changes for > the offheap work. All impl of cells deal with byte[] but when we change the > Hfileblocks/Readers to work purely with Buffers then the byte[] usage would > mean that always the data is copied to the onheap. Cell may need some > interface change to implement this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12282) Ensure Cells and its implementations work with Buffers also
[ https://issues.apache.org/jira/browse/HBASE-12282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188288#comment-14188288 ] ramkrishna.s.vasudevan commented on HBASE-12282: {code} +if (left.getQualifierArray() != null && right.getQualifierArray() != null) { + return Bytes.compareTo(left.getQualifierArray(), left.getQualifierOffset(), + left.getQualifierLength(), right.getQualifierArray(), right.getQualifierOffset(), + right.getQualifierLength()); +} else if (((left.getQualifierArray() != null && left.getQualifierBuffer() == null) && right +.getQualifierBuffer() != null)) { + return ByteBufferUtils.compareTo(left.getQualifierArray(), left.getQualifierOffset(), + left.getQualifierLength(), right.getQualifierBuffer(), right.getQualifierOffset(), + right.getQualifierLength()); +} else if (left.getQualifierBuffer() != null +&& (right.getQualifierBuffer() == null && right.getQualifierArray() != null)) { + return ByteBufferUtils.compareTo(left.getQualifierBuffer(), left.getQualifierOffset(), + left.getQualifierLength(), right.getQualifierArray(), right.getQualifierOffset(), + right.getQualifierLength()); +} else { + return ByteBufferUtils.compareTo(left.getQualifierBuffer(), left.getQualifierOffset(), + left.getQualifierLength(), right.getQualifierBuffer(), right.getQualifierOffset(), + right.getQualifierLength()); +} {code} So as done here we may have to use hasArray on both the left Cell and rightCell and do the comparison. > Ensure Cells and its implementations work with Buffers also > --- > > Key: HBASE-12282 > URL: https://issues.apache.org/jira/browse/HBASE-12282 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 0.99.1 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 0.99.2 > > Attachments: HBASE-12224_2.patch > > > This issue can be used to brainstorm and then do the necessary changes for > the offheap work. All impl of cells deal with byte[] but when we change the > Hfileblocks/Readers to work purely with Buffers then the byte[] usage would > mean that always the data is copied to the onheap. Cell may need some > interface change to implement this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12354) Update dependencies in time for 1.0 release
[ https://issues.apache.org/jira/browse/HBASE-12354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188316#comment-14188316 ] Nicolas Liochon commented on HBASE-12354: - +1, there is a +1 from Enis above as well. > Update dependencies in time for 1.0 release > --- > > Key: HBASE-12354 > URL: https://issues.apache.org/jira/browse/HBASE-12354 > Project: HBase > Issue Type: Sub-task > Components: dependencies >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 0.99.2 > > Attachments: 12354.txt, 12354v2.txt > > > Going through and updating egregiously old dependencies for 1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12375) LoadIncrementalHFiles fails to load data in table when CF name starts with '_'
[ https://issues.apache.org/jira/browse/HBASE-12375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188371#comment-14188371 ] Ashish Singhi commented on HBASE-12375: --- Thanks Matteo, for looking into the patch. bq. I think that by removing the check you'll fail in case of splits (but I haven't checked) No, it did not fail I tested manually the below scenario, 1. Create a table initially with 3 splits 2. Run bulkload 3. split the table into one more region 4. put some data into the table for that region 5. Run completebulkload bq. the problem is that the LoadIncrementalHFiles will create a "_tmp" directory. Yes, it does create the "_tmp" directory but inside the CF directory so it will not create any problem. We can see that from the logs generated after running the above mentioned scenario, {noformat} 2014-10-29 20:03:40,172 INFO [LoadIncrementalHFiles-0] mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://10.18.40.106:9000/s4/_d/_tmp/af37ac06db0f4a8ebe9ccd848d5864b7.top first=90 last=90 2014-10-29 20:03:40,172 INFO [LoadIncrementalHFiles-3] mapreduce.LoadIncrementalHFiles: Trying to load hfile=hdfs://10.18.40.106:9000/s4/_d/_tmp/af37ac06db0f4a8ebe9ccd848d5864b7.bottom first=5 last=67 {noformat} bq. In theory is enough adding to this patch the rename of "_tmp" to something like .tmp Do you still want me to do this ? bq. did you tried to run this patch with a set of files that requires splitting? Yes, as mentioned above > LoadIncrementalHFiles fails to load data in table when CF name starts with '_' > -- > > Key: HBASE-12375 > URL: https://issues.apache.org/jira/browse/HBASE-12375 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.5 >Reporter: Ashish Singhi >Assignee: Ashish Singhi >Priority: Minor > Attachments: HBASE-12375.patch > > > We do not restrict user from creating a table having column family starting > with '_'. > So when user creates a table in such a way then LoadIncrementalHFiles will > skip those family data to load into the table. > {code} > // Skip _logs, etc > if (familyDir.getName().startsWith("_")) continue; > {code} > I think we should remove that check as I do not see any _logs directory being > created by the bulkload tool in the output directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-10780) HFilePrettyPrinter#processFile should return immediately if file does not exists.
[ https://issues.apache.org/jira/browse/HBASE-10780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-10780: -- Status: Open (was: Patch Available) > HFilePrettyPrinter#processFile should return immediately if file does not > exists. > - > > Key: HBASE-10780 > URL: https://issues.apache.org/jira/browse/HBASE-10780 > Project: HBase > Issue Type: Bug > Components: HFile >Affects Versions: 0.94.11 >Reporter: Ashish Singhi >Assignee: Ashish Singhi >Priority: Minor > Attachments: HBASE-10780-v2.patch, HBASE-10780-v2.patch, > HBASE-10780.patch > > > HFilePrettyPrinter#processFile should return immediately if file does not > exists same like HLogPrettyPrinter#run > {code} > if (!fs.exists(file)) { > System.err.println("ERROR, file doesnt exist: " + file); > }{code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-10780) HFilePrettyPrinter#processFile should return immediately if file does not exists.
[ https://issues.apache.org/jira/browse/HBASE-10780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-10780: -- Attachment: HBASE-10780-v3.patch Reattaching the same patch to see if any actual test failure due to this patch. > HFilePrettyPrinter#processFile should return immediately if file does not > exists. > - > > Key: HBASE-10780 > URL: https://issues.apache.org/jira/browse/HBASE-10780 > Project: HBase > Issue Type: Bug > Components: HFile >Affects Versions: 0.94.11, 0.98.5 >Reporter: Ashish Singhi >Assignee: Ashish Singhi >Priority: Minor > Attachments: HBASE-10780-v2.patch, HBASE-10780-v2.patch, > HBASE-10780-v3.patch, HBASE-10780.patch > > > HFilePrettyPrinter#processFile should return immediately if file does not > exists same like HLogPrettyPrinter#run > {code} > if (!fs.exists(file)) { > System.err.println("ERROR, file doesnt exist: " + file); > }{code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-10780) HFilePrettyPrinter#processFile should return immediately if file does not exists.
[ https://issues.apache.org/jira/browse/HBASE-10780?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ashish Singhi updated HBASE-10780: -- Affects Version/s: 0.98.5 Status: Patch Available (was: Open) > HFilePrettyPrinter#processFile should return immediately if file does not > exists. > - > > Key: HBASE-10780 > URL: https://issues.apache.org/jira/browse/HBASE-10780 > Project: HBase > Issue Type: Bug > Components: HFile >Affects Versions: 0.98.5, 0.94.11 >Reporter: Ashish Singhi >Assignee: Ashish Singhi >Priority: Minor > Attachments: HBASE-10780-v2.patch, HBASE-10780-v2.patch, > HBASE-10780-v3.patch, HBASE-10780.patch > > > HFilePrettyPrinter#processFile should return immediately if file does not > exists same like HLogPrettyPrinter#run > {code} > if (!fs.exists(file)) { > System.err.println("ERROR, file doesnt exist: " + file); > }{code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12375) LoadIncrementalHFiles fails to load data in table when CF name starts with '_'
[ https://issues.apache.org/jira/browse/HBASE-12375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188401#comment-14188401 ] Matteo Bertozzi commented on HBASE-12375: - cool, thanks for verifying that. +1 on the patch If you want to do that, you can also change the _tmp stuff and get rid of all the if _ stuff otherwise the patch is already good enough for me. {code} protected List splitStoreFile(final LoadQueueItem item, ... // We use a '_' prefix which is ignored when walking directory trees above. final Path tmpDir = new Path(item.hfilePath.getParent(), "_tmp"); {code} > LoadIncrementalHFiles fails to load data in table when CF name starts with '_' > -- > > Key: HBASE-12375 > URL: https://issues.apache.org/jira/browse/HBASE-12375 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.5 >Reporter: Ashish Singhi >Assignee: Ashish Singhi >Priority: Minor > Attachments: HBASE-12375.patch > > > We do not restrict user from creating a table having column family starting > with '_'. > So when user creates a table in such a way then LoadIncrementalHFiles will > skip those family data to load into the table. > {code} > // Skip _logs, etc > if (familyDir.getName().startsWith("_")) continue; > {code} > I think we should remove that check as I do not see any _logs directory being > created by the bulkload tool in the output directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12279) Generated thrift files were generated with the wrong parameters
[ https://issues.apache.org/jira/browse/HBASE-12279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188422#comment-14188422 ] Niels Basjes commented on HBASE-12279: -- The patch I created for HBASE-12272 includes an explicit check that the configured version is actually used. So as long as the pom.xml says "0.9.0" trying it with 0.9.1 will fail, which is exactly the check we need here. As far as I can tell the HBASE-12272 patch should also work quite nicely on the older HBase where thrift 0.8.0 is still needed. > Generated thrift files were generated with the wrong parameters > --- > > Key: HBASE-12279 > URL: https://issues.apache.org/jira/browse/HBASE-12279 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.0, 0.98.0, 0.99.0 >Reporter: Niels Basjes > Fix For: 2.0.0, 0.98.8, 0.94.25, 0.99.2 > > Attachments: HBASE-12279-2014-10-16-v1.patch > > > It turns out that the java code generated from the thrift files have been > generated with the wrong settings. > Instead of the documented > ([thrift|http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/thrift/package-summary.html], > > [thrift2|http://hbase.apache.org/devapidocs/org/apache/hadoop/hbase/thrift2/package-summary.html]) > > {code} > thrift -strict --gen java:hashcode > {code} > the current files seem to be generated instead with > {code} > thrift -strict --gen java > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10780) HFilePrettyPrinter#processFile should return immediately if file does not exists.
[ https://issues.apache.org/jira/browse/HBASE-10780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188459#comment-14188459 ] Ted Yu commented on HBASE-10780: +1 > HFilePrettyPrinter#processFile should return immediately if file does not > exists. > - > > Key: HBASE-10780 > URL: https://issues.apache.org/jira/browse/HBASE-10780 > Project: HBase > Issue Type: Bug > Components: HFile >Affects Versions: 0.94.11, 0.98.5 >Reporter: Ashish Singhi >Assignee: Ashish Singhi >Priority: Minor > Attachments: HBASE-10780-v2.patch, HBASE-10780-v2.patch, > HBASE-10780-v3.patch, HBASE-10780.patch > > > HFilePrettyPrinter#processFile should return immediately if file does not > exists same like HLogPrettyPrinter#run > {code} > if (!fs.exists(file)) { > System.err.println("ERROR, file doesnt exist: " + file); > }{code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12282) Ensure Cells and its implementations work with Buffers also
[ https://issues.apache.org/jira/browse/HBASE-12282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188501#comment-14188501 ] Anoop Sam John commented on HBASE-12282: +1 for hasArray() The impl of BB backed Cell will throw UnsupportedOpException when calling getxxxArray() right? > Ensure Cells and its implementations work with Buffers also > --- > > Key: HBASE-12282 > URL: https://issues.apache.org/jira/browse/HBASE-12282 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 0.99.1 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 2.0.0, 0.99.2 > > Attachments: HBASE-12224_2.patch > > > This issue can be used to brainstorm and then do the necessary changes for > the offheap work. All impl of cells deal with byte[] but when we change the > Hfileblocks/Readers to work purely with Buffers then the byte[] usage would > mean that always the data is copied to the onheap. Cell may need some > interface change to implement this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-10780) HFilePrettyPrinter#processFile should return immediately if file does not exists.
[ https://issues.apache.org/jira/browse/HBASE-10780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188508#comment-14188508 ] Hadoop QA commented on HBASE-10780: --- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12677891/HBASE-10780-v3.patch against trunk revision . ATTACHMENT ID: 12677891 {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:red}-1 tests included{color}. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages. {color:green}+1 checkstyle{color}. The applied patch does not increase the total number of checkstyle errors {color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 2.0.3) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 lineLengths{color}. The patch does not introduce lines longer than 100 {color:green}+1 site{color}. The mvn site goal succeeds with this patch. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/11507//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11507//artifact/patchprocess/newPatchFindbugsWarningshbase-rest.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11507//artifact/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11507//artifact/patchprocess/newPatchFindbugsWarningshbase-client.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11507//artifact/patchprocess/newPatchFindbugsWarningshbase-annotations.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11507//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11507//artifact/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11507//artifact/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11507//artifact/patchprocess/newPatchFindbugsWarningshbase-protocol.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11507//artifact/patchprocess/newPatchFindbugsWarningshbase-thrift.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11507//artifact/patchprocess/newPatchFindbugsWarningshbase-examples.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/11507//artifact/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Checkstyle Errors: https://builds.apache.org/job/PreCommit-HBASE-Build/11507//artifact/patchprocess/checkstyle-aggregate.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/11507//console This message is automatically generated. > HFilePrettyPrinter#processFile should return immediately if file does not > exists. > - > > Key: HBASE-10780 > URL: https://issues.apache.org/jira/browse/HBASE-10780 > Project: HBase > Issue Type: Bug > Components: HFile >Affects Versions: 0.94.11, 0.98.5 >Reporter: Ashish Singhi >Assignee: Ashish Singhi >Priority: Minor > Attachments: HBASE-10780-v2.patch, HBASE-10780-v2.patch, > HBASE-10780-v3.patch, HBASE-10780.patch > > > HFilePrettyPrinter#processFile should return immediately if file does not > exists same like HLogPrettyPrinter#run > {code} > if (!fs.exists(file)) { > System.err.println("ERROR, file doesnt exist: " + file); > }{code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12324) Improve compaction speed and process for immutable short lived datasets
[ https://issues.apache.org/jira/browse/HBASE-12324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188518#comment-14188518 ] Sheetal Dolas commented on HBASE-12324: --- So to consolidate all inputs and comments, here is what I see * All compactions (including striped compactions) already have logic to delete TTL expired files when 'hbase.store.delete.expired.storefile' is set to 'true' (Reference: HBASE-5199 and HBASE-10141 ) * Archiving old files is based comparison between file timestamp and server time stamp. HBASE-5199 can be further improved to store and check latest TTL of any field in the file trailer and use that for comparison instead of file timestamp. This could be independent improvement from this thread. * Proposed OnlyDeleteExpiredFilesCompactionFiles has its own use case where use does not want to compact at all and just delete old data. * Cases where some compassion is needed (to avoid too many HFiles) can be addressed by striped compaction policy (as it already has smarter logic for deciding which files to compact as well as at already deletes TTL expired files before compaction.) * Declaring table/cf "immutable" - to make smarter decisions: This probably needs more exploration. * New question now aires is shall there be multiple compaction policies (only delete expired files, striped, immutable etc ) or should is all be consolidated under striped compaction with configuration parameters to enable/disable certain behavior. > Improve compaction speed and process for immutable short lived datasets > --- > > Key: HBASE-12324 > URL: https://issues.apache.org/jira/browse/HBASE-12324 > Project: HBase > Issue Type: New Feature > Components: Compaction >Affects Versions: 0.98.0, 0.96.0 >Reporter: Sheetal Dolas > Attachments: OnlyDeleteExpiredFilesCompactionPolicy.java > > > We have seen multiple cases where HBase is used to store immutable data and > the data lives for short period of time (few days) > On very high volume systems, major compactions become very costly and > slowdown ingestion rates. > In all such use cases (immutable data, high write rate and moderate read > rates and shorter ttl), avoiding any compactions and just deleting old data > brings lot of performance benefits. > We should have a compaction policy that can only delete/archive files older > than TTL and not compact any files. > Also attaching a patch that can do so. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-11861) Native MOB Compaction mechanisms.
[ https://issues.apache.org/jira/browse/HBASE-11861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-11861: --- Attachment: 141030-mob-compaction.pdf Attached is a pictorial design of the proposed core mob compaction mechanism. > Native MOB Compaction mechanisms. > - > > Key: HBASE-11861 > URL: https://issues.apache.org/jira/browse/HBASE-11861 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Affects Versions: 2.0.0 >Reporter: Jonathan Hsieh > Attachments: 141030-mob-compaction.pdf > > > Currently, the first cut of mob will have external processes to age off old > mob data (the ttl cleaner), and to compact away deleted or over written data > (the sweep tool). > From an operational point of view, having two external tools, especially one > that relies on MapReduce is undesirable. In this issue we'll tackle > integrating these into hbase without requiring external processes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12346) Scan's default auths behavior under Visibility labels
[ https://issues.apache.org/jira/browse/HBASE-12346?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188540#comment-14188540 ] Jerry He commented on HBASE-12346: -- [~anoop.hbase] explained well. Thanks, guys. > Scan's default auths behavior under Visibility labels > - > > Key: HBASE-12346 > URL: https://issues.apache.org/jira/browse/HBASE-12346 > Project: HBase > Issue Type: Bug > Components: API, security >Affects Versions: 0.98.7, 0.99.1 >Reporter: Jerry He > Fix For: 0.98.8, 0.99.2 > > Attachments: HBASE-12346-master-v2.patch, HBASE-12346-master.patch > > > In Visibility Labels security, a set of labels (auths) are administered and > associated with a user. > A user can normally only see cell data during scan that are part of the > user's label set (auths). > Scan uses setAuthorizations to indicates its wants to use the auths to access > the cells. > Similarly in the shell: > {code} > scan 'table1', AUTHORIZATIONS => ['private'] > {code} > But it is a surprise to find that setAuthorizations seems to be 'mandatory' > in the default visibility label security setting. Every scan needs to > setAuthorizations before the scan can get any cells even the cells are under > the labels the request user is part of. > The following steps will illustrate the issue: > Run as superuser. > {code} > 1. create a visibility label called 'private' > 2. create 'table1' > 3. put into 'table1' data and label the data as 'private' > 4. set_auths 'user1', 'private' > 5. grant 'user1', 'RW', 'table1' > {code} > Run as 'user1': > {code} > 1. scan 'table1' > This show no cells. > 2. scan 'table1', scan 'table1', AUTHORIZATIONS => ['private'] > This will show all the data. > {code} > I am not sure if this is expected by design or a bug. > But a more reasonable, more client application backward compatible, and less > surprising default behavior should probably look like this: > A scan's default auths, if its Authorizations attributes is not set > explicitly, should be all the auths the request user is administered and > allowed on the server. > If scan.setAuthorizations is used, then the server further filter the auths > during scan: use the input auths minus what is not in user's label set on the > server. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12373) Provide a command to list visibility labels
[ https://issues.apache.org/jira/browse/HBASE-12373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188548#comment-14188548 ] Jerry He commented on HBASE-12373: -- bq. It should be allowed only for a user who is having system label auth Yes. I will have a patch as soon as I can. Thanks! > Provide a command to list visibility labels > --- > > Key: HBASE-12373 > URL: https://issues.apache.org/jira/browse/HBASE-12373 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.98.7, 0.99.1 >Reporter: Jerry He >Priority: Minor > > A command to list visibility labels that are in place would be handy. > This is also in line with many of the other hbase list commands. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12313) Redo the hfile index length optimization so cell-based rather than serialized KV key
[ https://issues.apache.org/jira/browse/HBASE-12313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188657#comment-14188657 ] Anoop Sam John commented on HBASE-12313: bq.Put back old behavior. +1 for latest patch. Hope you can fix line lengths >100 on commit Thanks Stack > Redo the hfile index length optimization so cell-based rather than serialized > KV key > > > Key: HBASE-12313 > URL: https://issues.apache.org/jira/browse/HBASE-12313 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: stack >Assignee: stack > Attachments: > 0001-HBASE-12313-Redo-the-hfile-index-length-optimization.patch, > 0001-HBASE-12313-Redo-the-hfile-index-length-optimization.patch, > 0001-HBASE-12313-Redo-the-hfile-index-length-optimization.patch, > 0001-HBASE-12313-Redo-the-hfile-index-length-optimization.patch, > 0001-HBASE-12313-Redo-the-hfile-index-length-optimization.patch, > 12313v10.txt, 12313v5.txt, 12313v6.txt, 12313v8.txt > > > Trying to remove API that returns the 'key' of a KV serialized into a byte > array is thorny. > I tried to move over the first and last key serializations and the hfile > index entries to be cell but patch was turning massive. Here is a smaller > patch that just redoes the optimization that tries to find 'short' midpoints > between last key of last block and first key of next block so it is > Cell-based rather than byte array based (presuming Keys serialized in a > certain way). Adds unit tests which we didn't have before. > Also remove CellKey. Not needed... at least not yet. Its just utility for > toString. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12285) Builds are failing, possibly because of SUREFIRE-1091
[ https://issues.apache.org/jira/browse/HBASE-12285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188697#comment-14188697 ] stack commented on HBASE-12285: --- I set branch-1 and master back to DEBUG level logging (master was on INFO, branch-1 was on WARN-only). branch-1 has been blue overnight with a legit failure on occasion. Thought is that the HBASE-12353 which got rid of a bunch of spew fixes the branch-1 issue. Lets see. > Builds are failing, possibly because of SUREFIRE-1091 > - > > Key: HBASE-12285 > URL: https://issues.apache.org/jira/browse/HBASE-12285 > Project: HBase > Issue Type: Bug >Affects Versions: 1.0.0 >Reporter: Dima Spivak >Assignee: Dima Spivak >Priority: Blocker > Attachments: HBASE-12285_branch-1_v1.patch, > HBASE-12285_branch-1_v1.patch > > > Our branch-1 builds on builds.apache.org have been failing in recent days > after we switched over to an official version of Surefire a few days back > (HBASE-4955). The version we're using, 2.17, is hit by a bug > ([SUREFIRE-1091|https://jira.codehaus.org/browse/SUREFIRE-1091]) that results > in an IOException, which looks like what we're seeing on Jenkins. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-12293) Tests are logging too much
[ https://issues.apache.org/jira/browse/HBASE-12293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-12293. --- Resolution: Duplicate Resolving as dup of parent issue. Thats where the action is. > Tests are logging too much > -- > > Key: HBASE-12293 > URL: https://issues.apache.org/jira/browse/HBASE-12293 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Dima Spivak >Assignee: Dima Spivak >Priority: Minor > > In trying to solve HBASE-12285, it was pointed out that tests are writing too > much to output again. At best, this is a sloppy practice and, at worst, it > leaves us open to builds breaking when our test tools can't handle the flood. > If [~nkeywal] would be willing give me a little bit of mentoring on how he > dealt with this problem a few years back, I'd be happy to add it to my plate. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12354) Update dependencies in time for 1.0 release
[ https://issues.apache.org/jira/browse/HBASE-12354?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-12354: -- Resolution: Fixed Release Note: Updated dependencies. Of note, went from hadoop 2.2 to 2.5.1. Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks [~nkeywal] and [~enis] (missed this one). Pushed to branch-1+ > Update dependencies in time for 1.0 release > --- > > Key: HBASE-12354 > URL: https://issues.apache.org/jira/browse/HBASE-12354 > Project: HBase > Issue Type: Sub-task > Components: dependencies >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 0.99.2 > > Attachments: 12354.txt, 12354v2.txt > > > Going through and updating egregiously old dependencies for 1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12313) Redo the hfile index length optimization so cell-based rather than serialized KV key
[ https://issues.apache.org/jira/browse/HBASE-12313?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-12313: -- Resolution: Fixed Fix Version/s: 0.99.2 2.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Pushed to branch-1+. Thanks for reviews mighty [~anoopsamjohn] and [~ram_krish] I fixed long lines on commit. > Redo the hfile index length optimization so cell-based rather than serialized > KV key > > > Key: HBASE-12313 > URL: https://issues.apache.org/jira/browse/HBASE-12313 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 0.99.2 > > Attachments: > 0001-HBASE-12313-Redo-the-hfile-index-length-optimization.patch, > 0001-HBASE-12313-Redo-the-hfile-index-length-optimization.patch, > 0001-HBASE-12313-Redo-the-hfile-index-length-optimization.patch, > 0001-HBASE-12313-Redo-the-hfile-index-length-optimization.patch, > 0001-HBASE-12313-Redo-the-hfile-index-length-optimization.patch, > 12313v10.txt, 12313v5.txt, 12313v6.txt, 12313v8.txt > > > Trying to remove API that returns the 'key' of a KV serialized into a byte > array is thorny. > I tried to move over the first and last key serializations and the hfile > index entries to be cell but patch was turning massive. Here is a smaller > patch that just redoes the optimization that tries to find 'short' midpoints > between last key of last block and first key of next block so it is > Cell-based rather than byte array based (presuming Keys serialized in a > certain way). Adds unit tests which we didn't have before. > Also remove CellKey. Not needed... at least not yet. Its just utility for > toString. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-12372) [WINDOWS] Enable log4j configuration in hbase.cmd
[ https://issues.apache.org/jira/browse/HBASE-12372?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar resolved HBASE-12372. --- Resolution: Fixed Pushed to all 0.98+. > [WINDOWS] Enable log4j configuration in hbase.cmd > -- > > Key: HBASE-12372 > URL: https://issues.apache.org/jira/browse/HBASE-12372 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar >Priority: Trivial > Fix For: 2.0.0, 0.98.8, 0.99.2 > > Attachments: hbase-12372.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12285) Builds are failing, possibly because of SUREFIRE-1091
[ https://issues.apache.org/jira/browse/HBASE-12285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188825#comment-14188825 ] Hudson commented on HBASE-12285: SUCCESS: Integrated in HBase-TRUNK #5716 (See [https://builds.apache.org/job/HBase-TRUNK/5716/]) HBASE-12285 Builds are failing, possibly because of SUREFIRE-1091 ; ADDENDUM SETTING LOG LEVEL TO DEBUG AGAIN (stack: rev b240b00f4f751944869511756a3b739040769acb) * hbase-server/src/test/resources/log4j.properties > Builds are failing, possibly because of SUREFIRE-1091 > - > > Key: HBASE-12285 > URL: https://issues.apache.org/jira/browse/HBASE-12285 > Project: HBase > Issue Type: Bug >Affects Versions: 1.0.0 >Reporter: Dima Spivak >Assignee: Dima Spivak >Priority: Blocker > Attachments: HBASE-12285_branch-1_v1.patch, > HBASE-12285_branch-1_v1.patch > > > Our branch-1 builds on builds.apache.org have been failing in recent days > after we switched over to an official version of Surefire a few days back > (HBASE-4955). The version we're using, 2.17, is hit by a bug > ([SUREFIRE-1091|https://jira.codehaus.org/browse/SUREFIRE-1091]) that results > in an IOException, which looks like what we're seeing on Jenkins. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12376) HBaseAdmin leaks ZK connections if failure starting watchers (ConnectionLossException)
stack created HBASE-12376: - Summary: HBaseAdmin leaks ZK connections if failure starting watchers (ConnectionLossException) Key: HBASE-12376 URL: https://issues.apache.org/jira/browse/HBASE-12376 Project: HBase Issue Type: Bug Components: Zookeeper Affects Versions: 0.94.24, 0.98.7 Reporter: stack Assignee: stack Priority: Critical This is a 0.98 issue that some users have been running into mostly running Canary and for whatever reason, setup of zk connection fails, usually with a ConnectionLossException. End result is ugly leak zk connections. ZKWatcher created instances are just left hang out. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12376) HBaseAdmin leaks ZK connections if failure starting watchers (ConnectionLossException)
[ https://issues.apache.org/jira/browse/HBASE-12376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-12376: -- Attachment: 0001-12376-HBaseAdmin-leaks-ZK-connections-if-failure-sta.patch Patch for 0.98. Don't have issue in master/branch-1. Opened up CatalogTracker some so I could inject failure for a unit test. Reviews please. Locally 0.98 TestAdmin passes. > HBaseAdmin leaks ZK connections if failure starting watchers > (ConnectionLossException) > -- > > Key: HBASE-12376 > URL: https://issues.apache.org/jira/browse/HBASE-12376 > Project: HBase > Issue Type: Bug > Components: Zookeeper >Affects Versions: 0.98.7, 0.94.24 >Reporter: stack >Assignee: stack >Priority: Critical > Attachments: > 0001-12376-HBaseAdmin-leaks-ZK-connections-if-failure-sta.patch > > > This is a 0.98 issue that some users have been running into mostly running > Canary and for whatever reason, setup of zk connection fails, usually with a > ConnectionLossException. End result is ugly leak zk connections. ZKWatcher > created instances are just left hang out. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12376) HBaseAdmin leaks ZK connections if failure starting watchers (ConnectionLossException)
[ https://issues.apache.org/jira/browse/HBASE-12376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188870#comment-14188870 ] stack commented on HBASE-12376: --- There is no CatalogTracker in branch-1+ > HBaseAdmin leaks ZK connections if failure starting watchers > (ConnectionLossException) > -- > > Key: HBASE-12376 > URL: https://issues.apache.org/jira/browse/HBASE-12376 > Project: HBase > Issue Type: Bug > Components: Zookeeper >Affects Versions: 0.98.7, 0.94.24 >Reporter: stack >Assignee: stack >Priority: Critical > Attachments: > 0001-12376-HBaseAdmin-leaks-ZK-connections-if-failure-sta.patch > > > This is a 0.98 issue that some users have been running into mostly running > Canary and for whatever reason, setup of zk connection fails, usually with a > ConnectionLossException. End result is ugly leak zk connections. ZKWatcher > created instances are just left hang out. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-11819) Unit test for CoprocessorHConnection
[ https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188894#comment-14188894 ] stack commented on HBASE-11819: --- You create an admin. In future, close it inside a try/finally rather than just when you are done. +Admin admin = util.getHBaseAdmin(); +admin.createTable(htd, new byte[][] { rowSeperator1, rowSeperator2 }); +util.waitUntilAllRegionsAssigned(testTable); +admin.close(); Instead do... +Admin admin = util.getHBaseAdmin(); try { +admin.createTable(htd, new byte[][] { rowSeperator1, rowSeperator2 }); +util.waitUntilAllRegionsAssigned(testTable); } finally { admin.close... Just FYI. Same with HTable. See how the way you are making HTable is now deprecated? Avoid doing it the way you have it if you can. Otherwise patch looks good to me. Good for you [~apurtell]? > Unit test for CoprocessorHConnection > - > > Key: HBASE-11819 > URL: https://issues.apache.org/jira/browse/HBASE-11819 > Project: HBase > Issue Type: Test >Reporter: Andrew Purtell >Assignee: Talat UYARER >Priority: Minor > Labels: newbie++ > Fix For: 2.0.0, 0.98.8, 0.99.2 > > Attachments: HBASE-11819.patch > > > Add a unit test to hbase-server that exercises CoprocessorHConnection . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12285) Builds are failing, possibly because of SUREFIRE-1091
[ https://issues.apache.org/jira/browse/HBASE-12285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188893#comment-14188893 ] Dima Spivak commented on HBASE-12285: - A build of HBase-1.0 [just passed|https://builds.apache.org/job/HBase-1.0/380] (log level on branch-1 is back at DEBUG and the log-spew reduction from HBASE-12293 was present), so I think this is probably safe to resolve as fixed. What do you think, [~stack]? Since this is no longer blocking builds, we may want to hold off updating Surefire until 2.18 is officially released. > Builds are failing, possibly because of SUREFIRE-1091 > - > > Key: HBASE-12285 > URL: https://issues.apache.org/jira/browse/HBASE-12285 > Project: HBase > Issue Type: Bug >Affects Versions: 1.0.0 >Reporter: Dima Spivak >Assignee: Dima Spivak >Priority: Blocker > Attachments: HBASE-12285_branch-1_v1.patch, > HBASE-12285_branch-1_v1.patch > > > Our branch-1 builds on builds.apache.org have been failing in recent days > after we switched over to an official version of Surefire a few days back > (HBASE-4955). The version we're using, 2.17, is hit by a bug > ([SUREFIRE-1091|https://jira.codehaus.org/browse/SUREFIRE-1091]) that results > in an IOException, which looks like what we're seeing on Jenkins. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12354) Update dependencies in time for 1.0 release
[ https://issues.apache.org/jira/browse/HBASE-12354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188898#comment-14188898 ] Hudson commented on HBASE-12354: SUCCESS: Integrated in HBase-1.0 #380 (See [https://builds.apache.org/job/HBase-1.0/380/]) HBASE-12354 Update dependencies in time for 1.0 release (stack: rev 7aed6de9c84072df7cf4ce3cff43de76b65c48a1) * pom.xml * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java > Update dependencies in time for 1.0 release > --- > > Key: HBASE-12354 > URL: https://issues.apache.org/jira/browse/HBASE-12354 > Project: HBase > Issue Type: Sub-task > Components: dependencies >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 0.99.2 > > Attachments: 12354.txt, 12354v2.txt > > > Going through and updating egregiously old dependencies for 1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12285) Builds are failing, possibly because of SUREFIRE-1091
[ https://issues.apache.org/jira/browse/HBASE-12285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188897#comment-14188897 ] Hudson commented on HBASE-12285: SUCCESS: Integrated in HBase-1.0 #380 (See [https://builds.apache.org/job/HBase-1.0/380/]) HBASE-12285 Builds are failing, possibly because of SUREFIRE-1091 ; ADDENDUM SETTING LOG LEVEL TO DEBUG AGAIN (stack: rev 752c5460999ed474d45a7032b3ed986e7aa6749d) * hbase-server/src/test/resources/log4j.properties > Builds are failing, possibly because of SUREFIRE-1091 > - > > Key: HBASE-12285 > URL: https://issues.apache.org/jira/browse/HBASE-12285 > Project: HBase > Issue Type: Bug >Affects Versions: 1.0.0 >Reporter: Dima Spivak >Assignee: Dima Spivak >Priority: Blocker > Attachments: HBASE-12285_branch-1_v1.patch, > HBASE-12285_branch-1_v1.patch > > > Our branch-1 builds on builds.apache.org have been failing in recent days > after we switched over to an official version of Surefire a few days back > (HBASE-4955). The version we're using, 2.17, is hit by a bug > ([SUREFIRE-1091|https://jira.codehaus.org/browse/SUREFIRE-1091]) that results > in an IOException, which looks like what we're seeing on Jenkins. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12313) Redo the hfile index length optimization so cell-based rather than serialized KV key
[ https://issues.apache.org/jira/browse/HBASE-12313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188896#comment-14188896 ] Hudson commented on HBASE-12313: SUCCESS: Integrated in HBase-1.0 #380 (See [https://builds.apache.org/job/HBase-1.0/380/]) HBASE-12313 Redo the hfile index length optimization so cell-based rather than serialized KV key (stack: rev 6c39d36b32837bb2e114bc1919427c825d18042a) * hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/decode/PrefixTreeArrayScanner.java * hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java * hbase-common/src/test/java/org/apache/hadoop/hbase/TestCellUtil.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV3.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java * hbase-common/src/main/java/org/apache/hadoop/hbase/Cell.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * hbase-server/src/main/java/org/apache/hadoop/hbase/client/ClientSideRegionScanner.java * hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/decode/PrefixTreeCell.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java * hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ReplicationProtbufUtil.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java * hbase-common/src/main/java/org/apache/hadoop/hbase/CellKey.java * hbase-server/src/test/java/org/apache/hadoop/hbase/mapred/TestDriver.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/Increment.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java * hbase-common/src/test/java/org/apache/hadoop/hbase/TestCellComparator.java * hbase-client/src/test/java/org/apache/hadoop/hbase/ipc/TestIPCUtil.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV3.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java * hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileEncryption.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java > Redo the hfile index length optimization so cell-based rather than serialized > KV key > > > Key: HBASE-12313 > URL: https://issues.apache.org/jira/browse/HBASE-12313 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 0.99.2 > > Attachments: > 0001-HBASE-12313-Redo-the-hfile-index-length-optimization.patch, > 0001-HBASE-12313-Redo-the-hfile-index-length-optimization.patch, > 0001-HBASE-12313-Redo-the-hfile-index-length-optimization.patch, > 0001-HBASE-12313-Redo-the-hfile-index-length-optimization.patch, > 0001-HBASE-12313-Redo-the-hfile-index-length-optimization.patch, > 12313v10.txt, 12313v5.txt, 12313v6.txt, 12313v8.txt > > > Trying to remove API that returns the 'key' of a KV serialized into a byte > array is thorny. > I tried to move over the first and last key serializations and the hfile > index entries to be cell but patch was turning massive. Here is a smaller > patch that just redoes the optimization that tries to find 'short' midpoints > between last key of last block and first key of next block so it is > Cell-based rather than byte array based (presuming Keys serialized in a > certain way). Adds unit tests which we didn't have before. > Also remove CellKey. Not needed... at least not yet. Its just utility for > toString. -- This message was sent by At
[jira] [Commented] (HBASE-10462) Recategorize some of the client facing Public / Private interfaces
[ https://issues.apache.org/jira/browse/HBASE-10462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188903#comment-14188903 ] stack commented on HBASE-10462: --- Any movement on this one [~enis] What needs to be done to finish? Thanks. > Recategorize some of the client facing Public / Private interfaces > -- > > Key: HBASE-10462 > URL: https://issues.apache.org/jira/browse/HBASE-10462 > Project: HBase > Issue Type: Bug > Components: Client >Reporter: Enis Soztutar >Assignee: Enis Soztutar >Priority: Blocker > Fix For: 2.0.0, 0.99.2 > > Attachments: hbase-10462_wip1.patch > > > We should go over the list of InterfaceAudience.Public interfaces one more to > remove those that are NOT indeed public interfaces. > From current trunk, we should change these from public to private: > {code} > ReversedScannerCallable > ReversedClientScanner > ClientScanner (note that ResultScanner is public interface, while > ClientScanner should not be) > ClientSmallScanner > TableSnapshotScanner -> We need a way of constructing this since it cannot be > constructed from HConnection / HTable. Maybe a basic factory. > {code} > These are not marked: > {code} > Registry, > ZooKeeperRegistry > RpcRetryingCallerFactory > ZooKeeperKeepAliveConnection > AsyncProcess > DelegatingRetryingCallable > HConnectionKey > MasterKeepAliveConnection > MultiServerCallable > {code} > We can think about making these public interface: > {code} > ScanMetrics > {code} > Add javadoc to: > {code} > Query > {code} > We can add a test to find out all classes in client package to check for > interface mark. > We can extend this to brainstorm on the preferred API options. We probably > want the clients to use HTableInterface, instead of HTable everywhere. > HConnectionManager comes with bazillion methods which are not intended for > public use, etc. > Raising this as blocker to 1.0 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-9206) namespace permissions
[ https://issues.apache.org/jira/browse/HBASE-9206?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188907#comment-14188907 ] stack commented on HBASE-9206: -- [~toffer] Should we move out of 1.0? Would be good to have it in. > namespace permissions > - > > Key: HBASE-9206 > URL: https://issues.apache.org/jira/browse/HBASE-9206 > Project: HBase > Issue Type: Sub-task >Reporter: Francis Liu > Fix For: 0.99.2 > > > Now that we have namespaces let's address how we can give admins more > flexibility. > Let's list out the privileges we'd like. Then we can map it to existing > privileges and see if we need more. > So far we have: > 1. Modify namespace descriptor (ie quota, other values) > 2. create namespace > 3. delete namespace > 4. list tables in namespace > 5. create/drop tables in a namespace > 6. All namespace's tables create > 7. All namespace's tables write > 8. All namespace's tables execute > 9. All namespace's tables delete > 10. All namespace's tables admin > 1-3, is currently set to global admin only. Which seems acceptable to me. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12355) Update maven plugins
[ https://issues.apache.org/jira/browse/HBASE-12355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-12355: -- Attachment: 12355v7.txt What I am committing. It does NOT include updating surefire to 2.18-SNAPSHOT (nor does it add in apache SNAPSHOTs repo). Lets do that in different issue in one more directly related to the work going on over in "HBASE-12285 Builds are failing, possibly because of SUREFIRE-1091" Besides, we may not need to go to SNAPSHOT as it currently is looking (see HBASE-12285). > Update maven plugins > > > Key: HBASE-12355 > URL: https://issues.apache.org/jira/browse/HBASE-12355 > Project: HBase > Issue Type: Sub-task > Components: build >Reporter: stack >Assignee: stack > Fix For: 0.99.2 > > Attachments: 12355.txt, 12355v2.txt, 12355v3.txt, 12355v5.txt, > 12355v6.txt, 12355v6.txt, 12355v7.txt > > > Update maven plugins. Some are way old. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12376) HBaseAdmin leaks ZK connections if failure starting watchers (ConnectionLossException)
[ https://issues.apache.org/jira/browse/HBASE-12376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188918#comment-14188918 ] Sean Busbey commented on HBASE-12376: - {code} +} finally { + // If we did not succeed but created a catalogtracker, clean it up. CT has a ZK instance + // in it and we'll leak if we don't do the 'stop'. + if (!succeeded && ct != null) { +ct.stop(); +ct = null; + } {code} Wrap the ct.stop() call in a try/catch(RuntimeException) that logs a warning. Otherwise, if we end up in the finally block because of an exception (RuntimeException, ZooKeeperConnectionException, or IOException) and it should throw we'll overrun the original exception. {code} + /** + * For testing so can intercept the catalog tracker start. + * @param c + * @return Instance of CatalogTracker or exceptions if we fail + * @throws IOException + * @throws InterruptedException + */ {code} leave out the boilerplate javadoc param and throws. {code} + @VisibleForTesting + protected CatalogTracker startCatalogTracker(final CatalogTracker ct) + throws IOException, InterruptedException { {code} nit: this can be package-private and still be used in the test it's currently used in. {code} + CatalogTracker ct = doctoredAdmin.getCatalogTracker(); + assertFalse(ct.isStopped()); + ct.stop(); + assertTrue(ct.isStopped()); {code} use {{doctoredAdmin.cleanupCatalogTracker(ct)}} instead of {{ct.stop()}} since that's what the javadoc for getCatalogTracker says to do. > HBaseAdmin leaks ZK connections if failure starting watchers > (ConnectionLossException) > -- > > Key: HBASE-12376 > URL: https://issues.apache.org/jira/browse/HBASE-12376 > Project: HBase > Issue Type: Bug > Components: Zookeeper >Affects Versions: 0.98.7, 0.94.24 >Reporter: stack >Assignee: stack >Priority: Critical > Attachments: > 0001-12376-HBaseAdmin-leaks-ZK-connections-if-failure-sta.patch > > > This is a 0.98 issue that some users have been running into mostly running > Canary and for whatever reason, setup of zk connection fails, usually with a > ConnectionLossException. End result is ugly leak zk connections. ZKWatcher > created instances are just left hang out. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12355) Update maven plugins
[ https://issues.apache.org/jira/browse/HBASE-12355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-12355: -- Resolution: Fixed Fix Version/s: 2.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to branch-1+. Tried 0.98 but lots of diff so let it be. Let things like the new plugins catch stuff in master and branch-1 and we can then backport anything they find. > Update maven plugins > > > Key: HBASE-12355 > URL: https://issues.apache.org/jira/browse/HBASE-12355 > Project: HBase > Issue Type: Sub-task > Components: build >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 0.99.2 > > Attachments: 12355.txt, 12355v2.txt, 12355v3.txt, 12355v5.txt, > 12355v6.txt, 12355v6.txt, 12355v7.txt > > > Update maven plugins. Some are way old. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12121) maven release plugin does not allow for customized goals
[ https://issues.apache.org/jira/browse/HBASE-12121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enoch Hsu updated HBASE-12121: -- Fix Version/s: 1.0.0 > maven release plugin does not allow for customized goals > > > Key: HBASE-12121 > URL: https://issues.apache.org/jira/browse/HBASE-12121 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.98.6 >Reporter: Enoch Hsu >Assignee: Enoch Hsu >Priority: Minor > Fix For: 1.0.0 > > Attachments: HBASE-12121.patch > > > Inside the pom under the maven-release-plugin there is a configuration that > defines what the release-plugin uses like so > > > apache-release > > -Dmaven.test.skip.exec > pom.xml > > There is no property for goals so if the user passes in a goal from the > command line it will not get executed and the default behavior will be used > instead. > I propose to add in the following > ${goals} > This will allow custom release goal options -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12376) HBaseAdmin leaks ZK connections if failure starting watchers (ConnectionLossException)
[ https://issues.apache.org/jira/browse/HBASE-12376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-12376: -- Attachment: 0001-12376-HBaseAdmin-leaks-ZK-connections-if-failure-sta.version2.patch Thanks for nice review [~busbey] This version addresses all your comments. Regards bq. Wrap the ct.stop() call in a try/catch(RuntimeException) that logs a warning. So, you say RuntimeException in case an unexpected Exception (OOME or something?). The stop does pretty good job at catching all other crap. This a style you think we should institute throughout? Thanks. > HBaseAdmin leaks ZK connections if failure starting watchers > (ConnectionLossException) > -- > > Key: HBASE-12376 > URL: https://issues.apache.org/jira/browse/HBASE-12376 > Project: HBase > Issue Type: Bug > Components: Zookeeper >Affects Versions: 0.98.7, 0.94.24 >Reporter: stack >Assignee: stack >Priority: Critical > Attachments: > 0001-12376-HBaseAdmin-leaks-ZK-connections-if-failure-sta.patch, > 0001-12376-HBaseAdmin-leaks-ZK-connections-if-failure-sta.version2.patch > > > This is a 0.98 issue that some users have been running into mostly running > Canary and for whatever reason, setup of zk connection fails, usually with a > ConnectionLossException. End result is ugly leak zk connections. ZKWatcher > created instances are just left hang out. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12313) Redo the hfile index length optimization so cell-based rather than serialized KV key
[ https://issues.apache.org/jira/browse/HBASE-12313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188952#comment-14188952 ] Hudson commented on HBASE-12313: SUCCESS: Integrated in HBase-TRUNK #5717 (See [https://builds.apache.org/job/HBase-TRUNK/5717/]) HBASE-12313 Redo the hfile index length optimization so cell-based rather than serialized KV key (stack: rev 889333a6fd854cf27b552cf25ff711f2c50f8c08) * hbase-server/src/main/java/org/apache/hadoop/hbase/client/ClientSideRegionScanner.java * hbase-common/src/main/java/org/apache/hadoop/hbase/CellUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileWriterV2.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/Increment.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockIndex.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlockCompatibility.java * hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/decode/PrefixTreeCell.java * hbase-prefix-tree/src/main/java/org/apache/hadoop/hbase/codec/prefixtree/decode/PrefixTreeArrayScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestSeekTo.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/StoreScanner.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java * hbase-client/src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV3.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV2.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java * hbase-common/src/test/java/org/apache/hadoop/hbase/TestCellComparator.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/AbstractHFileWriter.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFilePrettyPrinter.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileEncryption.java * hbase-server/src/test/java/org/apache/hadoop/hbase/mapred/TestDriver.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RSRpcServices.java * hbase-common/src/main/java/org/apache/hadoop/hbase/CellKey.java * hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HStore.java * hbase-client/src/test/java/org/apache/hadoop/hbase/ipc/TestIPCUtil.java * hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java * hbase-common/src/main/java/org/apache/hadoop/hbase/Cell.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileBlock.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestChecksum.java * hbase-common/src/main/java/org/apache/hadoop/hbase/CellComparator.java * hbase-server/src/test/java/org/apache/hadoop/hbase/io/hfile/TestHFileWriterV3.java * hbase-common/src/test/java/org/apache/hadoop/hbase/TestCellUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ReplicationProtbufUtil.java * hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java > Redo the hfile index length optimization so cell-based rather than serialized > KV key > > > Key: HBASE-12313 > URL: https://issues.apache.org/jira/browse/HBASE-12313 > Project: HBase > Issue Type: Sub-task > Components: regionserver, Scanners >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 0.99.2 > > Attachments: > 0001-HBASE-12313-Redo-the-hfile-index-length-optimization.patch, > 0001-HBASE-12313-Redo-the-hfile-index-length-optimization.patch, > 0001-HBASE-12313-Redo-the-hfile-index-length-optimization.patch, > 0001-HBASE-12313-Redo-the-hfile-index-length-optimization.patch, > 0001-HBASE-12313-Redo-the-hfile-index-length-optimization.patch, > 12313v10.txt, 12313v5.txt, 12313v6.txt, 12313v8.txt > > > Trying to remove API that returns the 'key' of a KV serialized into a byte > array is thorny. > I tried to move over the first and last key serializations and the hfile > index entries to be cell but patch was turning massive. Here is a smaller > patch that just redoes the optimization that tries to find 'short' midpoints > between last key of last block and first key of next block so it is > Cell-based rather than byte array based (presuming Keys serialized in a > certain way). Adds unit tests which we didn't have before. > Also remove CellKey. Not needed... at least not yet. Its just utility for > toString. -- This message was sent
[jira] [Commented] (HBASE-12354) Update dependencies in time for 1.0 release
[ https://issues.apache.org/jira/browse/HBASE-12354?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188954#comment-14188954 ] Hudson commented on HBASE-12354: SUCCESS: Integrated in HBase-TRUNK #5717 (See [https://builds.apache.org/job/HBase-TRUNK/5717/]) HBASE-12354 Update dependencies in time for 1.0 release (stack: rev 7cfafe401e18522cbd92a8b06fb2ea380f71ced9) * hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestSplitLogManager.java * pom.xml > Update dependencies in time for 1.0 release > --- > > Key: HBASE-12354 > URL: https://issues.apache.org/jira/browse/HBASE-12354 > Project: HBase > Issue Type: Sub-task > Components: dependencies >Reporter: stack >Assignee: stack > Fix For: 2.0.0, 0.99.2 > > Attachments: 12354.txt, 12354v2.txt > > > Going through and updating egregiously old dependencies for 1.0. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12372) [WINDOWS] Enable log4j configuration in hbase.cmd
[ https://issues.apache.org/jira/browse/HBASE-12372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188953#comment-14188953 ] Hudson commented on HBASE-12372: SUCCESS: Integrated in HBase-TRUNK #5717 (See [https://builds.apache.org/job/HBase-TRUNK/5717/]) HBASE-12372 [WINDOWS] Enable log4j configuration in hbase.cmd (enis: rev 0fa43bd574f897f764749efcf0d42890aa44ff45) * bin/hbase.cmd > [WINDOWS] Enable log4j configuration in hbase.cmd > -- > > Key: HBASE-12372 > URL: https://issues.apache.org/jira/browse/HBASE-12372 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar >Priority: Trivial > Fix For: 2.0.0, 0.98.8, 0.99.2 > > Attachments: hbase-12372.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12377) HBaseAdmin#deleteTable fails when META region is moved around same time frame
Stephen Yuan Jiang created HBASE-12377: -- Summary: HBaseAdmin#deleteTable fails when META region is moved around same time frame Key: HBASE-12377 URL: https://issues.apache.org/jira/browse/HBASE-12377 Project: HBase Issue Type: Bug Components: Client Affects Versions: 0.98.4 Reporter: Stephen Yuan Jiang Assignee: Stephen Yuan Jiang Fix For: 2.0.0, 0.98.8, 0.99.2 This is the same issue that HBASE-10809 tried to address. The fix of HBASE-10809 refetch the latest meta location in retry-loop. However, there are 2 problems: (1). inside the retry loop, there is another try-catch block that would throw the exception before retry can kick in; (2). It looks like that HBaseAdmin::getFirstMetaServerForTable() always tries to get meta data from meta cache, which means if the meta cache is stale and out of date, retries would not solve the problem by fetch the right data. Here is the call stack of the issue: {noformat} 2014-10-27 10:11:58,495|beaver.machine|INFO|18218|140065036261120|MainThread|org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1 is not online on ip-172-31-0-48.ec2.internal,60020,1414403435009 2014-10-27 10:11:58,496|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2774) 2014-10-27 10:11:58,496|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4257) 2014-10-27 10:11:58,497|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3156) 2014-10-27 10:11:58,497|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29994) 2014-10-27 10:11:58,498|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2078) 2014-10-27 10:11:58,498|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) 2014-10-27 10:11:58,499|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) 2014-10-27 10:11:58,499|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) 2014-10-27 10:11:58,499|beaver.machine|INFO|18218|140065036261120|MainThread|at java.lang.Thread.run(Thread.java:745) 2014-10-27 10:11:58,500|beaver.machine|INFO|18218|140065036261120|MainThread| 2014-10-27 10:11:58,500|beaver.machine|INFO|18218|140065036261120|MainThread|at sun.reflect.GeneratedConstructorAccessor12.newInstance(Unknown Source) 2014-10-27 10:11:58,500|beaver.machine|INFO|18218|140065036261120|MainThread|at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 2014-10-27 10:11:58,501|beaver.machine|INFO|18218|140065036261120|MainThread|at java.lang.reflect.Constructor.newInstance(Constructor.java:526) 2014-10-27 10:11:58,501|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) 2014-10-27 10:11:58,502|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) 2014-10-27 10:11:58,502|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:306) 2014-10-27 10:11:58,502|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.client.HBaseAdmin.deleteTable(HBaseAdmin.java:699) 2014-10-27 10:11:58,503|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.client.HBaseAdmin.deleteTable(HBaseAdmin.java:654) 2014-10-27 10:11:58,503|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.IntegrationTestManyRegions.tearDown(IntegrationTestManyRegions.java:99) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12377) HBaseAdmin#deleteTable fails when META region is moved around the same timeframe
[ https://issues.apache.org/jira/browse/HBASE-12377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12377: --- Summary: HBaseAdmin#deleteTable fails when META region is moved around the same timeframe (was: HBaseAdmin#deleteTable fails when META region is moved around same time frame) > HBaseAdmin#deleteTable fails when META region is moved around the same > timeframe > > > Key: HBASE-12377 > URL: https://issues.apache.org/jira/browse/HBASE-12377 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.98.4 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > Fix For: 2.0.0, 0.98.8, 0.99.2 > > > This is the same issue that HBASE-10809 tried to address. The fix of > HBASE-10809 refetch the latest meta location in retry-loop. However, there > are 2 problems: (1). inside the retry loop, there is another try-catch block > that would throw the exception before retry can kick in; (2). It looks like > that HBaseAdmin::getFirstMetaServerForTable() always tries to get meta data > from meta cache, which means if the meta cache is stale and out of date, > retries would not solve the problem by fetch the right data. > Here is the call stack of the issue: > {noformat} > 2014-10-27 > 10:11:58,495|beaver.machine|INFO|18218|140065036261120|MainThread|org.apache.hadoop.hbase.NotServingRegionException: > org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1 is > not online on ip-172-31-0-48.ec2.internal,60020,1414403435009 > 2014-10-27 > 10:11:58,496|beaver.machine|INFO|18218|140065036261120|MainThread|at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2774) > 2014-10-27 > 10:11:58,496|beaver.machine|INFO|18218|140065036261120|MainThread|at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4257) > 2014-10-27 > 10:11:58,497|beaver.machine|INFO|18218|140065036261120|MainThread|at > org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3156) > 2014-10-27 > 10:11:58,497|beaver.machine|INFO|18218|140065036261120|MainThread|at > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29994) > 2014-10-27 > 10:11:58,498|beaver.machine|INFO|18218|140065036261120|MainThread|at > org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2078) > 2014-10-27 > 10:11:58,498|beaver.machine|INFO|18218|140065036261120|MainThread|at > org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) > 2014-10-27 > 10:11:58,499|beaver.machine|INFO|18218|140065036261120|MainThread|at > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) > 2014-10-27 > 10:11:58,499|beaver.machine|INFO|18218|140065036261120|MainThread|at > org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) > 2014-10-27 > 10:11:58,499|beaver.machine|INFO|18218|140065036261120|MainThread|at > java.lang.Thread.run(Thread.java:745) > 2014-10-27 10:11:58,500|beaver.machine|INFO|18218|140065036261120|MainThread| > 2014-10-27 > 10:11:58,500|beaver.machine|INFO|18218|140065036261120|MainThread|at > sun.reflect.GeneratedConstructorAccessor12.newInstance(Unknown Source) > 2014-10-27 > 10:11:58,500|beaver.machine|INFO|18218|140065036261120|MainThread|at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > 2014-10-27 > 10:11:58,501|beaver.machine|INFO|18218|140065036261120|MainThread|at > java.lang.reflect.Constructor.newInstance(Constructor.java:526) > 2014-10-27 > 10:11:58,501|beaver.machine|INFO|18218|140065036261120|MainThread|at > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) > 2014-10-27 > 10:11:58,502|beaver.machine|INFO|18218|140065036261120|MainThread|at > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) > 2014-10-27 > 10:11:58,502|beaver.machine|INFO|18218|140065036261120|MainThread|at > org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:306) > 2014-10-27 > 10:11:58,502|beaver.machine|INFO|18218|140065036261120|MainThread|at > org.apache.hadoop.hbase.client.HBaseAdmin.deleteTable(HBaseAdmin.java:699) > 2014-10-27 > 10:11:58,503|beaver.machine|INFO|18218|140065036261120|MainThread|at > org.apache.hadoop.hbase.client.HBaseAdmin.deleteTable(HBaseAdmin.java:654) > 2014-10-27 > 10:11:58,503|beaver.machine|INFO|18218|140065036261120|MainThread|at > org.apache.hadoop.hbase.IntegrationTestManyRegions.tearDown(IntegrationTestManyRegions.java:99) > {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12377) HBaseAdmin#deleteTable fails when META region is moved around the same timeframe
[ https://issues.apache.org/jira/browse/HBASE-12377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12377: --- Description: This is the same issue that HBASE-10809 tried to address. The fix of HBASE-10809 refetch the latest meta location in retry-loop. However, there are 2 problems: (1). inside the retry loop, there is another try-catch block that would throw the exception before retry can kick in; (2). It looks like that HBaseAdmin::getFirstMetaServerForTable() always tries to get meta data from meta cache, which means if the meta cache is stale and out of date, retries would not solve the problem by fetch the right data. Here is the call stack of the issue: {noformat} 2014-10-27 10:11:58,495|beaver.machine|INFO|18218|140065036261120|MainThread|org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1 is not online on ip-172-31-0-48.ec2.internal,60020,1414403435009 2014-10-27 10:11:58,496|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2774) 2014-10-27 10:11:58,496|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4257) 2014-10-27 10:11:58,497|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3156) 2014-10-27 10:11:58,497|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29994) 2014-10-27 10:11:58,498|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2078) 2014-10-27 10:11:58,498|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) 2014-10-27 10:11:58,499|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) 2014-10-27 10:11:58,499|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) 2014-10-27 10:11:58,499|beaver.machine|INFO|18218|140065036261120|MainThread|at java.lang.Thread.run(Thread.java:745) 2014-10-27 10:11:58,500|beaver.machine|INFO|18218|140065036261120|MainThread| 2014-10-27 10:11:58,500|beaver.machine|INFO|18218|140065036261120|MainThread|at sun.reflect.GeneratedConstructorAccessor12.newInstance(Unknown Source) 2014-10-27 10:11:58,500|beaver.machine|INFO|18218|140065036261120|MainThread|at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 2014-10-27 10:11:58,501|beaver.machine|INFO|18218|140065036261120|MainThread|at java.lang.reflect.Constructor.newInstance(Constructor.java:526) 2014-10-27 10:11:58,501|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) 2014-10-27 10:11:58,502|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) 2014-10-27 10:11:58,502|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:306) 2014-10-27 10:11:58,502|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.client.HBaseAdmin.deleteTable(HBaseAdmin.java:699) 2014-10-27 10:11:58,503|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.client.HBaseAdmin.deleteTable(HBaseAdmin.java:654) 2014-10-27 10:11:58,503|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.IntegrationTestManyRegions.tearDown(IntegrationTestManyRegions.java:99) {noformat} The META region was Online in RS1 when the delete table starts, it was moved to RS2 during the delete table operation. And the problem appears. was: This is the same issue that HBASE-10809 tried to address. The fix of HBASE-10809 refetch the latest meta location in retry-loop. However, there are 2 problems: (1). inside the retry loop, there is another try-catch block that would throw the exception before retry can kick in; (2). It looks like that HBaseAdmin::getFirstMetaServerForTable() always tries to get meta data from meta cache, which means if the meta cache is stale and out of date, retries would not solve the problem by fetch the right data. Here is the call stack of the issue: {noformat} 2014-10-27 10:11:58,495|beaver.machine|INFO|18218|140065036261120|MainThread|org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1 is not online on ip-172-31-0-48.ec2.
[jira] [Commented] (HBASE-9712) TestSplitLogManager still fails on occasion
[ https://issues.apache.org/jira/browse/HBASE-9712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188966#comment-14188966 ] Dima Spivak commented on HBASE-9712: This is hitting a lot of builds on [HBase-0.98|https://builds.apache.org/job/HBase-0.98/lastSuccessfulBuild/testReport/org.apache.hadoop.hbase.master/TestSplitLogManager/history/] but not [HBase-1.0|https://builds.apache.org/job/HBase-1.0/lastSuccessfulBuild/testReport/org.apache.hadoop.hbase.master/TestSplitLogManager/history/] or [HBase-TRUNK|https://builds.apache.org/job/HBase-TRUNK/lastSuccessfulBuild/testReport/org.apache.hadoop.hbase.master/TestSplitLogManager/history/]. Mind if I pick this one up, [~stack]? > TestSplitLogManager still fails on occasion > --- > > Key: HBASE-9712 > URL: https://issues.apache.org/jira/browse/HBASE-9712 > Project: HBase > Issue Type: Bug >Reporter: stack > Attachments: > org.apache.hadoop.hbase.master.TestSplitLogManager-output (1).txt > > > Opening this issue to keep account of failures. It failed for me locally > just now. > Failed tests: > testTaskResigned(org.apache.hadoop.hbase.master.TestSplitLogManager): > version1=2, version=2 > {code} > durruti:hbase stack$ more > hbase-server/target/surefire-reports/org.apache.hadoop.hbase.master.TestSplitLogManager.txt > --- > Test set: org.apache.hadoop.hbase.master.TestSplitLogManager > --- > Tests run: 14, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 86.697 sec > <<< FAILURE! > testTaskResigned(org.apache.hadoop.hbase.master.TestSplitLogManager) Time > elapsed: 0.004 sec <<< FAILURE! > java.lang.AssertionError: version1=2, version=2 > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.hbase.master.TestSplitLogManager.testTaskResigned(TestSplitLogManager.java:387) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:50) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:63) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:236) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:53) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:229) > at org.junit.runners.ParentRunner.run(ParentRunner.java:309) > at org.junit.runners.Suite.runChild(Suite.java:127) > at org.junit.runners.Suite.runChild(Suite.java:26) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:238) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:439) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303) > at java.util.concurrent.FutureTask.run(FutureTask.java:138) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918) > at java.lang.Thread.run(Thread.java:680) > {code} > Let me attach the log -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12376) HBaseAdmin leaks ZK connections if failure starting watchers (ConnectionLossException)
[ https://issues.apache.org/jira/browse/HBASE-12376?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188986#comment-14188986 ] Sean Busbey commented on HBASE-12376: - {quote} So, you say RuntimeException in case an unexpected Exception (OOME or something?). The stop does pretty good job at catching all other crap. This a style you think we should institute throughout? {quote} Yeah, something like that. It's a general code hygiene thing to guard against exceptions in finally blocks because of how it hides root causes. I'd be in favor of fixing it where we can. {code} +try { + ct.stop(); +} catch (RuntimeException re) { + LOG.error("In cleanup", re); +} {code} Since this is client side code, something with a stronger statement of action maybe something like: {code} LOG.error("Failed to clean up HBase's internal catalog tracker after a failed initialization. " + "We may have leaked network connections to ZooKeeper; they won't be cleaned up until " + "the JVM exits. If you see a large number of stale connections to ZooKeeper this is likely " + "the cause. The following exception details will be needed for assistance from the " + "HBase community.", re); {code} Actually now that I'm reading it, that's a bit long. :) Maybe an additional section for the ref guide section on troubleshooting related to ZooKeeper (currently 15.11) and then a link to it in the log message? Otherwise, +1 LGTM (I'm presuming you'll squash before pushing) > HBaseAdmin leaks ZK connections if failure starting watchers > (ConnectionLossException) > -- > > Key: HBASE-12376 > URL: https://issues.apache.org/jira/browse/HBASE-12376 > Project: HBase > Issue Type: Bug > Components: Zookeeper >Affects Versions: 0.98.7, 0.94.24 >Reporter: stack >Assignee: stack >Priority: Critical > Attachments: > 0001-12376-HBaseAdmin-leaks-ZK-connections-if-failure-sta.patch, > 0001-12376-HBaseAdmin-leaks-ZK-connections-if-failure-sta.version2.patch > > > This is a 0.98 issue that some users have been running into mostly running > Canary and for whatever reason, setup of zk connection fails, usually with a > ConnectionLossException. End result is ugly leak zk connections. ZKWatcher > created instances are just left hang out. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12372) [WINDOWS] Enable log4j configuration in hbase.cmd
[ https://issues.apache.org/jira/browse/HBASE-12372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188990#comment-14188990 ] Hudson commented on HBASE-12372: FAILURE: Integrated in HBase-1.0 #381 (See [https://builds.apache.org/job/HBase-1.0/381/]) HBASE-12372 [WINDOWS] Enable log4j configuration in hbase.cmd (enis: rev e79572fa1ceb30d337e6c3e883a8264dc1e66cfb) * bin/hbase.cmd > [WINDOWS] Enable log4j configuration in hbase.cmd > -- > > Key: HBASE-12372 > URL: https://issues.apache.org/jira/browse/HBASE-12372 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar >Priority: Trivial > Fix For: 2.0.0, 0.98.8, 0.99.2 > > Attachments: hbase-12372.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12372) [WINDOWS] Enable log4j configuration in hbase.cmd
[ https://issues.apache.org/jira/browse/HBASE-12372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14188993#comment-14188993 ] Hudson commented on HBASE-12372: FAILURE: Integrated in HBase-0.98 #636 (See [https://builds.apache.org/job/HBase-0.98/636/]) HBASE-12372 [WINDOWS] Enable log4j configuration in hbase.cmd (enis: rev b6d6db90daf0eb5e1d293d51264ad70f629d8ef9) * bin/hbase.cmd > [WINDOWS] Enable log4j configuration in hbase.cmd > -- > > Key: HBASE-12372 > URL: https://issues.apache.org/jira/browse/HBASE-12372 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar >Priority: Trivial > Fix For: 2.0.0, 0.98.8, 0.99.2 > > Attachments: hbase-12372.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12072) We are doing 35 x 35 retries for master operations
[ https://issues.apache.org/jira/browse/HBASE-12072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14189001#comment-14189001 ] Enis Soztutar commented on HBASE-12072: --- Can I get a review for this? > We are doing 35 x 35 retries for master operations > -- > > Key: HBASE-12072 > URL: https://issues.apache.org/jira/browse/HBASE-12072 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.6 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 0.99.2 > > Attachments: 12072-v1.txt, 12072-v2.txt, hbase-12072_v1.patch > > > For master requests, there are two retry mechanisms in effect. The first one > is from HBaseAdmin.executeCallable() > {code} > private V executeCallable(MasterCallable callable) throws > IOException { > RpcRetryingCaller caller = rpcCallerFactory.newCaller(); > try { > return caller.callWithRetries(callable); > } finally { > callable.close(); > } > } > {code} > And inside, the other one is from StubMaker.makeStub(): > {code} > /** >* Create a stub against the master. Retry if necessary. >* @return A stub to do intf against the master >* @throws MasterNotRunningException >*/ > @edu.umd.cs.findbugs.annotations.SuppressWarnings > (value="SWL_SLEEP_WITH_LOCK_HELD") > Object makeStub() throws MasterNotRunningException { > {code} > The tests will just hang for 10 min * 35 ~= 6hours. > {code} > 2014-09-23 16:19:05,151 INFO [main] > client.ConnectionManager$HConnectionImplementation: getMaster attempt 1 of 35 > failed; retrying after sleep of 100, exception=java.io.IOException: Can't get > master address from ZooKeeper; znode data == null > 2014-09-23 16:19:05,253 INFO [main] > client.ConnectionManager$HConnectionImplementation: getMaster attempt 2 of 35 > failed; retrying after sleep of 200, exception=java.io.IOException: Can't get > master address from ZooKeeper; znode data == null > 2014-09-23 16:19:05,456 INFO [main] > client.ConnectionManager$HConnectionImplementation: getMaster attempt 3 of 35 > failed; retrying after sleep of 300, exception=java.io.IOException: Can't get > master address from ZooKeeper; znode data == null > 2014-09-23 16:19:05,759 INFO [main] > client.ConnectionManager$HConnectionImplementation: getMaster attempt 4 of 35 > failed; retrying after sleep of 500, exception=java.io.IOException: Can't get > master address from ZooKeeper; znode data == null > 2014-09-23 16:19:06,262 INFO [main] > client.ConnectionManager$HConnectionImplementation: getMaster attempt 5 of 35 > failed; retrying after sleep of 1008, exception=java.io.IOException: Can't > get master address from ZooKeeper; znode data == null > 2014-09-23 16:19:07,273 INFO [main] > client.ConnectionManager$HConnectionImplementation: getMaster attempt 6 of 35 > failed; retrying after sleep of 2011, exception=java.io.IOException: Can't > get master address from ZooKeeper; znode data == null > 2014-09-23 16:19:09,286 INFO [main] > client.ConnectionManager$HConnectionImplementation: getMaster attempt 7 of 35 > failed; retrying after sleep of 4012, exception=java.io.IOException: Can't > get master address from ZooKeeper; znode data == null > 2014-09-23 16:19:13,303 INFO [main] > client.ConnectionManager$HConnectionImplementation: getMaster attempt 8 of 35 > failed; retrying after sleep of 10033, exception=java.io.IOException: Can't > get master address from ZooKeeper; znode data == null > 2014-09-23 16:19:23,343 INFO [main] > client.ConnectionManager$HConnectionImplementation: getMaster attempt 9 of 35 > failed; retrying after sleep of 10089, exception=java.io.IOException: Can't > get master address from ZooKeeper; znode data == null > 2014-09-23 16:19:33,439 INFO [main] > client.ConnectionManager$HConnectionImplementation: getMaster attempt 10 of > 35 failed; retrying after sleep of 10027, exception=java.io.IOException: > Can't get master address from ZooKeeper; znode data == null > 2014-09-23 16:19:43,473 INFO [main] > client.ConnectionManager$HConnectionImplementation: getMaster attempt 11 of > 35 failed; retrying after sleep of 10004, exception=java.io.IOException: > Can't get master address from ZooKeeper; znode data == null > 2014-09-23 16:19:53,485 INFO [main] > client.ConnectionManager$HConnectionImplementation: getMaster attempt 12 of > 35 failed; retrying after sleep of 20160, exception=java.io.IOException: > Can't get master address from ZooKeeper; znode data == null > 2014-09-23 16:20:13,656 INFO [main] > client.ConnectionManager$HConnectionImplementation: getMaster attempt 13 of > 35 failed; retrying after sleep of 20006, exception=java.io.IOException: > Can't get master address from ZooKeeper; znode data == null > 2014-09-23 16:2
[jira] [Updated] (HBASE-12072) We are doing 35 x 35 retries for master operations
[ https://issues.apache.org/jira/browse/HBASE-12072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Enis Soztutar updated HBASE-12072: -- Fix Version/s: 0.99.2 2.0.0 > We are doing 35 x 35 retries for master operations > -- > > Key: HBASE-12072 > URL: https://issues.apache.org/jira/browse/HBASE-12072 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.6 >Reporter: Enis Soztutar >Assignee: Enis Soztutar > Fix For: 2.0.0, 0.99.2 > > Attachments: 12072-v1.txt, 12072-v2.txt, hbase-12072_v1.patch > > > For master requests, there are two retry mechanisms in effect. The first one > is from HBaseAdmin.executeCallable() > {code} > private V executeCallable(MasterCallable callable) throws > IOException { > RpcRetryingCaller caller = rpcCallerFactory.newCaller(); > try { > return caller.callWithRetries(callable); > } finally { > callable.close(); > } > } > {code} > And inside, the other one is from StubMaker.makeStub(): > {code} > /** >* Create a stub against the master. Retry if necessary. >* @return A stub to do intf against the master >* @throws MasterNotRunningException >*/ > @edu.umd.cs.findbugs.annotations.SuppressWarnings > (value="SWL_SLEEP_WITH_LOCK_HELD") > Object makeStub() throws MasterNotRunningException { > {code} > The tests will just hang for 10 min * 35 ~= 6hours. > {code} > 2014-09-23 16:19:05,151 INFO [main] > client.ConnectionManager$HConnectionImplementation: getMaster attempt 1 of 35 > failed; retrying after sleep of 100, exception=java.io.IOException: Can't get > master address from ZooKeeper; znode data == null > 2014-09-23 16:19:05,253 INFO [main] > client.ConnectionManager$HConnectionImplementation: getMaster attempt 2 of 35 > failed; retrying after sleep of 200, exception=java.io.IOException: Can't get > master address from ZooKeeper; znode data == null > 2014-09-23 16:19:05,456 INFO [main] > client.ConnectionManager$HConnectionImplementation: getMaster attempt 3 of 35 > failed; retrying after sleep of 300, exception=java.io.IOException: Can't get > master address from ZooKeeper; znode data == null > 2014-09-23 16:19:05,759 INFO [main] > client.ConnectionManager$HConnectionImplementation: getMaster attempt 4 of 35 > failed; retrying after sleep of 500, exception=java.io.IOException: Can't get > master address from ZooKeeper; znode data == null > 2014-09-23 16:19:06,262 INFO [main] > client.ConnectionManager$HConnectionImplementation: getMaster attempt 5 of 35 > failed; retrying after sleep of 1008, exception=java.io.IOException: Can't > get master address from ZooKeeper; znode data == null > 2014-09-23 16:19:07,273 INFO [main] > client.ConnectionManager$HConnectionImplementation: getMaster attempt 6 of 35 > failed; retrying after sleep of 2011, exception=java.io.IOException: Can't > get master address from ZooKeeper; znode data == null > 2014-09-23 16:19:09,286 INFO [main] > client.ConnectionManager$HConnectionImplementation: getMaster attempt 7 of 35 > failed; retrying after sleep of 4012, exception=java.io.IOException: Can't > get master address from ZooKeeper; znode data == null > 2014-09-23 16:19:13,303 INFO [main] > client.ConnectionManager$HConnectionImplementation: getMaster attempt 8 of 35 > failed; retrying after sleep of 10033, exception=java.io.IOException: Can't > get master address from ZooKeeper; znode data == null > 2014-09-23 16:19:23,343 INFO [main] > client.ConnectionManager$HConnectionImplementation: getMaster attempt 9 of 35 > failed; retrying after sleep of 10089, exception=java.io.IOException: Can't > get master address from ZooKeeper; znode data == null > 2014-09-23 16:19:33,439 INFO [main] > client.ConnectionManager$HConnectionImplementation: getMaster attempt 10 of > 35 failed; retrying after sleep of 10027, exception=java.io.IOException: > Can't get master address from ZooKeeper; znode data == null > 2014-09-23 16:19:43,473 INFO [main] > client.ConnectionManager$HConnectionImplementation: getMaster attempt 11 of > 35 failed; retrying after sleep of 10004, exception=java.io.IOException: > Can't get master address from ZooKeeper; znode data == null > 2014-09-23 16:19:53,485 INFO [main] > client.ConnectionManager$HConnectionImplementation: getMaster attempt 12 of > 35 failed; retrying after sleep of 20160, exception=java.io.IOException: > Can't get master address from ZooKeeper; znode data == null > 2014-09-23 16:20:13,656 INFO [main] > client.ConnectionManager$HConnectionImplementation: getMaster attempt 13 of > 35 failed; retrying after sleep of 20006, exception=java.io.IOException: > Can't get master address from ZooKeeper; znode data == null > 2014-09-23 16:20:33,675 INFO [main] > client.Conne
[jira] [Commented] (HBASE-12377) HBaseAdmin#deleteTable fails when META region is moved around the same timeframe
[ https://issues.apache.org/jira/browse/HBASE-12377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14189017#comment-14189017 ] Enis Soztutar commented on HBASE-12377: --- I think HBASE-12072 is related since it unifies some paths so that we are using the "regular" rpc retrying mechanism instead of custom build ones inside HBaseAdmin. For this issue, the problem is that HBaseAdmin.deleteTable() does not use the regular scan rpc code (which handles retrying / meta cache, etc correctly) but instead does reinvent the stuff in a broken way. Another issue with this is that all this logic is in client side vs it should have been in the master side, but that is a different and much more involved issue. Can we do the patch so that it uses MetaReader or MetaScanner to obtain the list of regions for the table in the retry loop? > HBaseAdmin#deleteTable fails when META region is moved around the same > timeframe > > > Key: HBASE-12377 > URL: https://issues.apache.org/jira/browse/HBASE-12377 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 0.98.4 >Reporter: Stephen Yuan Jiang >Assignee: Stephen Yuan Jiang > Fix For: 2.0.0, 0.98.8, 0.99.2 > > > This is the same issue that HBASE-10809 tried to address. The fix of > HBASE-10809 refetch the latest meta location in retry-loop. However, there > are 2 problems: (1). inside the retry loop, there is another try-catch block > that would throw the exception before retry can kick in; (2). It looks like > that HBaseAdmin::getFirstMetaServerForTable() always tries to get meta data > from meta cache, which means if the meta cache is stale and out of date, > retries would not solve the problem by fetch the right data. > Here is the call stack of the issue: > {noformat} > 2014-10-27 > 10:11:58,495|beaver.machine|INFO|18218|140065036261120|MainThread|org.apache.hadoop.hbase.NotServingRegionException: > org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1 is > not online on ip-172-31-0-48.ec2.internal,60020,1414403435009 > 2014-10-27 > 10:11:58,496|beaver.machine|INFO|18218|140065036261120|MainThread|at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2774) > 2014-10-27 > 10:11:58,496|beaver.machine|INFO|18218|140065036261120|MainThread|at > org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4257) > 2014-10-27 > 10:11:58,497|beaver.machine|INFO|18218|140065036261120|MainThread|at > org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3156) > 2014-10-27 > 10:11:58,497|beaver.machine|INFO|18218|140065036261120|MainThread|at > org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29994) > 2014-10-27 > 10:11:58,498|beaver.machine|INFO|18218|140065036261120|MainThread|at > org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2078) > 2014-10-27 > 10:11:58,498|beaver.machine|INFO|18218|140065036261120|MainThread|at > org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) > 2014-10-27 > 10:11:58,499|beaver.machine|INFO|18218|140065036261120|MainThread|at > org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) > 2014-10-27 > 10:11:58,499|beaver.machine|INFO|18218|140065036261120|MainThread|at > org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) > 2014-10-27 > 10:11:58,499|beaver.machine|INFO|18218|140065036261120|MainThread|at > java.lang.Thread.run(Thread.java:745) > 2014-10-27 10:11:58,500|beaver.machine|INFO|18218|140065036261120|MainThread| > 2014-10-27 > 10:11:58,500|beaver.machine|INFO|18218|140065036261120|MainThread|at > sun.reflect.GeneratedConstructorAccessor12.newInstance(Unknown Source) > 2014-10-27 > 10:11:58,500|beaver.machine|INFO|18218|140065036261120|MainThread|at > sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) > 2014-10-27 > 10:11:58,501|beaver.machine|INFO|18218|140065036261120|MainThread|at > java.lang.reflect.Constructor.newInstance(Constructor.java:526) > 2014-10-27 > 10:11:58,501|beaver.machine|INFO|18218|140065036261120|MainThread|at > org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) > 2014-10-27 > 10:11:58,502|beaver.machine|INFO|18218|140065036261120|MainThread|at > org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) > 2014-10-27 > 10:11:58,502|beaver.machine|INFO|18218|140065036261120|MainThread|at > org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:306) > 2014-10-27 > 10:11:58,502|beaver.machine|INFO|18218|140065036261120|MainThread|at > org.apach
[jira] [Updated] (HBASE-12312) Another couple of createTable race conditions
[ https://issues.apache.org/jira/browse/HBASE-12312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-12312: --- Fix Version/s: 0.98.8 I applied this to 0.98 also > Another couple of createTable race conditions > - > > Key: HBASE-12312 > URL: https://issues.apache.org/jira/browse/HBASE-12312 > Project: HBase > Issue Type: Bug >Reporter: Dima Spivak >Assignee: Dima Spivak > Fix For: 2.0.0, 0.98.8, 0.99.2 > > Attachments: HBASE-12312_master_v1.patch, > HBASE-12312_master_v2.patch, HBASE-12312_master_v3 (1).patch, > HBASE-12312_master_v3.patch, HBASE-12312_master_v3.patch, > HBASE-12312_master_v3.patch, HBASE-12312_master_v4.patch > > > Found a couple more failing tests in TestAccessController and > TestScanEarlyTermination caused by my favorite race condition. :) Will post a > patch in a second. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12312) Another couple of createTable race conditions
[ https://issues.apache.org/jira/browse/HBASE-12312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-12312: --- Attachment: HBASE-12312-0.98.patch What I applied to 0.98. TestScanEarlyTermination passed for me 10 out of 10 times locally. Will watch it on Jenkins and our test rigs. > Another couple of createTable race conditions > - > > Key: HBASE-12312 > URL: https://issues.apache.org/jira/browse/HBASE-12312 > Project: HBase > Issue Type: Bug >Reporter: Dima Spivak >Assignee: Dima Spivak > Fix For: 2.0.0, 0.98.8, 0.99.2 > > Attachments: HBASE-12312-0.98.patch, HBASE-12312_master_v1.patch, > HBASE-12312_master_v2.patch, HBASE-12312_master_v3 (1).patch, > HBASE-12312_master_v3.patch, HBASE-12312_master_v3.patch, > HBASE-12312_master_v3.patch, HBASE-12312_master_v4.patch > > > Found a couple more failing tests in TestAccessController and > TestScanEarlyTermination caused by my favorite race condition. :) Will post a > patch in a second. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-12375) LoadIncrementalHFiles fails to load data in table when CF name starts with '_'
[ https://issues.apache.org/jira/browse/HBASE-12375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14189033#comment-14189033 ] Ted Yu commented on HBASE-12375: +1 > LoadIncrementalHFiles fails to load data in table when CF name starts with '_' > -- > > Key: HBASE-12375 > URL: https://issues.apache.org/jira/browse/HBASE-12375 > Project: HBase > Issue Type: Bug >Affects Versions: 0.98.5 >Reporter: Ashish Singhi >Assignee: Ashish Singhi >Priority: Minor > Attachments: HBASE-12375.patch > > > We do not restrict user from creating a table having column family starting > with '_'. > So when user creates a table in such a way then LoadIncrementalHFiles will > skip those family data to load into the table. > {code} > // Skip _logs, etc > if (familyDir.getName().startsWith("_")) continue; > {code} > I think we should remove that check as I do not see any _logs directory being > created by the bulkload tool in the output directory. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HBASE-9712) TestSplitLogManager still fails on occasion
[ https://issues.apache.org/jira/browse/HBASE-9712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14189035#comment-14189035 ] Dima Spivak commented on HBASE-9712: Some more debugging: all recent failures are caused by testLogFilesAreArchived (compare failure history of [testLogFilesAreArchived|https://builds.apache.org/job/HBase-0.98/lastSuccessfulBuild/testReport/junit/org.apache.hadoop.hbase.master/TestSplitLogManager/testLogFilesAreArchived/history/] and [TestSplitLogManager itself|https://builds.apache.org/job/HBase-0.98/lastSuccessfulBuild/testReport/org.apache.hadoop.hbase.master/TestSplitLogManager/history/]). Test output from a failed run suggests this may be legit. [Some output from a failed run|https://builds.apache.org/job/HBase-0.98/635/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.master.TestSplitLogManager-output.txt] suggests that things start going south after WAL splitting begins: {code} 2014-10-29 02:45:30,670 DEBUG [Thread-27] master.SplitLogManager(323): Scheduling batch of logs to split 2014-10-29 02:45:30,670 INFO [Thread-27] master.SplitLogManager(325): started splitting 1 logs in [/home/jenkins/jenkins-slave/workspace/HBase-0.98/hbase-server/target/test-data/5d71199c-6df9-4aa6-800f-9634b546ecef/testLogFilesAreArchived/595bcd1c-d510-464c-8207-33e8580aee96] 2014-10-29 02:45:30,672 WARN [Thread-31] master.TestSplitLogManager$5(527): org.apache.zookeeper.KeeperException$NoNodeException: KeeperErrorCode = NoNode for /hbase/splitWAL/595bcd1c-d510-464c-8207-33e8580aee96%2Ffoo%2C1%2C1 2014-10-29 02:45:30,672 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(703): put up splitlog task at znode /hbase/splitWAL/595bcd1c-d510-464c-8207-33e8580aee96%2Ffoo%2C1%2C1 2014-10-29 02:45:30,674 DEBUG [pool-1-thread-1-EventThread] master.SplitLogManager(745): task not yet acquired /hbase/splitWAL/595bcd1c-d510-464c-8207-33e8580aee96%2Ffoo%2C1%2C1 ver = 0 {code} For splits that occur successfully, the task is acquired quickly and completed within a few tens of milliseconds. For unsuccessful runs, the unassigned task looks to be resubmitted repeatedly by splitLogManagerTimeoutMonitor, but while tasks like /hbase/splitWAL/RESCAN01 enter into a DONE state, the splitlog one never does and times out after 60 s. Can someone more familiar with the SplitLogManager is doing here chime in? > TestSplitLogManager still fails on occasion > --- > > Key: HBASE-9712 > URL: https://issues.apache.org/jira/browse/HBASE-9712 > Project: HBase > Issue Type: Bug >Reporter: stack > Attachments: > org.apache.hadoop.hbase.master.TestSplitLogManager-output (1).txt > > > Opening this issue to keep account of failures. It failed for me locally > just now. > Failed tests: > testTaskResigned(org.apache.hadoop.hbase.master.TestSplitLogManager): > version1=2, version=2 > {code} > durruti:hbase stack$ more > hbase-server/target/surefire-reports/org.apache.hadoop.hbase.master.TestSplitLogManager.txt > --- > Test set: org.apache.hadoop.hbase.master.TestSplitLogManager > --- > Tests run: 14, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 86.697 sec > <<< FAILURE! > testTaskResigned(org.apache.hadoop.hbase.master.TestSplitLogManager) Time > elapsed: 0.004 sec <<< FAILURE! > java.lang.AssertionError: version1=2, version=2 > at org.junit.Assert.fail(Assert.java:88) > at org.junit.Assert.assertTrue(Assert.java:41) > at > org.apache.hadoop.hbase.master.TestSplitLogManager.testTaskResigned(TestSplitLogManager.java:387) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:271) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:70) > a
[jira] [Commented] (HBASE-11819) Unit test for CoprocessorHConnection
[ https://issues.apache.org/jira/browse/HBASE-11819?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14189048#comment-14189048 ] Andrew Purtell commented on HBASE-11819: Yes, lgtm, with Stack's suggestions. > Unit test for CoprocessorHConnection > - > > Key: HBASE-11819 > URL: https://issues.apache.org/jira/browse/HBASE-11819 > Project: HBase > Issue Type: Test >Reporter: Andrew Purtell >Assignee: Talat UYARER >Priority: Minor > Labels: newbie++ > Fix For: 2.0.0, 0.98.8, 0.99.2 > > Attachments: HBASE-11819.patch > > > Add a unit test to hbase-server that exercises CoprocessorHConnection . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12377) HBaseAdmin#deleteTable fails when META region is moved around the same timeframe
[ https://issues.apache.org/jira/browse/HBASE-12377?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen Yuan Jiang updated HBASE-12377: --- Description: This is the same issue that HBASE-10809 tried to address. The fix of HBASE-10809 refetch the latest meta location in retry-loop. However, there are 2 problems: (1). inside the retry loop, there is another try-catch block that would throw the exception before retry can kick in; (2). It looks like that HBaseAdmin::getFirstMetaServerForTable() always tries to get meta data from meta cache, which means if the meta cache is stale and out of date, retries would not solve the problem by fetching from the stale meta cache. Here is the call stack of the issue: {noformat} 2014-10-27 10:11:58,495|beaver.machine|INFO|18218|140065036261120|MainThread|org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1 is not online on ip-172-31-0-48.ec2.internal,60020,1414403435009 2014-10-27 10:11:58,496|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2774) 2014-10-27 10:11:58,496|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegion(HRegionServer.java:4257) 2014-10-27 10:11:58,497|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.regionserver.HRegionServer.scan(HRegionServer.java:3156) 2014-10-27 10:11:58,497|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:29994) 2014-10-27 10:11:58,498|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2078) 2014-10-27 10:11:58,498|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:108) 2014-10-27 10:11:58,499|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:114) 2014-10-27 10:11:58,499|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:94) 2014-10-27 10:11:58,499|beaver.machine|INFO|18218|140065036261120|MainThread|at java.lang.Thread.run(Thread.java:745) 2014-10-27 10:11:58,500|beaver.machine|INFO|18218|140065036261120|MainThread| 2014-10-27 10:11:58,500|beaver.machine|INFO|18218|140065036261120|MainThread|at sun.reflect.GeneratedConstructorAccessor12.newInstance(Unknown Source) 2014-10-27 10:11:58,500|beaver.machine|INFO|18218|140065036261120|MainThread|at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45) 2014-10-27 10:11:58,501|beaver.machine|INFO|18218|140065036261120|MainThread|at java.lang.reflect.Constructor.newInstance(Constructor.java:526) 2014-10-27 10:11:58,501|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106) 2014-10-27 10:11:58,502|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95) 2014-10-27 10:11:58,502|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.protobuf.ProtobufUtil.getRemoteException(ProtobufUtil.java:306) 2014-10-27 10:11:58,502|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.client.HBaseAdmin.deleteTable(HBaseAdmin.java:699) 2014-10-27 10:11:58,503|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.client.HBaseAdmin.deleteTable(HBaseAdmin.java:654) 2014-10-27 10:11:58,503|beaver.machine|INFO|18218|140065036261120|MainThread|at org.apache.hadoop.hbase.IntegrationTestManyRegions.tearDown(IntegrationTestManyRegions.java:99) {noformat} The META region was Online in RS1 when the delete table starts, it was moved to RS2 during the delete table operation. And the problem appears. was: This is the same issue that HBASE-10809 tried to address. The fix of HBASE-10809 refetch the latest meta location in retry-loop. However, there are 2 problems: (1). inside the retry loop, there is another try-catch block that would throw the exception before retry can kick in; (2). It looks like that HBaseAdmin::getFirstMetaServerForTable() always tries to get meta data from meta cache, which means if the meta cache is stale and out of date, retries would not solve the problem by fetch the right data. Here is the call stack of the issue: {noformat} 2014-10-27 10:11:58,495|beaver.machine|INFO|18218|140065036261120|MainThread|org.apache.hadoop.hbase.NotServingRegionException: org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1 is not online on ip-17
[jira] [Commented] (HBASE-12372) [WINDOWS] Enable log4j configuration in hbase.cmd
[ https://issues.apache.org/jira/browse/HBASE-12372?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14189062#comment-14189062 ] Hudson commented on HBASE-12372: SUCCESS: Integrated in HBase-0.98-on-Hadoop-1.1 #606 (See [https://builds.apache.org/job/HBase-0.98-on-Hadoop-1.1/606/]) HBASE-12372 [WINDOWS] Enable log4j configuration in hbase.cmd (enis: rev b6d6db90daf0eb5e1d293d51264ad70f629d8ef9) * bin/hbase.cmd > [WINDOWS] Enable log4j configuration in hbase.cmd > -- > > Key: HBASE-12372 > URL: https://issues.apache.org/jira/browse/HBASE-12372 > Project: HBase > Issue Type: Bug >Reporter: Enis Soztutar >Assignee: Enis Soztutar >Priority: Trivial > Fix For: 2.0.0, 0.98.8, 0.99.2 > > Attachments: hbase-12372.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HBASE-12378) Add a test to verify that the read-replica is able to read after a compaction
Matteo Bertozzi created HBASE-12378: --- Summary: Add a test to verify that the read-replica is able to read after a compaction Key: HBASE-12378 URL: https://issues.apache.org/jira/browse/HBASE-12378 Project: HBase Issue Type: Test Components: regionserver, Replication Affects Versions: 0.99.1, 2.0.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Fix For: 2.0.0, 0.99.2 Add a unit test that verify that the secondary read-replica is still able to read all the data even when the files on the primary are archived and the store file refresh is not executed. basically is to have a test that verifies that the file-link logic is not removed. (there are a couple of test that probably wants to do that.. but since they operate on small data they will never trigger the file-link reopen) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HBASE-12378) Add a test to verify that the read-replica is able to read after a compaction
[ https://issues.apache.org/jira/browse/HBASE-12378?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-12378: Attachment: HBASE-12378-v0.patch > Add a test to verify that the read-replica is able to read after a compaction > - > > Key: HBASE-12378 > URL: https://issues.apache.org/jira/browse/HBASE-12378 > Project: HBase > Issue Type: Test > Components: regionserver, Replication >Affects Versions: 2.0.0, 0.99.1 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi >Priority: Minor > Fix For: 2.0.0, 0.99.2 > > Attachments: HBASE-12378-v0.patch > > > Add a unit test that verify that the secondary read-replica is still able to > read all the data even when the files on the primary are archived and the > store file refresh is not executed. > basically is to have a test that verifies that the file-link logic is not > removed. > (there are a couple of test that probably wants to do that.. but since they > operate on small data they will never trigger the file-link reopen) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HBASE-12376) HBaseAdmin leaks ZK connections if failure starting watchers (ConnectionLossException)
[ https://issues.apache.org/jira/browse/HBASE-12376?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-12376. --- Resolution: Fixed Fix Version/s: 0.98.9 0.94.26 Hadoop Flags: Reviewed I liked your text [~busbey]; thats what I committed to 0.94 and 0.98 branches. Thanks for reviews. > HBaseAdmin leaks ZK connections if failure starting watchers > (ConnectionLossException) > -- > > Key: HBASE-12376 > URL: https://issues.apache.org/jira/browse/HBASE-12376 > Project: HBase > Issue Type: Bug > Components: Zookeeper >Affects Versions: 0.98.7, 0.94.24 >Reporter: stack >Assignee: stack >Priority: Critical > Fix For: 0.94.26, 0.98.9 > > Attachments: > 0001-12376-HBaseAdmin-leaks-ZK-connections-if-failure-sta.patch, > 0001-12376-HBaseAdmin-leaks-ZK-connections-if-failure-sta.version2.patch > > > This is a 0.98 issue that some users have been running into mostly running > Canary and for whatever reason, setup of zk connection fails, usually with a > ConnectionLossException. End result is ugly leak zk connections. ZKWatcher > created instances are just left hang out. -- This message was sent by Atlassian JIRA (v6.3.4#6332)