[jira] [Commented] (HBASE-6917) Trunk jdk7 build broke because we moved to zk 3.4.4
[ https://issues.apache.org/jira/browse/HBASE-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467555#comment-13467555 ] Andrew Purtell commented on HBASE-6917: --- Yep that's ZOOKEEPER-1550. > Trunk jdk7 build broke because we moved to zk 3.4.4 > --- > > Key: HBASE-6917 > URL: https://issues.apache.org/jira/browse/HBASE-6917 > Project: HBase > Issue Type: Bug > Components: test >Reporter: stack >Assignee: stack > Fix For: 0.96.0 > > Attachments: 6917.txt > > > Chatted w/ Mahadev and he confirmed issues running 3.4.4 w/ jdk7. Will be > fixed in zk3.4.5. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6918) Debugging to help figure what is different up on jenkins when TestHeapSize runs
[ https://issues.apache.org/jira/browse/HBASE-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467554#comment-13467554 ] Hudson commented on HBASE-6918: --- Integrated in HBase-TRUNK #3404 (See [https://builds.apache.org/job/HBase-TRUNK/3404/]) HBASE-6918 Debugging to help figure what is different up on jenkins when TestHeapSize runs (Revision 1392753) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java > Debugging to help figure what is different up on jenkins when TestHeapSize > runs > --- > > Key: HBASE-6918 > URL: https://issues.apache.org/jira/browse/HBASE-6918 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 0.96.0 >Reporter: stack >Assignee: stack > Fix For: 0.96.0 > > Attachments: 6918.txt > > > TestHeapSize will pass locally then fail up on jenkins. Add debugging to the > test that prints detail on environment and jvm in particular to can figure > what is different between two environments (it used to be that the d32 flag > was needed to mimic jenkins but now we are jdk7, the d32 flag doesn't work w/ > my oracle jvm... let me see what jenkins has). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6914) Scans/Gets/Mutations don't give a good error if the table is disabled.
[ https://issues.apache.org/jira/browse/HBASE-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467567#comment-13467567 ] Hadoop QA commented on HBASE-6914: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12547348/HBASE-6914-3.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified tests. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:red}-1 javadoc{color}. The javadoc tool appears to have generated 149 warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 findbugs{color}. The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2983//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2983//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2983//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2983//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2983//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2983//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2983//console This message is automatically generated. > Scans/Gets/Mutations don't give a good error if the table is disabled. > -- > > Key: HBASE-6914 > URL: https://issues.apache.org/jira/browse/HBASE-6914 > Project: HBase > Issue Type: Improvement >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 0.92.3, 0.94.3, 0.96.0 > > Attachments: HBASE-6914-0.patch, HBASE-6914-1.patch, > HBASE-6914-2.patch, HBASE-6914-3.patch > > > Scan a table that is disabled will have the client retry multiple times and > then will error out with NotServingRegionException. If the table is disabled > there's no need to re-try and the message should be more explicit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467575#comment-13467575 ] Hadoop QA commented on HBASE-5937: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12547291/HBASE-5937.v2.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 72 new or modified tests. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:red}-1 javadoc{color}. The javadoc tool appears to have generated 157 warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 findbugs{color}. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:red}-1 core tests{color}. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestHCM Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2984//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2984//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2984//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2984//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2984//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2984//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2984//console This message is automatically generated. > Refactor HLog into an interface. > > > Key: HBASE-5937 > URL: https://issues.apache.org/jira/browse/HBASE-5937 > Project: HBase > Issue Type: Sub-task >Reporter: Li Pi >Assignee: Flavio Junqueira >Priority: Minor > Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch, > HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, > HBASE-5937.v2.patch, > org.apache.hadoop.hbase.client.TestMultiParallel-output.txt > > > What the summary says. Create HLog interface. Make current implementation use > it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6805) Extend co-processor framework to provide observers for filter operations
[ https://issues.apache.org/jira/browse/HBASE-6805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467577#comment-13467577 ] Jason Dai commented on HBASE-6805: -- [~apurtell] The example in the updated patch file shows a possible example: the value of each cell stored in the table is automatically encrypted by the CP, which then needs to decrypt the cell before applying filter operations (filterKeyValue, transform, etc.). By implementing the filter CP, the encryption can be transparent to the user code. Similarly, for DOT, multiple fields are encoded in a single cell by the CP, and each field needs to be extracted before applying filter operations so that it can be transparent to the user. bq. If extending the CP hook model to internal filter methods, we must be deeply concerned about the costs of iterating CP hook lists during filtering/scanning. CPs extend the code path, first of all. Then, if hooks are registered, there will be method invocation and object allocation costs for _every_ filter operation, twice. While there are two method invocations for each filter operation, these method invocations are actually only called for the topmost filter (which FilterWrapper wraps), not for each filter contained in the chained FilterList or other composite filters. In our DOT benchmarking, these CP operations are never the hotspot in scanning. Having said that, CP operations could become a potential performance issue if we have a long list of CPs loaded. For instance, database trigger like CPs only execute upon data mutation (i.e., Put), but are still invoked for Get/Scan/Filter. One way to address this issue is that, instead of iterating the global _coprocessor_ set in these pre* & post* operations, the RegionCoprocessorHost can maintain several CP set, and iterate a different set in each different CP operation: one for region operations (preOpen/postOpen/preClose/...), one for update (prePut & postPut), one for read (preGet/postGet/preScannerOpen/...), and one for filter (preFilterKeyvalue/postFilterKeyvalue/); when loading each CP, it can be registered in appropriate sets (just as endpoints are registered in _Region.protocolHandlers_). > Extend co-processor framework to provide observers for filter operations > > > Key: HBASE-6805 > URL: https://issues.apache.org/jira/browse/HBASE-6805 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 0.96.0 >Reporter: Jason Dai > Attachments: extend_coprocessor.patch > > > There are several filter operations (e.g., filterKeyValue, filterRow, > transform, etc.) at the region server side that either exclude KVs from the > returned results, or transform the returned KV. We need to provide observers > (e.g., preFilterKeyValue and postFilterKeyValue) for these operations in the > same way as the observers for other data access operations (e.g., preGet and > postGet). This extension is needed to support DOT (e.g., extracting > individual fields from the document in the observers before passing them to > the related filter operations) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6317) Master clean start up and Partially enabled tables make region assignment inconsistent.
[ https://issues.apache.org/jira/browse/HBASE-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-6317: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the patch Rajesh Thanks for the review Stack, Jimmy and Ted. > Master clean start up and Partially enabled tables make region assignment > inconsistent. > --- > > Key: HBASE-6317 > URL: https://issues.apache.org/jira/browse/HBASE-6317 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: rajeshbabu > Fix For: 0.92.3, 0.94.3, 0.96.0 > > Attachments: HBASE-6317_94_3.patch, HBASE-6317_94.patch, > HBASE-6317_trunk_2.patch, HBASE-6317_trunk_3.patch, HBASE-6317_trunk_4.patch, > HBASE-6317_trunk_5.patch > > > If we have a table in partially enabled state (ENABLING) then on HMaster > restart we treat it as a clean cluster start up and do a bulk assign. > Currently in 0.94 bulk assign will not handle ALREADY_OPENED scenarios and it > leads to region assignment problems. Analysing more on this we found that we > have better way to handle these scenarios. > {code} > if (false == checkIfRegionBelongsToDisabled(regionInfo) > && false == checkIfRegionsBelongsToEnabling(regionInfo)) { > synchronized (this.regions) { > regions.put(regionInfo, regionLocation); > addToServers(regionLocation, regionInfo); > } > {code} > We dont add to regions map so that enable table handler can handle it. But > as nothing is added to regions map we think it as a clean cluster start up. > Will come up with a patch tomorrow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467589#comment-13467589 ] Flavio Junqueira commented on HBASE-5937: - I'll add an issue to the umbrella jira to revisit javadocs. I don't see how TestHCM could possibly have failed because of this patch. I'll fix the findbugs warnings. > Refactor HLog into an interface. > > > Key: HBASE-5937 > URL: https://issues.apache.org/jira/browse/HBASE-5937 > Project: HBase > Issue Type: Sub-task >Reporter: Li Pi >Assignee: Flavio Junqueira >Priority: Minor > Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch, > HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, > HBASE-5937.v2.patch, > org.apache.hadoop.hbase.client.TestMultiParallel-output.txt > > > What the summary says. Create HLog interface. Make current implementation use > it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6900) RegionScanner.reseek() creates NPE when a flush or compaction happens before the reseek.
[ https://issues.apache.org/jira/browse/HBASE-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467590#comment-13467590 ] ramkrishna.s.vasudevan commented on HBASE-6900: --- [~lhofhansl] Can you take a look at this? So that i can rebase the patch based on your suggestion. > RegionScanner.reseek() creates NPE when a flush or compaction happens before > the reseek. > > > Key: HBASE-6900 > URL: https://issues.apache.org/jira/browse/HBASE-6900 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Attachments: HBASE-6900_1.patch, HBASE-6900.patch > > > HBASE-5520 introduced reseek() on the RegionScanner. > Now when a scanner is created we have the StoreScanner heap. After this if a > flush or compaction happens parallely all the StoreScannerObservers are > cleared so that whenever a new next() call happens we tend to recreate the > scanner based on the latest store files. > The reseek() in StoreScanner expects the heap not to be null because always > reseek would be called from next() > {code} > public synchronized boolean reseek(KeyValue kv) throws IOException { > //Heap cannot be null, because this is only called from next() which > //guarantees that heap will never be null before this call. > if (explicitColumnQuery && lazySeekEnabledGlobally) { > return heap.requestSeek(kv, true, useRowColBloom); > } else { > return heap.reseek(kv); > } > } > {code} > Now when we call RegionScanner.reseek() directly using CPs we tend to get a > NPE. In our case it happened when a major compaction was going on. I will > also attach a testcase to show the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6317) Master clean start up and Partially enabled tables make region assignment inconsistent.
[ https://issues.apache.org/jira/browse/HBASE-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467601#comment-13467601 ] Hudson commented on HBASE-6317: --- Integrated in HBase-TRUNK #3405 (See [https://builds.apache.org/job/HBase-TRUNK/3405/]) HBASE-6317 Master clean start up and Partially enabled tables make region assignment inconsistent (RajeshBabu) (Revision 1392802) Result = FAILURE ramkrishna : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/EnableTableHandler.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java > Master clean start up and Partially enabled tables make region assignment > inconsistent. > --- > > Key: HBASE-6317 > URL: https://issues.apache.org/jira/browse/HBASE-6317 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: rajeshbabu > Fix For: 0.92.3, 0.94.3, 0.96.0 > > Attachments: HBASE-6317_94_3.patch, HBASE-6317_94.patch, > HBASE-6317_trunk_2.patch, HBASE-6317_trunk_3.patch, HBASE-6317_trunk_4.patch, > HBASE-6317_trunk_5.patch > > > If we have a table in partially enabled state (ENABLING) then on HMaster > restart we treat it as a clean cluster start up and do a bulk assign. > Currently in 0.94 bulk assign will not handle ALREADY_OPENED scenarios and it > leads to region assignment problems. Analysing more on this we found that we > have better way to handle these scenarios. > {code} > if (false == checkIfRegionBelongsToDisabled(regionInfo) > && false == checkIfRegionsBelongsToEnabling(regionInfo)) { > synchronized (this.regions) { > regions.put(regionInfo, regionLocation); > addToServers(regionLocation, regionInfo); > } > {code} > We dont add to regions map so that enable table handler can handle it. But > as nothing is added to regions map we think it as a clean cluster start up. > Will come up with a patch tomorrow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6317) Master clean start up and Partially enabled tables make region assignment inconsistent.
[ https://issues.apache.org/jira/browse/HBASE-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467624#comment-13467624 ] Hudson commented on HBASE-6317: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #203 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/203/]) HBASE-6317 Master clean start up and Partially enabled tables make region assignment inconsistent (RajeshBabu) (Revision 1392802) Result = FAILURE ramkrishna : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/EnableTableHandler.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java > Master clean start up and Partially enabled tables make region assignment > inconsistent. > --- > > Key: HBASE-6317 > URL: https://issues.apache.org/jira/browse/HBASE-6317 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: rajeshbabu > Fix For: 0.92.3, 0.94.3, 0.96.0 > > Attachments: HBASE-6317_94_3.patch, HBASE-6317_94.patch, > HBASE-6317_trunk_2.patch, HBASE-6317_trunk_3.patch, HBASE-6317_trunk_4.patch, > HBASE-6317_trunk_5.patch > > > If we have a table in partially enabled state (ENABLING) then on HMaster > restart we treat it as a clean cluster start up and do a bulk assign. > Currently in 0.94 bulk assign will not handle ALREADY_OPENED scenarios and it > leads to region assignment problems. Analysing more on this we found that we > have better way to handle these scenarios. > {code} > if (false == checkIfRegionBelongsToDisabled(regionInfo) > && false == checkIfRegionsBelongsToEnabling(regionInfo)) { > synchronized (this.regions) { > regions.put(regionInfo, regionLocation); > addToServers(regionLocation, regionInfo); > } > {code} > We dont add to regions map so that enable table handler can handle it. But > as nothing is added to regions map we think it as a clean cluster start up. > Will come up with a patch tomorrow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6918) Debugging to help figure what is different up on jenkins when TestHeapSize runs
[ https://issues.apache.org/jira/browse/HBASE-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467623#comment-13467623 ] Hudson commented on HBASE-6918: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #203 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/203/]) HBASE-6918 Debugging to help figure what is different up on jenkins when TestHeapSize runs (Revision 1392753) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java > Debugging to help figure what is different up on jenkins when TestHeapSize > runs > --- > > Key: HBASE-6918 > URL: https://issues.apache.org/jira/browse/HBASE-6918 > Project: HBase > Issue Type: Bug > Components: test >Affects Versions: 0.96.0 >Reporter: stack >Assignee: stack > Fix For: 0.96.0 > > Attachments: 6918.txt > > > TestHeapSize will pass locally then fail up on jenkins. Add debugging to the > test that prints detail on environment and jvm in particular to can figure > what is different between two environments (it used to be that the d32 flag > was needed to mimic jenkins but now we are jdk7, the d32 flag doesn't work w/ > my oracle jvm... let me see what jenkins has). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6915) String and ConcurrentHashMap sizes change on jdk7; makes TestHeapSize fail
[ https://issues.apache.org/jira/browse/HBASE-6915?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467622#comment-13467622 ] Hudson commented on HBASE-6915: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #203 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/203/]) HBASE-6915 String and ConcurrentHashMap sizes change on jdk7; makes TestHeapSize fail (Revision 1392731) Result = FAILURE stack : Files : * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/ClassSize.java > String and ConcurrentHashMap sizes change on jdk7; makes TestHeapSize fail > -- > > Key: HBASE-6915 > URL: https://issues.apache.org/jira/browse/HBASE-6915 > Project: HBase > Issue Type: Bug >Reporter: stack >Assignee: stack > Fix For: 0.96.0 > > Attachments: jdk7.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6917) Trunk jdk7 build broke because we moved to zk 3.4.4
[ https://issues.apache.org/jira/browse/HBASE-6917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467625#comment-13467625 ] Hudson commented on HBASE-6917: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #203 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/203/]) HBASE-6917 Trunk jdk7 build broke because we moved to zk 3.4.4 (Revision 1392746) Result = FAILURE stack : Files : * /hbase/trunk/pom.xml > Trunk jdk7 build broke because we moved to zk 3.4.4 > --- > > Key: HBASE-6917 > URL: https://issues.apache.org/jira/browse/HBASE-6917 > Project: HBase > Issue Type: Bug > Components: test >Reporter: stack >Assignee: stack > Fix For: 0.96.0 > > Attachments: 6917.txt > > > Chatted w/ Mahadev and he confirmed issues running 3.4.4 w/ jdk7. Will be > fixed in zk3.4.5. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HBASE-6317) Master clean start up and Partially enabled tables make region assignment inconsistent.
[ https://issues.apache.org/jira/browse/HBASE-6317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan reopened HBASE-6317: --- Not committed to other branches other than trunk.. Will resolve commiting in other branches also. > Master clean start up and Partially enabled tables make region assignment > inconsistent. > --- > > Key: HBASE-6317 > URL: https://issues.apache.org/jira/browse/HBASE-6317 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: rajeshbabu > Fix For: 0.92.3, 0.94.3, 0.96.0 > > Attachments: HBASE-6317_94_3.patch, HBASE-6317_94.patch, > HBASE-6317_trunk_2.patch, HBASE-6317_trunk_3.patch, HBASE-6317_trunk_4.patch, > HBASE-6317_trunk_5.patch > > > If we have a table in partially enabled state (ENABLING) then on HMaster > restart we treat it as a clean cluster start up and do a bulk assign. > Currently in 0.94 bulk assign will not handle ALREADY_OPENED scenarios and it > leads to region assignment problems. Analysing more on this we found that we > have better way to handle these scenarios. > {code} > if (false == checkIfRegionBelongsToDisabled(regionInfo) > && false == checkIfRegionsBelongsToEnabling(regionInfo)) { > synchronized (this.regions) { > regions.put(regionInfo, regionLocation); > addToServers(regionLocation, regionInfo); > } > {code} > We dont add to regions map so that enable table handler can handle it. But > as nothing is added to regions map we think it as a clean cluster start up. > Will come up with a patch tomorrow. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flavio Junqueira updated HBASE-5937: Status: Open (was: Patch Available) > Refactor HLog into an interface. > > > Key: HBASE-5937 > URL: https://issues.apache.org/jira/browse/HBASE-5937 > Project: HBase > Issue Type: Sub-task >Reporter: Li Pi >Assignee: Flavio Junqueira >Priority: Minor > Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch, > HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, > HBASE-5937.v2.patch, HBASE-5937.v3.patch, > org.apache.hadoop.hbase.client.TestMultiParallel-output.txt > > > What the summary says. Create HLog interface. Make current implementation use > it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flavio Junqueira updated HBASE-5937: Attachment: HBASE-5937.v3.patch > Refactor HLog into an interface. > > > Key: HBASE-5937 > URL: https://issues.apache.org/jira/browse/HBASE-5937 > Project: HBase > Issue Type: Sub-task >Reporter: Li Pi >Assignee: Flavio Junqueira >Priority: Minor > Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch, > HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, > HBASE-5937.v2.patch, HBASE-5937.v3.patch, > org.apache.hadoop.hbase.client.TestMultiParallel-output.txt > > > What the summary says. Create HLog interface. Make current implementation use > it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Flavio Junqueira updated HBASE-5937: Status: Patch Available (was: Open) I have checked the new findbugs warning and they haven't been introduced by any of the changes I made. They appear as new because I moved code around. I have looked into finxing them, though. I fixed one about a volatile variable by making it an AtomicInteger. The warning was about incrementing a volatile variable. There are two warnings about static variables being set to a mutable array in HLog. This is due to the use of Bytes.toBytes(). I didn't want to change that, so I left as is. There two other warnings related to this code excerpt: {noformat} synchronized (closeLogSyncer) { closeLogSyncer.set(true); closeLogSyncer.notifyAll(); } {noformat} where closeLogSyncer is an AtomicBoolean, so the warning is saying that we shouldn't synchronize. It sounds right to me, but I wanted to make sure that it is correct. I can create a jira to fix it if I get a confirmation. It is a pretty simple fix. > Refactor HLog into an interface. > > > Key: HBASE-5937 > URL: https://issues.apache.org/jira/browse/HBASE-5937 > Project: HBase > Issue Type: Sub-task >Reporter: Li Pi >Assignee: Flavio Junqueira >Priority: Minor > Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch, > HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, > HBASE-5937.v2.patch, HBASE-5937.v3.patch, > org.apache.hadoop.hbase.client.TestMultiParallel-output.txt > > > What the summary says. Create HLog interface. Make current implementation use > it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-5582) "No HServerInfo found for" should be a WARNING message
[ https://issues.apache.org/jira/browse/HBASE-5582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kevin Odell reassigned HBASE-5582: -- Assignee: Kevin Odell > "No HServerInfo found for" should be a WARNING message > -- > > Key: HBASE-5582 > URL: https://issues.apache.org/jira/browse/HBASE-5582 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 0.90.4 >Reporter: Shrijeet Paliwal >Assignee: Kevin Odell >Priority: Trivial > Labels: newbie > > The message from RegionServerTracker "No HServerInfo found for..." is easy to > miss. It should not be INFO. > From irc chat > {noformat} > jdcryans > JohnP789: can you grep for "No HServerInfo found for" in that log? > jdcryans > wait I see it > jdcryans > ok there's your problem > shrijeet_ > Yes it is there > shrijeet_ > jdcryans: it should be INFO, why? > jdcryans > it shouldn't be INFO, it's so easy to miss > jdcryans > it's not the first time we have to look super closely to figure this one out > shrijeet_ > yes , I will file a jira > jdcryans > in any case it's a mismatch in that machine's DNS config > shrijeet_ > anyways JohnP789 is waiting :) go on > JohnP789 > haha! > JohnP789 > yes... ??? :-) > jdcryans > the master is expecting a RS called > "localhost.localdomain,53875,1328924863478" > 17:26 jdcryans > but the RS calls itself "localhost,53875,1328924863478" > {noformat} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467813#comment-13467813 ] Hadoop QA commented on HBASE-5937: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12547389/HBASE-5937.v3.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 72 new or modified tests. {color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop 2.0 profile. {color:red}-1 javadoc{color}. The javadoc tool appears to have generated 157 warning messages. {color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings. {color:red}-1 findbugs{color}. The patch appears to introduce 5 new Findbugs (version 1.3.9) warnings. {color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings. {color:green}+1 core tests{color}. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2985//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2985//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2985//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2985//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2985//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2985//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2985//console This message is automatically generated. > Refactor HLog into an interface. > > > Key: HBASE-5937 > URL: https://issues.apache.org/jira/browse/HBASE-5937 > Project: HBase > Issue Type: Sub-task >Reporter: Li Pi >Assignee: Flavio Junqueira >Priority: Minor > Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch, > HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, > HBASE-5937.v2.patch, HBASE-5937.v3.patch, > org.apache.hadoop.hbase.client.TestMultiParallel-output.txt > > > What the summary says. Create HLog interface. Make current implementation use > it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467829#comment-13467829 ] Ted Yu commented on HBASE-5937: --- Nice to see a green QA run. {code} -@InterfaceAudience.Private -public class HLog implements Syncable { - static final Log LOG = LogFactory.getLog(HLog.class); - public static final byte [] METAFAMILY = Bytes.toBytes("METAFAMILY"); - static final byte [] METAROW = Bytes.toBytes("METAROW"); +public interface HLog { {code} Please restore annotation for HLog. I suggest using Public + Evolving. > Refactor HLog into an interface. > > > Key: HBASE-5937 > URL: https://issues.apache.org/jira/browse/HBASE-5937 > Project: HBase > Issue Type: Sub-task >Reporter: Li Pi >Assignee: Flavio Junqueira >Priority: Minor > Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch, > HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, > HBASE-5937.v2.patch, HBASE-5937.v3.patch, > org.apache.hadoop.hbase.client.TestMultiParallel-output.txt > > > What the summary says. Create HLog interface. Make current implementation use > it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6919) Remove unnecessary cast from Bytes.readVLong
James Taylor created HBASE-6919: --- Summary: Remove unnecessary cast from Bytes.readVLong Key: HBASE-6919 URL: https://issues.apache.org/jira/browse/HBASE-6919 Project: HBase Issue Type: Bug Reporter: James Taylor Priority: Minor Remove the throws IOException so that caller doesn't have to catch and ignore. public static long readVLong(final byte [] buffer, final int offset) throws IOException Also, add public static int readVInt(final byte [] buffer, final int offset) throws IOException { return (int)readVLong(buffer,offset); } and these are useful too: /** * Put long as variable length encoded number at the offset in * the result byte array. * @param vint Integer to make a vint of. * @param result buffer to put vint into * @return Vint length in bytes of vint */ public static int vintToBytes(byte[] result, int offset, final long vint) { long i = vint; if (i >= -112 && i <= 127) { result[offset] = (byte) i; return 1; } int len = -112; if (i < 0) { i ^= -1L; // take one's complement' len = -120; } long tmp = i; while (tmp != 0) { tmp = tmp >> 8; len--; } result[offset++] = (byte) len; len = (len < -120) ? -(len + 120) : -(len + 112); for (int idx = len; idx != 0; idx--) { int shiftbits = (idx - 1) * 8; long mask = 0xFFL << shiftbits; result[offset++] = (byte)((i & mask) >> shiftbits); } return len + 1; } /** * Decode a vint from the buffer pointed at to by ptr and * increment the offset of the ptr by the length of the * vint. * @param ptr a pointer to a byte array buffer * @return the decoded vint value as an int */ public static int vintFromBytes(ImmutableBytesWritable ptr) { return (int) vlongFromBytes(ptr); } /** * Decode a vint from the buffer pointed at to by ptr and * increment the offset of the ptr by the length of the * vint. * @param ptr a pointer to a byte array buffer * @return the decoded vint value as a long */ public static long vlongFromBytes(ImmutableBytesWritable ptr) { final byte [] buffer = ptr.get(); final int offset = ptr.getOffset(); byte firstByte = buffer[offset]; int len = WritableUtils.decodeVIntSize(firstByte); if (len == 1) { ptr.set(buffer, offset+1, ptr.getLength()); return firstByte; } long i = 0; for (int idx = 0; idx < len-1; idx++) { byte b = buffer[offset + 1 + idx]; i = i << 8; i = i | (b & 0xFF); } ptr.set(buffer, offset+len, ptr.getLength()); return (WritableUtils.isNegativeVInt(firstByte) ? ~i : i); } -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6914) Scans/Gets/Mutations don't give a good error if the table is disabled.
[ https://issues.apache.org/jira/browse/HBASE-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-6914: - Component/s: Client > Scans/Gets/Mutations don't give a good error if the table is disabled. > -- > > Key: HBASE-6914 > URL: https://issues.apache.org/jira/browse/HBASE-6914 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 0.92.3, 0.94.3, 0.96.0 > > Attachments: HBASE-6914-0.patch, HBASE-6914-1.patch, > HBASE-6914-2.patch, HBASE-6914-3.patch > > > Scan a table that is disabled will have the client retry multiple times and > then will error out with NotServingRegionException. If the table is disabled > there's no need to re-try and the message should be more explicit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6914) Scans/Gets/Mutations don't give a good error if the table is disabled.
[ https://issues.apache.org/jira/browse/HBASE-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467849#comment-13467849 ] stack commented on HBASE-6914: -- +1 on commit again. > Scans/Gets/Mutations don't give a good error if the table is disabled. > -- > > Key: HBASE-6914 > URL: https://issues.apache.org/jira/browse/HBASE-6914 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 0.92.3, 0.94.3, 0.96.0 > > Attachments: HBASE-6914-0.patch, HBASE-6914-1.patch, > HBASE-6914-2.patch, HBASE-6914-3.patch > > > Scan a table that is disabled will have the client retry multiple times and > then will error out with NotServingRegionException. If the table is disabled > there's no need to re-try and the message should be more explicit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6914) Scans/Gets/Mutations don't give a good error if the table is disabled.
[ https://issues.apache.org/jira/browse/HBASE-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467851#comment-13467851 ] stack commented on HBASE-6914: -- Thanks for digging in on the failed test in spite of my +1 > Scans/Gets/Mutations don't give a good error if the table is disabled. > -- > > Key: HBASE-6914 > URL: https://issues.apache.org/jira/browse/HBASE-6914 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 0.92.3, 0.94.3, 0.96.0 > > Attachments: HBASE-6914-0.patch, HBASE-6914-1.patch, > HBASE-6914-2.patch, HBASE-6914-3.patch > > > Scan a table that is disabled will have the client retry multiple times and > then will error out with NotServingRegionException. If the table is disabled > there's no need to re-try and the message should be more explicit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6919) Remove unnecessary cast from Bytes.readVLong
[ https://issues.apache.org/jira/browse/HBASE-6919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467852#comment-13467852 ] stack commented on HBASE-6919: -- Make a patch James and we'll commit. > Remove unnecessary cast from Bytes.readVLong > > > Key: HBASE-6919 > URL: https://issues.apache.org/jira/browse/HBASE-6919 > Project: HBase > Issue Type: Bug >Reporter: James Taylor >Priority: Minor > > Remove the throws IOException so that caller doesn't have to catch and ignore. > public static long readVLong(final byte [] buffer, final int offset) > throws IOException > Also, add > public static int readVInt(final byte [] buffer, final int offset) > throws IOException { > return (int)readVLong(buffer,offset); > } > and these are useful too: > /** > * Put long as variable length encoded number at the offset in > * the result byte array. > * @param vint Integer to make a vint of. > * @param result buffer to put vint into > * @return Vint length in bytes of vint > */ > public static int vintToBytes(byte[] result, int offset, final long vint) > { > long i = vint; > if (i >= -112 && i <= 127) { > result[offset] = (byte) i; > return 1; > } > int len = -112; > if (i < 0) { > i ^= -1L; // take one's complement' > len = -120; > } > long tmp = i; > while (tmp != 0) { > tmp = tmp >> 8; > len--; > } > result[offset++] = (byte) len; > len = (len < -120) ? -(len + 120) : -(len + 112); > for (int idx = len; idx != 0; idx--) { > int shiftbits = (idx - 1) * 8; > long mask = 0xFFL << shiftbits; > result[offset++] = (byte)((i & mask) >> shiftbits); > } > return len + 1; > } > /** > * Decode a vint from the buffer pointed at to by ptr and > * increment the offset of the ptr by the length of the > * vint. > * @param ptr a pointer to a byte array buffer > * @return the decoded vint value as an int > */ > public static int vintFromBytes(ImmutableBytesWritable ptr) { > return (int) vlongFromBytes(ptr); > } > > /** > * Decode a vint from the buffer pointed at to by ptr and > * increment the offset of the ptr by the length of the > * vint. > * @param ptr a pointer to a byte array buffer > * @return the decoded vint value as a long > */ > public static long vlongFromBytes(ImmutableBytesWritable ptr) { > final byte [] buffer = ptr.get(); > final int offset = ptr.getOffset(); > byte firstByte = buffer[offset]; > int len = WritableUtils.decodeVIntSize(firstByte); > if (len == 1) { > ptr.set(buffer, offset+1, ptr.getLength()); > return firstByte; > } > long i = 0; > for (int idx = 0; idx < len-1; idx++) { > byte b = buffer[offset + 1 + idx]; > i = i << 8; > i = i | (b & 0xFF); > } > ptr.set(buffer, offset+len, ptr.getLength()); > return (WritableUtils.isNegativeVInt(firstByte) ? ~i : i); > } -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467855#comment-13467855 ] stack commented on HBASE-5937: -- I can restore the annotation on commit. Will leave it private for now. > Refactor HLog into an interface. > > > Key: HBASE-5937 > URL: https://issues.apache.org/jira/browse/HBASE-5937 > Project: HBase > Issue Type: Sub-task >Reporter: Li Pi >Assignee: Flavio Junqueira >Priority: Minor > Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch, > HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, > HBASE-5937.v2.patch, HBASE-5937.v3.patch, > org.apache.hadoop.hbase.client.TestMultiParallel-output.txt > > > What the summary says. Create HLog interface. Make current implementation use > it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6920) On timeout connecting to master, client can get stuck and never make progress
Gregory Chanan created HBASE-6920: - Summary: On timeout connecting to master, client can get stuck and never make progress Key: HBASE-6920 URL: https://issues.apache.org/jira/browse/HBASE-6920 Project: HBase Issue Type: Bug Affects Versions: 0.94.2 Reporter: Gregory Chanan Assignee: Gregory Chanan Priority: Critical HBASE-5058 appears to have introduced an issue where a timeout in HConnection.getMaster() can cause the client to never be able to connect to the master. So, for example, an HBaseAdmin object can never successfully be initialized. The issue is here: {code} if (tryMaster.isMasterRunning()) { this.master = tryMaster; this.masterLock.notifyAll(); break; } {code} If isMasterRunning times out, it throws an UndeclaredThrowableException, which is already not ideal, because it can be returned to the application. But if the first call to getMaster succeeds, it will set masterChecked = true, which makes us never try to reconnect; that is, we will set this.master = null and just throw MasterNotRunningExceptions, without even trying to connect. I tried out a 94 client (actually a 92 client with some 94 patches) on a cluster with some network issues, and it would constantly get stuck as described above. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6920) On timeout connecting to master, client can get stuck and never make progress
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated HBASE-6920: -- Attachment: HBASE-6920.patch * Attached HBASE-6920.patch * Here is a patch for this issue and a test. The test is more complicated than I would like; I needed to introduce some functions / change some access controls to be able to use my own RPC Invoker. Any suggestions would be welcome. > On timeout connecting to master, client can get stuck and never make progress > - > > Key: HBASE-6920 > URL: https://issues.apache.org/jira/browse/HBASE-6920 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2 >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Critical > Attachments: HBASE-6920.patch > > > HBASE-5058 appears to have introduced an issue where a timeout in > HConnection.getMaster() can cause the client to never be able to connect to > the master. So, for example, an HBaseAdmin object can never successfully be > initialized. > The issue is here: > {code} > if (tryMaster.isMasterRunning()) { > this.master = tryMaster; > this.masterLock.notifyAll(); > break; > } > {code} > If isMasterRunning times out, it throws an UndeclaredThrowableException, > which is already not ideal, because it can be returned to the application. > But if the first call to getMaster succeeds, it will set masterChecked = > true, which makes us never try to reconnect; that is, we will set this.master > = null and just throw MasterNotRunningExceptions, without even trying to > connect. > I tried out a 94 client (actually a 92 client with some 94 patches) on a > cluster with some network issues, and it would constantly get stuck as > described above. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6920) On timeout connecting to master, client can get stuck and never make progress
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated HBASE-6920: -- Status: Patch Available (was: Open) > On timeout connecting to master, client can get stuck and never make progress > - > > Key: HBASE-6920 > URL: https://issues.apache.org/jira/browse/HBASE-6920 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2 >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Critical > Attachments: HBASE-6920.patch > > > HBASE-5058 appears to have introduced an issue where a timeout in > HConnection.getMaster() can cause the client to never be able to connect to > the master. So, for example, an HBaseAdmin object can never successfully be > initialized. > The issue is here: > {code} > if (tryMaster.isMasterRunning()) { > this.master = tryMaster; > this.masterLock.notifyAll(); > break; > } > {code} > If isMasterRunning times out, it throws an UndeclaredThrowableException, > which is already not ideal, because it can be returned to the application. > But if the first call to getMaster succeeds, it will set masterChecked = > true, which makes us never try to reconnect; that is, we will set this.master > = null and just throw MasterNotRunningExceptions, without even trying to > connect. > I tried out a 94 client (actually a 92 client with some 94 patches) on a > cluster with some network issues, and it would constantly get stuck as > described above. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master, client can get stuck and never make progress
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467873#comment-13467873 ] Gregory Chanan commented on HBASE-6920: --- Some other points: * I have not tried the client app that failed in the description with a patched client yet. I will do that soon. * This seems like a pretty serious issue (I literally could not get my app to work with a 94 client), though it's not clear how often users will see it, since it only happens on errors. It's also not new with 0.94.2 so I don't know if it needs to sink the RC. Any opinions? > On timeout connecting to master, client can get stuck and never make progress > - > > Key: HBASE-6920 > URL: https://issues.apache.org/jira/browse/HBASE-6920 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2 >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Critical > Attachments: HBASE-6920.patch > > > HBASE-5058 appears to have introduced an issue where a timeout in > HConnection.getMaster() can cause the client to never be able to connect to > the master. So, for example, an HBaseAdmin object can never successfully be > initialized. > The issue is here: > {code} > if (tryMaster.isMasterRunning()) { > this.master = tryMaster; > this.masterLock.notifyAll(); > break; > } > {code} > If isMasterRunning times out, it throws an UndeclaredThrowableException, > which is already not ideal, because it can be returned to the application. > But if the first call to getMaster succeeds, it will set masterChecked = > true, which makes us never try to reconnect; that is, we will set this.master > = null and just throw MasterNotRunningExceptions, without even trying to > connect. > I tried out a 94 client (actually a 92 client with some 94 patches) on a > cluster with some network issues, and it would constantly get stuck as > described above. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master, client can get stuck and never make progress
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467874#comment-13467874 ] Hadoop QA commented on HBASE-6920: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12547405/HBASE-6920.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified tests. {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2986//console This message is automatically generated. > On timeout connecting to master, client can get stuck and never make progress > - > > Key: HBASE-6920 > URL: https://issues.apache.org/jira/browse/HBASE-6920 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2 >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Critical > Attachments: HBASE-6920.patch > > > HBASE-5058 appears to have introduced an issue where a timeout in > HConnection.getMaster() can cause the client to never be able to connect to > the master. So, for example, an HBaseAdmin object can never successfully be > initialized. > The issue is here: > {code} > if (tryMaster.isMasterRunning()) { > this.master = tryMaster; > this.masterLock.notifyAll(); > break; > } > {code} > If isMasterRunning times out, it throws an UndeclaredThrowableException, > which is already not ideal, because it can be returned to the application. > But if the first call to getMaster succeeds, it will set masterChecked = > true, which makes us never try to reconnect; that is, we will set this.master > = null and just throw MasterNotRunningExceptions, without even trying to > connect. > I tried out a 94 client (actually a 92 client with some 94 patches) on a > cluster with some network issues, and it would constantly get stuck as > described above. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6900) RegionScanner.reseek() creates NPE when a flush or compaction happens before the reseek.
[ https://issues.apache.org/jira/browse/HBASE-6900?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467881#comment-13467881 ] Lars Hofhansl commented on HBASE-6900: -- [~ram_krish] Sorry for the delay. {code} {+if (!checkReseek() && heap != null) {} {code} Do we need to call checkReseek here? This is a somewhat expensive operation and we're now doing it for every reseek issued to a StoreScanner. Could we just check for {{heap == null}} in the beginning of the method and return {{false}} if so? > RegionScanner.reseek() creates NPE when a flush or compaction happens before > the reseek. > > > Key: HBASE-6900 > URL: https://issues.apache.org/jira/browse/HBASE-6900 > Project: HBase > Issue Type: Bug >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Attachments: HBASE-6900_1.patch, HBASE-6900.patch > > > HBASE-5520 introduced reseek() on the RegionScanner. > Now when a scanner is created we have the StoreScanner heap. After this if a > flush or compaction happens parallely all the StoreScannerObservers are > cleared so that whenever a new next() call happens we tend to recreate the > scanner based on the latest store files. > The reseek() in StoreScanner expects the heap not to be null because always > reseek would be called from next() > {code} > public synchronized boolean reseek(KeyValue kv) throws IOException { > //Heap cannot be null, because this is only called from next() which > //guarantees that heap will never be null before this call. > if (explicitColumnQuery && lazySeekEnabledGlobally) { > return heap.requestSeek(kv, true, useRowColBloom); > } else { > return heap.reseek(kv); > } > } > {code} > Now when we call RegionScanner.reseek() directly using CPs we tend to get a > NPE. In our case it happened when a major compaction was going on. I will > also attach a testcase to show the problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master, client can get stuck and never make progress
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467885#comment-13467885 ] Ted Yu commented on HBASE-6920: --- Great finding, Gregory. {code} + * @return true if should retry + */ +private boolean logGetMasterAttemptFailure(int tries, Exception e) { {code} Please rename the method to reflect the fact that it returns whether we should retry. {code} + LOG.info("getMaster attempt " + tries + " of " + numRetries + +" failed; retrying after sleep of " + +ConnectionUtils.getPauseTime(this.pause, tries), e); {code} Should the above log be at DEBUG level ? Comparing your code and the original, it seems that the return value of true actually means that we shouldn't retry. > On timeout connecting to master, client can get stuck and never make progress > - > > Key: HBASE-6920 > URL: https://issues.apache.org/jira/browse/HBASE-6920 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2 >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Critical > Attachments: HBASE-6920.patch > > > HBASE-5058 appears to have introduced an issue where a timeout in > HConnection.getMaster() can cause the client to never be able to connect to > the master. So, for example, an HBaseAdmin object can never successfully be > initialized. > The issue is here: > {code} > if (tryMaster.isMasterRunning()) { > this.master = tryMaster; > this.masterLock.notifyAll(); > break; > } > {code} > If isMasterRunning times out, it throws an UndeclaredThrowableException, > which is already not ideal, because it can be returned to the application. > But if the first call to getMaster succeeds, it will set masterChecked = > true, which makes us never try to reconnect; that is, we will set this.master > = null and just throw MasterNotRunningExceptions, without even trying to > connect. > I tried out a 94 client (actually a 92 client with some 94 patches) on a > cluster with some network issues, and it would constantly get stuck as > described above. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master, client can get stuck and never make progress
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467889#comment-13467889 ] Gregory Chanan commented on HBASE-6920: --- I don't have any opinion on the DEBUG level; if you think that's better I'll change it. Good catch on the return value, the javadoc is incorrect. > On timeout connecting to master, client can get stuck and never make progress > - > > Key: HBASE-6920 > URL: https://issues.apache.org/jira/browse/HBASE-6920 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2 >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Critical > Attachments: HBASE-6920.patch > > > HBASE-5058 appears to have introduced an issue where a timeout in > HConnection.getMaster() can cause the client to never be able to connect to > the master. So, for example, an HBaseAdmin object can never successfully be > initialized. > The issue is here: > {code} > if (tryMaster.isMasterRunning()) { > this.master = tryMaster; > this.masterLock.notifyAll(); > break; > } > {code} > If isMasterRunning times out, it throws an UndeclaredThrowableException, > which is already not ideal, because it can be returned to the application. > But if the first call to getMaster succeeds, it will set masterChecked = > true, which makes us never try to reconnect; that is, we will set this.master > = null and just throw MasterNotRunningExceptions, without even trying to > connect. > I tried out a 94 client (actually a 92 client with some 94 patches) on a > cluster with some network issues, and it would constantly get stuck as > described above. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master, client can get stuck and never make progress
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467891#comment-13467891 ] Ted Yu commented on HBASE-6920: --- {code} + /** Construct a client-side proxy object, specifying an InvocationHandler for testing purposes */ + VersionedProtocol getProxy(Class protocol, {code} Name the above method getProxyForTesting ? {code} + * RpcEngine that random throws a SocketTimeoutEngine for testing. {code} 'random' -> 'randomly' Please correct the spelling for SocketTimeoutEngine > On timeout connecting to master, client can get stuck and never make progress > - > > Key: HBASE-6920 > URL: https://issues.apache.org/jira/browse/HBASE-6920 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2 >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Critical > Attachments: HBASE-6920.patch > > > HBASE-5058 appears to have introduced an issue where a timeout in > HConnection.getMaster() can cause the client to never be able to connect to > the master. So, for example, an HBaseAdmin object can never successfully be > initialized. > The issue is here: > {code} > if (tryMaster.isMasterRunning()) { > this.master = tryMaster; > this.masterLock.notifyAll(); > break; > } > {code} > If isMasterRunning times out, it throws an UndeclaredThrowableException, > which is already not ideal, because it can be returned to the application. > But if the first call to getMaster succeeds, it will set masterChecked = > true, which makes us never try to reconnect; that is, we will set this.master > = null and just throw MasterNotRunningExceptions, without even trying to > connect. > I tried out a 94 client (actually a 92 client with some 94 patches) on a > cluster with some network issues, and it would constantly get stuck as > described above. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master, client can get stuck and never make progress
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467896#comment-13467896 ] Lars Hofhansl commented on HBASE-6920: -- Patch looks good. I don't quite grok the changes in HBaseRPC, RpcEngine, and WritableRpcEngine. Are those needed for this to work? > On timeout connecting to master, client can get stuck and never make progress > - > > Key: HBASE-6920 > URL: https://issues.apache.org/jira/browse/HBASE-6920 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2 >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Critical > Attachments: HBASE-6920.patch > > > HBASE-5058 appears to have introduced an issue where a timeout in > HConnection.getMaster() can cause the client to never be able to connect to > the master. So, for example, an HBaseAdmin object can never successfully be > initialized. > The issue is here: > {code} > if (tryMaster.isMasterRunning()) { > this.master = tryMaster; > this.masterLock.notifyAll(); > break; > } > {code} > If isMasterRunning times out, it throws an UndeclaredThrowableException, > which is already not ideal, because it can be returned to the application. > But if the first call to getMaster succeeds, it will set masterChecked = > true, which makes us never try to reconnect; that is, we will set this.master > = null and just throw MasterNotRunningExceptions, without even trying to > connect. > I tried out a 94 client (actually a 92 client with some 94 patches) on a > cluster with some network issues, and it would constantly get stuck as > described above. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master, client can get stuck and never make progress
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467898#comment-13467898 ] Lars Hofhansl commented on HBASE-6920: -- Also from the explanation I am not quite sure if this a problem in 0.92 as well. > On timeout connecting to master, client can get stuck and never make progress > - > > Key: HBASE-6920 > URL: https://issues.apache.org/jira/browse/HBASE-6920 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2 >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Critical > Attachments: HBASE-6920.patch > > > HBASE-5058 appears to have introduced an issue where a timeout in > HConnection.getMaster() can cause the client to never be able to connect to > the master. So, for example, an HBaseAdmin object can never successfully be > initialized. > The issue is here: > {code} > if (tryMaster.isMasterRunning()) { > this.master = tryMaster; > this.masterLock.notifyAll(); > break; > } > {code} > If isMasterRunning times out, it throws an UndeclaredThrowableException, > which is already not ideal, because it can be returned to the application. > But if the first call to getMaster succeeds, it will set masterChecked = > true, which makes us never try to reconnect; that is, we will set this.master > = null and just throw MasterNotRunningExceptions, without even trying to > connect. > I tried out a 94 client (actually a 92 client with some 94 patches) on a > cluster with some network issues, and it would constantly get stuck as > described above. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6921) String and ConcurrentHashMap sizes change on jdk7; makes TestHeapSize fail; second attempt
stack created HBASE-6921: Summary: String and ConcurrentHashMap sizes change on jdk7; makes TestHeapSize fail; second attempt Key: HBASE-6921 URL: https://issues.apache.org/jira/browse/HBASE-6921 Project: HBase Issue Type: Bug Components: test Reporter: stack Assignee: stack Fix For: 0.96.0 HBASE-6918 added debug. Jenkins runs jdk7 23.0-b21. Locally I have 23.6-b03. It looks like ConcurrentHashMap is a different size in these two jvm's (To be confirmed). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master, client can get stuck and never make progress
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467903#comment-13467903 ] Gregory Chanan commented on HBASE-6920: --- I think this is 0.94 only -- at least HBASE-5058 is only in 0.94. > On timeout connecting to master, client can get stuck and never make progress > - > > Key: HBASE-6920 > URL: https://issues.apache.org/jira/browse/HBASE-6920 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2 >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Critical > Attachments: HBASE-6920.patch > > > HBASE-5058 appears to have introduced an issue where a timeout in > HConnection.getMaster() can cause the client to never be able to connect to > the master. So, for example, an HBaseAdmin object can never successfully be > initialized. > The issue is here: > {code} > if (tryMaster.isMasterRunning()) { > this.master = tryMaster; > this.masterLock.notifyAll(); > break; > } > {code} > If isMasterRunning times out, it throws an UndeclaredThrowableException, > which is already not ideal, because it can be returned to the application. > But if the first call to getMaster succeeds, it will set masterChecked = > true, which makes us never try to reconnect; that is, we will set this.master > = null and just throw MasterNotRunningExceptions, without even trying to > connect. > I tried out a 94 client (actually a 92 client with some 94 patches) on a > cluster with some network issues, and it would constantly get stuck as > described above. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master, client can get stuck and never make progress
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467905#comment-13467905 ] Gregory Chanan commented on HBASE-6920: --- The changes in HBaseRPC, RpcEngine, and WritableRpcEngine are just for the test. I wanted to simulate throwing a SocketTimeoutException on a proxy call, but Mockito can't mock a proxy object as far as I can tell. So all those changes are just so I can setup my own RPC Invoker to run, which just randomly throws SocketTimeoutExceptions. > On timeout connecting to master, client can get stuck and never make progress > - > > Key: HBASE-6920 > URL: https://issues.apache.org/jira/browse/HBASE-6920 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2 >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Critical > Attachments: HBASE-6920.patch > > > HBASE-5058 appears to have introduced an issue where a timeout in > HConnection.getMaster() can cause the client to never be able to connect to > the master. So, for example, an HBaseAdmin object can never successfully be > initialized. > The issue is here: > {code} > if (tryMaster.isMasterRunning()) { > this.master = tryMaster; > this.masterLock.notifyAll(); > break; > } > {code} > If isMasterRunning times out, it throws an UndeclaredThrowableException, > which is already not ideal, because it can be returned to the application. > But if the first call to getMaster succeeds, it will set masterChecked = > true, which makes us never try to reconnect; that is, we will set this.master > = null and just throw MasterNotRunningExceptions, without even trying to > connect. > I tried out a 94 client (actually a 92 client with some 94 patches) on a > cluster with some network issues, and it would constantly get stuck as > described above. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6921) String and ConcurrentHashMap sizes change on jdk7; makes TestHeapSize fail; second attempt
[ https://issues.apache.org/jira/browse/HBASE-6921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6921: - Attachment: 6921.txt Remove jdk7 special handling and just use estimated sizes for String and ConcurrentHashMap instead. > String and ConcurrentHashMap sizes change on jdk7; makes TestHeapSize fail; > second attempt > -- > > Key: HBASE-6921 > URL: https://issues.apache.org/jira/browse/HBASE-6921 > Project: HBase > Issue Type: Bug > Components: test >Reporter: stack >Assignee: stack > Fix For: 0.96.0 > > Attachments: 6921.txt > > > HBASE-6918 added debug. Jenkins runs jdk7 23.0-b21. Locally I have > 23.6-b03. It looks like ConcurrentHashMap is a different size in these two > jvm's (To be confirmed). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6707) TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps
[ https://issues.apache.org/jira/browse/HBASE-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467909#comment-13467909 ] Jesse Yates commented on HBASE-6707: [~stack] are you sure you ran v4? I can't seem to find where that output would even occur in the patch... > TEST > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables > flaps > > > Key: HBASE-6707 > URL: https://issues.apache.org/jira/browse/HBASE-6707 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Sameer Vaishampayan >Assignee: Jesse Yates >Priority: Critical > Fix For: 0.96.0 > > Attachments: hbase-6707-v0.patch, hbase-6707-v1.patch, > hbase-6707-v2.patch, hbase-6707-v3.patch, hbase-6707-v4.patch > > > https://builds.apache.org/job/HBase-TRUNK/3293/ > Error Message > Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > Stacktrace > java.lang.AssertionError: Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > at org.junit.Assert.fail(Assert.java:93) > at org.junit.Assert.assertTrue(Assert.java:43) > at org.junit.Assert.assertNull(Assert.java:551) > at > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables(TestZooKeeperTableArchiveClient.java:291) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6707) TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps
[ https://issues.apache.org/jira/browse/HBASE-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467921#comment-13467921 ] Ted Yu commented on HBASE-6707: --- @Jesse: Do you mind updating review request on review board so that I can see the changes between the two patches ? Thanks > TEST > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables > flaps > > > Key: HBASE-6707 > URL: https://issues.apache.org/jira/browse/HBASE-6707 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Sameer Vaishampayan >Assignee: Jesse Yates >Priority: Critical > Fix For: 0.96.0 > > Attachments: hbase-6707-v0.patch, hbase-6707-v1.patch, > hbase-6707-v2.patch, hbase-6707-v3.patch, hbase-6707-v4.patch > > > https://builds.apache.org/job/HBase-TRUNK/3293/ > Error Message > Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > Stacktrace > java.lang.AssertionError: Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > at org.junit.Assert.fail(Assert.java:93) > at org.junit.Assert.assertTrue(Assert.java:43) > at org.junit.Assert.assertNull(Assert.java:551) > at > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables(TestZooKeeperTableArchiveClient.java:291) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6920) On timeout connecting to master, client can get stuck and never make progress
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Gregory Chanan updated HBASE-6920: -- Attachment: HBASE-6920-v2.patch v2, which fixes the javadoc issue (actually changes the return value of the code). I left the LOG as INFO, because that's what it was before; let me know if you disagree. > On timeout connecting to master, client can get stuck and never make progress > - > > Key: HBASE-6920 > URL: https://issues.apache.org/jira/browse/HBASE-6920 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2 >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Critical > Attachments: HBASE-6920.patch, HBASE-6920-v2.patch > > > HBASE-5058 appears to have introduced an issue where a timeout in > HConnection.getMaster() can cause the client to never be able to connect to > the master. So, for example, an HBaseAdmin object can never successfully be > initialized. > The issue is here: > {code} > if (tryMaster.isMasterRunning()) { > this.master = tryMaster; > this.masterLock.notifyAll(); > break; > } > {code} > If isMasterRunning times out, it throws an UndeclaredThrowableException, > which is already not ideal, because it can be returned to the application. > But if the first call to getMaster succeeds, it will set masterChecked = > true, which makes us never try to reconnect; that is, we will set this.master > = null and just throw MasterNotRunningExceptions, without even trying to > connect. > I tried out a 94 client (actually a 92 client with some 94 patches) on a > cluster with some network issues, and it would constantly get stuck as > described above. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master, client can get stuck and never make progress
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467926#comment-13467926 ] Hadoop QA commented on HBASE-6920: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12547415/HBASE-6920-v2.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 4 new or modified tests. {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2987//console This message is automatically generated. > On timeout connecting to master, client can get stuck and never make progress > - > > Key: HBASE-6920 > URL: https://issues.apache.org/jira/browse/HBASE-6920 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2 >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Critical > Attachments: HBASE-6920.patch, HBASE-6920-v2.patch > > > HBASE-5058 appears to have introduced an issue where a timeout in > HConnection.getMaster() can cause the client to never be able to connect to > the master. So, for example, an HBaseAdmin object can never successfully be > initialized. > The issue is here: > {code} > if (tryMaster.isMasterRunning()) { > this.master = tryMaster; > this.masterLock.notifyAll(); > break; > } > {code} > If isMasterRunning times out, it throws an UndeclaredThrowableException, > which is already not ideal, because it can be returned to the application. > But if the first call to getMaster succeeds, it will set masterChecked = > true, which makes us never try to reconnect; that is, we will set this.master > = null and just throw MasterNotRunningExceptions, without even trying to > connect. > I tried out a 94 client (actually a 92 client with some 94 patches) on a > cluster with some network issues, and it would constantly get stuck as > described above. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6707) TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps
[ https://issues.apache.org/jira/browse/HBASE-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467927#comment-13467927 ] Jesse Yates commented on HBASE-6707: @Ted - done. review can be found here: https://reviews.apache.org/r/7344/ > TEST > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables > flaps > > > Key: HBASE-6707 > URL: https://issues.apache.org/jira/browse/HBASE-6707 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Sameer Vaishampayan >Assignee: Jesse Yates >Priority: Critical > Fix For: 0.96.0 > > Attachments: hbase-6707-v0.patch, hbase-6707-v1.patch, > hbase-6707-v2.patch, hbase-6707-v3.patch, hbase-6707-v4.patch > > > https://builds.apache.org/job/HBase-TRUNK/3293/ > Error Message > Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > Stacktrace > java.lang.AssertionError: Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > at org.junit.Assert.fail(Assert.java:93) > at org.junit.Assert.assertTrue(Assert.java:43) > at org.junit.Assert.assertNull(Assert.java:551) > at > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables(TestZooKeeperTableArchiveClient.java:291) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master, client can get stuck and never make progress
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467933#comment-13467933 ] Lars Hofhansl commented on HBASE-6920: -- +1 on patch after addressing Ted's comments ... (although the RPC changes look a bit scary). > On timeout connecting to master, client can get stuck and never make progress > - > > Key: HBASE-6920 > URL: https://issues.apache.org/jira/browse/HBASE-6920 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2 >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Critical > Attachments: HBASE-6920.patch, HBASE-6920-v2.patch > > > HBASE-5058 appears to have introduced an issue where a timeout in > HConnection.getMaster() can cause the client to never be able to connect to > the master. So, for example, an HBaseAdmin object can never successfully be > initialized. > The issue is here: > {code} > if (tryMaster.isMasterRunning()) { > this.master = tryMaster; > this.masterLock.notifyAll(); > break; > } > {code} > If isMasterRunning times out, it throws an UndeclaredThrowableException, > which is already not ideal, because it can be returned to the application. > But if the first call to getMaster succeeds, it will set masterChecked = > true, which makes us never try to reconnect; that is, we will set this.master > = null and just throw MasterNotRunningExceptions, without even trying to > connect. > I tried out a 94 client (actually a 92 client with some 94 patches) on a > cluster with some network issues, and it would constantly get stuck as > described above. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6923) Create scanner benchmark
Karthik Ranganathan created HBASE-6923: -- Summary: Create scanner benchmark Key: HBASE-6923 URL: https://issues.apache.org/jira/browse/HBASE-6923 Project: HBase Issue Type: Improvement Reporter: Karthik Ranganathan Assignee: Karthik Ranganathan Create a simple program to benchmark performance/throughput of scanners, and print some results at the end. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6922) HBase scanner performance improvements
Karthik Ranganathan created HBASE-6922: -- Summary: HBase scanner performance improvements Key: HBASE-6922 URL: https://issues.apache.org/jira/browse/HBASE-6922 Project: HBase Issue Type: Umbrella Reporter: Karthik Ranganathan Assignee: Karthik Ranganathan Umbrella task for improving through in HBase scanners. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6923) Create scanner benchmark
[ https://issues.apache.org/jira/browse/HBASE-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Ranganathan updated HBASE-6923: --- Issue Type: Sub-task (was: Improvement) Parent: HBASE-6922 > Create scanner benchmark > > > Key: HBASE-6923 > URL: https://issues.apache.org/jira/browse/HBASE-6923 > Project: HBase > Issue Type: Sub-task >Reporter: Karthik Ranganathan >Assignee: Karthik Ranganathan > > Create a simple program to benchmark performance/throughput of scanners, and > print some results at the end. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6874) Implement prefetching for scanners
[ https://issues.apache.org/jira/browse/HBASE-6874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Ranganathan updated HBASE-6874: --- Issue Type: Sub-task (was: Improvement) Parent: HBASE-6922 > Implement prefetching for scanners > -- > > Key: HBASE-6874 > URL: https://issues.apache.org/jira/browse/HBASE-6874 > Project: HBase > Issue Type: Sub-task >Reporter: Karthik Ranganathan >Assignee: Karthik Ranganathan > > I did some quick experiments by scanning data that should be completely in > memory and found that adding pre-fetching increases the throughput by about > 50% from 26MB/s to 39MB/s. > The idea is to perform the next in a background thread, and keep the result > ready. When the scanner's next comes in, return the pre-computed result and > issue another background read. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6921) String and ConcurrentHashMap sizes change on jdk7; makes TestHeapSize fail; second attempt
[ https://issues.apache.org/jira/browse/HBASE-6921?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467936#comment-13467936 ] Hudson commented on HBASE-6921: --- Integrated in HBase-TRUNK #3406 (See [https://builds.apache.org/job/HBase-TRUNK/3406/]) HBASE-6921 String and ConcurrentHashMap sizes change on jdk7; makes TestHeapSize fail; second attempt (Revision 1393062) Result = FAILURE stack : Files : * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/ClassSize.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java > String and ConcurrentHashMap sizes change on jdk7; makes TestHeapSize fail; > second attempt > -- > > Key: HBASE-6921 > URL: https://issues.apache.org/jira/browse/HBASE-6921 > Project: HBase > Issue Type: Bug > Components: test >Reporter: stack >Assignee: stack > Fix For: 0.96.0 > > Attachments: 6921.txt > > > HBASE-6918 added debug. Jenkins runs jdk7 23.0-b21. Locally I have > 23.6-b03. It looks like ConcurrentHashMap is a different size in these two > jvm's (To be confirmed). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6770) Allow scanner setCaching to specify size instead of number of rows
[ https://issues.apache.org/jira/browse/HBASE-6770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Ranganathan updated HBASE-6770: --- Issue Type: Sub-task (was: Bug) Parent: HBASE-6922 > Allow scanner setCaching to specify size instead of number of rows > -- > > Key: HBASE-6770 > URL: https://issues.apache.org/jira/browse/HBASE-6770 > Project: HBase > Issue Type: Sub-task > Components: Client, regionserver >Reporter: Karthik Ranganathan >Assignee: Michal Gregorczyk > > Currently, we have the following api's to customize the behavior of scans: > setCaching() - how many rows to cache on client to speed up scans > setBatch() - max columns per row to return per row to prevent a very large > response. > Ideally, we should be able to specify a memory buffer size because: > 1. that would take care of both of these use cases. > 2. it does not need any knowledge of the size of the rows or cells, as the > final thing we are worried about is the available memory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master, client can get stuck and never make progress
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467937#comment-13467937 ] Gregory Chanan commented on HBASE-6920: --- Lars -- you have an idea for how to do the test without making RPC changes? I'd prefer that, but couldn't think of a way. > On timeout connecting to master, client can get stuck and never make progress > - > > Key: HBASE-6920 > URL: https://issues.apache.org/jira/browse/HBASE-6920 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2 >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Critical > Attachments: HBASE-6920.patch, HBASE-6920-v2.patch > > > HBASE-5058 appears to have introduced an issue where a timeout in > HConnection.getMaster() can cause the client to never be able to connect to > the master. So, for example, an HBaseAdmin object can never successfully be > initialized. > The issue is here: > {code} > if (tryMaster.isMasterRunning()) { > this.master = tryMaster; > this.masterLock.notifyAll(); > break; > } > {code} > If isMasterRunning times out, it throws an UndeclaredThrowableException, > which is already not ideal, because it can be returned to the application. > But if the first call to getMaster succeeds, it will set masterChecked = > true, which makes us never try to reconnect; that is, we will set this.master > = null and just throw MasterNotRunningExceptions, without even trying to > connect. > I tried out a 94 client (actually a 92 client with some 94 patches) on a > cluster with some network issues, and it would constantly get stuck as > described above. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6707) TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps
[ https://issues.apache.org/jira/browse/HBASE-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467939#comment-13467939 ] stack commented on HBASE-6707: -- [~jesse_yates] I must have done something wrong. Sorry about that. Seems fine running it in a loop here locally. Waiting on Ted's review before committing. > TEST > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables > flaps > > > Key: HBASE-6707 > URL: https://issues.apache.org/jira/browse/HBASE-6707 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Sameer Vaishampayan >Assignee: Jesse Yates >Priority: Critical > Fix For: 0.96.0 > > Attachments: hbase-6707-v0.patch, hbase-6707-v1.patch, > hbase-6707-v2.patch, hbase-6707-v3.patch, hbase-6707-v4.patch > > > https://builds.apache.org/job/HBase-TRUNK/3293/ > Error Message > Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > Stacktrace > java.lang.AssertionError: Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > at org.junit.Assert.fail(Assert.java:93) > at org.junit.Assert.assertTrue(Assert.java:43) > at org.junit.Assert.assertNull(Assert.java:551) > at > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables(TestZooKeeperTableArchiveClient.java:291) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6066) some low hanging read path improvement ideas
[ https://issues.apache.org/jira/browse/HBASE-6066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Ranganathan updated HBASE-6066: --- Issue Type: Sub-task (was: Improvement) Parent: HBASE-6922 > some low hanging read path improvement ideas > - > > Key: HBASE-6066 > URL: https://issues.apache.org/jira/browse/HBASE-6066 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: Kannan Muthukkaruppan >Assignee: Michal Gregorczyk >Priority: Critical > Labels: noob > Fix For: 0.96.0 > > Attachments: > 0001-jira-HBASE-6066-89-fb-Some-read-performance-improvem.patch, > metric-stringbuilder-fix.patch > > > I was running some single threaded scan performance tests for a table with > small sized rows that is fully cached. Some observations... > We seem to be doing several wasteful iterations over and/or building of > temporary lists. > 1) One such is the following code in HRegionServer.next(): > {code} >boolean moreRows = s.next(values, HRegion.METRIC_NEXTSIZE); >if (!values.isEmpty()) { > for (KeyValue kv : values) { --> wasteful in most > cases >currentScanResultSize += kv.heapSize(); >} >results.add(new Result(values)); > {code} > By default the "maxScannerResultSize" is Long.MAX_VALUE. In those cases, > we can avoid the unnecessary iteration to compute currentScanResultSize. > 2) An example of a wasteful temporary array, is "results" in > RegionScanner.next(). > {code} > results.clear(); > boolean returnResult = nextInternal(limit, metric); > outResults.addAll(results); > {code} > results then gets copied over to outResults via an addAll(). Not sure why we > can not directly collect the results in outResults. > 3) Another almost similar exmaple of a wasteful array is "results" in > StoreScanner.next(), which eventually also copies its results into > "outResults". > 4) Reduce overhead of "size metric" maintained in StoreScanner.next(). > {code} > if (metric != null) { > HRegion.incrNumericMetric(this.metricNamePrefix + metric, >copyKv.getLength()); > } > results.add(copyKv); > {code} > A single call to next() might fetch a lot of KVs. We can first add up the > size of those KVs in a local variable and then in a finally clause increment > the metric one shot, rather than updating AtomicLongs for each KV. > 5) RegionScanner.next() calls a helper RegionScanner.next() on the same > object. Both are synchronized methods. Synchronized methods calling nested > synchronized methods on the same object are probably adding some small > overhead. The inner next() calls isFilterDone() which is a also a > synchronized method. We should factor the code to avoid these nested > synchronized methods. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6912) Filters are not properly applied in certain cases
[ https://issues.apache.org/jira/browse/HBASE-6912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467947#comment-13467947 ] Alex Newman commented on HBASE-6912: Agreed, it would be unfortunate if we attempted, for instance to apply timestamp filters at that part of the code. > Filters are not properly applied in certain cases > - > > Key: HBASE-6912 > URL: https://issues.apache.org/jira/browse/HBASE-6912 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.1 >Reporter: Alex Newman > Fix For: 0.94.3, 0.96.0 > > Attachments: minimalTest.java > > > Steps to reproduce: > Create a table, load data into it. Flush the table. > Do a scan with > 1. Some filter which should not match the first entry in the scan > 2. Where one specifies a family and column. > You will notice that the first entry is returned even though it doesn't match > the filter. > It looks like the when the first KeyValue of a scan in the column from the > point of view of the code > HRegion.java > {code} > } else if (kv != null && !kv.isInternal() && filterRowKey(currentRow)) { > {code} > Is generated by > {code} > public static KeyValue createLastOnRow(final byte [] row, > final int roffset, final int rlength, final byte [] family, > final int foffset, final int flength, final byte [] qualifier, > final int qoffset, final int qlength) { return new KeyValue(row, roffset, > rlength, family, foffset, flength, qualifier, qoffset, qlength, > HConstants.OLDEST_TIMESTAMP, Type.Minimum, null, 0, 0); } > {code} > So it is always internal from that point of the code. > Only later from within > StoreScanner.java > {code} > public synchronized boolean next(List outResult, int limit, String > metric) throws IOException { > > LOOP: while((kv = this.heap.peek()) != null) { > {code} > ( The second time through) > Do we get the actual kv, with a proper type and timestamp. This seems to mess > with filtering. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6912) Filters are not properly applied in certain cases
[ https://issues.apache.org/jira/browse/HBASE-6912?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467946#comment-13467946 ] Lars Hofhansl commented on HBASE-6912: -- I have a patch, which fixes RowFilter. But I do not like it. Each "real" KV should through filterRowKey before it goes through filterKeyValue. That seems to be broken in some cases (which has to do with the lazy seeking we're doing now). > Filters are not properly applied in certain cases > - > > Key: HBASE-6912 > URL: https://issues.apache.org/jira/browse/HBASE-6912 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.1 >Reporter: Alex Newman > Fix For: 0.94.3, 0.96.0 > > Attachments: minimalTest.java > > > Steps to reproduce: > Create a table, load data into it. Flush the table. > Do a scan with > 1. Some filter which should not match the first entry in the scan > 2. Where one specifies a family and column. > You will notice that the first entry is returned even though it doesn't match > the filter. > It looks like the when the first KeyValue of a scan in the column from the > point of view of the code > HRegion.java > {code} > } else if (kv != null && !kv.isInternal() && filterRowKey(currentRow)) { > {code} > Is generated by > {code} > public static KeyValue createLastOnRow(final byte [] row, > final int roffset, final int rlength, final byte [] family, > final int foffset, final int flength, final byte [] qualifier, > final int qoffset, final int qlength) { return new KeyValue(row, roffset, > rlength, family, foffset, flength, qualifier, qoffset, qlength, > HConstants.OLDEST_TIMESTAMP, Type.Minimum, null, 0, 0); } > {code} > So it is always internal from that point of the code. > Only later from within > StoreScanner.java > {code} > public synchronized boolean next(List outResult, int limit, String > metric) throws IOException { > > LOOP: while((kv = this.heap.peek()) != null) { > {code} > ( The second time through) > Do we get the actual kv, with a proper type and timestamp. This seems to mess > with filtering. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6923) Create scanner benchmark
[ https://issues.apache.org/jira/browse/HBASE-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HBASE-6923: --- Attachment: TestStorePerformance.java Hey Karthik. Here's a test I wrote recently which may help you out. I also have some interesting results I'm hoping to share later this week. > Create scanner benchmark > > > Key: HBASE-6923 > URL: https://issues.apache.org/jira/browse/HBASE-6923 > Project: HBase > Issue Type: Sub-task >Reporter: Karthik Ranganathan >Assignee: Karthik Ranganathan > Attachments: TestStorePerformance.java > > > Create a simple program to benchmark performance/throughput of scanners, and > print some results at the end. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6914) Scans/Gets/Mutations don't give a good error if the table is disabled.
[ https://issues.apache.org/jira/browse/HBASE-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-6914: - Attachment: HBASE-6914-094-3.patch HBASE-6914-092-3.patch Attaching 0.92 and 0.94 patch versions before commit. > Scans/Gets/Mutations don't give a good error if the table is disabled. > -- > > Key: HBASE-6914 > URL: https://issues.apache.org/jira/browse/HBASE-6914 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 0.92.3, 0.94.3, 0.96.0 > > Attachments: HBASE-6914-092-3.patch, HBASE-6914-094-3.patch, > HBASE-6914-0.patch, HBASE-6914-1.patch, HBASE-6914-2.patch, HBASE-6914-3.patch > > > Scan a table that is disabled will have the client retry multiple times and > then will error out with NotServingRegionException. If the table is disabled > there's no need to re-try and the message should be more explicit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6914) Scans/Gets/Mutations don't give a good error if the table is disabled.
[ https://issues.apache.org/jira/browse/HBASE-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467956#comment-13467956 ] Hadoop QA commented on HBASE-6914: -- {color:red}-1 overall{color}. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12547421/HBASE-6914-094-3.patch against trunk revision . {color:green}+1 @author{color}. The patch does not contain any @author tags. {color:green}+1 tests included{color}. The patch appears to include 3 new or modified tests. {color:red}-1 patch{color}. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2988//console This message is automatically generated. > Scans/Gets/Mutations don't give a good error if the table is disabled. > -- > > Key: HBASE-6914 > URL: https://issues.apache.org/jira/browse/HBASE-6914 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 0.92.3, 0.94.3, 0.96.0 > > Attachments: HBASE-6914-092-3.patch, HBASE-6914-094-3.patch, > HBASE-6914-0.patch, HBASE-6914-1.patch, HBASE-6914-2.patch, HBASE-6914-3.patch > > > Scan a table that is disabled will have the client retry multiple times and > then will error out with NotServingRegionException. If the table is disabled > there's no need to re-try and the message should be more explicit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6878) DistributerLogSplit can fail to resubmit a task done if there is an exception during the log archiving
[ https://issues.apache.org/jira/browse/HBASE-6878?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13467980#comment-13467980 ] nkeywal commented on HBASE-6878: Thanks a lot Prakash. I understand I could change the CHECK to FORCE, but it won't actually change much in real cases. So let's mark this to wont fix :-). > DistributerLogSplit can fail to resubmit a task done if there is an exception > during the log archiving > -- > > Key: HBASE-6878 > URL: https://issues.apache.org/jira/browse/HBASE-6878 > Project: HBase > Issue Type: Bug > Components: master >Reporter: nkeywal >Priority: Minor > > The code in SplitLogManager# getDataSetWatchSuccess is: > {code} > if (slt.isDone()) { > LOG.info("task " + path + " entered state: " + slt.toString()); > if (taskFinisher != null && !ZKSplitLog.isRescanNode(watcher, path)) { > if (taskFinisher.finish(slt.getServerName(), > ZKSplitLog.getFileName(path)) == Status.DONE) { > setDone(path, SUCCESS); > } else { > resubmitOrFail(path, CHECK); > } > } else { > setDone(path, SUCCESS); > } > {code} > resubmitOrFail(path, CHECK); > should be > resubmitOrFail(path, FORCE); > Without it, the task won't be resubmitted if the delay is not reached, and > the task will be marked as failed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6878) DistributerLogSplit can fail to resubmit a task done if there is an exception during the log archiving
[ https://issues.apache.org/jira/browse/HBASE-6878?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nkeywal resolved HBASE-6878. Resolution: Won't Fix > DistributerLogSplit can fail to resubmit a task done if there is an exception > during the log archiving > -- > > Key: HBASE-6878 > URL: https://issues.apache.org/jira/browse/HBASE-6878 > Project: HBase > Issue Type: Bug > Components: master >Reporter: nkeywal >Priority: Minor > > The code in SplitLogManager# getDataSetWatchSuccess is: > {code} > if (slt.isDone()) { > LOG.info("task " + path + " entered state: " + slt.toString()); > if (taskFinisher != null && !ZKSplitLog.isRescanNode(watcher, path)) { > if (taskFinisher.finish(slt.getServerName(), > ZKSplitLog.getFileName(path)) == Status.DONE) { > setDone(path, SUCCESS); > } else { > resubmitOrFail(path, CHECK); > } > } else { > setDone(path, SUCCESS); > } > {code} > resubmitOrFail(path, CHECK); > should be > resubmitOrFail(path, FORCE); > Without it, the task won't be resubmitted if the delay is not reached, and > the task will be marked as failed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5937: - Resolution: Fixed Fix Version/s: 0.96.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk after running full suite and having it pass. Good on you Flavio. Please create new issues for problems found reviewing this patch. > Refactor HLog into an interface. > > > Key: HBASE-5937 > URL: https://issues.apache.org/jira/browse/HBASE-5937 > Project: HBase > Issue Type: Sub-task >Reporter: Li Pi >Assignee: Flavio Junqueira >Priority: Minor > Fix For: 0.96.0 > > Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch, > HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, > HBASE-5937.v2.patch, HBASE-5937.v3.patch, > org.apache.hadoop.hbase.client.TestMultiParallel-output.txt > > > What the summary says. Create HLog interface. Make current implementation use > it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6924) HBase Master spews into a loop if a user attempts to create a snappy table when snappy isn't properly configured
Alex Newman created HBASE-6924: -- Summary: HBase Master spews into a loop if a user attempts to create a snappy table when snappy isn't properly configured Key: HBASE-6924 URL: https://issues.apache.org/jira/browse/HBASE-6924 Project: HBase Issue Type: Bug Reporter: Alex Newman If a user attempts to create a table, for instance create 't1',{ NAME=>'c1', COMPRESSION=>'snappy'} on stock HBase(Without snappy setup), the master will spew this error in a loop 12/10/02 12:41:38 INFO handler.OpenRegionHandler: Opening of region {NAME => 't1,,1349206881317.2d34e32205ffe677496b03faa7e66063.', STARTKEY => '', ENDKEY => '', ENCODED => 2d34e32205ffe677496b03faa7e66063,} failed, marking as FAILED_OPEN in ZK 12/10/02 12:41:38 INFO regionserver.HRegionServer: Received request to open region: t1,,1349206881317.2d34e32205ffe677496b03faa7e66063. 12/10/02 12:41:38 INFO regionserver.HRegion: Setting up tabledescriptor config now ... 12/10/02 12:41:38 ERROR handler.OpenRegionHandler: Failed open of region=t1,,1349206881317.2d34e32205ffe677496b03faa7e66063., starting to roll back the global memstore size. java.io.IOException: Compression algorithm 'snappy' previously failed test. at org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:78) at org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:3822) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3811) at org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3761) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332) at org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108) at org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169) at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) at java.lang.Thread.run(Thread.java:662) Even after the shell is killed. In fact, even after a hbase reboot we will endlessly spew this painful and high overhead error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6924) HBase Master spews into a loop if a user attempts to create a snappy table when snappy isn't properly configured
[ https://issues.apache.org/jira/browse/HBASE-6924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468012#comment-13468012 ] Alex Newman commented on HBASE-6924: I assume the issue is that opening a region can often fail and we want some retry logic. > HBase Master spews into a loop if a user attempts to create a snappy table > when snappy isn't properly configured > > > Key: HBASE-6924 > URL: https://issues.apache.org/jira/browse/HBASE-6924 > Project: HBase > Issue Type: Bug >Reporter: Alex Newman > > If a user attempts to create a table, for instance > create 't1',{ NAME=>'c1', COMPRESSION=>'snappy'} > on stock HBase(Without snappy setup), > the master will spew this error in a loop > 12/10/02 12:41:38 INFO handler.OpenRegionHandler: Opening of region {NAME => > 't1,,1349206881317.2d34e32205ffe677496b03faa7e66063.', STARTKEY => '', ENDKEY > => '', ENCODED => 2d34e32205ffe677496b03faa7e66063,} failed, marking as > FAILED_OPEN in ZK > 12/10/02 12:41:38 INFO regionserver.HRegionServer: Received request to open > region: t1,,1349206881317.2d34e32205ffe677496b03faa7e66063. > 12/10/02 12:41:38 INFO regionserver.HRegion: Setting up tabledescriptor > config now ... > 12/10/02 12:41:38 ERROR handler.OpenRegionHandler: Failed open of > region=t1,,1349206881317.2d34e32205ffe677496b03faa7e66063., starting to roll > back the global memstore size. > java.io.IOException: Compression algorithm 'snappy' previously failed test. > at > org.apache.hadoop.hbase.util.CompressionTest.testCompression(CompressionTest.java:78) > at > org.apache.hadoop.hbase.regionserver.HRegion.checkCompressionCodecs(HRegion.java:3822) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3811) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3761) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:332) > at > org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:108) > at > org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:169) > at > java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908) > at java.lang.Thread.run(Thread.java:662) > Even after the shell is killed. > In fact, even after a hbase reboot we will endlessly spew this painful and > high overhead error. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6707) TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps
[ https://issues.apache.org/jira/browse/HBASE-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468033#comment-13468033 ] Ted Yu commented on HBASE-6707: --- The gist of this JIRA is about making sure that HFiles for table not configured to endure table drop should be deleted. I had a question w.r.t. the above which I noted on review board. > TEST > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables > flaps > > > Key: HBASE-6707 > URL: https://issues.apache.org/jira/browse/HBASE-6707 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Sameer Vaishampayan >Assignee: Jesse Yates >Priority: Critical > Fix For: 0.96.0 > > Attachments: hbase-6707-v0.patch, hbase-6707-v1.patch, > hbase-6707-v2.patch, hbase-6707-v3.patch, hbase-6707-v4.patch > > > https://builds.apache.org/job/HBase-TRUNK/3293/ > Error Message > Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > Stacktrace > java.lang.AssertionError: Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > at org.junit.Assert.fail(Assert.java:93) > at org.junit.Assert.assertTrue(Assert.java:43) > at org.junit.Assert.assertNull(Assert.java:551) > at > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables(TestZooKeeperTableArchiveClient.java:291) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6923) Create scanner benchmark
[ https://issues.apache.org/jira/browse/HBASE-6923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468039#comment-13468039 ] Karthik Ranganathan commented on HBASE-6923: Hey Todd, nice! I too have written a benchmark with interesting results. Would be interesting to compare :) > Create scanner benchmark > > > Key: HBASE-6923 > URL: https://issues.apache.org/jira/browse/HBASE-6923 > Project: HBase > Issue Type: Sub-task >Reporter: Karthik Ranganathan >Assignee: Karthik Ranganathan > Attachments: TestStorePerformance.java > > > Create a simple program to benchmark performance/throughput of scanners, and > print some results at the end. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6925) Change socket write size from 8K to 64K for HBaseServer
Karthik Ranganathan created HBASE-6925: -- Summary: Change socket write size from 8K to 64K for HBaseServer Key: HBASE-6925 URL: https://issues.apache.org/jira/browse/HBASE-6925 Project: HBase Issue Type: Improvement Reporter: Karthik Ranganathan Creating a JIRA for this, but the change is trivial: change NIO_BUFFER_LIMIT from 8K to 64K in HBaseServer. This seems to increase scan throughput. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6925) Change socket write size from 8K to 64K for HBaseServer
[ https://issues.apache.org/jira/browse/HBASE-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Ranganathan updated HBASE-6925: --- Issue Type: Sub-task (was: Improvement) Parent: HBASE-6922 > Change socket write size from 8K to 64K for HBaseServer > --- > > Key: HBASE-6925 > URL: https://issues.apache.org/jira/browse/HBASE-6925 > Project: HBase > Issue Type: Sub-task >Reporter: Karthik Ranganathan >Assignee: Karthik Ranganathan > > Creating a JIRA for this, but the change is trivial: change NIO_BUFFER_LIMIT > from 8K to 64K in HBaseServer. This seems to increase scan throughput. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6707) TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps
[ https://issues.apache.org/jira/browse/HBASE-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468042#comment-13468042 ] Ted Yu commented on HBASE-6707: --- I am sorry Jesse: HBASE-5699 has gone in and patch v4 is stale now. Once you provide rebased patch, I will run the test and confirm that the fix works. Thanks for your patience. > TEST > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables > flaps > > > Key: HBASE-6707 > URL: https://issues.apache.org/jira/browse/HBASE-6707 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Sameer Vaishampayan >Assignee: Jesse Yates >Priority: Critical > Fix For: 0.96.0 > > Attachments: hbase-6707-v0.patch, hbase-6707-v1.patch, > hbase-6707-v2.patch, hbase-6707-v3.patch, hbase-6707-v4.patch > > > https://builds.apache.org/job/HBase-TRUNK/3293/ > Error Message > Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > Stacktrace > java.lang.AssertionError: Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > at org.junit.Assert.fail(Assert.java:93) > at org.junit.Assert.assertTrue(Assert.java:43) > at org.junit.Assert.assertNull(Assert.java:551) > at > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables(TestZooKeeperTableArchiveClient.java:291) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6925) Change socket write size from 8K to 64K for HBaseServer
[ https://issues.apache.org/jira/browse/HBASE-6925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Ranganathan reassigned HBASE-6925: -- Assignee: Karthik Ranganathan > Change socket write size from 8K to 64K for HBaseServer > --- > > Key: HBASE-6925 > URL: https://issues.apache.org/jira/browse/HBASE-6925 > Project: HBase > Issue Type: Improvement >Reporter: Karthik Ranganathan >Assignee: Karthik Ranganathan > > Creating a JIRA for this, but the change is trivial: change NIO_BUFFER_LIMIT > from 8K to 64K in HBaseServer. This seems to increase scan throughput. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6707) TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps
[ https://issues.apache.org/jira/browse/HBASE-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468043#comment-13468043 ] Jesse Yates commented on HBASE-6707: @Ted - which jira? That one's still open... I want to make sure my changes make sense. > TEST > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables > flaps > > > Key: HBASE-6707 > URL: https://issues.apache.org/jira/browse/HBASE-6707 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Sameer Vaishampayan >Assignee: Jesse Yates >Priority: Critical > Fix For: 0.96.0 > > Attachments: hbase-6707-v0.patch, hbase-6707-v1.patch, > hbase-6707-v2.patch, hbase-6707-v3.patch, hbase-6707-v4.patch > > > https://builds.apache.org/job/HBase-TRUNK/3293/ > Error Message > Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > Stacktrace > java.lang.AssertionError: Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > at org.junit.Assert.fail(Assert.java:93) > at org.junit.Assert.assertTrue(Assert.java:43) > at org.junit.Assert.assertNull(Assert.java:551) > at > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables(TestZooKeeperTableArchiveClient.java:291) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6707) TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps
[ https://issues.apache.org/jira/browse/HBASE-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468047#comment-13468047 ] Ted Yu commented on HBASE-6707: --- Pardon me. Looks like Stack entered wrong JIRA: {code} r1393126 | stack | 2012-10-02 12:29:19 -0700 (Tue, 02 Oct 2012) | 1 line HBASE-5699 Refactor HLog into an interface {code} What he integrated was HBASE-5937 > TEST > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables > flaps > > > Key: HBASE-6707 > URL: https://issues.apache.org/jira/browse/HBASE-6707 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Sameer Vaishampayan >Assignee: Jesse Yates >Priority: Critical > Fix For: 0.96.0 > > Attachments: hbase-6707-v0.patch, hbase-6707-v1.patch, > hbase-6707-v2.patch, hbase-6707-v3.patch, hbase-6707-v4.patch > > > https://builds.apache.org/job/HBase-TRUNK/3293/ > Error Message > Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > Stacktrace > java.lang.AssertionError: Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > at org.junit.Assert.fail(Assert.java:93) > at org.junit.Assert.assertTrue(Assert.java:43) > at org.junit.Assert.assertNull(Assert.java:551) > at > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables(TestZooKeeperTableArchiveClient.java:291) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6926) Cleanup some of the javadoc warnings.
stack created HBASE-6926: Summary: Cleanup some of the javadoc warnings. Key: HBASE-6926 URL: https://issues.apache.org/jira/browse/HBASE-6926 Project: HBase Issue Type: Bug Components: documentation Reporter: stack Assignee: stack Fix For: 0.96.0 Attachments: javadocleanup.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6926) Cleanup some of the javadoc warnings.
[ https://issues.apache.org/jira/browse/HBASE-6926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6926: - Attachment: javadocleanup.txt Makes all but hbase-server module clean. Still a bunch to do. > Cleanup some of the javadoc warnings. > - > > Key: HBASE-6926 > URL: https://issues.apache.org/jira/browse/HBASE-6926 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: stack >Assignee: stack > Fix For: 0.96.0 > > Attachments: javadocleanup.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6926) Cleanup some of the javadoc warnings.
[ https://issues.apache.org/jira/browse/HBASE-6926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-6926. -- Resolution: Fixed I committed these fixups. > Cleanup some of the javadoc warnings. > - > > Key: HBASE-6926 > URL: https://issues.apache.org/jira/browse/HBASE-6926 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: stack >Assignee: stack > Fix For: 0.96.0 > > Attachments: javadocleanup.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6914) Scans/Gets/Mutations don't give a good error if the table is disabled.
[ https://issues.apache.org/jira/browse/HBASE-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-6914: - Resolution: Fixed Release Note: Scans/Gets/Mutations will now throw a DoNotRetryException, with an explanation, when the tageted table has been disabled. Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Scans/Gets/Mutations don't give a good error if the table is disabled. > -- > > Key: HBASE-6914 > URL: https://issues.apache.org/jira/browse/HBASE-6914 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 0.92.3, 0.94.3, 0.96.0 > > Attachments: HBASE-6914-092-3.patch, HBASE-6914-094-3.patch, > HBASE-6914-0.patch, HBASE-6914-1.patch, HBASE-6914-2.patch, HBASE-6914-3.patch > > > Scan a table that is disabled will have the client retry multiple times and > then will error out with NotServingRegionException. If the table is disabled > there's no need to re-try and the message should be more explicit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-3661) Handle empty qualifier better in shell for increments
[ https://issues.apache.org/jira/browse/HBASE-3661?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Drzal reassigned HBASE-3661: Assignee: Michael Drzal > Handle empty qualifier better in shell for increments > - > > Key: HBASE-3661 > URL: https://issues.apache.org/jira/browse/HBASE-3661 > Project: HBase > Issue Type: Improvement > Components: shell >Affects Versions: 0.92.0 >Reporter: Lars George >Assignee: Michael Drzal >Priority: Minor > > When trying to increment a counter using the examples, which specify no > *explicit* qualifier you get an error: > {code} > hbase(main):014:0> incr 'testtable', 'cnt1', 'colfam1', 1 > ERROR: org.apache.hadoop.hbase.client.RetriesExhaustedException: Trying to > contact region server 10.0.0.57:51640 for region > testtable,,1300267113942.cd2e7925140eb414d519621e384fb654., row 'cnt1', but > failed after 7 attempts. > Exceptions: > java.io.IOException: java.io.IOException: java.lang.NullPointerException > at > org.apache.hadoop.hbase.regionserver.ColumnCount.(ColumnCount.java:47) > at > org.apache.hadoop.hbase.regionserver.ExplicitColumnTracker.(ExplicitColumnTracker.java:69) > at > org.apache.hadoop.hbase.regionserver.ScanQueryMatcher.(ScanQueryMatcher.java:93) > at > org.apache.hadoop.hbase.regionserver.StoreScanner.(StoreScanner.java:65) > at > org.apache.hadoop.hbase.regionserver.Store.getScanner(Store.java:1436) > at > org.apache.hadoop.hbase.regionserver.HRegion$RegionScanner.(HRegion.java:2412) > at > org.apache.hadoop.hbase.regionserver.HRegion.instantiateInternalScanner(HRegion.java:1185) > at > org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1171) > at > org.apache.hadoop.hbase.regionserver.HRegion.getScanner(HRegion.java:1155) > at > org.apache.hadoop.hbase.regionserver.HRegion.getLastIncrement(HRegion.java:3087) > at > org.apache.hadoop.hbase.regionserver.HRegion.incrementColumnValue(HRegion.java:3312) > at > org.apache.hadoop.hbase.regionserver.HRegionServer.incrementColumnValue(HRegionServer.java:2570) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.apache.hadoop.hbase.ipc.WritableRpcEngine$Server.call(WritableRpcEngine.java:309) > at > org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1060) > Here is some help for this command: > Increments a cell 'value' at specified table/row/column coordinates. > To increment a cell value in table 't1' at row 'r1' under column > 'c1' by 1 (can be omitted) or 10 do: > hbase> incr 't1', 'r1', 'c1' > hbase> incr 't1', 'r1', 'c1', 1 > hbase> incr 't1', 'r1', 'c1', 10 > {code} > Handle this more gracefully (printing 5 stacktraces is ugly), improve the > help to specify what is needed more clearly. Or fix the server side to > support this, if this makes sense, and therefore never triggering this issue. > Adding a qualifier makes it work: > {code} > hbase(main):015:0> incr 'testtable', 'cnt1', 'colfam1:test', 1 > COUNTER VALUE = 1 > hbase(main):016:0> incr 'testtable', 'cnt1', 'colfam1:test', 1 > COUNTER VALUE = 2 > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6921) String and ConcurrentHashMap sizes change on jdk7; makes TestHeapSize fail; second attempt
[ https://issues.apache.org/jira/browse/HBASE-6921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-6921. -- Resolution: Fixed This seems to fix the jdk7 issues. Resolving. > String and ConcurrentHashMap sizes change on jdk7; makes TestHeapSize fail; > second attempt > -- > > Key: HBASE-6921 > URL: https://issues.apache.org/jira/browse/HBASE-6921 > Project: HBase > Issue Type: Bug > Components: test >Reporter: stack >Assignee: stack > Fix For: 0.96.0 > > Attachments: 6921.txt > > > HBASE-6918 added debug. Jenkins runs jdk7 23.0-b21. Locally I have > 23.6-b03. It looks like ConcurrentHashMap is a different size in these two > jvm's (To be confirmed). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6707) TEST org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables flaps
[ https://issues.apache.org/jira/browse/HBASE-6707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468063#comment-13468063 ] Jesse Yates commented on HBASE-6707: [~te...@apache.org][~stack] looks like stack accidentally rolled in the latest version of this patch with HBASE-5937. So feel free to test it Ted(?) since its in. We can always revert if not good. Only thing missing from the patch to trunk is removal of CheckedArchivingHFileCleaner.java > TEST > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables > flaps > > > Key: HBASE-6707 > URL: https://issues.apache.org/jira/browse/HBASE-6707 > Project: HBase > Issue Type: Bug > Components: test >Reporter: Sameer Vaishampayan >Assignee: Jesse Yates >Priority: Critical > Fix For: 0.96.0 > > Attachments: hbase-6707-v0.patch, hbase-6707-v1.patch, > hbase-6707-v2.patch, hbase-6707-v3.patch, hbase-6707-v4.patch > > > https://builds.apache.org/job/HBase-TRUNK/3293/ > Error Message > Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > Stacktrace > java.lang.AssertionError: Archived HFiles > (hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam) > should have gotten deleted, but didn't, remaining > files:[hdfs://localhost:59986/user/jenkins/hbase/.archive/otherTable/01ced3b55d7220a9c460273a4a57b198/fam/fc872572a1f5443eb55b6e2567cfeb1c] > at org.junit.Assert.fail(Assert.java:93) > at org.junit.Assert.assertTrue(Assert.java:43) > at org.junit.Assert.assertNull(Assert.java:551) > at > org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient.testMultipleTables(TestZooKeeperTableArchiveClient.java:291) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[ https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468066#comment-13468066 ] Ted Yu commented on HBASE-5937: --- I am confused as to the scope of the commit: {code} $ svn log hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/LongTermArchivingHFileCleaner.java | head r1393126 | stack | 2012-10-02 12:29:19 -0700 (Tue, 02 Oct 2012) | 1 line HBASE-5699 Refactor HLog into an interface {code} First, JIRA number was wrong. Second, LongTermArchivingHFileCleaner.java wasn't even in Flavio's patch. Maybe one version of patch for HBASE-6707 was integrated by chance ? > Refactor HLog into an interface. > > > Key: HBASE-5937 > URL: https://issues.apache.org/jira/browse/HBASE-5937 > Project: HBase > Issue Type: Sub-task >Reporter: Li Pi >Assignee: Flavio Junqueira >Priority: Minor > Fix For: 0.96.0 > > Attachments: 5937-hlog-with-javadoc.txt, HBASE-5937.patch, > HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, HBASE-5937.patch, > HBASE-5937.v2.patch, HBASE-5937.v3.patch, > org.apache.hadoop.hbase.client.TestMultiParallel-output.txt > > > What the summary says. Create HLog interface. Make current implementation use > it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4060) Making region assignment more robust
[ https://issues.apache.org/jira/browse/HBASE-4060?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-4060: - Fix Version/s: (was: 0.96.0) Moving out of 0.96.0. Needs to be worked through more. Won't be done for 0.96.0 > Making region assignment more robust > > > Key: HBASE-4060 > URL: https://issues.apache.org/jira/browse/HBASE-4060 > Project: HBase > Issue Type: Bug >Reporter: Ted Yu >Assignee: ramkrishna.s.vasudevan > > From Eran Kutner: > My concern is that the region allocation process seems to rely too much on > timing considerations and doesn't seem to take enough measures to guarantee > conflicts do not occur. I understand that in a distributed environment, when > you don't get a timely response from a remote machine you can't know for > sure if it did or did not receive the request, however there are things that > can be done to mitigate this and reduce the conflict time significantly. For > example, when I run dbck it knows that some regions are multiply assigned, > the master could do the same and try to resolve the conflict. Another > approach would be to handle late responses, even if the response from the > remote machine arrives after it was assumed to be dead the master should > have enough information to know it had created a conflict by assigning the > region to another server. An even better solution, I think, is for the RS to > periodically test that it is indeed the rightful owner of every region it > holds and relinquish control over the region if it's not. > Obviously a state where two RSs hold the same region is pathological and can > lead to data loss, as demonstrated in my case. The system should be able to > actively protect itself against such a scenario. It probably doesn't need > saying but there is really nothing worse for a data storage system than data > loss. > In my case the problem didn't happen in the initial phase but after > disabling and enabling a table with about 12K regions. > For more background information, see 'Errors after major compaction' > discussion on u...@hbase.apache.org -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6480) If callQueueSize exceed maxQueueSize, all call will be rejected, do not reject priorityCall
[ https://issues.apache.org/jira/browse/HBASE-6480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468069#comment-13468069 ] stack commented on HBASE-6480: -- [~aoxiang] These should be some upper bound. What is your operational experience running with this patch? The idea is good. This call is probably expensive ' callSize = call.getSize();' What happens when client gets 'callTooBig'? It retries? Can it wait before it retries? > If callQueueSize exceed maxQueueSize, all call will be rejected, do not > reject priorityCall > > > Key: HBASE-6480 > URL: https://issues.apache.org/jira/browse/HBASE-6480 > Project: HBase > Issue Type: Bug >Reporter: binlijin > Fix For: 0.94.3, 0.96.0 > > Attachments: HBASE-6480-94.patch, HBASE-6480-trunk.patch > > > Current if the callQueueSize exceed maxQueueSize, all call will be rejected, > Should we let the priority Call pass through? > Current: > {code} > if ((callSize + callQueueSize.get()) > maxQueueSize) { > Call callTooBig = xxx > return ; > } > if (priorityCallQueue != null && getQosLevel(param) > highPriorityLevel) { > priorityCallQueue.put(call); > updateCallQueueLenMetrics(priorityCallQueue); > } else { > callQueue.put(call); // queue the call; maybe blocked here > updateCallQueueLenMetrics(callQueue); > } > {code} > Should we change it to : > {code} > if (priorityCallQueue != null && getQosLevel(param) > highPriorityLevel) { > priorityCallQueue.put(call); > updateCallQueueLenMetrics(priorityCallQueue); > } else { > if ((callSize + callQueueSize.get()) > maxQueueSize) { >Call callTooBig = xxx >return ; > } > callQueue.put(call); // queue the call; maybe blocked here > updateCallQueueLenMetrics(callQueue); > } > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6491) add limit function at ClientScanner
[ https://issues.apache.org/jira/browse/HBASE-6491?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6491: - Fix Version/s: (was: 0.96.0) Moving out till questions answered. > add limit function at ClientScanner > --- > > Key: HBASE-6491 > URL: https://issues.apache.org/jira/browse/HBASE-6491 > Project: HBase > Issue Type: New Feature > Components: Client >Affects Versions: 0.96.0 >Reporter: ronghai.ma >Assignee: ronghai.ma > Labels: patch > Attachments: ClientScanner.java, HBASE-6491.patch > > > Add a new method in ClientScanner to implement a function like LIMIT in MySQL. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6920) On timeout connecting to master, client can get stuck and never make progress
[ https://issues.apache.org/jira/browse/HBASE-6920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468084#comment-13468084 ] Lars Hofhansl commented on HBASE-6920: -- Hmm... Not offhand. Looking at the patch again... Will this actually fix the problem? The problem is that masterChecked is true when we successfully connected to a master once. If true, we do not try again. So we were successful once, we will never enter the retry loop (since masterChecked == true)... Right? I wonder whether another approach would be to set masterChecked = false at the same spot where master is set to null, but only do that if master was not-null before. > On timeout connecting to master, client can get stuck and never make progress > - > > Key: HBASE-6920 > URL: https://issues.apache.org/jira/browse/HBASE-6920 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.2 >Reporter: Gregory Chanan >Assignee: Gregory Chanan >Priority: Critical > Attachments: HBASE-6920.patch, HBASE-6920-v2.patch > > > HBASE-5058 appears to have introduced an issue where a timeout in > HConnection.getMaster() can cause the client to never be able to connect to > the master. So, for example, an HBaseAdmin object can never successfully be > initialized. > The issue is here: > {code} > if (tryMaster.isMasterRunning()) { > this.master = tryMaster; > this.masterLock.notifyAll(); > break; > } > {code} > If isMasterRunning times out, it throws an UndeclaredThrowableException, > which is already not ideal, because it can be returned to the application. > But if the first call to getMaster succeeds, it will set masterChecked = > true, which makes us never try to reconnect; that is, we will set this.master > = null and just throw MasterNotRunningExceptions, without even trying to > connect. > I tried out a 94 client (actually a 92 client with some 94 patches) on a > cluster with some network issues, and it would constantly get stuck as > described above. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6805) Extend co-processor framework to provide observers for filter operations
[ https://issues.apache.org/jira/browse/HBASE-6805?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468086#comment-13468086 ] Gary Helmling commented on HBASE-6805: -- Looking at the encryption example here, it seems like you could provide that with the existing coprocessor hooks: Option A: # In {{EncryptingRegionObserver.preScannerOpen()}}, if any Filter is set on the Scan object, wrap it in a custom {{DecryptingFilterWrapper}}. This would just decrypt the KVs before passing them on to the client provided Filter, essentially doing the same work your example preFilterXXX methods are doing. # In {{EncryptingRegionObserver.postScannerNext()}}, again decrypt the final KVs being returned to the client, same as your example. The duplicate decryption here seems unnecessary, but it should give you the same results as your provided example, without the need to add a batch of pre/postFilterXXX hooks to RegionObservers. Option B: # In {{EncryptingRegionObserver.preStoreScannerOpen()}} return a custom KeyValueScanner implementation that extends or wraps the default StoreScanner implementation. Note that this would still be a little tricky since filters are applied down in ScanQueryMatcher. For decryption what you would really want is to hook in above the StoreFileScanners and MemStoreScanners used internally by StoreScanner, but below the ScanQueryMatcher operations, so that you can decrypt each KV once as it's read. Seems like that would currently require duplicating a fair amount of StoreScanner functionality. Maybe something needs to be added to better hook in to this data reading layer? The main issue I see is that the added hooks fuzz the line between Filters and RegionObservers and their areas of responsibility. It doesn't seem like we should really need pre/postFilterXXX hooks, because that's what filters are supposed to provide. And of course adding more Observer hooks does have a cost in increasing complexity of the coprocessor interfaces and added overhead (especially in hot code paths). Are there really cases that require the pre/postFilter hooks that can't be accomplished by having a RegionObserver wrap gets/scans with it's own Filter implementation that coordinates with the RegionObserver instance? > Extend co-processor framework to provide observers for filter operations > > > Key: HBASE-6805 > URL: https://issues.apache.org/jira/browse/HBASE-6805 > Project: HBase > Issue Type: Sub-task > Components: Coprocessors >Affects Versions: 0.96.0 >Reporter: Jason Dai > Attachments: extend_coprocessor.patch > > > There are several filter operations (e.g., filterKeyValue, filterRow, > transform, etc.) at the region server side that either exclude KVs from the > returned results, or transform the returned KV. We need to provide observers > (e.g., preFilterKeyValue and postFilterKeyValue) for these operations in the > same way as the observers for other data access operations (e.g., preGet and > postGet). This extension is needed to support DOT (e.g., extracting > individual fields from the document in the observers before passing them to > the related filter operations) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5699) Run with > 1 WAL in HRegionServer
[ https://issues.apache.org/jira/browse/HBASE-5699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468087#comment-13468087 ] Hudson commented on HBASE-5699: --- Integrated in HBase-TRUNK #3408 (See [https://builds.apache.org/job/HBase-TRUNK/3408/]) HBASE-5699 Refactor HLog into an interface (Revision 1393126) Result = FAILURE stack : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/backup/example/LongTermArchivingHFileCleaner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/fs/HFileSystem.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/HLogInputFormat.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/WALPlayer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/CleanerChore.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/HFileCleaner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/cleaner/LogCleaner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/Compressor.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/FSHLog.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogFactory.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogMetrics.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogPrettyPrinter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogReader.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/SequenceFileLogWriter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALActionsListener.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/WALCoprocessorHost.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/HMerge.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/util/MetaUtils.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/backup/example/TestZooKeeperTableArchiveClient.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestRowProcessorEndpoint.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestWALObserver.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/fs/TestBlockReorder.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/mapreduce/TestHLogRecordReader.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestDistributedLogSplitting.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCacheOnWriteInSchema.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCompactSelection.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestSplitTransaction.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/FaultySequenceFileLogReader.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogPerformanceEvaluation.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/HLogUtilsForTests.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLog.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLo
[jira] [Commented] (HBASE-6914) Scans/Gets/Mutations don't give a good error if the table is disabled.
[ https://issues.apache.org/jira/browse/HBASE-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468088#comment-13468088 ] Hudson commented on HBASE-6914: --- Integrated in HBase-TRUNK #3408 (See [https://builds.apache.org/job/HBase-TRUNK/3408/]) HBASE-6914 Scans/Gets/Mutations don't give a good error if the table is disabled. (Revision 1393163) Result = FAILURE eclark : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java > Scans/Gets/Mutations don't give a good error if the table is disabled. > -- > > Key: HBASE-6914 > URL: https://issues.apache.org/jira/browse/HBASE-6914 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 0.92.3, 0.94.3, 0.96.0 > > Attachments: HBASE-6914-092-3.patch, HBASE-6914-094-3.patch, > HBASE-6914-0.patch, HBASE-6914-1.patch, HBASE-6914-2.patch, HBASE-6914-3.patch > > > Scan a table that is disabled will have the client retry multiple times and > then will error out with NotServingRegionException. If the table is disabled > there's no need to re-try and the message should be more explicit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6926) Cleanup some of the javadoc warnings.
[ https://issues.apache.org/jira/browse/HBASE-6926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468089#comment-13468089 ] Hudson commented on HBASE-6926: --- Integrated in HBase-TRUNK #3408 (See [https://builds.apache.org/job/HBase-TRUNK/3408/]) HBASE-6926 Cleanup some of the javadoc warnings (Revision 1393168) Result = FAILURE stack : Files : * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/KeyValue.java * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java * /hbase/trunk/hbase-hadoop1-compat/src/main/java/org/apache/hadoop/hbase/metrics/BaseMetricsSourceImpl.java * /hbase/trunk/hbase-hadoop2-compat/src/main/java/org/apache/hadoop/metrics2/lib/DynamicMetricsRegistry.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/ClusterId.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HBaseException.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HBaseIOException.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HColumnDescriptor.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HServerInfo.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/HTableDescriptor.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/MasterAdminProtocol.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/RegionTransition.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HConnection.java > Cleanup some of the javadoc warnings. > - > > Key: HBASE-6926 > URL: https://issues.apache.org/jira/browse/HBASE-6926 > Project: HBase > Issue Type: Bug > Components: documentation >Reporter: stack >Assignee: stack > Fix For: 0.96.0 > > Attachments: javadocleanup.txt > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6479) HFileReaderV1 caching the same parent META block could cause server abot when splitting
[ https://issues.apache.org/jira/browse/HBASE-6479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6479: - Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk. Thanks Chunhui (would need more work getting this into 0.94) > HFileReaderV1 caching the same parent META block could cause server abot when > splitting > --- > > Key: HBASE-6479 > URL: https://issues.apache.org/jira/browse/HBASE-6479 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.0 >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.96.0 > > Attachments: HBASE-6479.patch, HBASE-6479v2.patch, test.patch > > > If the hfile's version is 1 now, when splitting, two daughters would > loadBloomfilter concurrently in the open progress. Because their META block > is the same one(parent's META block), the following expection would be > thrown when doing HFileReaderV1#getMetaBlock > {code} > java.io.IOException: Failed > null-daughterOpener=af73f8c9a9b409531ac211a9a7f92eba > at > org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughters(SplitTransaction.java:367) > at > org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:453) > at > org.apache.hadoop.hbase.regionserver.TestSplitTransaction.testWholesomeSplit(TestSplitTransaction.java:225) > at > org.apache.hadoop.hbase.regionserver.TestSplitTransaction.testWholesomeSplitWithHFileV1(TestSplitTransaction.java:203) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47) > at org.junit.rules.RunRules.evaluate(RunRules.java:18) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) > at org.junit.runners.ParentRunner.run(ParentRunner.java:300) > at > org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:49) > at > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197) > Caused by: java.io.IOException: java.io.IOException: > java.lang.RuntimeException: Cached an already cached block > at > org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:540) > at > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:463) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3784) > at > org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughterRegion(SplitTransaction.java:506) > at > org.apache.hadoop.hbase.regionserver.SplitTransaction$DaughterOpener.run(SplitTransaction.java:486) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.io.IOException: java.lang.RuntimeException: Cached an already > cached block > at > org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:424) > at org.apache.hadoop.hbase.regionserver.
[jira] [Updated] (HBASE-6919) Remove unnecessary cast from Bytes.readVLong
[ https://issues.apache.org/jira/browse/HBASE-6919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-6919: - Fix Version/s: 0.96.0 0.94.3 > Remove unnecessary cast from Bytes.readVLong > > > Key: HBASE-6919 > URL: https://issues.apache.org/jira/browse/HBASE-6919 > Project: HBase > Issue Type: Bug >Reporter: James Taylor >Priority: Minor > Fix For: 0.94.3, 0.96.0 > > > Remove the throws IOException so that caller doesn't have to catch and ignore. > public static long readVLong(final byte [] buffer, final int offset) > throws IOException > Also, add > public static int readVInt(final byte [] buffer, final int offset) > throws IOException { > return (int)readVLong(buffer,offset); > } > and these are useful too: > /** > * Put long as variable length encoded number at the offset in > * the result byte array. > * @param vint Integer to make a vint of. > * @param result buffer to put vint into > * @return Vint length in bytes of vint > */ > public static int vintToBytes(byte[] result, int offset, final long vint) > { > long i = vint; > if (i >= -112 && i <= 127) { > result[offset] = (byte) i; > return 1; > } > int len = -112; > if (i < 0) { > i ^= -1L; // take one's complement' > len = -120; > } > long tmp = i; > while (tmp != 0) { > tmp = tmp >> 8; > len--; > } > result[offset++] = (byte) len; > len = (len < -120) ? -(len + 120) : -(len + 112); > for (int idx = len; idx != 0; idx--) { > int shiftbits = (idx - 1) * 8; > long mask = 0xFFL << shiftbits; > result[offset++] = (byte)((i & mask) >> shiftbits); > } > return len + 1; > } > /** > * Decode a vint from the buffer pointed at to by ptr and > * increment the offset of the ptr by the length of the > * vint. > * @param ptr a pointer to a byte array buffer > * @return the decoded vint value as an int > */ > public static int vintFromBytes(ImmutableBytesWritable ptr) { > return (int) vlongFromBytes(ptr); > } > > /** > * Decode a vint from the buffer pointed at to by ptr and > * increment the offset of the ptr by the length of the > * vint. > * @param ptr a pointer to a byte array buffer > * @return the decoded vint value as a long > */ > public static long vlongFromBytes(ImmutableBytesWritable ptr) { > final byte [] buffer = ptr.get(); > final int offset = ptr.getOffset(); > byte firstByte = buffer[offset]; > int len = WritableUtils.decodeVIntSize(firstByte); > if (len == 1) { > ptr.set(buffer, offset+1, ptr.getLength()); > return firstByte; > } > long i = 0; > for (int idx = 0; idx < len-1; idx++) { > byte b = buffer[offset + 1 + idx]; > i = i << 8; > i = i | (b & 0xFF); > } > ptr.set(buffer, offset+len, ptr.getLength()); > return (WritableUtils.isNegativeVInt(firstByte) ? ~i : i); > } -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6479) HFileReaderV1 caching the same parent META block could cause server abot when splitting
[ https://issues.apache.org/jira/browse/HBASE-6479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6479: - Attachment: 6479v2.txt Here is what I committed. Had to massage a little to get it in. Thanks for the patch Chunhui. > HFileReaderV1 caching the same parent META block could cause server abot when > splitting > --- > > Key: HBASE-6479 > URL: https://issues.apache.org/jira/browse/HBASE-6479 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.0 >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.96.0 > > Attachments: 6479v2.txt, HBASE-6479.patch, HBASE-6479v2.patch, > test.patch > > > If the hfile's version is 1 now, when splitting, two daughters would > loadBloomfilter concurrently in the open progress. Because their META block > is the same one(parent's META block), the following expection would be > thrown when doing HFileReaderV1#getMetaBlock > {code} > java.io.IOException: Failed > null-daughterOpener=af73f8c9a9b409531ac211a9a7f92eba > at > org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughters(SplitTransaction.java:367) > at > org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:453) > at > org.apache.hadoop.hbase.regionserver.TestSplitTransaction.testWholesomeSplit(TestSplitTransaction.java:225) > at > org.apache.hadoop.hbase.regionserver.TestSplitTransaction.testWholesomeSplitWithHFileV1(TestSplitTransaction.java:203) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47) > at org.junit.rules.RunRules.evaluate(RunRules.java:18) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) > at org.junit.runners.ParentRunner.run(ParentRunner.java:300) > at > org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:49) > at > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197) > Caused by: java.io.IOException: java.io.IOException: > java.lang.RuntimeException: Cached an already cached block > at > org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:540) > at > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:463) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3784) > at > org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughterRegion(SplitTransaction.java:506) > at > org.apache.hadoop.hbase.regionserver.SplitTransaction$DaughterOpener.run(SplitTransaction.java:486) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.io.IOException: java.lang.RuntimeException: Cached an already > cached block > at > org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:424) > at org.apache.hadoop.hbase.regionserver.Store.(Store.ja
[jira] [Updated] (HBASE-6479) HFileReaderV1 caching the same parent META block could cause server abort when splitting
[ https://issues.apache.org/jira/browse/HBASE-6479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-6479: -- Summary: HFileReaderV1 caching the same parent META block could cause server abort when splitting (was: HFileReaderV1 caching the same parent META block could cause server abot when splitting) > HFileReaderV1 caching the same parent META block could cause server abort > when splitting > > > Key: HBASE-6479 > URL: https://issues.apache.org/jira/browse/HBASE-6479 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.0 >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.96.0 > > Attachments: 6479v2.txt, HBASE-6479.patch, HBASE-6479v2.patch, > test.patch > > > If the hfile's version is 1 now, when splitting, two daughters would > loadBloomfilter concurrently in the open progress. Because their META block > is the same one(parent's META block), the following expection would be > thrown when doing HFileReaderV1#getMetaBlock > {code} > java.io.IOException: Failed > null-daughterOpener=af73f8c9a9b409531ac211a9a7f92eba > at > org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughters(SplitTransaction.java:367) > at > org.apache.hadoop.hbase.regionserver.SplitTransaction.execute(SplitTransaction.java:453) > at > org.apache.hadoop.hbase.regionserver.TestSplitTransaction.testWholesomeSplit(TestSplitTransaction.java:225) > at > org.apache.hadoop.hbase.regionserver.TestSplitTransaction.testWholesomeSplitWithHFileV1(TestSplitTransaction.java:203) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) > at java.lang.reflect.Method.invoke(Method.java:597) > at > org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:45) > at > org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:15) > at > org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:42) > at > org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:20) > at > org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:28) > at > org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:30) > at org.junit.rules.TestWatcher$1.evaluate(TestWatcher.java:47) > at org.junit.rules.RunRules.evaluate(RunRules.java:18) > at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:263) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:68) > at > org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:47) > at org.junit.runners.ParentRunner$3.run(ParentRunner.java:231) > at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:60) > at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:229) > at org.junit.runners.ParentRunner.access$000(ParentRunner.java:50) > at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:222) > at org.junit.runners.ParentRunner.run(ParentRunner.java:300) > at > org.eclipse.jdt.internal.junit4.runner.JUnit4TestReference.run(JUnit4TestReference.java:49) > at > org.eclipse.jdt.internal.junit.runner.TestExecution.run(TestExecution.java:38) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:467) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.runTests(RemoteTestRunner.java:683) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.run(RemoteTestRunner.java:390) > at > org.eclipse.jdt.internal.junit.runner.RemoteTestRunner.main(RemoteTestRunner.java:197) > Caused by: java.io.IOException: java.io.IOException: > java.lang.RuntimeException: Cached an already cached block > at > org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:540) > at > org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:463) > at > org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:3784) > at > org.apache.hadoop.hbase.regionserver.SplitTransaction.openDaughterRegion(SplitTransaction.java:506) > at > org.apache.hadoop.hbase.regionserver.SplitTransaction$DaughterOpener.run(SplitTransaction.java:486) > at java.lang.Thread.run(Thread.java:662) > Caused by: java.io.IOException: java.lang.RuntimeException: Cached an already > cached block > at > org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:4
[jira] [Created] (HBASE-6927) WrongFS using HRegionInfo.getTableDesc() and different fs for hbase.root and fs.defaultFS
Matteo Bertozzi created HBASE-6927: -- Summary: WrongFS using HRegionInfo.getTableDesc() and different fs for hbase.root and fs.defaultFS Key: HBASE-6927 URL: https://issues.apache.org/jira/browse/HBASE-6927 Project: HBase Issue Type: Bug Affects Versions: 0.94.1, 0.92.2, 0.96.0 Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Calling HRegionInfo.getTableDesc() with different fs schema for hbase.root and fs.defaultFS raises "IllegalArgumentException: Wrong FS" exception. HRegionInfo.getTableDesc() is called only by bin/region_mover.rb to get the table name and can be easily replaced, getTableDesc() is also deprecated. The main problem is that getTableDesc() doesn't replace fs.defaultFS with hbase.root as all the other hbase code (all the code does this, except getTableDesc) {code} Configuration c = HBaseConfiguration.create(); c.set("fs.defaultFS", c.get(HConstants.HBASE_DIR)); c.set("fs.default.name", c.get(HConstants.HBASE_DIR)); {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6927) WrongFS using HRegionInfo.getTableDesc() and different fs for hbase.root and fs.defaultFS
[ https://issues.apache.org/jira/browse/HBASE-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-6927: --- Attachment: HBASE-6927-v0.patch > WrongFS using HRegionInfo.getTableDesc() and different fs for hbase.root and > fs.defaultFS > - > > Key: HBASE-6927 > URL: https://issues.apache.org/jira/browse/HBASE-6927 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.2, 0.94.1, 0.96.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Attachments: HBASE-6927-v0.patch > > > Calling HRegionInfo.getTableDesc() with different fs schema for hbase.root > and fs.defaultFS raises "IllegalArgumentException: Wrong FS" exception. > HRegionInfo.getTableDesc() is called only by bin/region_mover.rb to get the > table name and can be easily replaced, getTableDesc() is also deprecated. > The main problem is that getTableDesc() doesn't replace fs.defaultFS with > hbase.root as all the other hbase code (all the code does this, except > getTableDesc) > {code} > Configuration c = HBaseConfiguration.create(); > c.set("fs.defaultFS", c.get(HConstants.HBASE_DIR)); > c.set("fs.default.name", c.get(HConstants.HBASE_DIR)); > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6927) WrongFS using HRegionInfo.getTableDesc() and different fs for hbase.root and fs.defaultFS
[ https://issues.apache.org/jira/browse/HBASE-6927?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matteo Bertozzi updated HBASE-6927: --- Status: Patch Available (was: Open) > WrongFS using HRegionInfo.getTableDesc() and different fs for hbase.root and > fs.defaultFS > - > > Key: HBASE-6927 > URL: https://issues.apache.org/jira/browse/HBASE-6927 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.1, 0.92.2, 0.96.0 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Attachments: HBASE-6927-v0.patch > > > Calling HRegionInfo.getTableDesc() with different fs schema for hbase.root > and fs.defaultFS raises "IllegalArgumentException: Wrong FS" exception. > HRegionInfo.getTableDesc() is called only by bin/region_mover.rb to get the > table name and can be easily replaced, getTableDesc() is also deprecated. > The main problem is that getTableDesc() doesn't replace fs.defaultFS with > hbase.root as all the other hbase code (all the code does this, except > getTableDesc) > {code} > Configuration c = HBaseConfiguration.create(); > c.set("fs.defaultFS", c.get(HConstants.HBASE_DIR)); > c.set("fs.default.name", c.get(HConstants.HBASE_DIR)); > {code} -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6770) Allow scanner setCaching to specify size instead of number of rows
[ https://issues.apache.org/jira/browse/HBASE-6770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Karthik Ranganathan reassigned HBASE-6770: -- Assignee: Chen Jin (was: Michal Gregorczyk) > Allow scanner setCaching to specify size instead of number of rows > -- > > Key: HBASE-6770 > URL: https://issues.apache.org/jira/browse/HBASE-6770 > Project: HBase > Issue Type: Sub-task > Components: Client, regionserver >Reporter: Karthik Ranganathan >Assignee: Chen Jin > > Currently, we have the following api's to customize the behavior of scans: > setCaching() - how many rows to cache on client to speed up scans > setBatch() - max columns per row to return per row to prevent a very large > response. > Ideally, we should be able to specify a memory buffer size because: > 1. that would take care of both of these use cases. > 2. it does not need any knowledge of the size of the rows or cells, as the > final thing we are worried about is the available memory. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6914) Scans/Gets/Mutations don't give a good error if the table is disabled.
[ https://issues.apache.org/jira/browse/HBASE-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468113#comment-13468113 ] Hudson commented on HBASE-6914: --- Integrated in HBase-0.92 #592 (See [https://builds.apache.org/job/HBase-0.92/592/]) HBASE-6914 Scans/Gets/Mutations don't give a good error if the table is disabled. (Revision 1393166) Result = FAILURE eclark : Files : * /hbase/branches/0.92/CHANGES.txt * /hbase/branches/0.92/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java * /hbase/branches/0.92/src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java > Scans/Gets/Mutations don't give a good error if the table is disabled. > -- > > Key: HBASE-6914 > URL: https://issues.apache.org/jira/browse/HBASE-6914 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 0.92.3, 0.94.3, 0.96.0 > > Attachments: HBASE-6914-092-3.patch, HBASE-6914-094-3.patch, > HBASE-6914-0.patch, HBASE-6914-1.patch, HBASE-6914-2.patch, HBASE-6914-3.patch > > > Scan a table that is disabled will have the client retry multiple times and > then will error out with NotServingRegionException. If the table is disabled > there's no need to re-try and the message should be more explicit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6914) Scans/Gets/Mutations don't give a good error if the table is disabled.
[ https://issues.apache.org/jira/browse/HBASE-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13468115#comment-13468115 ] Ted Yu commented on HBASE-6914: --- 0.92 build #592 seems to indicate that 0.92 build is broken. > Scans/Gets/Mutations don't give a good error if the table is disabled. > -- > > Key: HBASE-6914 > URL: https://issues.apache.org/jira/browse/HBASE-6914 > Project: HBase > Issue Type: Improvement > Components: Client >Reporter: Elliott Clark >Assignee: Elliott Clark > Fix For: 0.92.3, 0.94.3, 0.96.0 > > Attachments: HBASE-6914-092-3.patch, HBASE-6914-094-3.patch, > HBASE-6914-0.patch, HBASE-6914-1.patch, HBASE-6914-2.patch, HBASE-6914-3.patch > > > Scan a table that is disabled will have the client retry multiple times and > then will error out with NotServingRegionException. If the table is disabled > there's no need to re-try and the message should be more explicit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira