[jira] [Updated] (HBASE-6698) Refactor checkAndPut and checkAndDelete to use doMiniBatchMutation
[ https://issues.apache.org/jira/browse/HBASE-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-6698: -- Attachment: (was: HBASE-6698_3.patch) Refactor checkAndPut and checkAndDelete to use doMiniBatchMutation -- Key: HBASE-6698 URL: https://issues.apache.org/jira/browse/HBASE-6698 Project: HBase Issue Type: Improvement Reporter: ramkrishna.s.vasudevan Fix For: 0.96.0 Attachments: HBASE-6698_1.patch, HBASE-6698_2.patch, HBASE-6698.patch Currently the checkAndPut and checkAndDelete api internally calls the internalPut and internalDelete. May be we can just call doMiniBatchMutation only. This will help in future like if we have some hooks and the CP handles certain cases in the doMiniBatchMutation the same can be done while doing a put thro checkAndPut or while doing a delete thro checkAndDelete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-879) When dfs restarts or moves blocks around, hbase regionservers don't notice
[ https://issues.apache.org/jira/browse/HBASE-879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449443#comment-13449443 ] stack commented on HBASE-879: - I'm fine closing this. The issue Michael talks of in original comment is not an issue any more; the DFSClient should pick up blocks in new location getting new block locations list from namenode. Andrew's comment on hbase being able to survive a dfs restart under it is still to be fixed (we have other issues to cover this though) When dfs restarts or moves blocks around, hbase regionservers don't notice -- Key: HBASE-879 URL: https://issues.apache.org/jira/browse/HBASE-879 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.18.1, 0.19.0 Reporter: Michael Bieniosek Since the hbase regionservers use a DFSClient to keep handles open to the dfs, if the dfs blocks move around (typically because of a dfs restart, but can also happen if datanodes die or blocks get shuffled around), the regionserver will be unable to service the region. It would be nice if the DFSClient that the regionservers use could notice this case and refresh the block list. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6698) Refactor checkAndPut and checkAndDelete to use doMiniBatchMutation
[ https://issues.apache.org/jira/browse/HBASE-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449444#comment-13449444 ] ramkrishna.s.vasudevan commented on HBASE-6698: --- Deleted the attachment as the attached had one problem while creating patch. Refactor checkAndPut and checkAndDelete to use doMiniBatchMutation -- Key: HBASE-6698 URL: https://issues.apache.org/jira/browse/HBASE-6698 Project: HBase Issue Type: Improvement Reporter: ramkrishna.s.vasudevan Fix For: 0.96.0 Attachments: HBASE-6698_1.patch, HBASE-6698_2.patch, HBASE-6698.patch Currently the checkAndPut and checkAndDelete api internally calls the internalPut and internalDelete. May be we can just call doMiniBatchMutation only. This will help in future like if we have some hooks and the CP handles certain cases in the doMiniBatchMutation the same can be done while doing a put thro checkAndPut or while doing a delete thro checkAndDelete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6495) HBaseAdmin shouldn't expect HConnection to be an HConnectionImplementation
[
https://issues.apache.org/jira/browse/HBASE-6495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449451#comment-13449451
]
Hudson commented on HBASE-6495:
---
Integrated in HBase-TRUNK #3309 (See
[https://builds.apache.org/job/HBase-TRUNK/3309/])
HBASE-6495 HBaseAdmin shouldn't expect HConnection to be an
HConnectionImplementation (Revision 1381467)
Result = FAILURE
stack :
Files :
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HConnection.java
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
HBaseAdmin shouldn't expect HConnection to be an HConnectionImplementation
--
Key: HBASE-6495
URL: https://issues.apache.org/jira/browse/HBASE-6495
Project: HBase
Issue Type: Bug
Affects Versions: 0.96.0, 0.94.1
Reporter: Jesse Yates
Assignee: Jesse Yates
Fix For: 0.96.0
Attachments: hbase-6495-v0.patch
Currently, the HBaseAdmin has a constructor that takes an HConnection, but
then immediately casts it to an HConnectionManager.HConnectionImplementation:
{code}
public HBaseAdmin(HConnection connection)
throws MasterNotRunningException, ZooKeeperConnectionException {
this.conf = connection.getConfiguration();
// We want the real class, without showing it our public interface,
// hence the cast.
this.connection =
(HConnectionManager.HConnectionImplementation)connection;
{code}
However, this breaks the explicit contract in the javadocs and makes it
basically impossible to mock out the hbaseadmin.
We need to either make the hbaseadmin use a basic HConnection and optimize
for cases where its smarter or bring up the couple of methods in
HConnectionManager.HConnectionImplementation to the HConnection interface.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449453#comment-13449453 ] stack commented on HBASE-6659: -- bq. They are all consumed in org.apache.hadoop.hbase.master namespace. Since ServerManager doesn't implement interface, currently I don't see how LastFlushedSequenceIds can be encapsulated within ServerManager. Sorry. I don't follow what you are saying above. updateLastFlushedSequenceIds is implemented in HMaster and used by ServerManager. So ServerManager can use updateLastFlushedSequenceIds you must add it to MasterServices. Can you better encapsulate this new sequenceid to region accounting by moving it all into ServerManager so you do not have to do this back and forth between SM and HM and you can avoid adding more facility to the MasterServices Interface? Looking at updateLastFlushedSequenceIds implementation I do not see an obvious reason why we could not but I may be missing something. bq. We pass reference of RegionServerServices from region server to SplitLogWorker, then to HLogSplitter. HLogSplitter would be able to consume this new method. The above is basically right (We'd pass a reference to a LastSequenceId Interface -- though the actual instance may be a RegionServerServices that implements LastSequenceId). Do you think this makes sense? Would it be a PITA to do? We'd pass this Interface instead of passing in the crazy RegionServerStatusProtocol masterRef (A LastSequenceId would be easy to mock, a RegionServerStatusProtocol would be a pain) Port HBASE-6508 Filter out edits at log split time -- Key: HBASE-6659 URL: https://issues.apache.org/jira/browse/HBASE-6659 Project: HBase Issue Type: Bug Reporter: Zhihong Ted Yu Assignee: Zhihong Ted Yu Fix For: 0.96.0 Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, 6508-v7.txt, 6508-v7.txt HBASE-6508 is for 0.89-fb branch. This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6720) Optionally limit number of regions balanced in each balancer run
[ https://issues.apache.org/jira/browse/HBASE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449459#comment-13449459 ] stack commented on HBASE-6720: -- Ok, then I change to a +1 on patch (with amendment of HConstant addition on commit) given Elliott's comment. bq. Imo way too much balancing logic has leaked into the HMaster class. I would think that timing and per table balancing should be moved into the Balancer. Yes. We need to add more to the Balancer Interface? Optionally limit number of regions balanced in each balancer run Key: HBASE-6720 URL: https://issues.apache.org/jira/browse/HBASE-6720 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Fix For: 0.96.0, 0.94.3 Attachments: 6720-0.96-v1.txt See discussion on HBASE-3866 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6698) Refactor checkAndPut and checkAndDelete to use doMiniBatchMutation
[ https://issues.apache.org/jira/browse/HBASE-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Priyadarshini updated HBASE-6698: - Attachment: HBASE-6698_3.patch Refactor checkAndPut and checkAndDelete to use doMiniBatchMutation -- Key: HBASE-6698 URL: https://issues.apache.org/jira/browse/HBASE-6698 Project: HBase Issue Type: Improvement Reporter: ramkrishna.s.vasudevan Fix For: 0.96.0 Attachments: HBASE-6698_1.patch, HBASE-6698_2.patch, HBASE-6698_3.patch, HBASE-6698.patch Currently the checkAndPut and checkAndDelete api internally calls the internalPut and internalDelete. May be we can just call doMiniBatchMutation only. This will help in future like if we have some hooks and the CP handles certain cases in the doMiniBatchMutation the same can be done while doing a put thro checkAndPut or while doing a delete thro checkAndDelete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6720) Optionally limit number of regions balanced in each balancer run
[ https://issues.apache.org/jira/browse/HBASE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449463#comment-13449463 ] Elliott Clark commented on HBASE-6720: -- I'm -1 The patch doesn't work if per table balancing is turned off. The patch also fails if the number of regions that need to be moved in one table is larger than the limit. Optionally limit number of regions balanced in each balancer run Key: HBASE-6720 URL: https://issues.apache.org/jira/browse/HBASE-6720 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Fix For: 0.96.0, 0.94.3 Attachments: 6720-0.96-v1.txt See discussion on HBASE-3866 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6713) Stopping meta/root RS may take 50mins when it is in region-splitting
[
https://issues.apache.org/jira/browse/HBASE-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449464#comment-13449464
]
chunhui shen commented on HBASE-6713:
-
{code}
else if (abortRequested) {
if (this.fsOk) {
closeAllRegions(abortRequested); // Don't leave any open file handles
}
} else {
closeAllRegions(abortRequested);
}{code}
As the above, we will closeAllRegions if(!abortRequested||this.fsOk), so we
should do the same for MetaTable regions.
Adding this.compactSplitThread = null is in order to not do
this.compactSplitThread.join() in the HRegionserver#join
Stopping meta/root RS may take 50mins when it is in region-splitting
Key: HBASE-6713
URL: https://issues.apache.org/jira/browse/HBASE-6713
Project: HBase
Issue Type: Bug
Components: regionserver
Affects Versions: 0.94.1
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.3
Attachments: HBASE-6713.patch
When we stop the RS carrying ROOT/META, if it is in the splitting for some
region, the whole stopping process may take 50 mins.
The reason is :
1.ROOT/META region is closed when stopping the regionserver
2.The Split Transaction failed updating META and it will retry
3.The retry num is 100, and the total time is about 50 mins as default;
This configuration is set by
HConnectionManager#setServerSideHConnectionRetries
I think 50 mins is too long to acceptable, my suggested solution is closing
MetaTable regions after the compact/split thread is closed
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6713) Stopping meta/root RS may take 50mins when it is in region-splitting
[ https://issues.apache.org/jira/browse/HBASE-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-6713: Attachment: HBASE-6713v2.patch Updating the patch as Ted's comment Stopping meta/root RS may take 50mins when it is in region-splitting Key: HBASE-6713 URL: https://issues.apache.org/jira/browse/HBASE-6713 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.94.1 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.3 Attachments: HBASE-6713.patch, HBASE-6713v2.patch When we stop the RS carrying ROOT/META, if it is in the splitting for some region, the whole stopping process may take 50 mins. The reason is : 1.ROOT/META region is closed when stopping the regionserver 2.The Split Transaction failed updating META and it will retry 3.The retry num is 100, and the total time is about 50 mins as default; This configuration is set by HConnectionManager#setServerSideHConnectionRetries I think 50 mins is too long to acceptable, my suggested solution is closing MetaTable regions after the compact/split thread is closed -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5498) Secure Bulk Load
[ https://issues.apache.org/jira/browse/HBASE-5498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Francis Liu updated HBASE-5498: --- Attachment: HBASE-5498_94.patch Here's close to the final patch. It's just missing unit tests for secure bulk load. Putting it up for initial comments. After thinking more about dealing with failure scenarios. I changed the design a bit. The secure bulk load RPC now mimics the existing bulkLoadHFiles() api. Failure became easier to deal with if all the necessary checks are done prior to staging an HFile for the actual bulk load. Given the similarity between the secure and non-secure apis. We should probably consider integrating the secure bulkload RPC into the non-security classes (ie HRegionServer, HRegion, etc.) in 0.96. Which will streamline the implementation. Usage of secure mode is now done under the covers. LoadIncrementalHFiles will automatically switch to using the secure mode if hbase security is enabled. Secure Bulk Load Key: HBASE-5498 URL: https://issues.apache.org/jira/browse/HBASE-5498 Project: HBase Issue Type: Improvement Components: mapred, security Reporter: Francis Liu Assignee: Francis Liu Fix For: 0.96.0 Attachments: HBASE-5498_94.patch, HBASE-5498_draft_94.patch, HBASE-5498_draft.patch Design doc: https://cwiki.apache.org/confluence/display/HCATALOG/HBase+Secure+Bulk+Load Short summary: Security as it stands does not cover the bulkLoadHFiles() feature. Users calling this method will bypass ACLs. Also loading is made more cumbersome in a secure setting because of hdfs privileges. bulkLoadHFiles() moves the data from user's directory to the hbase directory, which would require certain write access privileges set. Our solution is to create a coprocessor which makes use of AuthManager to verify if a user has write access to the table. If so, launches a MR job as the hbase user to do the importing (ie rewrite from text to hfiles). One tricky part this job will have to do is impersonate the calling user when reading the input files. We can do this by expecting the user to pass an hdfs delegation token as part of the secureBulkLoad() coprocessor call and extend an inputformat to make use of that token. The output is written to a temporary directory accessible only by hbase and then bulkloadHFiles() is called. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6476) Replace all occurrances of System.currentTimeMillis() with EnvironmentEdge equivalent
[
https://issues.apache.org/jira/browse/HBASE-6476?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449489#comment-13449489
]
nkeywal commented on HBASE-6476:
I think we're actually on the same line :-).
In src/main code, we should NEVER use System.currentTimeMillis().
In src/test code, we sometimes use it to measure the time spent in the
function. For example:
{noformat}
// some test code
final long max = System.currentTimeMillis() + 1;
while (!condition){ // condition is modified by another thread from src/main
Assert.assertTrue( System.currentTimeMillis() max, condition not met after
10s, exiting);
Thread.sleep(1)
}
// more test code
{noformat}
I've wrote quite a lot of code like this. It helps the test to fail fast and
clearly. In a way, that comes from JUnit/Surefire limitations. But at the end
of the day, there is no reason to use EnvEdge here. On the contrary, we should
NOT use it here. Because we could imagine a test that needs both to have a
specific EnvEdge to fake the time for the src/main method, but still need to
check that we're not spending too much 'real' time in it.
What do you think?
Replace all occurrances of System.currentTimeMillis() with EnvironmentEdge
equivalent
-
Key: HBASE-6476
URL: https://issues.apache.org/jira/browse/HBASE-6476
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Assignee: Lars Hofhansl
Priority: Minor
Fix For: 0.96.0
Attachments: 6476.txt, 6476-v2.txt, 6476-v2.txt
There are still some areas where System.currentTimeMillis() is used in HBase.
In order to make all parts of the code base testable and (potentially) to be
able to configure HBase's notion of time, this should be generally be
replaced with EnvironmentEdgeManager.currentTimeMillis().
How hard would it be to add a maven task that checks for that, so we do not
introduce System.currentTimeMillis back in the future?
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6698) Refactor checkAndPut and checkAndDelete to use doMiniBatchMutation
[ https://issues.apache.org/jira/browse/HBASE-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449493#comment-13449493 ] Hadoop QA commented on HBASE-6698: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12543996/HBASE-6698_3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. -1 javadoc. The javadoc tool appears to have generated 108 warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2799//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2799//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2799//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2799//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2799//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2799//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2799//console This message is automatically generated. Refactor checkAndPut and checkAndDelete to use doMiniBatchMutation -- Key: HBASE-6698 URL: https://issues.apache.org/jira/browse/HBASE-6698 Project: HBase Issue Type: Improvement Reporter: ramkrishna.s.vasudevan Fix For: 0.96.0 Attachments: HBASE-6698_1.patch, HBASE-6698_2.patch, HBASE-6698_3.patch, HBASE-6698.patch Currently the checkAndPut and checkAndDelete api internally calls the internalPut and internalDelete. May be we can just call doMiniBatchMutation only. This will help in future like if we have some hooks and the CP handles certain cases in the doMiniBatchMutation the same can be done while doing a put thro checkAndPut or while doing a delete thro checkAndDelete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6713) Stopping meta/root RS may take 50mins when it is in region-splitting
[ https://issues.apache.org/jira/browse/HBASE-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449518#comment-13449518 ] Hadoop QA commented on HBASE-6713: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12543997/HBASE-6713v2.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. -1 javadoc. The javadoc tool appears to have generated 108 warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.backup.example.TestZooKeeperTableArchiveClient org.apache.hadoop.hbase.catalog.TestMetaReaderEditor Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2800//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2800//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2800//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2800//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2800//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2800//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2800//console This message is automatically generated. Stopping meta/root RS may take 50mins when it is in region-splitting Key: HBASE-6713 URL: https://issues.apache.org/jira/browse/HBASE-6713 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.94.1 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.3 Attachments: HBASE-6713.patch, HBASE-6713v2.patch When we stop the RS carrying ROOT/META, if it is in the splitting for some region, the whole stopping process may take 50 mins. The reason is : 1.ROOT/META region is closed when stopping the regionserver 2.The Split Transaction failed updating META and it will retry 3.The retry num is 100, and the total time is about 50 mins as default; This configuration is set by HConnectionManager#setServerSideHConnectionRetries I think 50 mins is too long to acceptable, my suggested solution is closing MetaTable regions after the compact/split thread is closed -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5631) hbck should handle case where .tableinfo file is missing.
[ https://issues.apache.org/jira/browse/HBASE-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449540#comment-13449540 ] Jonathan Hsieh commented on HBASE-5631: --- I haven't encountered this situation but for a first cut we should do the bare minimum to get a functional table again. I agree that at a bare minimum we can infer the table name and the colfam names from the dir structure. Once those are up, a user could alter the table to fix the other parameters. A second cut (follow on jira issue) could probably look at the newest hfile in each colfam and get some of the parameters (compression settings, encodings, etc). hbck should handle case where .tableinfo file is missing. - Key: HBASE-5631 URL: https://issues.apache.org/jira/browse/HBASE-5631 Project: HBase Issue Type: Improvement Components: hbck Affects Versions: 0.92.2, 0.94.0, 0.96.0 Reporter: Jonathan Hsieh Assignee: Jie Huang Attachments: hbase-5631.patch 0.92+ branches have a .tableinfo file which could be missing from hdfs. hbck should be able to detect and repair this properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5631) hbck should handle case where .tableinfo file is missing.
[ https://issues.apache.org/jira/browse/HBASE-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-5631: - Attachment: hbase-5631-v1.patch Updated the patch file. 1. if get that htd from cache, create a new .tableinfo accordingly. 2. else create a default .tableinfo file with the correct table name, column family and default properties. Hint the end-user to modify that default .tableinfo file if necessary. 3. add the unit test according to Jon's comments. @Jonathon, What do you think? hbck should handle case where .tableinfo file is missing. - Key: HBASE-5631 URL: https://issues.apache.org/jira/browse/HBASE-5631 Project: HBase Issue Type: Improvement Components: hbck Affects Versions: 0.92.2, 0.94.0, 0.96.0 Reporter: Jonathan Hsieh Assignee: Jie Huang Attachments: hbase-5631.patch, hbase-5631-v1.patch 0.92+ branches have a .tableinfo file which could be missing from hdfs. hbck should be able to detect and repair this properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5631) hbck should handle case where .tableinfo file is missing.
[ https://issues.apache.org/jira/browse/HBASE-5631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449604#comment-13449604 ] Jie Huang commented on HBASE-5631: -- Oops. Sorry to miss your latest comment. For the second cut, I haven't done in this patch version. We may try to figure out all those properties we can get from the HFile. Thanks Jonathan. hbck should handle case where .tableinfo file is missing. - Key: HBASE-5631 URL: https://issues.apache.org/jira/browse/HBASE-5631 Project: HBase Issue Type: Improvement Components: hbck Affects Versions: 0.92.2, 0.94.0, 0.96.0 Reporter: Jonathan Hsieh Assignee: Jie Huang Attachments: hbase-5631.patch, hbase-5631-v1.patch 0.92+ branches have a .tableinfo file which could be missing from hdfs. hbck should be able to detect and repair this properly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6324) Direct API calls from embedded Thrift server to regionserver
[ https://issues.apache.org/jira/browse/HBASE-6324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449629#comment-13449629 ] Michael Drzal commented on HBASE-6324: -- Looking at the facebook review, it appears that this mostly done. [~mikhail] what else needs to get done to close this out here? Direct API calls from embedded Thrift server to regionserver Key: HBASE-6324 URL: https://issues.apache.org/jira/browse/HBASE-6324 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin When handling Thrift calls in the regionserver we should not go through RPC to talk to the local regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6327) HLog can be null when create table
[ https://issues.apache.org/jira/browse/HBASE-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449631#comment-13449631 ] Michael Drzal commented on HBASE-6327: -- I'm confused about the state of this jira. [~zhi...@ebaysf.com] said that he added in the patch, but the state is patch available with comments after the fix. Can this be closed out? HLog can be null when create table -- Key: HBASE-6327 URL: https://issues.apache.org/jira/browse/HBASE-6327 Project: HBase Issue Type: Bug Reporter: ShiXing Assignee: ShiXing Fix For: 0.96.0 Attachments: 6327.txt, createTableFailedMaster.log, HBASE-6327-trunk-V1.patch As HBASE-4010 discussed, the HLog can be null. We have meet createTable failed because the no use hlog. When createHReagion, the HLog.LogSyncer is run sync(), in under layer it call the DFSClient.DFSOutputStream.sync(). Then the hlog.closeAndDelete() was called，firstly the HLog.close() will interrupt the LogSyncer, and interrupt DFSClient.DFSOutputStream.sync().The DFSClient.DFSOutputStream will store the exception and throw it when we called DFSClient.close(). The HLog.close() call the writer.close()/DFSClient.close() after interrupt the LogSyncer. And there is no catch exception for the close(). So the Master throw exception to the client. There is no need to throw this exception, further， the hlog is no use. Our cluster is 0.90, the logs is attached, after closing hlog writer, there is no log for the createTable(). The trunk and 0.92, 0.94, we used just one hlog, and if the exception happends, the client will got createTable failed, but indeed ,we expect all the regions for the table can also be assigned. I will give the patch for this later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6327) HLog can be null when create table
[ https://issues.apache.org/jira/browse/HBASE-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-6327: -- Resolution: Fixed Status: Resolved (was: Patch Available) HLog can be null when create table -- Key: HBASE-6327 URL: https://issues.apache.org/jira/browse/HBASE-6327 Project: HBase Issue Type: Bug Reporter: ShiXing Assignee: ShiXing Fix For: 0.96.0 Attachments: 6327.txt, createTableFailedMaster.log, HBASE-6327-trunk-V1.patch As HBASE-4010 discussed, the HLog can be null. We have meet createTable failed because the no use hlog. When createHReagion, the HLog.LogSyncer is run sync(), in under layer it call the DFSClient.DFSOutputStream.sync(). Then the hlog.closeAndDelete() was called，firstly the HLog.close() will interrupt the LogSyncer, and interrupt DFSClient.DFSOutputStream.sync().The DFSClient.DFSOutputStream will store the exception and throw it when we called DFSClient.close(). The HLog.close() call the writer.close()/DFSClient.close() after interrupt the LogSyncer. And there is no catch exception for the close(). So the Master throw exception to the client. There is no need to throw this exception, further， the hlog is no use. Our cluster is 0.90, the logs is attached, after closing hlog writer, there is no log for the createTable(). The trunk and 0.92, 0.94, we used just one hlog, and if the exception happends, the client will got createTable failed, but indeed ,we expect all the regions for the table can also be assigned. I will give the patch for this later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6338) Cache Method in RPC handler
[ https://issues.apache.org/jira/browse/HBASE-6338?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449634#comment-13449634 ] Michael Drzal commented on HBASE-6338: -- [~aoxiang] this seems to have stalled. Cache Method in RPC handler --- Key: HBASE-6338 URL: https://issues.apache.org/jira/browse/HBASE-6338 Project: HBase Issue Type: Improvement Reporter: binlijin Attachments: HBASE-6338-90-2.patch, HBASE-6338-90.patch, HBASE-6338-92-2.patch, HBASE-6338-92.patch, HBASE-6338-94-2.patch, HBASE-6338-94.patch, HBASE-6338-trunk-2.patch, HBASE-6338-trunk.patch Every call in rpc handler a Method will be created, if we cache the method will improve a little. I test with 0.90, Average Class.getMethod(String name, Class... parameterTypes) cost 4780 ns , if we cache it cost 2620 ns. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5937) Refactor HLog into an interface.
[
https://issues.apache.org/jira/browse/HBASE-5937?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449640#comment-13449640
]
Flavio Junqueira commented on HBASE-5937:
-
I have moved static finals to HLog as suggested.
I have also looked at making getReader/getWriter part of HLog. I picked
getReader and extracted the classes using calling it. For each class, I tried
to determine if we can instantiate HLog or have already an instance of it. Here
is a summary:
# HLogInputFormat: Not clear how to instantiate HLog
# HLogPrettyPrinter: Executed through main calls in FSHLog and
HLogPrettyPrinter, so maybe we could pass necessary parameters
# HLogSplitter: Have all parameters
# HRegion: Have HLog object
# ReplicationSource: Not clear how to instantiate HLog
I was also wondering if there are important side-effects in the case we use the
factory to get an HLog object just to get a reader or a writer. I have looked
into the main constructor of FSHLog and I haven't been able to convince myself
that there is a way of executing it without throwing an exception unnecessarily
or having side-effects.
The errors and failures I'm getting for tests now are:
{noformat}
Failed tests:
testRetrying(org.apache.hadoop.hbase.catalog.TestMetaReaderEditor): reader:
count=2, t=null
testExceptionFromCoprocessorDuringPut(org.apache.hadoop.hbase.coprocessor.TestRegionServerCoprocessorExceptionWithAbort):
The put should have failed, as the coprocessor is buggy
testWALCoprocessorReplay(org.apache.hadoop.hbase.coprocessor.TestWALObserver)
test2727(org.apache.hadoop.hbase.regionserver.wal.TestWALReplay)
testReplayEditsWrittenViaHRegion(org.apache.hadoop.hbase.regionserver.wal.TestWALReplay)
testReplayEditsAfterPartialFlush(org.apache.hadoop.hbase.regionserver.wal.TestWALReplay):
expected:30 but was:20
testReplayEditsWrittenIntoWAL(org.apache.hadoop.hbase.regionserver.wal.TestWALReplay):
Flushcount=0
test2727(org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed)
testReplayEditsWrittenViaHRegion(org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed)
testReplayEditsAfterPartialFlush(org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed):
expected:30 but was:20
testReplayEditsWrittenIntoWAL(org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed):
Flushcount=0
{noformat}
{noformat}
testZKClosingNodeVersionMismatch(org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler):
KeeperErrorCode = NodeExists for
/hbase/unassigned/67f6e4f3629fa0bb34fbd5cb604ff44e
testCloseRegion(org.apache.hadoop.hbase.regionserver.handler.TestCloseRegionHandler):
KeeperErrorCode = NodeExists for
/hbase/unassigned/e2b47297b32db6a95e2a310be053172f
testDataCorrectnessReplayingRecoveredEdits(org.apache.hadoop.hbase.regionserver.TestHRegionOnCluster):
Failed 1 action: ExecutionException: 1 time,
testReplayEditsAfterRegionMovedWithMultiCF(org.apache.hadoop.hbase.regionserver.wal.TestWALReplay):
Failed after attempts=10, exceptions:(..)
testReplayEditsAfterRegionMovedWithMultiCF(org.apache.hadoop.hbase.regionserver.wal.TestWALReplayCompressed):
Failed after attempts=10, exceptions:(..)
{noformat}
Refactor HLog into an interface.
Key: HBASE-5937
URL: https://issues.apache.org/jira/browse/HBASE-5937
Project: HBase
Issue Type: Sub-task
Reporter: Li Pi
Assignee: Flavio Junqueira
Priority: Minor
What the summary says. Create HLog interface. Make current implementation use
it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6352) Add copy method in Bytes
[ https://issues.apache.org/jira/browse/HBASE-6352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449645#comment-13449645 ] Michael Drzal commented on HBASE-6352: -- Can this be closed? Add copy method in Bytes Key: HBASE-6352 URL: https://issues.apache.org/jira/browse/HBASE-6352 Project: HBase Issue Type: Improvement Components: util Affects Versions: 0.94.0 Reporter: Jean-Marc Spaggiari Priority: Minor Labels: Bytes, Util Attachments: HBASE_JIRA_6352.patch, HBASE_JIRA_6352_v2.patch, HBASE_JIRA_6352_v3.patch, HBASE_JIRA_6352_v4.patch Original Estimate: 1h Remaining Estimate: 1h Having a copy method into Bytes might be nice to reduce client code size and improve readability. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6352) Add copy method in Bytes
[ https://issues.apache.org/jira/browse/HBASE-6352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449648#comment-13449648 ] Jean-Marc Spaggiari commented on HBASE-6352: Hi Michael, I don't know if the patch has been applied, so I don't know if it can be closed or not. JM Add copy method in Bytes Key: HBASE-6352 URL: https://issues.apache.org/jira/browse/HBASE-6352 Project: HBase Issue Type: Improvement Components: util Affects Versions: 0.94.0 Reporter: Jean-Marc Spaggiari Priority: Minor Labels: Bytes, Util Attachments: HBASE_JIRA_6352.patch, HBASE_JIRA_6352_v2.patch, HBASE_JIRA_6352_v3.patch, HBASE_JIRA_6352_v4.patch Original Estimate: 1h Remaining Estimate: 1h Having a copy method into Bytes might be nice to reduce client code size and improve readability. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3869) RegionServer metrics - add read and write byte-transfer statistics
[ https://issues.apache.org/jira/browse/HBASE-3869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449650#comment-13449650 ] Doug Meil commented on HBASE-3869: -- What's a specific example KV-size type metrics? RegionServer metrics - add read and write byte-transfer statistics -- Key: HBASE-3869 URL: https://issues.apache.org/jira/browse/HBASE-3869 Project: HBase Issue Type: Improvement Reporter: Doug Meil Priority: Minor It would be beneficial to have the data transfer weight of reads and writes per region server. HBASE-3647 split out the read/write metric requests from the uber-request metric - which is great. But there isn't a notion of data transfer weight and this is why it's important: the read metrics are effectively RPC-based. Thus, with a scan caching of 500, there is 1 RPC call every 500 rows read (and 1 'read' metric increment). And this metric doesn't indicate how much data is being transferred (e.g., a read with 50 attributes will probably cost a lot more than a read with 5 attributes). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449738#comment-13449738 ] Francis Liu commented on HBASE-6721: GroupInfo may be retrieved internally through an instance of the GroupInfoManager. This shouldn't leak through the interface. On a similar note Stack was suggesting something different to making it pluggable in HBASE-6723. RegionServer Group based Assignment --- Key: HBASE-6721 URL: https://issues.apache.org/jira/browse/HBASE-6721 Project: HBase Issue Type: New Feature Reporter: Francis Liu Assignee: Vandana Ayyalasomayajula Fix For: 0.96.0 Attachments: HBASE-6721-DesigDoc.pdf In multi-tenant deployments of HBase, it is likely that a RegionServer will be serving out regions from a number of different tables owned by various client applications. Being able to group a subset of running RegionServers and assign specific tables to it, provides a client application a level of isolation and resource allocation. The proposal essentially is to have an AssignmentManager which is aware of RegionServer groups and assigns tables to region servers based on groupings. Load balancing will occur on a per group basis as well. This is essentially a simplification of the approach taken in HBASE-4120. See attached document. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6723) Make AssignmentManager pluggable
[ https://issues.apache.org/jira/browse/HBASE-6723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449746#comment-13449746 ] Francis Liu commented on HBASE-6723: Good suggestion Stack. We'll look into how might that look like. Make AssignmentManager pluggable Key: HBASE-6723 URL: https://issues.apache.org/jira/browse/HBASE-6723 Project: HBase Issue Type: Sub-task Reporter: Francis Liu -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6725) Check and Put can fail when using locks
Nicolas Thiébaud created HBASE-6725: --- Summary: Check and Put can fail when using locks Key: HBASE-6725 URL: https://issues.apache.org/jira/browse/HBASE-6725 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.4 Environment: tested on mac os x Reporter: Nicolas Thiébaud When using CAP with an empty array as expected value and using a lock on the corresponding row, hbase may allow a Put even if the value was already there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6725) Check and Put can fail when using locks
[ https://issues.apache.org/jira/browse/HBASE-6725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nicolas Thiébaud updated HBASE-6725: Attachment: CAPwithLocks.zip Example test case Check and Put can fail when using locks --- Key: HBASE-6725 URL: https://issues.apache.org/jira/browse/HBASE-6725 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.4 Environment: tested on mac os x Reporter: Nicolas Thiébaud Attachments: CAPwithLocks.zip When using CAP with an empty array as expected value and using a lock on the corresponding row, hbase may allow a Put even if the value was already there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu reassigned HBASE-6659: - Assignee: Ted Yu (was: Zhihong Ted Yu) Port HBASE-6508 Filter out edits at log split time -- Key: HBASE-6659 URL: https://issues.apache.org/jira/browse/HBASE-6659 Project: HBase Issue Type: Bug Reporter: Zhihong Ted Yu Assignee: Ted Yu Fix For: 0.96.0 Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, 6508-v7.txt, 6508-v7.txt HBASE-6508 is for 0.89-fb branch. This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-6659: -- Attachment: 6508-v8.txt Patch v8 is also on review board. Hopefully I have addressed Stack's comments. Port HBASE-6508 Filter out edits at log split time -- Key: HBASE-6659 URL: https://issues.apache.org/jira/browse/HBASE-6659 Project: HBase Issue Type: Bug Reporter: Zhihong Ted Yu Assignee: Ted Yu Fix For: 0.96.0 Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, 6508-v7.txt, 6508-v8.txt HBASE-6508 is for 0.89-fb branch. This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-6659: -- Attachment: (was: 6508-v7.txt) Port HBASE-6508 Filter out edits at log split time -- Key: HBASE-6659 URL: https://issues.apache.org/jira/browse/HBASE-6659 Project: HBase Issue Type: Bug Reporter: Zhihong Ted Yu Assignee: Ted Yu Fix For: 0.96.0 Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, 6508-v7.txt, 6508-v8.txt HBASE-6508 is for 0.89-fb branch. This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6352) Add copy method in Bytes
[ https://issues.apache.org/jira/browse/HBASE-6352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6352: - Attachment: HBASE_JIRA_6352_v5.patch What I applied. Add copy method in Bytes Key: HBASE-6352 URL: https://issues.apache.org/jira/browse/HBASE-6352 Project: HBase Issue Type: Improvement Components: util Affects Versions: 0.94.0 Reporter: Jean-Marc Spaggiari Priority: Minor Labels: Bytes, Util Attachments: HBASE_JIRA_6352.patch, HBASE_JIRA_6352_v2.patch, HBASE_JIRA_6352_v3.patch, HBASE_JIRA_6352_v4.patch, HBASE_JIRA_6352_v5.patch Original Estimate: 1h Remaining Estimate: 1h Having a copy method into Bytes might be nice to reduce client code size and improve readability. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6524) Hooks for hbase tracing
[ https://issues.apache.org/jira/browse/HBASE-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449788#comment-13449788 ] stack commented on HBASE-6524: -- @Jonathan Was wondering if we need to add a bit of doc on how to make tracing work to the refguide. In general, this is a 'cool feature' (tm) and it would be a shame if it was buried for the want of someone talking it up. Hooks for hbase tracing --- Key: HBASE-6524 URL: https://issues.apache.org/jira/browse/HBASE-6524 Project: HBase Issue Type: Sub-task Reporter: Jonathan Leavitt Assignee: Jonathan Leavitt Fix For: 0.96.0 Attachments: 6524.addendum, 6524-v2.txt, 6524v3.txt, createTableTrace.png, hbase-6524.diff Includes the hooks that use [htrace|http://www.github.com/cloudera/htrace] library to add dapper-like tracing to hbase. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449791#comment-13449791 ] Ted Yu commented on HBASE-6721: --- bq. through an instance of the GroupInfoManager If I understand correctly, GroupInfoManager singleton would be used inside new AssignmentManager. RegionServer Group based Assignment --- Key: HBASE-6721 URL: https://issues.apache.org/jira/browse/HBASE-6721 Project: HBase Issue Type: New Feature Reporter: Francis Liu Assignee: Vandana Ayyalasomayajula Fix For: 0.96.0 Attachments: HBASE-6721-DesigDoc.pdf In multi-tenant deployments of HBase, it is likely that a RegionServer will be serving out regions from a number of different tables owned by various client applications. Being able to group a subset of running RegionServers and assign specific tables to it, provides a client application a level of isolation and resource allocation. The proposal essentially is to have an AssignmentManager which is aware of RegionServer groups and assigns tables to region servers based on groupings. Load balancing will occur on a per group basis as well. This is essentially a simplification of the approach taken in HBASE-4120. See attached document. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6352) Add copy method in Bytes
[ https://issues.apache.org/jira/browse/HBASE-6352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-6352: - Resolution: Fixed Fix Version/s: 0.96.0 Assignee: Jean-Marc Spaggiari Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Applied to trunk. Thanks for the patch Jean-Marc. I was unconvinced it added anything at first but looking through code, it could save us a couple of lines in lots of places. I added you as a contributor. I assigned this issue to you. Are there others we should assign you? Thanks Michael for bringing this back to our attention. Add copy method in Bytes Key: HBASE-6352 URL: https://issues.apache.org/jira/browse/HBASE-6352 Project: HBase Issue Type: Improvement Components: util Affects Versions: 0.94.0 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Minor Labels: Bytes, Util Fix For: 0.96.0 Attachments: HBASE_JIRA_6352.patch, HBASE_JIRA_6352_v2.patch, HBASE_JIRA_6352_v3.patch, HBASE_JIRA_6352_v4.patch, HBASE_JIRA_6352_v5.patch Original Estimate: 1h Remaining Estimate: 1h Having a copy method into Bytes might be nice to reduce client code size and improve readability. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6715) TestFromClientSide.testCacheOnWriteEvictOnClose is flaky
[
https://issues.apache.org/jira/browse/HBASE-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449798#comment-13449798
]
Jimmy Xiang commented on HBASE-6715:
Sure, committed it to trunk and 0.94. We will see if the issue is gone.
TestFromClientSide.testCacheOnWriteEvictOnClose is flaky
Key: HBASE-6715
URL: https://issues.apache.org/jira/browse/HBASE-6715
Project: HBase
Issue Type: Test
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
Attachments: trunk-6715.patch
Occasionally, this test fails:
{noformat}
expected:2049 but was:2069
Stacktrace
java.lang.AssertionError: expected:2049 but was:2069
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at
org.apache.hadoop.hbase.client.TestFromClientSide.testCacheOnWriteEvictOnClose(TestFromClientSide.java:4248)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{noformat}
It could be because there is other thread still accessing the cache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449805#comment-13449805 ] Hadoop QA commented on HBASE-6659: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12544072/6508-v8.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. -1 javadoc. The javadoc tool appears to have generated 107 warning messages. -1 javac. The applied patch generated 5 javac compiler warnings (more than the trunk's current 4 warnings). -1 findbugs. The patch appears to introduce 7 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2803//console This message is automatically generated. Port HBASE-6508 Filter out edits at log split time -- Key: HBASE-6659 URL: https://issues.apache.org/jira/browse/HBASE-6659 Project: HBase Issue Type: Bug Reporter: Zhihong Ted Yu Assignee: Ted Yu Fix For: 0.96.0 Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, 6508-v7.txt, 6508-v8.txt HBASE-6508 is for 0.89-fb branch. This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[
https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449808#comment-13449808
]
stack commented on HBASE-6659:
--
+1
To me at least, we encapsulate the sequenceid counting inside ServerManager
rather than have it spread between SM and HM (also, don't need to add an
esoteric method to the generic MasterServices Interface). HLogSplitter is
passed a specific Interface for getting last sequenceid rather than a protocol
implementation. I think this better. Thanks for entertaining my reviews.
On commit remove '+ * Copyright The Apache Software Foundation' and change this
'+ * @param master' (param name is different).
This change seems gratuitous but it is from original patch so its ok:
{code}
-regionInfo.getTableName(), completeSequenceId,
+regionInfo.getTableName(), completeSeqId,
{code}
Are the imports in MasterServices needed? (they don't seem to be referred to by
this patch). Remove on commit.
Port HBASE-6508 Filter out edits at log split time
--
Key: HBASE-6659
URL: https://issues.apache.org/jira/browse/HBASE-6659
Project: HBase
Issue Type: Bug
Reporter: Zhihong Ted Yu
Assignee: Ted Yu
Fix For: 0.96.0
Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt,
6508-v7.txt, 6508-v8.txt
HBASE-6508 is for 0.89-fb branch.
This JIRA ports the feature to trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6726) Port HBASE-4465 'Lazy-seek optimization for StoreFile scanners' to 0.92
Ted Yu created HBASE-6726: - Summary: Port HBASE-4465 'Lazy-seek optimization for StoreFile scanners' to 0.92 Key: HBASE-6726 URL: https://issues.apache.org/jira/browse/HBASE-6726 Project: HBase Issue Type: Bug Reporter: Ted Yu This is expected to significantly reduce the amount of disk IO -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449809#comment-13449809 ] stack commented on HBASE-6659: -- Maybe check out the TestHLogMethods failure... unlikely it this change but worth a look I'd say. Port HBASE-6508 Filter out edits at log split time -- Key: HBASE-6659 URL: https://issues.apache.org/jira/browse/HBASE-6659 Project: HBase Issue Type: Bug Reporter: Zhihong Ted Yu Assignee: Ted Yu Fix For: 0.96.0 Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, 6508-v7.txt, 6508-v8.txt HBASE-6508 is for 0.89-fb branch. This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6720) Optionally limit number of regions balanced in each balancer run
[ https://issues.apache.org/jira/browse/HBASE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449811#comment-13449811 ] stack commented on HBASE-6720: -- Thanks Elliott. How so? Won't all be well on the next go around of the balancer? Optionally limit number of regions balanced in each balancer run Key: HBASE-6720 URL: https://issues.apache.org/jira/browse/HBASE-6720 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Fix For: 0.96.0, 0.94.3 Attachments: 6720-0.96-v1.txt See discussion on HBASE-3866 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-6659: -- Attachment: 6508-v9.txt Patch v9 should be close to the final version - fixed compilation error. Port HBASE-6508 Filter out edits at log split time -- Key: HBASE-6659 URL: https://issues.apache.org/jira/browse/HBASE-6659 Project: HBase Issue Type: Bug Reporter: Zhihong Ted Yu Assignee: Ted Yu Fix For: 0.96.0 Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, 6508-v7.txt, 6508-v8.txt, 6508-v9.txt HBASE-6508 is for 0.89-fb branch. This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449817#comment-13449817 ] stack commented on HBASE-6659: -- I took a quick look. +1. Port HBASE-6508 Filter out edits at log split time -- Key: HBASE-6659 URL: https://issues.apache.org/jira/browse/HBASE-6659 Project: HBase Issue Type: Bug Reporter: Zhihong Ted Yu Assignee: Ted Yu Fix For: 0.96.0 Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, 6508-v7.txt, 6508-v8.txt, 6508-v9.txt HBASE-6508 is for 0.89-fb branch. This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6726) Port HBASE-4465 'Lazy-seek optimization for StoreFile scanners' to 0.92
[ https://issues.apache.org/jira/browse/HBASE-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449818#comment-13449818 ] stack commented on HBASE-6726: -- We should be putting bug fixes into point releases? (This is marked as a bug but seems like a feature to me). Port HBASE-4465 'Lazy-seek optimization for StoreFile scanners' to 0.92 --- Key: HBASE-6726 URL: https://issues.apache.org/jira/browse/HBASE-6726 Project: HBase Issue Type: Bug Reporter: Ted Yu This is expected to significantly reduce the amount of disk IO -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449819#comment-13449819 ] ramkrishna.s.vasudevan commented on HBASE-6659: --- @Ted Nice useful backport. Thanks for taking it up. Port HBASE-6508 Filter out edits at log split time -- Key: HBASE-6659 URL: https://issues.apache.org/jira/browse/HBASE-6659 Project: HBase Issue Type: Bug Reporter: Zhihong Ted Yu Assignee: Ted Yu Fix For: 0.96.0 Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, 6508-v7.txt, 6508-v8.txt, 6508-v9.txt HBASE-6508 is for 0.89-fb branch. This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6698) Refactor checkAndPut and checkAndDelete to use doMiniBatchMutation
[ https://issues.apache.org/jira/browse/HBASE-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449821#comment-13449821 ] ramkrishna.s.vasudevan commented on HBASE-6698: --- Latest patch attached. Pls provide your reviews/comments. Refactor checkAndPut and checkAndDelete to use doMiniBatchMutation -- Key: HBASE-6698 URL: https://issues.apache.org/jira/browse/HBASE-6698 Project: HBase Issue Type: Improvement Reporter: ramkrishna.s.vasudevan Fix For: 0.96.0 Attachments: HBASE-6698_1.patch, HBASE-6698_2.patch, HBASE-6698_3.patch, HBASE-6698.patch Currently the checkAndPut and checkAndDelete api internally calls the internalPut and internalDelete. May be we can just call doMiniBatchMutation only. This will help in future like if we have some hooks and the CP handles certain cases in the doMiniBatchMutation the same can be done while doing a put thro checkAndPut or while doing a delete thro checkAndDelete. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-6659: -- Status: Open (was: Patch Available) Port HBASE-6508 Filter out edits at log split time -- Key: HBASE-6659 URL: https://issues.apache.org/jira/browse/HBASE-6659 Project: HBase Issue Type: Bug Reporter: Zhihong Ted Yu Assignee: Ted Yu Fix For: 0.96.0 Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, 6508-v7.txt, 6508-v8.txt, 6508-v9.txt HBASE-6508 is for 0.89-fb branch. This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[
https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449824#comment-13449824
]
Ted Yu commented on HBASE-6659:
---
{code}
Running org.apache.hadoop.hbase.mapreduce.TestHLogRecordReader
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.17 sec
Running org.apache.hadoop.hbase.regionserver.wal.TestHLog
Tests run: 11, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 77.567 sec
Running org.apache.hadoop.hbase.regionserver.wal.TestHLogFiltering
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 40.173 sec
Running org.apache.hadoop.hbase.regionserver.wal.TestHLogMethods
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.719 sec
Running org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit
Tests run: 30, Failures: 0, Errors: 0, Skipped: 1, Time elapsed: 234.204 sec
Results :
Tests run: 47, Failures: 0, Errors: 0, Skipped: 1
{code}
Going to commit.
Port HBASE-6508 Filter out edits at log split time
--
Key: HBASE-6659
URL: https://issues.apache.org/jira/browse/HBASE-6659
Project: HBase
Issue Type: Bug
Reporter: Zhihong Ted Yu
Assignee: Ted Yu
Fix For: 0.96.0
Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt,
6508-v7.txt, 6508-v8.txt, 6508-v9.txt
HBASE-6508 is for 0.89-fb branch.
This JIRA ports the feature to trunk.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6060) Regions's in OPENING state from failed regionservers takes a long time to recover
[ https://issues.apache.org/jira/browse/HBASE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449825#comment-13449825 ] ramkrishna.s.vasudevan commented on HBASE-6060: --- @Stack Recently Jon's mail thread on rewriting AM altogether implies that the scenario mentioned here also will be covered ? Regions's in OPENING state from failed regionservers takes a long time to recover - Key: HBASE-6060 URL: https://issues.apache.org/jira/browse/HBASE-6060 Project: HBase Issue Type: Bug Components: master, regionserver Reporter: Enis Soztutar Assignee: rajeshbabu Fix For: 0.96.0, 0.92.3, 0.94.3 Attachments: 6060-94-v3.patch, 6060-94-v4_1.patch, 6060-94-v4_1.patch, 6060-94-v4.patch, 6060_alternative_suggestion.txt, 6060_suggestion2_based_off_v3.patch, 6060_suggestion_based_off_v3.patch, 6060_suggestion_toassign_rs_wentdown_beforerequest.patch, 6060-trunk_2.patch, 6060-trunk_3.patch, 6060-trunk.patch, 6060-trunk.patch, HBASE-6060-92.patch, HBASE-6060-94.patch, HBASE-6060_latest.patch, HBASE-6060_latest.patch, HBASE-6060_latest.patch, HBASE-6060-trunk_4.patch, HBASE-6060_trunk_5.patch we have seen a pattern in tests, that the regions are stuck in OPENING state for a very long time when the region server who is opening the region fails. My understanding of the process: - master calls rs to open the region. If rs is offline, a new plan is generated (a new rs is chosen). RegionState is set to PENDING_OPEN (only in master memory, zk still shows OFFLINE). See HRegionServer.openRegion(), HMaster.assign() - RegionServer, starts opening a region, changes the state in znode. But that znode is not ephemeral. (see ZkAssign) - Rs transitions zk node from OFFLINE to OPENING. See OpenRegionHandler.process() - rs then opens the region, and changes znode from OPENING to OPENED - when rs is killed between OPENING and OPENED states, then zk shows OPENING state, and the master just waits for rs to change the region state, but since rs is down, that wont happen. - There is a AssignmentManager.TimeoutMonitor, which does exactly guard against these kind of conditions. It periodically checks (every 10 sec by default) the regions in transition to see whether they timedout (hbase.master.assignment.timeoutmonitor.timeout). Default timeout is 30 min, which explains what you and I are seeing. - ServerShutdownHandler in Master does not reassign regions in OPENING state, although it handles other states. Lowering that threshold from the configuration is one option, but still I think we can do better. Will investigate more. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449828#comment-13449828 ] Ted Yu commented on HBASE-6659: --- Integrated to trunk. Thanks for the review, Stack and Ram. Port HBASE-6508 Filter out edits at log split time -- Key: HBASE-6659 URL: https://issues.apache.org/jira/browse/HBASE-6659 Project: HBase Issue Type: Bug Reporter: Zhihong Ted Yu Assignee: Ted Yu Fix For: 0.96.0 Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, 6508-v7.txt, 6508-v8.txt, 6508-v9.txt HBASE-6508 is for 0.89-fb branch. This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6060) Regions's in OPENING state from failed regionservers takes a long time to recover
[ https://issues.apache.org/jira/browse/HBASE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449830#comment-13449830 ] stack commented on HBASE-6060: -- @Ram He's not at 'total rewrite' yet (smile). What you think of that thread Ram? Regions's in OPENING state from failed regionservers takes a long time to recover - Key: HBASE-6060 URL: https://issues.apache.org/jira/browse/HBASE-6060 Project: HBase Issue Type: Bug Components: master, regionserver Reporter: Enis Soztutar Assignee: rajeshbabu Fix For: 0.96.0, 0.92.3, 0.94.3 Attachments: 6060-94-v3.patch, 6060-94-v4_1.patch, 6060-94-v4_1.patch, 6060-94-v4.patch, 6060_alternative_suggestion.txt, 6060_suggestion2_based_off_v3.patch, 6060_suggestion_based_off_v3.patch, 6060_suggestion_toassign_rs_wentdown_beforerequest.patch, 6060-trunk_2.patch, 6060-trunk_3.patch, 6060-trunk.patch, 6060-trunk.patch, HBASE-6060-92.patch, HBASE-6060-94.patch, HBASE-6060_latest.patch, HBASE-6060_latest.patch, HBASE-6060_latest.patch, HBASE-6060-trunk_4.patch, HBASE-6060_trunk_5.patch we have seen a pattern in tests, that the regions are stuck in OPENING state for a very long time when the region server who is opening the region fails. My understanding of the process: - master calls rs to open the region. If rs is offline, a new plan is generated (a new rs is chosen). RegionState is set to PENDING_OPEN (only in master memory, zk still shows OFFLINE). See HRegionServer.openRegion(), HMaster.assign() - RegionServer, starts opening a region, changes the state in znode. But that znode is not ephemeral. (see ZkAssign) - Rs transitions zk node from OFFLINE to OPENING. See OpenRegionHandler.process() - rs then opens the region, and changes znode from OPENING to OPENED - when rs is killed between OPENING and OPENED states, then zk shows OPENING state, and the master just waits for rs to change the region state, but since rs is down, that wont happen. - There is a AssignmentManager.TimeoutMonitor, which does exactly guard against these kind of conditions. It periodically checks (every 10 sec by default) the regions in transition to see whether they timedout (hbase.master.assignment.timeoutmonitor.timeout). Default timeout is 30 min, which explains what you and I are seeing. - ServerShutdownHandler in Master does not reassign regions in OPENING state, although it handles other states. Lowering that threshold from the configuration is one option, but still I think we can do better. Will investigate more. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6726) Port HBASE-4465 'Lazy-seek optimization for StoreFile scanners' to 0.92
[ https://issues.apache.org/jira/browse/HBASE-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449837#comment-13449837 ] Ted Yu commented on HBASE-6726: --- There were 145 improvements in 0.92.2: https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12319888styleName=TextprojectId=12310753Create=Createatl_token=A5KQ-2QAV-T4JA-FDED|20dd5912cca928036b1ca12dd6edb463220c0a85|lin And there were 21 new features in 0.92.2 I think this feature is very important for our users. Port HBASE-4465 'Lazy-seek optimization for StoreFile scanners' to 0.92 --- Key: HBASE-6726 URL: https://issues.apache.org/jira/browse/HBASE-6726 Project: HBase Issue Type: Bug Reporter: Ted Yu This is expected to significantly reduce the amount of disk IO -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6726) Port HBASE-4465 'Lazy-seek optimization for StoreFile scanners' to 0.92
[ https://issues.apache.org/jira/browse/HBASE-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-6726: -- Issue Type: Task (was: Bug) Port HBASE-4465 'Lazy-seek optimization for StoreFile scanners' to 0.92 --- Key: HBASE-6726 URL: https://issues.apache.org/jira/browse/HBASE-6726 Project: HBase Issue Type: Task Reporter: Ted Yu This is expected to significantly reduce the amount of disk IO -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6726) Port HBASE-4465 'Lazy-seek optimization for StoreFile scanners' to 0.92
[ https://issues.apache.org/jira/browse/HBASE-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449845#comment-13449845 ] stack commented on HBASE-6726: -- I think we should be careful what we backport, especially to old branches. We cannot afford to ship a point release that is less stable than those that precede it, especially one made from an old branch. Backporting features, especially invasive big changes, is one sure way of destabilizing code base. Port HBASE-4465 'Lazy-seek optimization for StoreFile scanners' to 0.92 --- Key: HBASE-6726 URL: https://issues.apache.org/jira/browse/HBASE-6726 Project: HBase Issue Type: Task Reporter: Ted Yu This is expected to significantly reduce the amount of disk IO -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[ https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449844#comment-13449844 ] Ted Yu commented on HBASE-6721: --- bq. This is similar to hbase cluster sizing for fault tolerance. I think what I wanted to say was that we may need to reserve some region servers from the Default pool for emergency scenario so that each group can have decent size all the time. RegionServer Group based Assignment --- Key: HBASE-6721 URL: https://issues.apache.org/jira/browse/HBASE-6721 Project: HBase Issue Type: New Feature Reporter: Francis Liu Assignee: Vandana Ayyalasomayajula Fix For: 0.96.0 Attachments: HBASE-6721-DesigDoc.pdf In multi-tenant deployments of HBase, it is likely that a RegionServer will be serving out regions from a number of different tables owned by various client applications. Being able to group a subset of running RegionServers and assign specific tables to it, provides a client application a level of isolation and resource allocation. The proposal essentially is to have an AssignmentManager which is aware of RegionServer groups and assigns tables to region servers based on groupings. Load balancing will occur on a per group basis as well. This is essentially a simplification of the approach taken in HBASE-4120. See attached document. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6726) Port HBASE-4465 'Lazy-seek optimization for StoreFile scanners' to 0.92
[ https://issues.apache.org/jira/browse/HBASE-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449846#comment-13449846 ] stack commented on HBASE-6726: -- Seems like a discussion better done out on dev list. Port HBASE-4465 'Lazy-seek optimization for StoreFile scanners' to 0.92 --- Key: HBASE-6726 URL: https://issues.apache.org/jira/browse/HBASE-6726 Project: HBase Issue Type: Task Reporter: Ted Yu This is expected to significantly reduce the amount of disk IO -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[
https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449848#comment-13449848
]
Francis Liu commented on HBASE-6721:
{quote}If I understand correctly, GroupInfoManager singleton would be used
inside new AssignmentManager{quote}
Yep it will create an instance of it internally.
RegionServer Group based Assignment
---
Key: HBASE-6721
URL: https://issues.apache.org/jira/browse/HBASE-6721
Project: HBase
Issue Type: New Feature
Reporter: Francis Liu
Assignee: Vandana Ayyalasomayajula
Fix For: 0.96.0
Attachments: HBASE-6721-DesigDoc.pdf
In multi-tenant deployments of HBase, it is likely that a RegionServer will
be serving out regions from a number of different tables owned by various
client applications. Being able to group a subset of running RegionServers
and assign specific tables to it, provides a client application a level of
isolation and resource allocation.
The proposal essentially is to have an AssignmentManager which is aware of
RegionServer groups and assigns tables to region servers based on groupings.
Load balancing will occur on a per group basis as well.
This is essentially a simplification of the approach taken in HBASE-4120. See
attached document.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6726) Port HBASE-4465 'Lazy-seek optimization for StoreFile scanners' to 0.92
[ https://issues.apache.org/jira/browse/HBASE-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449849#comment-13449849 ] Ted Yu commented on HBASE-6726: --- Will post to dev@ list. This feature has been in production @ Facebook and EBay. It is a stable feature. Port HBASE-4465 'Lazy-seek optimization for StoreFile scanners' to 0.92 --- Key: HBASE-6726 URL: https://issues.apache.org/jira/browse/HBASE-6726 Project: HBase Issue Type: Task Reporter: Ted Yu This is expected to significantly reduce the amount of disk IO -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6327) HLog can be null when create table
[ https://issues.apache.org/jira/browse/HBASE-6327?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449852#comment-13449852 ] stack commented on HBASE-6327: -- @Michael This patch is in the code base so yeah, should be resolved (though it seems like there was some late dissent...post-commit) HLog can be null when create table -- Key: HBASE-6327 URL: https://issues.apache.org/jira/browse/HBASE-6327 Project: HBase Issue Type: Bug Reporter: ShiXing Assignee: ShiXing Fix For: 0.96.0 Attachments: 6327.txt, createTableFailedMaster.log, HBASE-6327-trunk-V1.patch As HBASE-4010 discussed, the HLog can be null. We have meet createTable failed because the no use hlog. When createHReagion, the HLog.LogSyncer is run sync(), in under layer it call the DFSClient.DFSOutputStream.sync(). Then the hlog.closeAndDelete() was called，firstly the HLog.close() will interrupt the LogSyncer, and interrupt DFSClient.DFSOutputStream.sync().The DFSClient.DFSOutputStream will store the exception and throw it when we called DFSClient.close(). The HLog.close() call the writer.close()/DFSClient.close() after interrupt the LogSyncer. And there is no catch exception for the close(). So the Master throw exception to the client. There is no need to throw this exception, further， the hlog is no use. Our cluster is 0.90, the logs is attached, after closing hlog writer, there is no log for the createTable(). The trunk and 0.92, 0.94, we used just one hlog, and if the exception happends, the client will got createTable failed, but indeed ,we expect all the regions for the table can also be assigned. I will give the patch for this later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6721) RegionServer Group based Assignment
[
https://issues.apache.org/jira/browse/HBASE-6721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449854#comment-13449854
]
Francis Liu commented on HBASE-6721:
{quote}I think what I wanted to say was that we may need to reserve some region
servers from the Default pool for emergency scenario so that each group can
have decent size all the time.{quote}
I see, that could be something we could add later on. There are some scenarios
we have to think more thoroughly about. A conservative group sizing and manual
intervention should be good enough for now? Let's get the basics fully baked
first.
RegionServer Group based Assignment
---
Key: HBASE-6721
URL: https://issues.apache.org/jira/browse/HBASE-6721
Project: HBase
Issue Type: New Feature
Reporter: Francis Liu
Assignee: Vandana Ayyalasomayajula
Fix For: 0.96.0
Attachments: HBASE-6721-DesigDoc.pdf
In multi-tenant deployments of HBase, it is likely that a RegionServer will
be serving out regions from a number of different tables owned by various
client applications. Being able to group a subset of running RegionServers
and assign specific tables to it, provides a client application a level of
isolation and resource allocation.
The proposal essentially is to have an AssignmentManager which is aware of
RegionServer groups and assigns tables to region servers based on groupings.
Load balancing will occur on a per group basis as well.
This is essentially a simplification of the approach taken in HBASE-4120. See
attached document.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6726) Port HBASE-4465 'Lazy-seek optimization for StoreFile scanners' to 0.92
[ https://issues.apache.org/jira/browse/HBASE-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449853#comment-13449853 ] stack commented on HBASE-6726: -- Your comment is specious. FB are not running 0.92. Neither are EBay last I heard and if it is a 0.92, its not an Apache 0.92? Port HBASE-4465 'Lazy-seek optimization for StoreFile scanners' to 0.92 --- Key: HBASE-6726 URL: https://issues.apache.org/jira/browse/HBASE-6726 Project: HBase Issue Type: Task Reporter: Ted Yu This is expected to significantly reduce the amount of disk IO -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6324) Direct API calls from embedded Thrift server to regionserver
[ https://issues.apache.org/jira/browse/HBASE-6324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449856#comment-13449856 ] stack commented on HBASE-6324: -- My guess is that we need to forward port this facility from the 89-fb branch Michael. Direct API calls from embedded Thrift server to regionserver Key: HBASE-6324 URL: https://issues.apache.org/jira/browse/HBASE-6324 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin When handling Thrift calls in the regionserver we should not go through RPC to talk to the local regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6726) Port HBASE-4465 'Lazy-seek optimization for StoreFile scanners' to 0.92
[ https://issues.apache.org/jira/browse/HBASE-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449859#comment-13449859 ] Ted Yu commented on HBASE-6726: --- This feature was deployed to a production cluster running code base very close to 0.92. From earlier discussion, my understanding is that it is easier for a feature to be accepted by Apache HBase if someone has put the feature in their production environment. Port HBASE-4465 'Lazy-seek optimization for StoreFile scanners' to 0.92 --- Key: HBASE-6726 URL: https://issues.apache.org/jira/browse/HBASE-6726 Project: HBase Issue Type: Task Reporter: Ted Yu This is expected to significantly reduce the amount of disk IO -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (HBASE-6726) Port HBASE-4465 'Lazy-seek optimization for StoreFile scanners' to 0.92
[ https://issues.apache.org/jira/browse/HBASE-6726?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449859#comment-13449859 ] Ted Yu edited comment on HBASE-6726 at 9/7/12 4:58 AM: --- This feature was deployed to a production cluster @ EBay running code base very close to 0.92. From earlier discussion, my understanding is that it is easier for a feature to be accepted by Apache HBase if someone has put the feature in their production environment. was (Author: yuzhih...@gmail.com): This feature was deployed to a production cluster running code base very close to 0.92. From earlier discussion, my understanding is that it is easier for a feature to be accepted by Apache HBase if someone has put the feature in their production environment. Port HBASE-4465 'Lazy-seek optimization for StoreFile scanners' to 0.92 --- Key: HBASE-6726 URL: https://issues.apache.org/jira/browse/HBASE-6726 Project: HBase Issue Type: Task Reporter: Ted Yu This is expected to significantly reduce the amount of disk IO -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6727) [89-fb] allow HBaseServers's callqueue to be better configurable to avoid OOMs
Kannan Muthukkaruppan created HBASE-6727: Summary: [89-fb] allow HBaseServers's callqueue to be better configurable to avoid OOMs Key: HBASE-6727 URL: https://issues.apache.org/jira/browse/HBASE-6727 Project: HBase Issue Type: Bug Reporter: Kannan Muthukkaruppan Assignee: Kannan Muthukkaruppan The callQueue size (where requests get queued up if all handlers are busy) is a LinkedBlockingQueue of size 100*# of handlers. So, with say 300 handler threads, the call queue can have upto 30k entries queued up. If the requests are large enough-- then this can result in OOM or severe GC pauses. Ideally, we should allow this param to be separately configurable independent of the # of handlers; and better still be able to specify a memory size based limit, instead of a # of entries based limit. [I have not looked at the trunk version for this issue. So it may or may not be relevant there.] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6720) Optionally limit number of regions balanced in each balancer run
[ https://issues.apache.org/jira/browse/HBASE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449870#comment-13449870 ] Elliott Clark commented on HBASE-6720: -- No you're right it will work the next time around. I just mistook where the if statement was located. Optionally limit number of regions balanced in each balancer run Key: HBASE-6720 URL: https://issues.apache.org/jira/browse/HBASE-6720 Project: HBase Issue Type: Bug Reporter: Lars Hofhansl Fix For: 0.96.0, 0.94.3 Attachments: 6720-0.96-v1.txt See discussion on HBASE-3866 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6725) Check and Put can fail when using locks
[ https://issues.apache.org/jira/browse/HBASE-6725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449875#comment-13449875 ] Ted Yu commented on HBASE-6725: --- @Nicolas: Can you format ConcurrentCAPWithLocks class as a proper test so that we can easily verify this bug ? Thanks Check and Put can fail when using locks --- Key: HBASE-6725 URL: https://issues.apache.org/jira/browse/HBASE-6725 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.90.4 Environment: tested on mac os x Reporter: Nicolas Thiébaud Attachments: CAPwithLocks.zip When using CAP with an empty array as expected value and using a lock on the corresponding row, hbase may allow a Put even if the value was already there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6352) Add copy method in Bytes
[ https://issues.apache.org/jira/browse/HBASE-6352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449884#comment-13449884 ] Hudson commented on HBASE-6352: --- Integrated in HBase-TRUNK #3310 (See [https://builds.apache.org/job/HBase-TRUNK/3310/]) HBASE-6352 Add copy method in Bytes (Revision 1381670) Result = FAILURE stack : Files : * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/util/Bytes.java * /hbase/trunk/hbase-common/src/test/java/org/apache/hadoop/hbase/util/TestBytes.java Add copy method in Bytes Key: HBASE-6352 URL: https://issues.apache.org/jira/browse/HBASE-6352 Project: HBase Issue Type: Improvement Components: util Affects Versions: 0.94.0 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Minor Labels: Bytes, Util Fix For: 0.96.0 Attachments: HBASE_JIRA_6352.patch, HBASE_JIRA_6352_v2.patch, HBASE_JIRA_6352_v3.patch, HBASE_JIRA_6352_v4.patch, HBASE_JIRA_6352_v5.patch Original Estimate: 1h Remaining Estimate: 1h Having a copy method into Bytes might be nice to reduce client code size and improve readability. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6715) TestFromClientSide.testCacheOnWriteEvictOnClose is flaky
[
https://issues.apache.org/jira/browse/HBASE-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449885#comment-13449885
]
Hudson commented on HBASE-6715:
---
Integrated in HBase-TRUNK #3310 (See
[https://builds.apache.org/job/HBase-TRUNK/3310/])
HBASE-6715 TestFromClientSide.testCacheOnWriteEvictOnClose is flaky
(Revision 1381671)
Result = FAILURE
jxiang :
Files :
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
TestFromClientSide.testCacheOnWriteEvictOnClose is flaky
Key: HBASE-6715
URL: https://issues.apache.org/jira/browse/HBASE-6715
Project: HBase
Issue Type: Test
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
Attachments: trunk-6715.patch
Occasionally, this test fails:
{noformat}
expected:2049 but was:2069
Stacktrace
java.lang.AssertionError: expected:2049 but was:2069
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at
org.apache.hadoop.hbase.client.TestFromClientSide.testCacheOnWriteEvictOnClose(TestFromClientSide.java:4248)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{noformat}
It could be because there is other thread still accessing the cache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6508) [0.89-fb] Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449886#comment-13449886 ] Hudson commented on HBASE-6508: --- Integrated in HBase-TRUNK #3310 (See [https://builds.apache.org/job/HBase-TRUNK/3310/]) HBASE-6659 Port HBASE-6508 Filter out edits at log split time (Revision 1381684) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/RegionLoad.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LastSequenceId.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java * /hbase/trunk/hbase-server/src/main/protobuf/RegionServerStatus.proto * /hbase/trunk/hbase-server/src/main/protobuf/hbase.proto * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java [0.89-fb] Filter out edits at log split time Key: HBASE-6508 URL: https://issues.apache.org/jira/browse/HBASE-6508 Project: HBase Issue Type: Improvement Components: master, regionserver, wal Affects Versions: 0.89-fb Reporter: Alex Feinberg Assignee: Alex Feinberg Fix For: 0.89-fb At log splitting time, we can filter out many edits if we have a conservative estimate of what was saved last in each region. This patch does the following: 1) When a region server flushes a MemStore to HFile, store the last flushed sequence id for the region in a map. 2) Send the map to master it as a part of the region server report. 3) Adds an RPC call in HMasterRegionInterface to allow a region server to query the last last flushed sequence id for a region. 4) Skips any log entry with sequence id lower than last flushed sequence id for the region during log split time. 5) When a region is removed from a region server, removed the the entry for that region from the map, so that it isn't sent during the next report. This can reduce downtime when a regionserver goes down quite a bit. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449887#comment-13449887 ] Hudson commented on HBASE-6659: --- Integrated in HBase-TRUNK #3310 (See [https://builds.apache.org/job/HBase-TRUNK/3310/]) HBASE-6659 Port HBASE-6508 Filter out edits at log split time (Revision 1381684) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/RegionLoad.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/LastSequenceId.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java * /hbase/trunk/hbase-server/src/main/protobuf/RegionServerStatus.proto * /hbase/trunk/hbase-server/src/main/protobuf/hbase.proto * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogFiltering.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java Port HBASE-6508 Filter out edits at log split time -- Key: HBASE-6659 URL: https://issues.apache.org/jira/browse/HBASE-6659 Project: HBase Issue Type: Bug Reporter: Zhihong Ted Yu Assignee: Ted Yu Fix For: 0.96.0 Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, 6508-v7.txt, 6508-v8.txt, 6508-v9.txt HBASE-6508 is for 0.89-fb branch. This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6720) Optionally limit number of regions balanced in each balancer run
[
https://issues.apache.org/jira/browse/HBASE-6720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449889#comment-13449889
]
Ted Yu commented on HBASE-6720:
---
We already have cutoff time for region balancing:
{code}
rpCount++;
if (rpCount plans.size()
// if performing next balance exceeds cutoff time, exit the loop
(System.currentTimeMillis() + (totalRegPlanExecTime / rpCount))
cutoffTime) {
LOG.debug(No more balancing till next balance run;
maximumBalanceTime= +
maximumBalanceTime);
break;
}
{code}
Is the above not enough ?
Optionally limit number of regions balanced in each balancer run
Key: HBASE-6720
URL: https://issues.apache.org/jira/browse/HBASE-6720
Project: HBase
Issue Type: Bug
Reporter: Lars Hofhansl
Fix For: 0.96.0, 0.94.3
Attachments: 6720-0.96-v1.txt
See discussion on HBASE-3866
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6060) Regions's in OPENING state from failed regionservers takes a long time to recover
[ https://issues.apache.org/jira/browse/HBASE-6060?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449903#comment-13449903 ] ramkrishna.s.vasudevan commented on HBASE-6060: --- @Stack, Jon's comments makes sense. Lot of thought process is needed though. A better FSM may be needed. Surely can try it out. Regions's in OPENING state from failed regionservers takes a long time to recover - Key: HBASE-6060 URL: https://issues.apache.org/jira/browse/HBASE-6060 Project: HBase Issue Type: Bug Components: master, regionserver Reporter: Enis Soztutar Assignee: rajeshbabu Fix For: 0.96.0, 0.92.3, 0.94.3 Attachments: 6060-94-v3.patch, 6060-94-v4_1.patch, 6060-94-v4_1.patch, 6060-94-v4.patch, 6060_alternative_suggestion.txt, 6060_suggestion2_based_off_v3.patch, 6060_suggestion_based_off_v3.patch, 6060_suggestion_toassign_rs_wentdown_beforerequest.patch, 6060-trunk_2.patch, 6060-trunk_3.patch, 6060-trunk.patch, 6060-trunk.patch, HBASE-6060-92.patch, HBASE-6060-94.patch, HBASE-6060_latest.patch, HBASE-6060_latest.patch, HBASE-6060_latest.patch, HBASE-6060-trunk_4.patch, HBASE-6060_trunk_5.patch we have seen a pattern in tests, that the regions are stuck in OPENING state for a very long time when the region server who is opening the region fails. My understanding of the process: - master calls rs to open the region. If rs is offline, a new plan is generated (a new rs is chosen). RegionState is set to PENDING_OPEN (only in master memory, zk still shows OFFLINE). See HRegionServer.openRegion(), HMaster.assign() - RegionServer, starts opening a region, changes the state in znode. But that znode is not ephemeral. (see ZkAssign) - Rs transitions zk node from OFFLINE to OPENING. See OpenRegionHandler.process() - rs then opens the region, and changes znode from OPENING to OPENED - when rs is killed between OPENING and OPENED states, then zk shows OPENING state, and the master just waits for rs to change the region state, but since rs is down, that wont happen. - There is a AssignmentManager.TimeoutMonitor, which does exactly guard against these kind of conditions. It periodically checks (every 10 sec by default) the regions in transition to see whether they timedout (hbase.master.assignment.timeoutmonitor.timeout). Default timeout is 30 min, which explains what you and I are seeing. - ServerShutdownHandler in Master does not reassign regions in OPENING state, although it handles other states. Lowering that threshold from the configuration is one option, but still I think we can do better. Will investigate more. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6715) TestFromClientSide.testCacheOnWriteEvictOnClose is flaky
[
https://issues.apache.org/jira/browse/HBASE-6715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449908#comment-13449908
]
Hudson commented on HBASE-6715:
---
Integrated in HBase-0.94 #451 (See
[https://builds.apache.org/job/HBase-0.94/451/])
HBASE-6715 TestFromClientSide.testCacheOnWriteEvictOnClose is flaky
(Revision 1381678)
Result = FAILURE
jxiang :
Files :
*
/hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java
TestFromClientSide.testCacheOnWriteEvictOnClose is flaky
Key: HBASE-6715
URL: https://issues.apache.org/jira/browse/HBASE-6715
Project: HBase
Issue Type: Test
Reporter: Jimmy Xiang
Assignee: Jimmy Xiang
Priority: Minor
Attachments: trunk-6715.patch
Occasionally, this test fails:
{noformat}
expected:2049 but was:2069
Stacktrace
java.lang.AssertionError: expected:2049 but was:2069
at org.junit.Assert.fail(Assert.java:93)
at org.junit.Assert.failNotEquals(Assert.java:647)
at org.junit.Assert.assertEquals(Assert.java:128)
at org.junit.Assert.assertEquals(Assert.java:472)
at org.junit.Assert.assertEquals(Assert.java:456)
at
org.apache.hadoop.hbase.client.TestFromClientSide.testCacheOnWriteEvictOnClose(TestFromClientSide.java:4248)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
{noformat}
It could be because there is other thread still accessing the cache.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6729) Fix broken test - TestSplitLogWorker
Michal Gregorczyk created HBASE-6729: Summary: Fix broken test - TestSplitLogWorker Key: HBASE-6729 URL: https://issues.apache.org/jira/browse/HBASE-6729 Project: HBase Issue Type: Test Reporter: Michal Gregorczyk Assignee: Michal Gregorczyk 4 out of 5 tests in TestSplitLogWorker fail due to testing different strings for equality (eg. owned rs vs owned rs Worker-0). Comparison is made to check if task is owned by correct SpliLogWorker, and name of the worker is not rs but rs Worker-0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6637) Move DaemonThreadFactory into Threads and Threads to hbase-common
[ https://issues.apache.org/jira/browse/HBASE-6637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-6637: --- Attachment: hbase-6637-r1.patch Updated the current patch to trunk (to include the replication change). I think its ready to go Move DaemonThreadFactory into Threads and Threads to hbase-common - Key: HBASE-6637 URL: https://issues.apache.org/jira/browse/HBASE-6637 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: Jesse Yates Assignee: Jesse Yates Priority: Minor Fix For: 0.96.0 Attachments: hbase-6637-r1.patch, hbase-6637-v0.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6730) Enable rolling averages in StochasticLoadBalancer
Elliott Clark created HBASE-6730: Summary: Enable rolling averages in StochasticLoadBalancer Key: HBASE-6730 URL: https://issues.apache.org/jira/browse/HBASE-6730 Project: HBase Issue Type: Improvement Reporter: Elliott Clark Now that all of the ServerLoad transitions into pb are done. the load balancer should get more updates about the current state of the cluster. This should be used to allow StochasticLoadBalancer to get rolling averages. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6698) Refactor checkAndPut and checkAndDelete to use doMiniBatchMutation
[
https://issues.apache.org/jira/browse/HBASE-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449924#comment-13449924
]
Ted Yu commented on HBASE-6698:
---
PreCommit build #2799 timed out.
Please run latest patch through test suite.
nit:
{code}
+batchOp[0] = new PairMutation, Integer(((Put) w), lid);
{code}
The cast to Put should be to Mutation, right ?
Refactor checkAndPut and checkAndDelete to use doMiniBatchMutation
--
Key: HBASE-6698
URL: https://issues.apache.org/jira/browse/HBASE-6698
Project: HBase
Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Fix For: 0.96.0
Attachments: HBASE-6698_1.patch, HBASE-6698_2.patch,
HBASE-6698_3.patch, HBASE-6698.patch
Currently the checkAndPut and checkAndDelete api internally calls the
internalPut and internalDelete. May be we can just call doMiniBatchMutation
only. This will help in future like if we have some hooks and the CP
handles certain cases in the doMiniBatchMutation the same can be done while
doing a put thro checkAndPut or while doing a delete thro checkAndDelete.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6324) Direct API calls from embedded Thrift server to regionserver
[ https://issues.apache.org/jira/browse/HBASE-6324?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449925#comment-13449925 ] Michael Drzal commented on HBASE-6324: -- [~stack] thanks for the explanation. That makes sense. Direct API calls from embedded Thrift server to regionserver Key: HBASE-6324 URL: https://issues.apache.org/jira/browse/HBASE-6324 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Mikhail Bautin When handling Thrift calls in the regionserver we should not go through RPC to talk to the local regionserver. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6713) Stopping META/ROOT RS may take 50mins when some region is splitting
[ https://issues.apache.org/jira/browse/HBASE-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-6713: -- Hadoop Flags: Reviewed Summary: Stopping META/ROOT RS may take 50mins when some region is splitting (was: Stopping meta/root RS may take 50mins when it is in region-splitting) Stopping META/ROOT RS may take 50mins when some region is splitting --- Key: HBASE-6713 URL: https://issues.apache.org/jira/browse/HBASE-6713 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.94.1 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.3 Attachments: HBASE-6713.patch, HBASE-6713v2.patch When we stop the RS carrying ROOT/META, if it is in the splitting for some region, the whole stopping process may take 50 mins. The reason is : 1.ROOT/META region is closed when stopping the regionserver 2.The Split Transaction failed updating META and it will retry 3.The retry num is 100, and the total time is about 50 mins as default; This configuration is set by HConnectionManager#setServerSideHConnectionRetries I think 50 mins is too long to acceptable, my suggested solution is closing MetaTable regions after the compact/split thread is closed -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6713) Stopping META/ROOT RS may take 50mins when some region is splitting
[ https://issues.apache.org/jira/browse/HBASE-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-6713: -- Fix Version/s: 0.92.3 Stopping META/ROOT RS may take 50mins when some region is splitting --- Key: HBASE-6713 URL: https://issues.apache.org/jira/browse/HBASE-6713 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.94.1 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.92.3, 0.94.3 Attachments: HBASE-6713.patch, HBASE-6713v2.patch When we stop the RS carrying ROOT/META, if it is in the splitting for some region, the whole stopping process may take 50 mins. The reason is : 1.ROOT/META region is closed when stopping the regionserver 2.The Split Transaction failed updating META and it will retry 3.The retry num is 100, and the total time is about 50 mins as default; This configuration is set by HConnectionManager#setServerSideHConnectionRetries I think 50 mins is too long to acceptable, my suggested solution is closing MetaTable regions after the compact/split thread is closed -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6713) Stopping META/ROOT RS may take 50mins when some region is splitting
[ https://issues.apache.org/jira/browse/HBASE-6713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-6713: -- Attachment: 6713v3.patch Patch v3 corrects a few spelling mistakes. Stopping META/ROOT RS may take 50mins when some region is splitting --- Key: HBASE-6713 URL: https://issues.apache.org/jira/browse/HBASE-6713 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.94.1 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.92.3, 0.94.3 Attachments: 6713v3.patch, HBASE-6713.patch, HBASE-6713v2.patch When we stop the RS carrying ROOT/META, if it is in the splitting for some region, the whole stopping process may take 50 mins. The reason is : 1.ROOT/META region is closed when stopping the regionserver 2.The Split Transaction failed updating META and it will retry 3.The retry num is 100, and the total time is about 50 mins as default; This configuration is set by HConnectionManager#setServerSideHConnectionRetries I think 50 mins is too long to acceptable, my suggested solution is closing MetaTable regions after the compact/split thread is closed -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6698) Refactor checkAndPut and checkAndDelete to use doMiniBatchMutation
[
https://issues.apache.org/jira/browse/HBASE-6698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13449972#comment-13449972
]
Ted Yu commented on HBASE-6698:
---
nit: line is too long:
{code}
+if
(batchMutate[0].getOperationStatusCode().equals(OperationStatusCode.SANITY_CHECK_FAILURE))
{
{code}
The code snippet of creating PairMutation, Integer, calling batchMutate(),
etc is repeated multiple times. Consider refactoring into a utility method.
Refactor checkAndPut and checkAndDelete to use doMiniBatchMutation
--
Key: HBASE-6698
URL: https://issues.apache.org/jira/browse/HBASE-6698
Project: HBase
Issue Type: Improvement
Reporter: ramkrishna.s.vasudevan
Fix For: 0.96.0
Attachments: HBASE-6698_1.patch, HBASE-6698_2.patch,
HBASE-6698_3.patch, HBASE-6698.patch
Currently the checkAndPut and checkAndDelete api internally calls the
internalPut and internalDelete. May be we can just call doMiniBatchMutation
only. This will help in future like if we have some hooks and the CP
handles certain cases in the doMiniBatchMutation the same can be done while
doing a put thro checkAndPut or while doing a delete thro checkAndDelete.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6731) Port HBASE-6537 'Race between balancer and disable table can lead to inconsistent cluster' to 0.92
Ted Yu created HBASE-6731: - Summary: Port HBASE-6537 'Race between balancer and disable table can lead to inconsistent cluster' to 0.92 Key: HBASE-6731 URL: https://issues.apache.org/jira/browse/HBASE-6731 Project: HBase Issue Type: Bug Reporter: Ted Yu Fix For: 0.92.3 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6732) Reduce scope of synchronized block in HBaseClient.Connection#sendParam.
Michal Gregorczyk created HBASE-6732: Summary: Reduce scope of synchronized block in HBaseClient.Connection#sendParam. Key: HBASE-6732 URL: https://issues.apache.org/jira/browse/HBASE-6732 Project: HBase Issue Type: Test Reporter: Michal Gregorczyk Assignee: Michal Gregorczyk sendParam method synchronizes on out stream (so that only one thread writes to socket). Right now it prepares (eg. compress) message under this lock as well. Lock should be taken only for sending. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Work started] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HBASE-6659 started by Ted Yu. Port HBASE-6508 Filter out edits at log split time -- Key: HBASE-6659 URL: https://issues.apache.org/jira/browse/HBASE-6659 Project: HBase Issue Type: Bug Reporter: Zhihong Ted Yu Assignee: Ted Yu Fix For: 0.96.0 Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, 6508-v7.txt, 6508-v8.txt, 6508-v9.txt HBASE-6508 is for 0.89-fb branch. This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-6659: -- Status: Patch Available (was: In Progress) Port HBASE-6508 Filter out edits at log split time -- Key: HBASE-6659 URL: https://issues.apache.org/jira/browse/HBASE-6659 Project: HBase Issue Type: Bug Reporter: Zhihong Ted Yu Assignee: Ted Yu Fix For: 0.96.0 Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, 6508-v7.txt, 6508-v8.txt, 6508-v9.txt HBASE-6508 is for 0.89-fb branch. This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6659) Port HBASE-6508 Filter out edits at log split time
[ https://issues.apache.org/jira/browse/HBASE-6659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-6659: -- Resolution: Fixed Status: Resolved (was: Patch Available) Port HBASE-6508 Filter out edits at log split time -- Key: HBASE-6659 URL: https://issues.apache.org/jira/browse/HBASE-6659 Project: HBase Issue Type: Bug Reporter: Zhihong Ted Yu Assignee: Ted Yu Fix For: 0.96.0 Attachments: 6508-v2.txt, 6508-v3.txt, 6508-v4.txt, 6508-v5.txt, 6508-v7.txt, 6508-v8.txt, 6508-v9.txt HBASE-6508 is for 0.89-fb branch. This JIRA ports the feature to trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6409) Create histogram class for metrics 2
[ https://issues.apache.org/jira/browse/HBASE-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-6409: - Attachment: HBASE-6409-1.patch * Made threads used by MetricsExecutor daemon threads so they are shut down with hbase. * Change the name of splitTime and splitSize to better reflect that they are describing splitting hlogs and not splitting regions. Create histogram class for metrics 2 Key: HBASE-6409 URL: https://issues.apache.org/jira/browse/HBASE-6409 Project: HBase Issue Type: Sub-task Affects Versions: 0.96.0 Reporter: Elliott Clark Assignee: Elliott Clark Priority: Blocker Attachments: HBASE-6409-0.patch, HBASE-6409-1.patch Create the replacement for MetricsHistogram and PersistantTimeVaryingRate classes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6409) Create histogram class for metrics 2
[ https://issues.apache.org/jira/browse/HBASE-6409?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elliott Clark updated HBASE-6409: - Attachment: HBASE-6409-2.patch git format-patch doesn't work when moving files. Create histogram class for metrics 2 Key: HBASE-6409 URL: https://issues.apache.org/jira/browse/HBASE-6409 Project: HBase Issue Type: Sub-task Affects Versions: 0.96.0 Reporter: Elliott Clark Assignee: Elliott Clark Priority: Blocker Attachments: HBASE-6409-0.patch, HBASE-6409-1.patch, HBASE-6409-2.patch Create the replacement for MetricsHistogram and PersistantTimeVaryingRate classes. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6637) Move DaemonThreadFactory into Threads and Threads to hbase-common
[ https://issues.apache.org/jira/browse/HBASE-6637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13450040#comment-13450040 ] stack commented on HBASE-6637: -- +1 on patch. Thanks for moving stuff back to common. Will wait on hadoopqa Move DaemonThreadFactory into Threads and Threads to hbase-common - Key: HBASE-6637 URL: https://issues.apache.org/jira/browse/HBASE-6637 Project: HBase Issue Type: Bug Affects Versions: 0.96.0 Reporter: Jesse Yates Assignee: Jesse Yates Priority: Minor Fix For: 0.96.0 Attachments: hbase-6637-r1.patch, hbase-6637-v0.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6729) Fix broken test - TestSplitLogWorker
[ https://issues.apache.org/jira/browse/HBASE-6729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13450042#comment-13450042 ] stack commented on HBASE-6729: -- Good one Michal. How do we fix it or do you have a patch? Fix broken test - TestSplitLogWorker Key: HBASE-6729 URL: https://issues.apache.org/jira/browse/HBASE-6729 Project: HBase Issue Type: Test Reporter: Michal Gregorczyk Assignee: Michal Gregorczyk 4 out of 5 tests in TestSplitLogWorker fail due to testing different strings for equality (eg. owned rs vs owned rs Worker-0). Comparison is made to check if task is owned by correct SpliLogWorker, and name of the worker is not rs but rs Worker-0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6729) Fix broken test - TestSplitLogWorker
[ https://issues.apache.org/jira/browse/HBASE-6729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13450043#comment-13450043 ] Michal Gregorczyk commented on HBASE-6729: -- I have one. It is being reviewed right now, but I can post it anyway. Fix broken test - TestSplitLogWorker Key: HBASE-6729 URL: https://issues.apache.org/jira/browse/HBASE-6729 Project: HBase Issue Type: Test Reporter: Michal Gregorczyk Assignee: Michal Gregorczyk 4 out of 5 tests in TestSplitLogWorker fail due to testing different strings for equality (eg. owned rs vs owned rs Worker-0). Comparison is made to check if task is owned by correct SpliLogWorker, and name of the worker is not rs but rs Worker-0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6728) [89-fb] prevent OOM possibility due to per connection responseQueue being unbounded
[ https://issues.apache.org/jira/browse/HBASE-6728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13450050#comment-13450050 ] stack commented on HBASE-6728: -- I looked at trunk. We have same issue. responseQueue is a LinkedList. We keep a count of entries but we do not put an upper bound on how many items we add. [89-fb] prevent OOM possibility due to per connection responseQueue being unbounded --- Key: HBASE-6728 URL: https://issues.apache.org/jira/browse/HBASE-6728 Project: HBase Issue Type: Bug Reporter: Kannan Muthukkaruppan Assignee: Kannan Muthukkaruppan The per connection responseQueue is an unbounded queue. The request handler threads today try to send the response in line, but if things start to backup, the response is sent via a per connection responder thread. The intermediate queue used in between has no bound either, and can be another source of OOMs. [Have not looked at this issue in trunk. So it may or may not be applicable there.] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6729) Fix broken test - TestSplitLogWorker
[ https://issues.apache.org/jira/browse/HBASE-6729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michal Gregorczyk updated HBASE-6729: - Status: Patch Available (was: Open) Fix broken test - TestSplitLogWorker Key: HBASE-6729 URL: https://issues.apache.org/jira/browse/HBASE-6729 Project: HBase Issue Type: Test Reporter: Michal Gregorczyk Assignee: Michal Gregorczyk 4 out of 5 tests in TestSplitLogWorker fail due to testing different strings for equality (eg. owned rs vs owned rs Worker-0). Comparison is made to check if task is owned by correct SpliLogWorker, and name of the worker is not rs but rs Worker-0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6729) Fix broken test - TestSplitLogWorker
[ https://issues.apache.org/jira/browse/HBASE-6729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michal Gregorczyk updated HBASE-6729: - Status: Open (was: Patch Available) Fix broken test - TestSplitLogWorker Key: HBASE-6729 URL: https://issues.apache.org/jira/browse/HBASE-6729 Project: HBase Issue Type: Test Reporter: Michal Gregorczyk Assignee: Michal Gregorczyk 4 out of 5 tests in TestSplitLogWorker fail due to testing different strings for equality (eg. owned rs vs owned rs Worker-0). Comparison is made to check if task is owned by correct SpliLogWorker, and name of the worker is not rs but rs Worker-0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6727) [89-fb] allow HBaseServers's callqueue to be better configurable to avoid OOMs
[ https://issues.apache.org/jira/browse/HBASE-6727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13450053#comment-13450053 ] stack commented on HBASE-6727: -- Check out HBASE-3813 Kannan. [89-fb] allow HBaseServers's callqueue to be better configurable to avoid OOMs -- Key: HBASE-6727 URL: https://issues.apache.org/jira/browse/HBASE-6727 Project: HBase Issue Type: Bug Reporter: Kannan Muthukkaruppan Assignee: Kannan Muthukkaruppan The callQueue size (where requests get queued up if all handlers are busy) is a LinkedBlockingQueue of size 100*# of handlers. So, with say 300 handler threads, the call queue can have upto 30k entries queued up. If the requests are large enough-- then this can result in OOM or severe GC pauses. Ideally, we should allow this param to be separately configurable independent of the # of handlers; and better still be able to specify a memory size based limit, instead of a # of entries based limit. [I have not looked at the trunk version for this issue. So it may or may not be relevant there.] -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6729) Fix broken test - TestSplitLogWorker
[ https://issues.apache.org/jira/browse/HBASE-6729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michal Gregorczyk updated HBASE-6729: - Attachment: HBASE-6729.patch Fix broken test - TestSplitLogWorker Key: HBASE-6729 URL: https://issues.apache.org/jira/browse/HBASE-6729 Project: HBase Issue Type: Test Reporter: Michal Gregorczyk Assignee: Michal Gregorczyk Attachments: HBASE-6729.patch 4 out of 5 tests in TestSplitLogWorker fail due to testing different strings for equality (eg. owned rs vs owned rs Worker-0). Comparison is made to check if task is owned by correct SpliLogWorker, and name of the worker is not rs but rs Worker-0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6729) Fix broken test - TestSplitLogWorker
[ https://issues.apache.org/jira/browse/HBASE-6729?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13450058#comment-13450058 ] stack commented on HBASE-6729: -- Does the patch apply for you Michal? It doesn't apply on trunk nor on 0.94 branch. What you working against? Thanks. Fix broken test - TestSplitLogWorker Key: HBASE-6729 URL: https://issues.apache.org/jira/browse/HBASE-6729 Project: HBase Issue Type: Test Reporter: Michal Gregorczyk Assignee: Michal Gregorczyk Attachments: HBASE-6729.patch 4 out of 5 tests in TestSplitLogWorker fail due to testing different strings for equality (eg. owned rs vs owned rs Worker-0). Comparison is made to check if task is owned by correct SpliLogWorker, and name of the worker is not rs but rs Worker-0. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
