[jira] [Commented] (HBASE-6626) Add a chapter on HDFS in the troubleshooting section of the HBase reference guide.
[ https://issues.apache.org/jira/browse/HBASE-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580500#comment-13580500 ] James Kinley commented on HBASE-6626: - This is really useful. A couple of suggestions: it might be easier to digest if you describe the various HDFS communication points along the lines of the write path. For example: * Client issues Put to a region server, which writes the edit to the WAL, which appends to HLog in HDFS ** This uses DFSClient ** HLog writes are pipelined to X number of datanodes - based on replication factor ** HLogs are rolled at intervals (60 minutes), which requires interaction with HDFS ** Indicate which timeout and retry options apply here ** ... * Same for HFiles... In addition to this, and the typical error messages, it would also be great to describe what to look out for in certain scenarios. For example: * What happens when a datanode goes bad (you've touched on this already) ** How does this affect a region server? ** What to look out for in the logs? ** What configuration options apply? * What happens when the active namenode goes bad, and what happens during namenode failover? ** A region server needs to communicate with the namenode when it rolls it's WAL, flushes it's store files, etc ** In between, it only communicates with the datanodes to append to the WAL ** What configuration options apply? ** What to look out for in the logs? Thanks, James. > Add a chapter on HDFS in the troubleshooting section of the HBase reference > guide. > -- > > Key: HBASE-6626 > URL: https://issues.apache.org/jira/browse/HBASE-6626 > Project: HBase > Issue Type: Improvement > Components: documentation >Affects Versions: 0.96.0 >Reporter: nkeywal >Assignee: Doug Meil >Priority: Blocker > Attachments: troubleshooting.txt > > > I looked mainly at the major failure case, but here is what I have: > New sub chapter in the existing chapter "Troubleshooting and Debugging > HBase": "HDFS & HBASE" > 1) HDFS & HBase > 2) Connection related settings > 2.1) Number of retries > 2.2) Timeouts > 3) Log samples > 1) HDFS & HBase > HBase uses HDFS to store its HFile, i.e. the core HBase files and the > Write-Ahead-Logs, i.e. the files that will be used to restore the data after > a crash. > In both cases, the reliability of HBase comes from the fact that HDFS writes > the data to multiple locations. To be efficient, HBase needs the data to be > available locally, hence it's highly recommended to have the HDFS datanode on > the same machines as the HBase Region Servers. > Detailed information on how HDFS works can be found at [1]. > Important features are: > - HBase is a client application of HDFS, i.e. uses the HDFS DFSClient class. > This class can appears in HBase logs with other HDFS client related logs. > - Some HDFS settings are HDFS-server-side, i.e. must be set on the HDFS > side, while some other are HDFS-client-side, i.e. must be set in HBase, while > some other must be set in both places. > - the HDFS writes are pipelined from one datanode to another. When writing, > there are communications between: > - HBase and HDFS namenode, through the HDFS client classes. > - HBase and HDFS datanodes, through the HDFS client classes. > - HDFS datanode between themselves: issues on these communications are in > HDFS logs, not HBase. HDFS writes are always local when possible. As a > consequence, there should not be much write error in HBase Region Servers: > they write to the local datanode. If this datanode can't replicate the > blocks, it will appear in its logs, not in the region servers logs. > - datanodes can be contacted through the ipc.Client interface (once again > this class can shows up in HBase logs) and the data transfer interface > (usually shows up as the DataNode class in the HBase logs). There are on > different ports (defaults being: 50010 and 50020). > - To understand exactly what's going on, you must look that the HDFS log > files as well: HBase logs represent the client side. > - With the default setting, HDFS needs 630s to mark a datanode as dead. For > this reason, this node will still be tried by HBase or by other datanodes > when writing and reading until HDFS definitively decides it's dead. This will > add some extras lines in the logs. This monitoring is performed by the > NameNode. > - The HDFS clients (i.e. HBase using HDFS client code) don't fully rely on > the NameNode, but can mark temporally a node as dead if they had an error > when they tried to use it. > 2) Settings for retries and timeouts > 2.1) Retries > ipc.client.connect.max.retries > Default 10 > Indicates the number of retries a client will make to establish
[jira] [Commented] (HBASE-6626) Add a chapter on HDFS in the troubleshooting section of the HBase reference guide.
[ https://issues.apache.org/jira/browse/HBASE-6626?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580533#comment-13580533 ] nkeywal commented on HBASE-6626: [~varunsharma] Sorry, I didn't see your comment before. For writes, the decision to mark a datanode as dead is taken locally. For a new write (i.e. no connection established), yes it will be the connect timeout that will be used. IIRC, it retries 3 times in this case. The dead node is per client file handler, so if you open a new file, you may go to the same server and fail again. [~jrkinley] Yes, I agree with your comments. I plan to write an integration test with a namenode failure and to document the setting from HDFS-3703 from an HBase point of view. > Add a chapter on HDFS in the troubleshooting section of the HBase reference > guide. > -- > > Key: HBASE-6626 > URL: https://issues.apache.org/jira/browse/HBASE-6626 > Project: HBase > Issue Type: Improvement > Components: documentation >Affects Versions: 0.96.0 >Reporter: nkeywal >Assignee: Doug Meil >Priority: Blocker > Attachments: troubleshooting.txt > > > I looked mainly at the major failure case, but here is what I have: > New sub chapter in the existing chapter "Troubleshooting and Debugging > HBase": "HDFS & HBASE" > 1) HDFS & HBase > 2) Connection related settings > 2.1) Number of retries > 2.2) Timeouts > 3) Log samples > 1) HDFS & HBase > HBase uses HDFS to store its HFile, i.e. the core HBase files and the > Write-Ahead-Logs, i.e. the files that will be used to restore the data after > a crash. > In both cases, the reliability of HBase comes from the fact that HDFS writes > the data to multiple locations. To be efficient, HBase needs the data to be > available locally, hence it's highly recommended to have the HDFS datanode on > the same machines as the HBase Region Servers. > Detailed information on how HDFS works can be found at [1]. > Important features are: > - HBase is a client application of HDFS, i.e. uses the HDFS DFSClient class. > This class can appears in HBase logs with other HDFS client related logs. > - Some HDFS settings are HDFS-server-side, i.e. must be set on the HDFS > side, while some other are HDFS-client-side, i.e. must be set in HBase, while > some other must be set in both places. > - the HDFS writes are pipelined from one datanode to another. When writing, > there are communications between: > - HBase and HDFS namenode, through the HDFS client classes. > - HBase and HDFS datanodes, through the HDFS client classes. > - HDFS datanode between themselves: issues on these communications are in > HDFS logs, not HBase. HDFS writes are always local when possible. As a > consequence, there should not be much write error in HBase Region Servers: > they write to the local datanode. If this datanode can't replicate the > blocks, it will appear in its logs, not in the region servers logs. > - datanodes can be contacted through the ipc.Client interface (once again > this class can shows up in HBase logs) and the data transfer interface > (usually shows up as the DataNode class in the HBase logs). There are on > different ports (defaults being: 50010 and 50020). > - To understand exactly what's going on, you must look that the HDFS log > files as well: HBase logs represent the client side. > - With the default setting, HDFS needs 630s to mark a datanode as dead. For > this reason, this node will still be tried by HBase or by other datanodes > when writing and reading until HDFS definitively decides it's dead. This will > add some extras lines in the logs. This monitoring is performed by the > NameNode. > - The HDFS clients (i.e. HBase using HDFS client code) don't fully rely on > the NameNode, but can mark temporally a node as dead if they had an error > when they tried to use it. > 2) Settings for retries and timeouts > 2.1) Retries > ipc.client.connect.max.retries > Default 10 > Indicates the number of retries a client will make to establish a server > connection. Not taken into account if the error is a SocketTimeout. In this > case the number of retries is 45 (fixed on branch, HADOOP-7932 or in > HADOOP-7397). For SASL, the number of retries is hard-coded to 15. Can be > increased, especially if the socket timeouts have been lowered. > ipc.client.connect.max.retries.on.timeouts > Default 45 > If you have HADOOP-7932, max number of retries on timeout. Counter is > different than ipc.client.connect.max.retries so if you mix the socket errors > you will get 55 retries with the default values. Could be lowered, once it is > available. With HADOOP-7397 ipc.client.connect.max.retries is reused so there > would be 10 tries. > dfs.client.
[jira] [Commented] (HBASE-7867) setPreallocSize is different with COMMENT in setupTestEnv in MiniZooKeeperCluster.java
[
https://issues.apache.org/jira/browse/HBASE-7867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580568#comment-13580568
]
Hudson commented on HBASE-7867:
---
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #411 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/411/])
HBASE-7867 setPreallocSize is different with COMMENT in setupTestEnv in
MiniZooKeeperCluster.java (DaeMyung Kang) (Revision 1447106)
Result = FAILURE
larsh :
Files :
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/zookeeper/MiniZooKeeperCluster.java
> setPreallocSize is different with COMMENT in setupTestEnv in
> MiniZooKeeperCluster.java
> --
>
> Key: HBASE-7867
> URL: https://issues.apache.org/jira/browse/HBASE-7867
> Project: HBase
> Issue Type: Bug
>Reporter: DaeMyung Kang
>Assignee: DaeMyung Kang
>Priority: Trivial
> Fix For: 0.96.0, 0.94.6
>
> Attachments: hbase-7867.patch, hbase-7867.patch
>
>
> setupTestEnv() just sets 100 bytes not 100k bytes
> {code}
> private static void setupTestEnv() {
> // / XXX: From o.a.zk.t.ClientBaseprivate static void setupTestEnv() {
> // during the tests we run with 100K prealloc in the logs.
> // on windows systems prealloc of 64M was seen to take ~15seconds
> // resulting in test failure (client timeout on first session).
> // set env and directly in order to handle static init/gc issues
> System.setProperty("zookeeper.preAllocSize", "100");
> FileTxnLog.setPreallocSize(100);
> }
> {code}
> to reduce too many call padLogFile(), I think 100k is correct.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7801) Allow a deferred sync option per Mutation.
[ https://issues.apache.org/jira/browse/HBASE-7801?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580575#comment-13580575 ] Anoop Sam John commented on HBASE-7801: --- +1 We can add in release notes about how to achieve the different use cases. bq. Increment is not supported as it does not extend Mutation (but it could be supported if we make an PB change). For 0.96 we have a JIRA issue for making Increment also a Mutation right? I forgot the issue id. > Allow a deferred sync option per Mutation. > -- > > Key: HBASE-7801 > URL: https://issues.apache.org/jira/browse/HBASE-7801 > Project: HBase > Issue Type: Sub-task >Reporter: Lars Hofhansl >Assignee: Lars Hofhansl > Fix For: 0.96.0, 0.94.6 > > Attachments: 7801-0.94-v1.txt, 7801-0.94-v2.txt, 7801-0.96-v1.txt > > > Won't have time for parent. But a deferred sync option on a per operation > basis comes up quite frequently. > In 0.96 this can be handled cleanly via protobufs and 0.94 we can have a > special mutation attribute. > For batch operation we'd take the safest sync option of any of the mutations. > I.e. if there is at least one that wants to be flushed we'd sync the batch, > if there's none of those but at least one that wants deferred flush we defer > flush the batch, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7868) HFile performance regression between 0.92 and 0.94
[ https://issues.apache.org/jira/browse/HBASE-7868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-7868: --- Attachment: performances.pdf > HFile performance regression between 0.92 and 0.94 > -- > > Key: HBASE-7868 > URL: https://issues.apache.org/jira/browse/HBASE-7868 > Project: HBase > Issue Type: Bug > Components: io >Affects Versions: 0.94.5 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 0.94.6 > > Attachments: FilteredScan.png, hfileperf-graphs.png, > performances.pdf, performances.pdf > > > By HFilePerformanceEvaluation seems that 0.94 is slower then 0.92 > Looking at the profiler for the Scan path, seems that most of the time, > compared to 92, is spent in the metrics dictionary lookup. [~eclark] pointed > out the new per family/block metrics. > By commenting the metrics call in HFileReaderV2, the performance seems to get > better, but maybe metrics is not the only problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580581#comment-13580581 ] chunhui shen commented on HBASE-7403: - I'm waiting HBASE-7721 to commit. Current patch use BlockingMetaScannerVisitor to provide consistency view of META entries of table, it is similar with split > Online Merge > > > Key: HBASE-7403 > URL: https://issues.apache.org/jira/browse/HBASE-7403 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.94.3 >Reporter: chunhui shen >Assignee: chunhui shen >Priority: Critical > Fix For: 0.96.0, 0.94.6 > > Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, > 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, > hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, > hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, > hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, > hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, > hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, > merge region.pdf > > > The feature of this online merge: > 1.Online,no necessary to disable table > 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 > 3.Easy to call merege request, no need to input a long region name, only > encoded name enough > 4.No limit when operation, you don't need to tabke care the events like > Server Dead, Balance, Split, Disabing/Enabing table, no need to take care > whether you send a wrong merge request, it has alread done for you > 5.Only little offline time for two merging regions > Usage: > 1.Tool: > bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show] > > 2.API: static void MergeManager#createMergeRequest > We need merge in the following cases： > 1.Region hole or region overlap, can’t be fix by hbck > 2.Region become empty because of TTL and not reasonable Rowkey design > 3.Region is always empty or very small because of presplit when create table > 4.Too many empty or small regions would reduce the system performance(e.g. > mslab) > Current merge tools only support offline and are not able to redo if > exception is thrown in the process of merging, causing a dirty data > For online system, we need a online merge. > This implement logic of this patch for Online Merge is : > For example, merge regionA and regionB into regionC > 1.Offline the two regions A and B > 2.Merge the two regions in the HDFS(Create regionC’s directory, move > regionA’s and regionB’s file to regionC’s directory, delete regionA’s and > regionB’s directory) > 3.Add the merged regionC to .META. > 4.Assign the merged regionC > As design of this patch , once we do the merge work in the HDFS,we could redo > it until successful if it throws exception or abort or server restart, but > couldn’t be rolled back. > It depends on > Use zookeeper to record the transaction journal state, make redo easier > Use zookeeper to send/receive merge request > Merge transaction is executed on the master > Support calling merge request through API or shell tool > About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7868) HFile performance regression between 0.92 and 0.94
[ https://issues.apache.org/jira/browse/HBASE-7868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580582#comment-13580582 ] Jean-Marc Spaggiari commented on HBASE-7868: [~lhofhansl] Lars, I'm running in standalone mode for those tests, so I don't think the network is really impacting the restuls. Also, HBase is stopped between each iteration, so cache is not really helping one iteration versus another one. And last, I'm deleting zll ZooKeeper and HBase data between each iteration too. I have attached an updated report with more results. I'm currently running each test 10 times and taking the 80th percentil, but I can increase that and run each test 20 or 50 times if that helps. I'm now running randomSeekScan. Results will come over the day. > HFile performance regression between 0.92 and 0.94 > -- > > Key: HBASE-7868 > URL: https://issues.apache.org/jira/browse/HBASE-7868 > Project: HBase > Issue Type: Bug > Components: io >Affects Versions: 0.94.5 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 0.94.6 > > Attachments: FilteredScan.png, hfileperf-graphs.png, > performances.pdf, performances.pdf > > > By HFilePerformanceEvaluation seems that 0.94 is slower then 0.92 > Looking at the profiler for the Scan path, seems that most of the time, > compared to 92, is spent in the metrics dictionary lookup. [~eclark] pointed > out the new per family/block metrics. > By commenting the metrics call in HFileReaderV2, the performance seems to get > better, but maybe metrics is not the only problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7870) Delete(byte [] row, long timestamp) constructor is deprecate and should not.
Jean-Marc Spaggiari created HBASE-7870: -- Summary: Delete(byte [] row, long timestamp) constructor is deprecate and should not. Key: HBASE-7870 URL: https://issues.apache.org/jira/browse/HBASE-7870 Project: HBase Issue Type: Bug Affects Versions: 0.94.5 Reporter: Jean-Marc Spaggiari Assignee: Jean-Marc Spaggiari Priority: Critical Most probably a cut&past issue. Patch is coming. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7870) Delete(byte [] row, long timestamp) constructor is deprecate and should not.
[ https://issues.apache.org/jira/browse/HBASE-7870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-7870: --- Attachment: HBASE-7870-v0-0.94.patch > Delete(byte [] row, long timestamp) constructor is deprecate and should not. > > > Key: HBASE-7870 > URL: https://issues.apache.org/jira/browse/HBASE-7870 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.5 >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari >Priority: Critical > Attachments: HBASE-7870-v0-0.94.patch > > > Most probably a cut&past issue. Patch is coming. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7870) Delete(byte [] row, long timestamp) constructor is deprecate and should not.
[ https://issues.apache.org/jira/browse/HBASE-7870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-7870: --- Status: Patch Available (was: Open) Fixing: public Delete(byte [] row) which should not use lockID constructor public Delete(byte [] row, long timestamp) comments which are making it deprecated when it should not since not using LockID. > Delete(byte [] row, long timestamp) constructor is deprecate and should not. > > > Key: HBASE-7870 > URL: https://issues.apache.org/jira/browse/HBASE-7870 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.5 >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari >Priority: Critical > Attachments: HBASE-7870-v0-0.94.patch > > > Most probably a cut&past issue. Patch is coming. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7870) Delete(byte [] row, long timestamp) constructor is deprecate and should not.
[
https://issues.apache.org/jira/browse/HBASE-7870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580601#comment-13580601
]
Hadoop QA commented on HBASE-7870:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12569808/HBASE-7870-v0-0.94.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4453//console
This message is automatically generated.
> Delete(byte [] row, long timestamp) constructor is deprecate and should not.
>
>
> Key: HBASE-7870
> URL: https://issues.apache.org/jira/browse/HBASE-7870
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.5
>Reporter: Jean-Marc Spaggiari
>Assignee: Jean-Marc Spaggiari
>Priority: Critical
> Attachments: HBASE-7870-v0-0.94.patch
>
>
> Most probably a cut&past issue. Patch is coming.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7846) Add support for merging implicit regions in Merge tool
[ https://issues.apache.org/jira/browse/HBASE-7846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-7846: --- Assignee: Jean-Marc Spaggiari > Add support for merging implicit regions in Merge tool > -- > > Key: HBASE-7846 > URL: https://issues.apache.org/jira/browse/HBASE-7846 > Project: HBase > Issue Type: Improvement > Components: util >Reporter: Kaufman Ng >Assignee: Jean-Marc Spaggiari >Priority: Minor > Attachments: HBASE-7846-v0-trunk.patch > > > Currently org.apache.hadoop.hbase.util.Merge needs 2 region names to be > explicitly specified to perform a merge. This can be cumbersome. > One idea for improvement is to have Merge to figure out all the adjacent > regions and perform the merges. > For example: > regions before merge: row-10, row-20, row-30, row-40, row-50 > regions after merge: row-10, row-30, row-50 > In the above example, region names of "row-10" and "row-20" are merged to > become a new bigger region of "row-10". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7846) Add support for merging implicit regions in Merge tool
[ https://issues.apache.org/jira/browse/HBASE-7846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-7846: --- Attachment: HBASE-7846-v0-trunk.patch > Add support for merging implicit regions in Merge tool > -- > > Key: HBASE-7846 > URL: https://issues.apache.org/jira/browse/HBASE-7846 > Project: HBase > Issue Type: Improvement > Components: util >Reporter: Kaufman Ng >Assignee: Jean-Marc Spaggiari >Priority: Minor > Attachments: HBASE-7846-v0-trunk.patch > > > Currently org.apache.hadoop.hbase.util.Merge needs 2 region names to be > explicitly specified to perform a merge. This can be cumbersome. > One idea for improvement is to have Merge to figure out all the adjacent > regions and perform the merges. > For example: > regions before merge: row-10, row-20, row-30, row-40, row-50 > regions after merge: row-10, row-30, row-50 > In the above example, region names of "row-10" and "row-20" are merged to > become a new bigger region of "row-10". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7846) Add support for merging implicit regions in Merge tool
[ https://issues.apache.org/jira/browse/HBASE-7846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580604#comment-13580604 ] Jean-Marc Spaggiari commented on HBASE-7846: Attached patch for trunk. Adding 2 parameters to the merge tool. --all to merge all the regions 2 by 2 and --allrecurse to merge all the regions 2 by 2 and recuse until there is only one region remaining. Patch for 0.94 ready too. Will post it when trunk will be built. > Add support for merging implicit regions in Merge tool > -- > > Key: HBASE-7846 > URL: https://issues.apache.org/jira/browse/HBASE-7846 > Project: HBase > Issue Type: Improvement > Components: util >Reporter: Kaufman Ng >Assignee: Jean-Marc Spaggiari >Priority: Minor > Attachments: HBASE-7846-v0-trunk.patch > > > Currently org.apache.hadoop.hbase.util.Merge needs 2 region names to be > explicitly specified to perform a merge. This can be cumbersome. > One idea for improvement is to have Merge to figure out all the adjacent > regions and perform the merges. > For example: > regions before merge: row-10, row-20, row-30, row-40, row-50 > regions after merge: row-10, row-30, row-50 > In the above example, region names of "row-10" and "row-20" are merged to > become a new bigger region of "row-10". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7846) Add support for merging implicit regions in Merge tool
[ https://issues.apache.org/jira/browse/HBASE-7846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-7846: --- Status: Patch Available (was: Open) > Add support for merging implicit regions in Merge tool > -- > > Key: HBASE-7846 > URL: https://issues.apache.org/jira/browse/HBASE-7846 > Project: HBase > Issue Type: Improvement > Components: util >Reporter: Kaufman Ng >Assignee: Jean-Marc Spaggiari >Priority: Minor > Attachments: HBASE-7846-v0-trunk.patch > > > Currently org.apache.hadoop.hbase.util.Merge needs 2 region names to be > explicitly specified to perform a merge. This can be cumbersome. > One idea for improvement is to have Merge to figure out all the adjacent > regions and perform the merges. > For example: > regions before merge: row-10, row-20, row-30, row-40, row-50 > regions after merge: row-10, row-30, row-50 > In the above example, region names of "row-10" and "row-20" are merged to > become a new bigger region of "row-10". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7846) Add support for merging implicit regions in Merge tool
[
https://issues.apache.org/jira/browse/HBASE-7846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580626#comment-13580626
]
Hadoop QA commented on HBASE-7846:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12569810/HBASE-7846-v0-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4454//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4454//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4454//console
This message is automatically generated.
> Add support for merging implicit regions in Merge tool
> --
>
> Key: HBASE-7846
> URL: https://issues.apache.org/jira/browse/HBASE-7846
> Project: HBase
> Issue Type: Improvement
> Components: util
>Reporter: Kaufman Ng
>Assignee: Jean-Marc Spaggiari
>Priority: Minor
> Attachments: HBASE-7846-v0-trunk.patch
>
>
> Currently org.apache.hadoop.hbase.util.Merge needs 2 region names to be
> explicitly specified to perform a merge. This can be cumbersome.
> One idea for improvement is to have Merge to figure out all the adjacent
> regions and perform the merges.
> For example:
> regions before merge: row-10, row-20, row-30, row-40, row-50
> regions after merge: row-10, row-30, row-50
> In the above example, region names of "row-10" and "row-20" are merged to
> become a new bigger region of "row-10".
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7870) Delete(byte [] row, long timestamp) constructor is deprecate and should not.
[ https://issues.apache.org/jira/browse/HBASE-7870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580649#comment-13580649 ] Ted Yu commented on HBASE-7870: --- Good catch. +1 > Delete(byte [] row, long timestamp) constructor is deprecate and should not. > > > Key: HBASE-7870 > URL: https://issues.apache.org/jira/browse/HBASE-7870 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.5 >Reporter: Jean-Marc Spaggiari >Assignee: Jean-Marc Spaggiari >Priority: Critical > Attachments: HBASE-7870-v0-0.94.patch > > > Most probably a cut&past issue. Patch is coming. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7866) TestSplitTransactionOnCluster.testSplitBeforeSettingSplittingInZK failed 3 times in a row
[
https://issues.apache.org/jira/browse/HBASE-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan updated HBASE-7866:
--
Attachment: HBASE-7866_0.94.patch
Modified the testcase to handle this scenario. Even by the new wait introduced
if the znode does not get deleted atleast the assertion will tell us the
failure was due to Node already existing while creating SPLIT node.
> TestSplitTransactionOnCluster.testSplitBeforeSettingSplittingInZK failed 3
> times in a row
> -
>
> Key: HBASE-7866
> URL: https://issues.apache.org/jira/browse/HBASE-7866
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
> Fix For: 0.96.0, 0.94.6
>
> Attachments: HBASE-7866_0.94.patch
>
>
> Looks like the jenkins machines are flaky/slow again, causing this test to
> fail. Same stacktrace all three times:
> {code}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:92)
> at org.junit.Assert.assertTrue(Assert.java:43)
> at org.junit.Assert.assertTrue(Assert.java:54)
> at
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster.testSplitBeforeSettingSplittingInZK(TestSplitTransactionOnCluster.java:656)
> at
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster.testSplitBeforeSettingSplittingInZK(TestSplitTransactionOnCluster.java:608)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7868) HFile performance regression between 0.92 and 0.94
[
https://issues.apache.org/jira/browse/HBASE-7868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580702#comment-13580702
]
Lars Hofhansl commented on HBASE-7868:
--
JM, what I am saying is that
{{bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation --rows=10
filterScan 1}}
opens 10 scanners and calls next until there are no more rows. That would
"never" finish unless your each scan returns almost no data. You are not
testing scan performance with that, but RPC/Network performance (if you're
testing locally you're testing context switch performance through the lo
interface).
Do you seed the table with by calling SequentialWriteTest before running each
read test?
(I find that I have to do that, and otherwise there is just an empty test table)
The performance improvement after 0.94.0 that I measure this way is in line
with other tests I did before.
> HFile performance regression between 0.92 and 0.94
> --
>
> Key: HBASE-7868
> URL: https://issues.apache.org/jira/browse/HBASE-7868
> Project: HBase
> Issue Type: Bug
> Components: io
>Affects Versions: 0.94.5
>Reporter: Matteo Bertozzi
>Assignee: Matteo Bertozzi
> Fix For: 0.94.6
>
> Attachments: FilteredScan.png, hfileperf-graphs.png,
> performances.pdf, performances.pdf
>
>
> By HFilePerformanceEvaluation seems that 0.94 is slower then 0.92
> Looking at the profiler for the Scan path, seems that most of the time,
> compared to 92, is spent in the metrics dictionary lookup. [~eclark] pointed
> out the new per family/block metrics.
> By commenting the metrics call in HFileReaderV2, the performance seems to get
> better, but maybe metrics is not the only problem.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7866) TestSplitTransactionOnCluster.testSplitBeforeSettingSplittingInZK failed 3 times in a row
[
https://issues.apache.org/jira/browse/HBASE-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580703#comment-13580703
]
Lars Hofhansl commented on HBASE-7866:
--
Patch looks good.
> TestSplitTransactionOnCluster.testSplitBeforeSettingSplittingInZK failed 3
> times in a row
> -
>
> Key: HBASE-7866
> URL: https://issues.apache.org/jira/browse/HBASE-7866
> Project: HBase
> Issue Type: Bug
>Reporter: Lars Hofhansl
> Fix For: 0.96.0, 0.94.6
>
> Attachments: HBASE-7866_0.94.patch
>
>
> Looks like the jenkins machines are flaky/slow again, causing this test to
> fail. Same stacktrace all three times:
> {code}
> java.lang.AssertionError
> at org.junit.Assert.fail(Assert.java:92)
> at org.junit.Assert.assertTrue(Assert.java:43)
> at org.junit.Assert.assertTrue(Assert.java:54)
> at
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster.testSplitBeforeSettingSplittingInZK(TestSplitTransactionOnCluster.java:656)
> at
> org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster.testSplitBeforeSettingSplittingInZK(TestSplitTransactionOnCluster.java:608)
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7869) Provide way to not start LogSyncer thread
[ https://issues.apache.org/jira/browse/HBASE-7869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580704#comment-13580704 ] ramkrishna.s.vasudevan commented on HBASE-7869: --- So the fix will be something like check if the configured value is hbase.regionserver.optionallogflushinterval is 0 and if so do not start the thread right? Just asking, why not introduce a parameter hbase.regionserver.uselogsyncer=false. If this is false do not start the log syncer thread. And also we should avoid the wait statement that happens on closeLogSyncer while close() is happening if LogSyncer is not started. > Provide way to not start LogSyncer thread > - > > Key: HBASE-7869 > URL: https://issues.apache.org/jira/browse/HBASE-7869 > Project: HBase > Issue Type: Bug > Components: regionserver, wal >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.96.0, 0.94.6 > > > In our usecase we want all the Mutations on all the tables to be immediately > written and synced to WAL. We dont want the LogSyncer thread to be running in > such a case. > In minbatch operation we write entries to WAL and we use postWALWrite and > write some more WAL entries(Only write no sync). We want all these written > data to be synced as one unit (at the sync step in doMiniBatchOperation) But > if the LogSyncer thread is running in the system it can come and do the sync > at any point in time which we dont want in our case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7868) HFile performance regression between 0.92 and 0.94
[ https://issues.apache.org/jira/browse/HBASE-7868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580710#comment-13580710 ] Jean-Marc Spaggiari commented on HBASE-7868: Hi Lars, I see your point! I looked at the table while the test was running and it's empty. I was relying on the PerformanceEvaluation class to create the data it needs for its tests... Which is not the case. Now --rows=100 is running way slower, more realistic. Thanks for correting me. I will update my script to make sure SequentialWriteTest is called before all read/scan tests... > HFile performance regression between 0.92 and 0.94 > -- > > Key: HBASE-7868 > URL: https://issues.apache.org/jira/browse/HBASE-7868 > Project: HBase > Issue Type: Bug > Components: io >Affects Versions: 0.94.5 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 0.94.6 > > Attachments: FilteredScan.png, hfileperf-graphs.png, > performances.pdf, performances.pdf > > > By HFilePerformanceEvaluation seems that 0.94 is slower then 0.92 > Looking at the profiler for the Scan path, seems that most of the time, > compared to 92, is spent in the metrics dictionary lookup. [~eclark] pointed > out the new per family/block metrics. > By commenting the metrics call in HFileReaderV2, the performance seems to get > better, but maybe metrics is not the only problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7871) HBase can be stuck in the shutdown
nkeywal created HBASE-7871:
--
Summary: HBase can be stuck in the shutdown
Key: HBASE-7871
URL: https://issues.apache.org/jira/browse/HBASE-7871
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.96.0
Reporter: nkeywal
Attachments: TestStartStop.java
The attached test fails ~1% of the the time on 0.96. It seems it does not fail
on 0.94.5. It's simple: a table creation and some puts.
I attach the stack. Logs says nothing it seems.
The suspicious part is:
{noformat}
"RS_CLOSE_REGION-localhost,57575,1361197489166-2" prio=10
tid=0x7fb0c8775800 nid=0x61ac runnable [0x7fb09f272000]
java.lang.Thread.State: RUNNABLE
at java.util.TreeMap.fixAfterDeletion(TreeMap.java:2193)
at java.util.TreeMap.deleteEntry(TreeMap.java:2151)
at java.util.TreeMap.remove(TreeMap.java:585)
at java.util.TreeSet.remove(TreeSet.java:259)
at
org.apache.hadoop.hbase.regionserver.MetricsRegionAggregateSourceImpl.deregister(MetricsRegionAggregateSourceImpl.java:55)
at
org.apache.hadoop.hbase.regionserver.MetricsRegionSourceImpl.close(MetricsRegionSourceImpl.java:86)
at
org.apache.hadoop.hbase.regionserver.MetricsRegion.close(MetricsRegion.java:40)
at
org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1063)
at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:969)
- locked <0x0006944e2558> (a java.lang.Object)
at
org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:146)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:203)
at
java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7871) HBase can be stuck in the shutdown
[
https://issues.apache.org/jira/browse/HBASE-7871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nkeywal updated HBASE-7871:
---
Attachment: TestStartStop.java
> HBase can be stuck in the shutdown
> --
>
> Key: HBASE-7871
> URL: https://issues.apache.org/jira/browse/HBASE-7871
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.96.0
>Reporter: nkeywal
> Attachments: TestStartStop.java
>
>
> The attached test fails ~1% of the the time on 0.96. It seems it does not
> fail on 0.94.5. It's simple: a table creation and some puts.
> I attach the stack. Logs says nothing it seems.
> The suspicious part is:
> {noformat}
> "RS_CLOSE_REGION-localhost,57575,1361197489166-2" prio=10
> tid=0x7fb0c8775800 nid=0x61ac runnable [0x7fb09f272000]
>java.lang.Thread.State: RUNNABLE
> at java.util.TreeMap.fixAfterDeletion(TreeMap.java:2193)
> at java.util.TreeMap.deleteEntry(TreeMap.java:2151)
> at java.util.TreeMap.remove(TreeMap.java:585)
> at java.util.TreeSet.remove(TreeSet.java:259)
> at
> org.apache.hadoop.hbase.regionserver.MetricsRegionAggregateSourceImpl.deregister(MetricsRegionAggregateSourceImpl.java:55)
> at
> org.apache.hadoop.hbase.regionserver.MetricsRegionSourceImpl.close(MetricsRegionSourceImpl.java:86)
> at
> org.apache.hadoop.hbase.regionserver.MetricsRegion.close(MetricsRegion.java:40)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1063)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:969)
> - locked <0x0006944e2558> (a java.lang.Object)
> at
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:146)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:203)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7871) HBase can be stuck in the shutdown
[
https://issues.apache.org/jira/browse/HBASE-7871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nkeywal updated HBASE-7871:
---
Attachment: s1.txt
> HBase can be stuck in the shutdown
> --
>
> Key: HBASE-7871
> URL: https://issues.apache.org/jira/browse/HBASE-7871
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.96.0
>Reporter: nkeywal
> Attachments: s1.txt, TestStartStop.java
>
>
> The attached test fails ~1% of the the time on 0.96. It seems it does not
> fail on 0.94.5. It's simple: a table creation and some puts.
> I attach the stack. Logs says nothing it seems.
> The suspicious part is:
> {noformat}
> "RS_CLOSE_REGION-localhost,57575,1361197489166-2" prio=10
> tid=0x7fb0c8775800 nid=0x61ac runnable [0x7fb09f272000]
>java.lang.Thread.State: RUNNABLE
> at java.util.TreeMap.fixAfterDeletion(TreeMap.java:2193)
> at java.util.TreeMap.deleteEntry(TreeMap.java:2151)
> at java.util.TreeMap.remove(TreeMap.java:585)
> at java.util.TreeSet.remove(TreeSet.java:259)
> at
> org.apache.hadoop.hbase.regionserver.MetricsRegionAggregateSourceImpl.deregister(MetricsRegionAggregateSourceImpl.java:55)
> at
> org.apache.hadoop.hbase.regionserver.MetricsRegionSourceImpl.close(MetricsRegionSourceImpl.java:86)
> at
> org.apache.hadoop.hbase.regionserver.MetricsRegion.close(MetricsRegion.java:40)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1063)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:969)
> - locked <0x0006944e2558> (a java.lang.Object)
> at
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:146)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:203)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7865) HBase regionserver never stops when running `bin/stop-hbase.sh` on master
[ https://issues.apache.org/jira/browse/HBASE-7865?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580715#comment-13580715 ] nkeywal commented on HBASE-7865: FYI, I tried a simple test. 0.95 fails but not .94.5. See HBASE-7871. > HBase regionserver never stops when running `bin/stop-hbase.sh` on master > - > > Key: HBASE-7865 > URL: https://issues.apache.org/jira/browse/HBASE-7865 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.5 >Reporter: Jean-Marc Spaggiari > Attachments: hbase-root-master-hbasetest.log, > hbase-root-master-hbasetest.out.4, jstack_hbasetest.txt, jstack_node1.txt, > jstack_node3.txt, jstack_node7.txt > > > I faced 3 regions (out of 8) never stopping today. This is pretty bad because > the script is supposed to wait until all the RS stopped to re-start > everything, therefor, servers are never going back online. > HBASE-7838 will help with that and will kill the RSs. But that will not > really solve the root cause. > Attached are the jstack for the 3 servers. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7868) HFile performance regression between 0.92 and 0.94
[ https://issues.apache.org/jira/browse/HBASE-7868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580721#comment-13580721 ] Lars Hofhansl commented on HBASE-7868: -- I think PE needs some TLC. In it's current form it is not that useful (to say it bluntly), and it is 100% not obvious how to use it (I had to look at the source code to figure out what the filterScan test is supposed to do). At the very least we should add to the help text that one should seed the table first with a SequentialWriteTest picking the right of rows. The HBase wiki seems to imply that SequentialWriteTest is automatically run unless we run it in M/R mode, but it looks like that is not true. The help text does say to run FilterScanTest with --rows=20, not entirely sure why. > HFile performance regression between 0.92 and 0.94 > -- > > Key: HBASE-7868 > URL: https://issues.apache.org/jira/browse/HBASE-7868 > Project: HBase > Issue Type: Bug > Components: io >Affects Versions: 0.94.5 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 0.94.6 > > Attachments: FilteredScan.png, hfileperf-graphs.png, > performances.pdf, performances.pdf > > > By HFilePerformanceEvaluation seems that 0.94 is slower then 0.92 > Looking at the profiler for the Scan path, seems that most of the time, > compared to 92, is spent in the metrics dictionary lookup. [~eclark] pointed > out the new per family/block metrics. > By commenting the metrics call in HFileReaderV2, the performance seems to get > better, but maybe metrics is not the only problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7868) HFile performance regression between 0.92 and 0.94
[ https://issues.apache.org/jira/browse/HBASE-7868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580726#comment-13580726 ] Andrew Purtell commented on HBASE-7868: --- bq. I think PE needs some TLC. I've been using TestHFilePerformance for HFile microbenchmarking. It's passable but could use some TLC too. For minicluster "full system tests", variations on extensions to TestMiniClusterLoadXXX. > HFile performance regression between 0.92 and 0.94 > -- > > Key: HBASE-7868 > URL: https://issues.apache.org/jira/browse/HBASE-7868 > Project: HBase > Issue Type: Bug > Components: io >Affects Versions: 0.94.5 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 0.94.6 > > Attachments: FilteredScan.png, hfileperf-graphs.png, > performances.pdf, performances.pdf > > > By HFilePerformanceEvaluation seems that 0.94 is slower then 0.92 > Looking at the profiler for the Scan path, seems that most of the time, > compared to 92, is spent in the metrics dictionary lookup. [~eclark] pointed > out the new per family/block metrics. > By commenting the metrics call in HFileReaderV2, the performance seems to get > better, but maybe metrics is not the only problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7868) HFile performance regression between 0.92 and 0.94
[ https://issues.apache.org/jira/browse/HBASE-7868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580727#comment-13580727 ] Jean-Marc Spaggiari commented on HBASE-7868: Regarding --rows=20 it's because default value is 10240x1024... Which will take forever. 20 is a bit small. I'm using 100 now and test is running about 9 minutes. Which is correct. I agree that PE should be updated to call SequentialWriteTest first when required, or at least this should be specified in the documentation. I never looked at TestHFilePerformance... But I will ;) > HFile performance regression between 0.92 and 0.94 > -- > > Key: HBASE-7868 > URL: https://issues.apache.org/jira/browse/HBASE-7868 > Project: HBase > Issue Type: Bug > Components: io >Affects Versions: 0.94.5 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 0.94.6 > > Attachments: FilteredScan.png, hfileperf-graphs.png, > performances.pdf, performances.pdf > > > By HFilePerformanceEvaluation seems that 0.94 is slower then 0.92 > Looking at the profiler for the Scan path, seems that most of the time, > compared to 92, is spent in the metrics dictionary lookup. [~eclark] pointed > out the new per family/block metrics. > By commenting the metrics call in HFileReaderV2, the performance seems to get > better, but maybe metrics is not the only problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7868) HFile performance regression between 0.92 and 0.94
[ https://issues.apache.org/jira/browse/HBASE-7868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580743#comment-13580743 ] Matt Corgan commented on HBASE-7868: I have a decent start on a benchmark that tests many different combinations of inputs like blockSize, encoding, compression, keyLength, commonPrefixLength, valueLength. You can either generate fake test data or provide an existing HFile. It tests scans and seeks and outputs a summary of performance and memory/disk usage at the end so you can find the best settings for your use case. It's lurking somewhere in my git repo. I was planning to dig it up at the meetup tomorrow and get it working again. Maybe we can combine all these benchmarks somehow. > HFile performance regression between 0.92 and 0.94 > -- > > Key: HBASE-7868 > URL: https://issues.apache.org/jira/browse/HBASE-7868 > Project: HBase > Issue Type: Bug > Components: io >Affects Versions: 0.94.5 >Reporter: Matteo Bertozzi >Assignee: Matteo Bertozzi > Fix For: 0.94.6 > > Attachments: FilteredScan.png, hfileperf-graphs.png, > performances.pdf, performances.pdf > > > By HFilePerformanceEvaluation seems that 0.94 is slower then 0.92 > Looking at the profiler for the Scan path, seems that most of the time, > compared to 92, is spent in the metrics dictionary lookup. [~eclark] pointed > out the new per family/block metrics. > By commenting the metrics call in HFileReaderV2, the performance seems to get > better, but maybe metrics is not the only problem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7725) Add generic attributes to CP initiated compaction request AND latch on compaction completion
[ https://issues.apache.org/jira/browse/HBASE-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-7725: --- Attachment: hbase-7725-v5.patch Attaching patch with test (thanks for the suggestion [~lhofhansl]!). Had to do a little patch to HRegionServer so it could be mocked out in the test - wrapping the direct access to the compactSplitThread in a getter (3 lines). I'd like to commit on Tuesday. > Add generic attributes to CP initiated compaction request AND latch on > compaction completion > > > Key: HBASE-7725 > URL: https://issues.apache.org/jira/browse/HBASE-7725 > Project: HBase > Issue Type: Bug > Components: Compaction, Coprocessors, regionserver >Reporter: Jesse Yates >Assignee: Jesse Yates > Fix For: 0.96.0, 0.94.6 > > Attachments: example.java, hbase-7725_0.94-v0.patch, > hbase-7725-v0.patch, hbase-7725-v1.patch, hbase-7725-v3.patch, > hbase-7725-v4.patch, hbase-7725-v5.patch, > hbase-7725_with-attributes-0.94-v0.patch, > hbase-7725_with-attributes-0.94-v1.patch > > > You can request that a compaction be started, but you can't be sure when that > compaction request completes. This is a simple update to the > CompactionRequest interface and the compact-split thread on the RS that > doesn't actually impact the RS exposed interface. > This is particularly useful for CPs so they can control starting/running a > compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7725) Add generic attributes to CP initiated compaction request AND latch on compaction completion
[ https://issues.apache.org/jira/browse/HBASE-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580770#comment-13580770 ] Andrew Purtell commented on HBASE-7725: --- This looks good. +1 if the new test passes > Add generic attributes to CP initiated compaction request AND latch on > compaction completion > > > Key: HBASE-7725 > URL: https://issues.apache.org/jira/browse/HBASE-7725 > Project: HBase > Issue Type: Bug > Components: Compaction, Coprocessors, regionserver >Reporter: Jesse Yates >Assignee: Jesse Yates > Fix For: 0.96.0, 0.94.6 > > Attachments: example.java, hbase-7725_0.94-v0.patch, > hbase-7725-v0.patch, hbase-7725-v1.patch, hbase-7725-v3.patch, > hbase-7725-v4.patch, hbase-7725-v5.patch, > hbase-7725_with-attributes-0.94-v0.patch, > hbase-7725_with-attributes-0.94-v1.patch > > > You can request that a compaction be started, but you can't be sure when that > compaction request completes. This is a simple update to the > CompactionRequest interface and the compact-split thread on the RS that > doesn't actually impact the RS exposed interface. > This is particularly useful for CPs so they can control starting/running a > compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7725) Add generic attributes to CP initiated compaction request AND latch on compaction completion
[ https://issues.apache.org/jira/browse/HBASE-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-7725: --- Status: Patch Available (was: Open) > Add generic attributes to CP initiated compaction request AND latch on > compaction completion > > > Key: HBASE-7725 > URL: https://issues.apache.org/jira/browse/HBASE-7725 > Project: HBase > Issue Type: Bug > Components: Compaction, Coprocessors, regionserver >Reporter: Jesse Yates >Assignee: Jesse Yates > Fix For: 0.96.0, 0.94.6 > > Attachments: example.java, hbase-7725_0.94-v0.patch, > hbase-7725-v0.patch, hbase-7725-v1.patch, hbase-7725-v3.patch, > hbase-7725-v4.patch, hbase-7725-v5.patch, > hbase-7725_with-attributes-0.94-v0.patch, > hbase-7725_with-attributes-0.94-v1.patch > > > You can request that a compaction be started, but you can't be sure when that > compaction request completes. This is a simple update to the > CompactionRequest interface and the compact-split thread on the RS that > doesn't actually impact the RS exposed interface. > This is particularly useful for CPs so they can control starting/running a > compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7725) Add generic attributes to CP initiated compaction request AND latch on compaction completion
[
https://issues.apache.org/jira/browse/HBASE-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580806#comment-13580806
]
Hadoop QA commented on HBASE-7725:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12569835/hbase-7725-v5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 6
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.coprocessor.TestRegionObserverInterface
org.apache.hadoop.hbase.io.encoding.TestLoadAndSwitchEncodeOnDisk
org.apache.hadoop.hbase.io.encoding.TestChangingEncoding
org.apache.hadoop.hbase.backup.TestHFileArchiving
org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor
org.apache.hadoop.hbase.regionserver.TestCompactionState
org.apache.hadoop.hbase.client.TestShell
org.apache.hadoop.hbase.mapreduce.TestHFileOutputFormat
org.apache.hadoop.hbase.regionserver.TestEndToEndSplitTransaction
org.apache.hadoop.hbase.regionserver.TestSplitTransactionOnCluster
org.apache.hadoop.hbase.client.TestFromClientSide
org.apache.hadoop.hbase.client.TestFromClientSide3
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4455//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4455//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4455//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4455//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4455//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4455//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4455//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4455//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4455//console
This message is automatically generated.
> Add generic attributes to CP initiated compaction request AND latch on
> compaction completion
>
>
> Key: HBASE-7725
> URL: https://issues.apache.org/jira/browse/HBASE-7725
> Project: HBase
> Issue Type: Bug
> Components: Compaction, Coprocessors, regionserver
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Fix For: 0.96.0, 0.94.6
>
> Attachments: example.java, hbase-7725_0.94-v0.patch,
> hbase-7725-v0.patch, hbase-7725-v1.patch, hbase-7725-v3.patch,
> hbase-7725-v4.patch, hbase-7725-v5.patch,
> hbase-7725_with-attributes-0.94-v0.patch,
> hbase-7725_with-attributes-0.94-v1.patch
>
>
> You can request that a compaction be started, but you can't be sure when that
> compaction request completes. This is a simple update to the
> CompactionRequest interface and the compact-split thread on the RS that
> doesn't actually impact the RS exposed interface.
> This is particularly useful for CPs so they can control starting/running a
> compaction.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7872) Delete(byte [] row, long timestamp) is marked as deprecated
James Taylor created HBASE-7872:
---
Summary: Delete(byte [] row, long timestamp) is marked as
deprecated
Key: HBASE-7872
URL: https://issues.apache.org/jira/browse/HBASE-7872
Project: HBase
Issue Type: Bug
Affects Versions: 0.94.5
Reporter: James Taylor
Priority: Minor
I see that the constructor: public Delete(byte [] row, long timestamp, RowLock
rowLock) was deprecated, but the following was also deprecated with a partial
comment that looks like a copy/paste error, no?
/**
* Create a Delete operation for the specified row and timestamp.
*
* If no further operations are done, this will delete all columns in all
* families of the specified row with a timestamp less than or equal to the
* specified timestamp.
*
* This timestamp is ONLY used for a delete row operation. If specifying
* families or columns, you must specify each timestamp individually.
* @param row row key
* @param timestamp maximum version timestamp (only for delete row)
* @param rowLock previously acquired row lock, or null
* @deprecated {@link RowLock} is deprecated, use #de
*/
public Delete(byte [] row, long timestamp) {
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7725) Add generic attributes to CP initiated compaction request AND latch on compaction completion
[
https://issues.apache.org/jira/browse/HBASE-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580816#comment-13580816
]
Andrew Purtell commented on HBASE-7725:
---
-1 Some NPEs out of compaction code:
{noformat}
Caused by: org.apache.hadoop.ipc.RemoteException: java.io.IOException:
java.lang.NullPointerException
at
org.apache.hadoop.hbase.regionserver.CompactSplitThread.requestCompaction(CompactSplitThread.java:217)
at
org.apache.hadoop.hbase.regionserver.HRegionServer.compactRegion(HRegionServer.java:3633)
{noformat}
> Add generic attributes to CP initiated compaction request AND latch on
> compaction completion
>
>
> Key: HBASE-7725
> URL: https://issues.apache.org/jira/browse/HBASE-7725
> Project: HBase
> Issue Type: Bug
> Components: Compaction, Coprocessors, regionserver
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Fix For: 0.96.0, 0.94.6
>
> Attachments: example.java, hbase-7725_0.94-v0.patch,
> hbase-7725-v0.patch, hbase-7725-v1.patch, hbase-7725-v3.patch,
> hbase-7725-v4.patch, hbase-7725-v5.patch,
> hbase-7725_with-attributes-0.94-v0.patch,
> hbase-7725_with-attributes-0.94-v1.patch
>
>
> You can request that a compaction be started, but you can't be sure when that
> compaction request completes. This is a simple update to the
> CompactionRequest interface and the compact-split thread on the RS that
> doesn't actually impact the RS exposed interface.
> This is particularly useful for CPs so they can control starting/running a
> compaction.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7872) Delete(byte [] row, long timestamp) is marked as deprecated
[
https://issues.apache.org/jira/browse/HBASE-7872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580819#comment-13580819
]
Jean-Marc Spaggiari commented on HBASE-7872:
Is this a duplicate of HBASE-7870?
> Delete(byte [] row, long timestamp) is marked as deprecated
> ---
>
> Key: HBASE-7872
> URL: https://issues.apache.org/jira/browse/HBASE-7872
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.5
>Reporter: James Taylor
>Priority: Minor
>
> I see that the constructor: public Delete(byte [] row, long timestamp,
> RowLock rowLock) was deprecated, but the following was also deprecated with a
> partial comment that looks like a copy/paste error, no?
> /**
>* Create a Delete operation for the specified row and timestamp.
>*
>* If no further operations are done, this will delete all columns in all
>* families of the specified row with a timestamp less than or equal to the
>* specified timestamp.
>*
>* This timestamp is ONLY used for a delete row operation. If specifying
>* families or columns, you must specify each timestamp individually.
>* @param row row key
>* @param timestamp maximum version timestamp (only for delete row)
>* @param rowLock previously acquired row lock, or null
>* @deprecated {@link RowLock} is deprecated, use #de
>*/
> public Delete(byte [] row, long timestamp) {
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7221) RowKey utility class for rowkey construction
[
https://issues.apache.org/jira/browse/HBASE-7221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580829#comment-13580829
]
Doug Meil commented on HBASE-7221:
--
Hey folks, can I ask you to give the v5 patch a review? Thanks!
> RowKey utility class for rowkey construction
>
>
> Key: HBASE-7221
> URL: https://issues.apache.org/jira/browse/HBASE-7221
> Project: HBase
> Issue Type: Improvement
> Components: util
>Reporter: Doug Meil
>Assignee: Doug Meil
>Priority: Minor
> Attachments: HBASE_7221.patch, hbase-common_hbase_7221_2.patch,
> hbase-common_hbase_7221_v3.patch, hbase-common_hbase_7221_v4.patch,
> hbase-server_hbase_7221_v5.patch
>
>
> A common question in the dist-lists is how to construct rowkeys, particularly
> composite keys. Put/Get/Scan specifies byte[] as the rowkey, but it's up to
> you to sensibly populate that byte-array, and that's where things tend to go
> off the rails.
> The intent of this RowKey utility class isn't meant to add functionality into
> Put/Get/Scan, but rather make it simpler for folks to construct said arrays.
> Example:
> {code}
>RowKey key = RowKey.create(RowKey.SIZEOF_MD5_HASH + RowKey.SIZEOF_LONG);
>key.addHash(a);
>key.add(b);
>byte bytes[] = key.getBytes();
> {code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-7872) Delete(byte [] row, long timestamp) is marked as deprecated
[
https://issues.apache.org/jira/browse/HBASE-7872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
James Taylor resolved HBASE-7872.
-
Resolution: Duplicate
Duplicate of HBASE-7870
> Delete(byte [] row, long timestamp) is marked as deprecated
> ---
>
> Key: HBASE-7872
> URL: https://issues.apache.org/jira/browse/HBASE-7872
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.94.5
>Reporter: James Taylor
>Priority: Minor
>
> I see that the constructor: public Delete(byte [] row, long timestamp,
> RowLock rowLock) was deprecated, but the following was also deprecated with a
> partial comment that looks like a copy/paste error, no?
> /**
>* Create a Delete operation for the specified row and timestamp.
>*
>* If no further operations are done, this will delete all columns in all
>* families of the specified row with a timestamp less than or equal to the
>* specified timestamp.
>*
>* This timestamp is ONLY used for a delete row operation. If specifying
>* families or columns, you must specify each timestamp individually.
>* @param row row key
>* @param timestamp maximum version timestamp (only for delete row)
>* @param rowLock previously acquired row lock, or null
>* @deprecated {@link RowLock} is deprecated, use #de
>*/
> public Delete(byte [] row, long timestamp) {
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7846) Add support for merging implicit regions in Merge tool
[ https://issues.apache.org/jira/browse/HBASE-7846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-7846: --- Status: Open (was: Patch Available) > Add support for merging implicit regions in Merge tool > -- > > Key: HBASE-7846 > URL: https://issues.apache.org/jira/browse/HBASE-7846 > Project: HBase > Issue Type: Improvement > Components: util >Reporter: Kaufman Ng >Assignee: Jean-Marc Spaggiari >Priority: Minor > Attachments: HBASE-7846-v0-trunk.patch, HBASE-7846-v1-trunk.patch > > > Currently org.apache.hadoop.hbase.util.Merge needs 2 region names to be > explicitly specified to perform a merge. This can be cumbersome. > One idea for improvement is to have Merge to figure out all the adjacent > regions and perform the merges. > For example: > regions before merge: row-10, row-20, row-30, row-40, row-50 > regions after merge: row-10, row-30, row-50 > In the above example, region names of "row-10" and "row-20" are merged to > become a new bigger region of "row-10". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7846) Add support for merging implicit regions in Merge tool
[ https://issues.apache.org/jira/browse/HBASE-7846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-7846: --- Attachment: HBASE-7846-v1-trunk.patch > Add support for merging implicit regions in Merge tool > -- > > Key: HBASE-7846 > URL: https://issues.apache.org/jira/browse/HBASE-7846 > Project: HBase > Issue Type: Improvement > Components: util >Reporter: Kaufman Ng >Assignee: Jean-Marc Spaggiari >Priority: Minor > Attachments: HBASE-7846-v0-trunk.patch, HBASE-7846-v1-trunk.patch > > > Currently org.apache.hadoop.hbase.util.Merge needs 2 region names to be > explicitly specified to perform a merge. This can be cumbersome. > One idea for improvement is to have Merge to figure out all the adjacent > regions and perform the merges. > For example: > regions before merge: row-10, row-20, row-30, row-40, row-50 > regions after merge: row-10, row-30, row-50 > In the above example, region names of "row-10" and "row-20" are merged to > become a new bigger region of "row-10". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7846) Add support for merging implicit regions in Merge tool
[ https://issues.apache.org/jira/browse/HBASE-7846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jean-Marc Spaggiari updated HBASE-7846: --- Status: Patch Available (was: Open) Fixed lineLengths issue. There is no unit test class for util.Merge. I tested it locally by splitting and merging many different tables. > Add support for merging implicit regions in Merge tool > -- > > Key: HBASE-7846 > URL: https://issues.apache.org/jira/browse/HBASE-7846 > Project: HBase > Issue Type: Improvement > Components: util >Reporter: Kaufman Ng >Assignee: Jean-Marc Spaggiari >Priority: Minor > Attachments: HBASE-7846-v0-trunk.patch, HBASE-7846-v1-trunk.patch > > > Currently org.apache.hadoop.hbase.util.Merge needs 2 region names to be > explicitly specified to perform a merge. This can be cumbersome. > One idea for improvement is to have Merge to figure out all the adjacent > regions and perform the merges. > For example: > regions before merge: row-10, row-20, row-30, row-40, row-50 > regions after merge: row-10, row-30, row-50 > In the above example, region names of "row-10" and "row-20" are merged to > become a new bigger region of "row-10". -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7846) Add support for merging implicit regions in Merge tool
[
https://issues.apache.org/jira/browse/HBASE-7846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580888#comment-13580888
]
Hadoop QA commented on HBASE-7846:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12569862/HBASE-7846-v1-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestZKBasedOpenCloseRegion
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4456//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4456//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4456//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4456//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4456//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4456//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4456//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4456//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4456//console
This message is automatically generated.
> Add support for merging implicit regions in Merge tool
> --
>
> Key: HBASE-7846
> URL: https://issues.apache.org/jira/browse/HBASE-7846
> Project: HBase
> Issue Type: Improvement
> Components: util
>Reporter: Kaufman Ng
>Assignee: Jean-Marc Spaggiari
>Priority: Minor
> Attachments: HBASE-7846-v0-trunk.patch, HBASE-7846-v1-trunk.patch
>
>
> Currently org.apache.hadoop.hbase.util.Merge needs 2 region names to be
> explicitly specified to perform a merge. This can be cumbersome.
> One idea for improvement is to have Merge to figure out all the adjacent
> regions and perform the merges.
> For example:
> regions before merge: row-10, row-20, row-30, row-40, row-50
> regions after merge: row-10, row-30, row-50
> In the above example, region names of "row-10" and "row-20" are merged to
> become a new bigger region of "row-10".
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7725) Add generic attributes to CP initiated compaction request AND latch on compaction completion
[ https://issues.apache.org/jira/browse/HBASE-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-7725: --- Attachment: hbase-7725-v6.patch Attaching patch with the fix (Forgot an else and just dropped through the if :-/). Failed tests passed locally. Lets try this again... > Add generic attributes to CP initiated compaction request AND latch on > compaction completion > > > Key: HBASE-7725 > URL: https://issues.apache.org/jira/browse/HBASE-7725 > Project: HBase > Issue Type: Bug > Components: Compaction, Coprocessors, regionserver >Reporter: Jesse Yates >Assignee: Jesse Yates > Fix For: 0.96.0, 0.94.6 > > Attachments: example.java, hbase-7725_0.94-v0.patch, > hbase-7725-v0.patch, hbase-7725-v1.patch, hbase-7725-v3.patch, > hbase-7725-v4.patch, hbase-7725-v5.patch, hbase-7725-v6.patch, > hbase-7725_with-attributes-0.94-v0.patch, > hbase-7725_with-attributes-0.94-v1.patch > > > You can request that a compaction be started, but you can't be sure when that > compaction request completes. This is a simple update to the > CompactionRequest interface and the compact-split thread on the RS that > doesn't actually impact the RS exposed interface. > This is particularly useful for CPs so they can control starting/running a > compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7869) Provide way to not start LogSyncer thread
[ https://issues.apache.org/jira/browse/HBASE-7869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580928#comment-13580928 ] Anoop Sam John commented on HBASE-7869: --- [~ram_krish] New config item may not be needed. When hbase.regionserver.optionallogflushinterval is set as 0, means you need an immediate sync for all the operations. The thread is starting for the deferred sync. So in this case thread wont be useful at all and so dont start it at all. Yes at the close() time no need to go for the thread close part and join. > Provide way to not start LogSyncer thread > - > > Key: HBASE-7869 > URL: https://issues.apache.org/jira/browse/HBASE-7869 > Project: HBase > Issue Type: Bug > Components: regionserver, wal >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.96.0, 0.94.6 > > > In our usecase we want all the Mutations on all the tables to be immediately > written and synced to WAL. We dont want the LogSyncer thread to be running in > such a case. > In minbatch operation we write entries to WAL and we use postWALWrite and > write some more WAL entries(Only write no sync). We want all these written > data to be synced as one unit (at the sync step in doMiniBatchOperation) But > if the LogSyncer thread is running in the system it can come and do the sync > at any point in time which we dont want in our case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7869) Provide way to not start LogSyncer thread
[ https://issues.apache.org/jira/browse/HBASE-7869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-7869: -- Attachment: HBASE-7869_Trunk.patch > Provide way to not start LogSyncer thread > - > > Key: HBASE-7869 > URL: https://issues.apache.org/jira/browse/HBASE-7869 > Project: HBase > Issue Type: Bug > Components: regionserver, wal >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.96.0, 0.94.6 > > Attachments: HBASE-7869_Trunk.patch > > > In our usecase we want all the Mutations on all the tables to be immediately > written and synced to WAL. We dont want the LogSyncer thread to be running in > such a case. > In minbatch operation we write entries to WAL and we use postWALWrite and > write some more WAL entries(Only write no sync). We want all these written > data to be synced as one unit (at the sync step in doMiniBatchOperation) But > if the LogSyncer thread is running in the system it can come and do the sync > at any point in time which we dont want in our case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7869) Provide way to not start LogSyncer thread
[ https://issues.apache.org/jira/browse/HBASE-7869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John updated HBASE-7869: -- Status: Patch Available (was: Open) Run HLog related tests locally and passed. Let QA run. I will see any more changes needed in other parts as we dont have syncer thread > Provide way to not start LogSyncer thread > - > > Key: HBASE-7869 > URL: https://issues.apache.org/jira/browse/HBASE-7869 > Project: HBase > Issue Type: Bug > Components: regionserver, wal >Reporter: Anoop Sam John >Assignee: Anoop Sam John > Fix For: 0.96.0, 0.94.6 > > Attachments: HBASE-7869_Trunk.patch > > > In our usecase we want all the Mutations on all the tables to be immediately > written and synced to WAL. We dont want the LogSyncer thread to be running in > such a case. > In minbatch operation we write entries to WAL and we use postWALWrite and > write some more WAL entries(Only write no sync). We want all these written > data to be synced as one unit (at the sync step in doMiniBatchOperation) But > if the LogSyncer thread is running in the system it can come and do the sync > at any point in time which we dont want in our case. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7725) Add generic attributes to CP initiated compaction request AND latch on compaction completion
[ https://issues.apache.org/jira/browse/HBASE-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-7725: -- Status: Open (was: Patch Available) > Add generic attributes to CP initiated compaction request AND latch on > compaction completion > > > Key: HBASE-7725 > URL: https://issues.apache.org/jira/browse/HBASE-7725 > Project: HBase > Issue Type: Bug > Components: Compaction, Coprocessors, regionserver >Reporter: Jesse Yates >Assignee: Jesse Yates > Fix For: 0.96.0, 0.94.6 > > Attachments: example.java, hbase-7725_0.94-v0.patch, > hbase-7725-v0.patch, hbase-7725-v1.patch, hbase-7725-v3.patch, > hbase-7725-v4.patch, hbase-7725-v5.patch, hbase-7725-v6.patch, > hbase-7725_with-attributes-0.94-v0.patch, > hbase-7725_with-attributes-0.94-v1.patch > > > You can request that a compaction be started, but you can't be sure when that > compaction request completes. This is a simple update to the > CompactionRequest interface and the compact-split thread on the RS that > doesn't actually impact the RS exposed interface. > This is particularly useful for CPs so they can control starting/running a > compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7725) Add generic attributes to CP initiated compaction request AND latch on compaction completion
[ https://issues.apache.org/jira/browse/HBASE-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-7725: -- Status: Patch Available (was: Open) > Add generic attributes to CP initiated compaction request AND latch on > compaction completion > > > Key: HBASE-7725 > URL: https://issues.apache.org/jira/browse/HBASE-7725 > Project: HBase > Issue Type: Bug > Components: Compaction, Coprocessors, regionserver >Reporter: Jesse Yates >Assignee: Jesse Yates > Fix For: 0.96.0, 0.94.6 > > Attachments: example.java, hbase-7725_0.94-v0.patch, > hbase-7725-v0.patch, hbase-7725-v1.patch, hbase-7725-v3.patch, > hbase-7725-v4.patch, hbase-7725-v5.patch, hbase-7725-v6.patch, > hbase-7725_with-attributes-0.94-v0.patch, > hbase-7725_with-attributes-0.94-v1.patch > > > You can request that a compaction be started, but you can't be sure when that > compaction request completes. This is a simple update to the > CompactionRequest interface and the compact-split thread on the RS that > doesn't actually impact the RS exposed interface. > This is particularly useful for CPs so they can control starting/running a > compaction. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7835) Implementation of the log splitting without creating intermediate files
[ https://issues.apache.org/jira/browse/HBASE-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jeffrey Zhong updated HBASE-7835: - Attachment: hbase-7006_2.patch Fixes test failures and rework the master node recovery logic. Thanks, -Jeffrey > Implementation of the log splitting without creating intermediate files > > > Key: HBASE-7835 > URL: https://issues.apache.org/jira/browse/HBASE-7835 > Project: HBase > Issue Type: Sub-task > Components: regionserver >Reporter: Jeffrey Zhong >Assignee: Jeffrey Zhong > Fix For: 0.96.0 > > Attachments: hbase-7006_1.patch, hbase-7006_2.patch > > > The sub task is to check in a separate patch for major implementation for the > proposal. > More checkins will include a new "replay" command and more metrics support. > Thanks, > -Jeffrey -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7725) Add generic attributes to CP initiated compaction request AND latch on compaction completion
[
https://issues.apache.org/jira/browse/HBASE-7725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580950#comment-13580950
]
Hadoop QA commented on HBASE-7725:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12569873/hbase-7725-v6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 6
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.wal.TestHLog
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4457//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4457//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4457//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4457//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4457//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4457//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4457//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4457//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4457//console
This message is automatically generated.
> Add generic attributes to CP initiated compaction request AND latch on
> compaction completion
>
>
> Key: HBASE-7725
> URL: https://issues.apache.org/jira/browse/HBASE-7725
> Project: HBase
> Issue Type: Bug
> Components: Compaction, Coprocessors, regionserver
>Reporter: Jesse Yates
>Assignee: Jesse Yates
> Fix For: 0.96.0, 0.94.6
>
> Attachments: example.java, hbase-7725_0.94-v0.patch,
> hbase-7725-v0.patch, hbase-7725-v1.patch, hbase-7725-v3.patch,
> hbase-7725-v4.patch, hbase-7725-v5.patch, hbase-7725-v6.patch,
> hbase-7725_with-attributes-0.94-v0.patch,
> hbase-7725_with-attributes-0.94-v1.patch
>
>
> You can request that a compaction be started, but you can't be sure when that
> compaction request completes. This is a simple update to the
> CompactionRequest interface and the compact-split thread on the RS that
> doesn't actually impact the RS exposed interface.
> This is particularly useful for CPs so they can control starting/running a
> compaction.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7873) Extract common logic between TakeSnapshotHandler and RestoreSnapshotHandler
Ted Yu created HBASE-7873: - Summary: Extract common logic between TakeSnapshotHandler and RestoreSnapshotHandler Key: HBASE-7873 URL: https://issues.apache.org/jira/browse/HBASE-7873 Project: HBase Issue Type: Sub-task Reporter: Ted Yu This came up from Jesse during review: Feel like there ought to be come commonality here with the RestoreHandler. Maybe a util state storage class? Not a deal breaker by any means, just thought it might help make both a little more concise -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7858) cleanup before merging snapshots branch to trunk
[ https://issues.apache.org/jira/browse/HBASE-7858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ted Yu updated HBASE-7858: -- Attachment: 7858-v8.txt Patch v8 removes verifyLogs() which is not used. Also addresses race condition in submitProcedure() > cleanup before merging snapshots branch to trunk > > > Key: HBASE-7858 > URL: https://issues.apache.org/jira/browse/HBASE-7858 > Project: HBase > Issue Type: Sub-task >Reporter: Ted Yu >Assignee: Ted Yu > Attachments: 7858-v1.txt, 7858-v2.txt, 7858-v3.txt, 7858-v4.txt, > 7858-v5.txt, 7858-v6.txt, 7858-v7.txt, 7858-v8.txt > > > There have been a lot of review comments from > https://reviews.apache.org/r/9416 > Since our goal of merging snapshot feature to trunk would preserve revision > history, a separate JIRA is needed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7869) Provide way to not start LogSyncer thread
[
https://issues.apache.org/jira/browse/HBASE-7869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580959#comment-13580959
]
Hadoop QA commented on HBASE-7869:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12569876/HBASE-7869_Trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4458//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4458//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4458//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4458//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4458//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4458//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4458//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4458//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4458//console
This message is automatically generated.
> Provide way to not start LogSyncer thread
> -
>
> Key: HBASE-7869
> URL: https://issues.apache.org/jira/browse/HBASE-7869
> Project: HBase
> Issue Type: Bug
> Components: regionserver, wal
>Reporter: Anoop Sam John
>Assignee: Anoop Sam John
> Fix For: 0.96.0, 0.94.6
>
> Attachments: HBASE-7869_Trunk.patch
>
>
> In our usecase we want all the Mutations on all the tables to be immediately
> written and synced to WAL. We dont want the LogSyncer thread to be running in
> such a case.
> In minbatch operation we write entries to WAL and we use postWALWrite and
> write some more WAL entries(Only write no sync). We want all these written
> data to be synced as one unit (at the sync step in doMiniBatchOperation) But
> if the LogSyncer thread is running in the system it can come and do the sync
> at any point in time which we dont want in our case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7835) Implementation of the log splitting without creating intermediate files
[
https://issues.apache.org/jira/browse/HBASE-7835?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580966#comment-13580966
]
Hadoop QA commented on HBASE-7835:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12569884/hbase-7006_2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 12 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 2
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4459//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4459//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4459//console
This message is automatically generated.
> Implementation of the log splitting without creating intermediate files
>
>
> Key: HBASE-7835
> URL: https://issues.apache.org/jira/browse/HBASE-7835
> Project: HBase
> Issue Type: Sub-task
> Components: regionserver
>Reporter: Jeffrey Zhong
>Assignee: Jeffrey Zhong
> Fix For: 0.96.0
>
> Attachments: hbase-7006_1.patch, hbase-7006_2.patch
>
>
> The sub task is to check in a separate patch for major implementation for the
> proposal.
> More checkins will include a new "replay" command and more metrics support.
> Thanks,
> -Jeffrey
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7874) Allow RegionServer to abort "Put" if it detects that the HBase Client got SocketTimeoutException and disconnected.
Maryann Xue created HBASE-7874: -- Summary: Allow RegionServer to abort "Put" if it detects that the HBase Client got SocketTimeoutException and disconnected. Key: HBASE-7874 URL: https://issues.apache.org/jira/browse/HBASE-7874 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.94.5 Reporter: Maryann Xue Assignee: Maryann Xue Priority: Minor Usually, when regionserver cannot catch up with the put load given by the client, what happens is region server starts to block update requests from the client until required resource has been reclaimed (i.e. memstore has been flushed). But in more severe situations, the blocking time gets so long that the client begins to have SocketTimeoutException and then decides to retry, while in fact the updates are written into memstore later after they are unblocked. Even though the client has something like a binary rollback for retry intervals, this can still lead to a vicious circle, leaving the client to have very low throughput. Think we can enable an option to allow regionserver to check if the client has disconnected (just like what we do in scan) after coming back from "blocking", so that the regionserver has the same view as the client on whether updates are successfully committed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7875) introduce a compaction switch in HBase Shell
Liang Xie created HBASE-7875:
Summary: introduce a compaction switch in HBase Shell
Key: HBASE-7875
URL: https://issues.apache.org/jira/browse/HBASE-7875
Project: HBase
Issue Type: New Feature
Components: shell
Affects Versions: 0.96.0
Reporter: Liang Xie
Assignee: Liang Xie
Priority: Trivial
Attachments: HBASE-7875.txt
A trivial patch to add an per-table compaction switch. it would be helpful for
OPS or manual testing in real cluster, etc.
Usage:
hbase(main):016:0> alter 'YCSBTest', {METHOD => 'table_att', COMPACTION_ENABLE
=> 'FALSE'}
Updating all regions with the new schema...
64/64 regions updated.
Done.
0 row(s) in 1.1290 seconds
hbase(main):017:0> describe 'YCSBTest'
DESCRIPTION
ENABLED
{NAME => 'YCSBTest', COMPACTION_ENABLE => 'false', SPLIT_POLICY =>
'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPoli true
cy', FAMILIES => [{NAME => 'test', DATA_BLOCK_ENCODING => 'NONE', BLOOMFILTER
=> 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'S
NAPPY', VERSIONS => '1', TTL => '2147483647', MIN_VERSIONS => '0',
KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', ENCODE_ON_DI
SK => 'true', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}
1 row(s) in 0.0210 seconds
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7875) introduce a compaction switch in HBase Shell
[
https://issues.apache.org/jira/browse/HBASE-7875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liang Xie updated HBASE-7875:
-
Attachment: HBASE-7875.txt
> introduce a compaction switch in HBase Shell
>
>
> Key: HBASE-7875
> URL: https://issues.apache.org/jira/browse/HBASE-7875
> Project: HBase
> Issue Type: New Feature
> Components: shell
>Affects Versions: 0.96.0
>Reporter: Liang Xie
>Assignee: Liang Xie
>Priority: Trivial
> Attachments: HBASE-7875.txt
>
>
> A trivial patch to add an per-table compaction switch. it would be helpful
> for OPS or manual testing in real cluster, etc.
> Usage:
> hbase(main):016:0> alter 'YCSBTest', {METHOD => 'table_att',
> COMPACTION_ENABLE => 'FALSE'}
> Updating all regions with the new schema...
> 64/64 regions updated.
> Done.
> 0 row(s) in 1.1290 seconds
> hbase(main):017:0> describe 'YCSBTest'
> DESCRIPTION
> ENABLED
>
> {NAME => 'YCSBTest', COMPACTION_ENABLE => 'false', SPLIT_POLICY =>
> 'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPoli true
>
> cy', FAMILIES => [{NAME => 'test', DATA_BLOCK_ENCODING => 'NONE',
> BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'S
>
> NAPPY', VERSIONS => '1', TTL => '2147483647', MIN_VERSIONS => '0',
> KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', ENCODE_ON_DI
>
> SK => 'true', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}
>
>
> 1 row(s) in 0.0210 seconds
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7875) introduce a compaction switch in HBase Shell
[
https://issues.apache.org/jira/browse/HBASE-7875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Liang Xie updated HBASE-7875:
-
Status: Patch Available (was: Open)
> introduce a compaction switch in HBase Shell
>
>
> Key: HBASE-7875
> URL: https://issues.apache.org/jira/browse/HBASE-7875
> Project: HBase
> Issue Type: New Feature
> Components: shell
>Affects Versions: 0.96.0
>Reporter: Liang Xie
>Assignee: Liang Xie
>Priority: Trivial
> Attachments: HBASE-7875.txt
>
>
> A trivial patch to add an per-table compaction switch. it would be helpful
> for OPS or manual testing in real cluster, etc.
> Usage:
> hbase(main):016:0> alter 'YCSBTest', {METHOD => 'table_att',
> COMPACTION_ENABLE => 'FALSE'}
> Updating all regions with the new schema...
> 64/64 regions updated.
> Done.
> 0 row(s) in 1.1290 seconds
> hbase(main):017:0> describe 'YCSBTest'
> DESCRIPTION
> ENABLED
>
> {NAME => 'YCSBTest', COMPACTION_ENABLE => 'false', SPLIT_POLICY =>
> 'org.apache.hadoop.hbase.regionserver.ConstantSizeRegionSplitPoli true
>
> cy', FAMILIES => [{NAME => 'test', DATA_BLOCK_ENCODING => 'NONE',
> BLOOMFILTER => 'NONE', REPLICATION_SCOPE => '0', COMPRESSION => 'S
>
> NAPPY', VERSIONS => '1', TTL => '2147483647', MIN_VERSIONS => '0',
> KEEP_DELETED_CELLS => 'false', BLOCKSIZE => '65536', ENCODE_ON_DI
>
> SK => 'true', IN_MEMORY => 'false', BLOCKCACHE => 'true'}]}
>
>
> 1 row(s) in 0.0210 seconds
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7876) Got exception when splitting a region that contains no storefile
Maryann Xue created HBASE-7876: -- Summary: Got exception when splitting a region that contains no storefile Key: HBASE-7876 URL: https://issues.apache.org/jira/browse/HBASE-7876 Project: HBase Issue Type: Bug Components: regionserver Affects Versions: 0.94.5 Reporter: Maryann Xue Assignee: Maryann Xue We should allow a region to split successfully even if it does not yet have storefiles. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7874) Allow RegionServer to abort "Put" if it detects that the HBase Client got SocketTimeoutException and disconnected.
[ https://issues.apache.org/jira/browse/HBASE-7874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maryann Xue updated HBASE-7874: --- Attachment: HBASE-7874-0.94.patch add "check channel closed" in HRegion#batchMutate() > Allow RegionServer to abort "Put" if it detects that the HBase Client got > SocketTimeoutException and disconnected. > -- > > Key: HBASE-7874 > URL: https://issues.apache.org/jira/browse/HBASE-7874 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.94.5 >Reporter: Maryann Xue >Assignee: Maryann Xue >Priority: Minor > Attachments: HBASE-7874-0.94.patch > > > Usually, when regionserver cannot catch up with the put load given by the > client, what happens is region server starts to block update requests from > the client until required resource has been reclaimed (i.e. memstore has been > flushed). > But in more severe situations, the blocking time gets so long that the client > begins to have SocketTimeoutException and then decides to retry, while in > fact the updates are written into memstore later after they are unblocked. > Even though the client has something like a binary rollback for retry > intervals, this can still lead to a vicious circle, leaving the client to > have very low throughput. > Think we can enable an option to allow regionserver to check if the client > has disconnected (just like what we do in scan) after coming back from > "blocking", so that the regionserver has the same view as the client on > whether updates are successfully committed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7874) Allow RegionServer to abort "Put" if it detects that the HBase Client got SocketTimeoutException and disconnected.
[ https://issues.apache.org/jira/browse/HBASE-7874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maryann Xue updated HBASE-7874: --- Status: Patch Available (was: Open) > Allow RegionServer to abort "Put" if it detects that the HBase Client got > SocketTimeoutException and disconnected. > -- > > Key: HBASE-7874 > URL: https://issues.apache.org/jira/browse/HBASE-7874 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.94.5 >Reporter: Maryann Xue >Assignee: Maryann Xue >Priority: Minor > Attachments: HBASE-7874-0.94.patch > > > Usually, when regionserver cannot catch up with the put load given by the > client, what happens is region server starts to block update requests from > the client until required resource has been reclaimed (i.e. memstore has been > flushed). > But in more severe situations, the blocking time gets so long that the client > begins to have SocketTimeoutException and then decides to retry, while in > fact the updates are written into memstore later after they are unblocked. > Even though the client has something like a binary rollback for retry > intervals, this can still lead to a vicious circle, leaving the client to > have very low throughput. > Think we can enable an option to allow regionserver to check if the client > has disconnected (just like what we do in scan) after coming back from > "blocking", so that the regionserver has the same view as the client on > whether updates are successfully committed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7876) Got exception when manually triggers a split on an empty region
[ https://issues.apache.org/jira/browse/HBASE-7876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maryann Xue updated HBASE-7876: --- Summary: Got exception when manually triggers a split on an empty region (was: Got exception when splitting a region that contains no storefile) > Got exception when manually triggers a split on an empty region > --- > > Key: HBASE-7876 > URL: https://issues.apache.org/jira/browse/HBASE-7876 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.5 >Reporter: Maryann Xue >Assignee: Maryann Xue > > We should allow a region to split successfully even if it does not yet have > storefiles. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7876) Got exception when manually triggers a split on an empty region
[ https://issues.apache.org/jira/browse/HBASE-7876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maryann Xue updated HBASE-7876: --- Priority: Minor (was: Major) > Got exception when manually triggers a split on an empty region > --- > > Key: HBASE-7876 > URL: https://issues.apache.org/jira/browse/HBASE-7876 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.5 >Reporter: Maryann Xue >Assignee: Maryann Xue >Priority: Minor > > We should allow a region to split successfully even if it does not yet have > storefiles. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7876) Got exception when manually triggers a split on an empty region
[ https://issues.apache.org/jira/browse/HBASE-7876?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Maryann Xue updated HBASE-7876: --- Attachment: HBASE-7876-0.94.patch "return" if no storefile. > Got exception when manually triggers a split on an empty region > --- > > Key: HBASE-7876 > URL: https://issues.apache.org/jira/browse/HBASE-7876 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.5 >Reporter: Maryann Xue >Assignee: Maryann Xue >Priority: Minor > Attachments: HBASE-7876-0.94.patch > > > We should allow a region to split successfully even if it does not yet have > storefiles. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7876) Got exception when manually triggers a split on an empty region
[ https://issues.apache.org/jira/browse/HBASE-7876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580994#comment-13580994 ] ramkrishna.s.vasudevan commented on HBASE-7876: --- @Maryann HBASE-6853 introduced this behaviour. Check out the comment by Stack in https://issues.apache.org/jira/secure/EditComment!default.jspa?id=12608605&commentId=13460866. Infact we had two options either to succeed the split or throw exception and finally agreed to throw exception. > Got exception when manually triggers a split on an empty region > --- > > Key: HBASE-7876 > URL: https://issues.apache.org/jira/browse/HBASE-7876 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.5 >Reporter: Maryann Xue >Assignee: Maryann Xue >Priority: Minor > Attachments: HBASE-7876-0.94.patch > > > We should allow a region to split successfully even if it does not yet have > storefiles. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7876) Got exception when manually triggers a split on an empty region
[ https://issues.apache.org/jira/browse/HBASE-7876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580997#comment-13580997 ] Anoop Sam John commented on HBASE-7876: --- Specific use case you have [~maryannxue]? Mind sharing that? I am remembering the discussion happened with HBASE-6853 > Got exception when manually triggers a split on an empty region > --- > > Key: HBASE-7876 > URL: https://issues.apache.org/jira/browse/HBASE-7876 > Project: HBase > Issue Type: Bug > Components: regionserver >Affects Versions: 0.94.5 >Reporter: Maryann Xue >Assignee: Maryann Xue >Priority: Minor > Attachments: HBASE-7876-0.94.patch > > > We should allow a region to split successfully even if it does not yet have > storefiles. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7495) parallel seek in StoreScanner
[
https://issues.apache.org/jira/browse/HBASE-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13580998#comment-13580998
]
chunhui shen commented on HBASE-7495:
-
>From the above test results, I think this feature is useful for the case where
>ops is low and latency is high.
But will it become worse under some cases? e.g. thousands of ops with some
block cache hit ratio
Since we couldn't make it as a default confiuration, how about per-table or
per-family configuration? It may be better to enable parallel seeking
> parallel seek in StoreScanner
> -
>
> Key: HBASE-7495
> URL: https://issues.apache.org/jira/browse/HBASE-7495
> Project: HBase
> Issue Type: Bug
> Components: Scanners
>Affects Versions: 0.94.3, 0.96.0
>Reporter: Liang Xie
>Assignee: Liang Xie
> Attachments: 7495-v12.txt, HBASE-7495-0.94.txt, HBASE-7495.txt,
> HBASE-7495.txt, HBASE-7495.txt, HBASE-7495-v10.txt, HBASE-7495-v11.txt,
> HBASE-7495-v2.txt, HBASE-7495-v3.txt, HBASE-7495-v4.txt, HBASE-7495-v4.txt,
> HBASE-7495-v5.txt, HBASE-7495-v6.txt, HBASE-7495-v7.txt, HBASE-7495-v8.txt,
> HBASE-7495-v9.txt
>
>
> seems there's a potential improvable space before doing scanner.next:
> {code:title=StoreScanner.java|borderStyle=solid}
> if (explicitColumnQuery && lazySeekEnabledGlobally) {
> for (KeyValueScanner scanner : scanners) {
> scanner.requestSeek(matcher.getStartKey(), false, true);
> }
> } else {
> for (KeyValueScanner scanner : scanners) {
> scanner.seek(matcher.getStartKey());
> }
> }
> {code}
> we can do scanner.requestSeek or scanner.seek in parallel, instead of current
> serialization, to reduce latency for special case.
> Any ideas on it ? I'll have a try if the comments/suggestions are positive：）
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7495) parallel seek in StoreScanner
[
https://issues.apache.org/jira/browse/HBASE-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13581011#comment-13581011
]
Liang Xie commented on HBASE-7495:
--
[~zjusch], yeh, good idea, i can't agree more.
but to be honest,i'd like to do per-table or other granularity configue in
another jira, to make parallel-seeking feature more configurable. it's just
boring to rebase again(if conflicts with new patches) for me:)
is it suitable to merge Ted's last rebased v12 patch into trunk now ?
Thanks very much:)
> parallel seek in StoreScanner
> -
>
> Key: HBASE-7495
> URL: https://issues.apache.org/jira/browse/HBASE-7495
> Project: HBase
> Issue Type: Bug
> Components: Scanners
>Affects Versions: 0.94.3, 0.96.0
>Reporter: Liang Xie
>Assignee: Liang Xie
> Attachments: 7495-v12.txt, HBASE-7495-0.94.txt, HBASE-7495.txt,
> HBASE-7495.txt, HBASE-7495.txt, HBASE-7495-v10.txt, HBASE-7495-v11.txt,
> HBASE-7495-v2.txt, HBASE-7495-v3.txt, HBASE-7495-v4.txt, HBASE-7495-v4.txt,
> HBASE-7495-v5.txt, HBASE-7495-v6.txt, HBASE-7495-v7.txt, HBASE-7495-v8.txt,
> HBASE-7495-v9.txt
>
>
> seems there's a potential improvable space before doing scanner.next:
> {code:title=StoreScanner.java|borderStyle=solid}
> if (explicitColumnQuery && lazySeekEnabledGlobally) {
> for (KeyValueScanner scanner : scanners) {
> scanner.requestSeek(matcher.getStartKey(), false, true);
> }
> } else {
> for (KeyValueScanner scanner : scanners) {
> scanner.seek(matcher.getStartKey());
> }
> }
> {code}
> we can do scanner.requestSeek or scanner.seek in parallel, instead of current
> serialization, to reduce latency for special case.
> Any ideas on it ? I'll have a try if the comments/suggestions are positive：）
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7495) parallel seek in StoreScanner
[
https://issues.apache.org/jira/browse/HBASE-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13581013#comment-13581013
]
Liang Xie commented on HBASE-7495:
--
Yes, the parallel-seeking feature is disabled by default in current patch.
> parallel seek in StoreScanner
> -
>
> Key: HBASE-7495
> URL: https://issues.apache.org/jira/browse/HBASE-7495
> Project: HBase
> Issue Type: Bug
> Components: Scanners
>Affects Versions: 0.94.3, 0.96.0
>Reporter: Liang Xie
>Assignee: Liang Xie
> Attachments: 7495-v12.txt, HBASE-7495-0.94.txt, HBASE-7495.txt,
> HBASE-7495.txt, HBASE-7495.txt, HBASE-7495-v10.txt, HBASE-7495-v11.txt,
> HBASE-7495-v2.txt, HBASE-7495-v3.txt, HBASE-7495-v4.txt, HBASE-7495-v4.txt,
> HBASE-7495-v5.txt, HBASE-7495-v6.txt, HBASE-7495-v7.txt, HBASE-7495-v8.txt,
> HBASE-7495-v9.txt
>
>
> seems there's a potential improvable space before doing scanner.next:
> {code:title=StoreScanner.java|borderStyle=solid}
> if (explicitColumnQuery && lazySeekEnabledGlobally) {
> for (KeyValueScanner scanner : scanners) {
> scanner.requestSeek(matcher.getStartKey(), false, true);
> }
> } else {
> for (KeyValueScanner scanner : scanners) {
> scanner.seek(matcher.getStartKey());
> }
> }
> {code}
> we can do scanner.requestSeek or scanner.seek in parallel, instead of current
> serialization, to reduce latency for special case.
> Any ideas on it ? I'll have a try if the comments/suggestions are positive：）
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7495) parallel seek in StoreScanner
[
https://issues.apache.org/jira/browse/HBASE-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13581029#comment-13581029
]
Ted Yu commented on HBASE-7495:
---
Will integrate this tomorrow morning if there is no further review comment.
> parallel seek in StoreScanner
> -
>
> Key: HBASE-7495
> URL: https://issues.apache.org/jira/browse/HBASE-7495
> Project: HBase
> Issue Type: Bug
> Components: Scanners
>Affects Versions: 0.94.3, 0.96.0
>Reporter: Liang Xie
>Assignee: Liang Xie
> Attachments: 7495-v12.txt, HBASE-7495-0.94.txt, HBASE-7495.txt,
> HBASE-7495.txt, HBASE-7495.txt, HBASE-7495-v10.txt, HBASE-7495-v11.txt,
> HBASE-7495-v2.txt, HBASE-7495-v3.txt, HBASE-7495-v4.txt, HBASE-7495-v4.txt,
> HBASE-7495-v5.txt, HBASE-7495-v6.txt, HBASE-7495-v7.txt, HBASE-7495-v8.txt,
> HBASE-7495-v9.txt
>
>
> seems there's a potential improvable space before doing scanner.next:
> {code:title=StoreScanner.java|borderStyle=solid}
> if (explicitColumnQuery && lazySeekEnabledGlobally) {
> for (KeyValueScanner scanner : scanners) {
> scanner.requestSeek(matcher.getStartKey(), false, true);
> }
> } else {
> for (KeyValueScanner scanner : scanners) {
> scanner.seek(matcher.getStartKey());
> }
> }
> {code}
> we can do scanner.requestSeek or scanner.seek in parallel, instead of current
> serialization, to reduce latency for special case.
> Any ideas on it ? I'll have a try if the comments/suggestions are positive：）
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7845) optimize hfile index size like leveldb's ByteWiseComparatorImpl::FindShortestSeparator() style
[ https://issues.apache.org/jira/browse/HBASE-7845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Xie updated HBASE-7845: - Assignee: Liang Xie > optimize hfile index size like leveldb's > ByteWiseComparatorImpl::FindShortestSeparator() style > -- > > Key: HBASE-7845 > URL: https://issues.apache.org/jira/browse/HBASE-7845 > Project: HBase > Issue Type: Improvement > Components: HFile >Affects Versions: 0.96.0 >Reporter: Liang Xie >Assignee: Liang Xie > > Leveldb uses ByteWiseComparatorImpl::FindShortestSeparator() & > FindShortSuccessor() to reduce index key size, it would be helpful under > special conditions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-7877) KeyPrefixRegionSplitPolicy and DelimitedKeyPrefixRegionSplitPolicy splits are not always optimal
Robert Dyer created HBASE-7877: -- Summary: KeyPrefixRegionSplitPolicy and DelimitedKeyPrefixRegionSplitPolicy splits are not always optimal Key: HBASE-7877 URL: https://issues.apache.org/jira/browse/HBASE-7877 Project: HBase Issue Type: Improvement Components: regionserver Affects Versions: 0.94.5, 0.96.0 Reporter: Robert Dyer Priority: Minor With KeyPrefixRegionSplitPolicy (and now DelimitedKeyPrefixRegionSplitPolicy), if a split would break a group of keys it is modified to become the first key in the group's range. This is not always optimal. If the distribution of keys are such that the group containing the split has half the keys in the region, then no split will occur. The best solution would be to compute both the current key group's first key (what the current implementation does) as well as the next key group's first key and then choosing which of the two is closest to the original split point. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7845) optimize hfile index size like leveldb's ByteWiseComparatorImpl::FindShortestSeparator() style
[
https://issues.apache.org/jira/browse/HBASE-7845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13581048#comment-13581048
]
Liang Xie commented on HBASE-7845:
--
After a further investigation, i know:
1)For leveldb, the index key was a "faked" key which larger or equals than
current data block's last key, and smaller than the next data block's first key
2)For HFileV2, we use the first key in current data block, there's a discussion
before:HBASE-4443
IMHO, a "faked" last key has more benefit than mentioned in HBASE-4443. e.g,
there's a good example from leveldb's comments:
{quote}
// We do not emit the index entry for a block until we have seen the
// first key for the next data block. This allows us to use shorter
// keys in the index block. For example, consider a block boundary
// between the keys "the quick brown fox" and "the who". We can use
// "the r" as the key for the index block entry since it is >= all
// entries in the first block and < all entries in subsequent
// blocks.
{quote}
I'd like to have a try to make a patch these days on it. Any comments and
advices will be highly appreciated:)
> optimize hfile index size like leveldb's
> ByteWiseComparatorImpl::FindShortestSeparator() style
> --
>
> Key: HBASE-7845
> URL: https://issues.apache.org/jira/browse/HBASE-7845
> Project: HBase
> Issue Type: Improvement
> Components: HFile
>Affects Versions: 0.96.0
>Reporter: Liang Xie
>Assignee: Liang Xie
>
> Leveldb uses ByteWiseComparatorImpl::FindShortestSeparator() &
> FindShortSuccessor() to reduce index key size, it would be helpful under
> special conditions.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7845) optimize hfile index size like leveldb's ByteWiseComparatorImpl::FindShortestSeparator() style
[ https://issues.apache.org/jira/browse/HBASE-7845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13581051#comment-13581051 ] Matt Corgan commented on HBASE-7845: I don't really have time to work on it, but using prefix-tree format would be nearly ideal for index blocks. > optimize hfile index size like leveldb's > ByteWiseComparatorImpl::FindShortestSeparator() style > -- > > Key: HBASE-7845 > URL: https://issues.apache.org/jira/browse/HBASE-7845 > Project: HBase > Issue Type: Improvement > Components: HFile >Affects Versions: 0.96.0 >Reporter: Liang Xie >Assignee: Liang Xie > > Leveldb uses ByteWiseComparatorImpl::FindShortestSeparator() & > FindShortSuccessor() to reduce index key size, it would be helpful under > special conditions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7874) Allow RegionServer to abort "Put" if it detects that the HBase Client got SocketTimeoutException and disconnected.
[
https://issues.apache.org/jira/browse/HBASE-7874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13581053#comment-13581053
]
Hadoop QA commented on HBASE-7874:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12569893/HBASE-7874-0.94.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4461//console
This message is automatically generated.
> Allow RegionServer to abort "Put" if it detects that the HBase Client got
> SocketTimeoutException and disconnected.
> --
>
> Key: HBASE-7874
> URL: https://issues.apache.org/jira/browse/HBASE-7874
> Project: HBase
> Issue Type: Improvement
> Components: regionserver
>Affects Versions: 0.94.5
>Reporter: Maryann Xue
>Assignee: Maryann Xue
>Priority: Minor
> Attachments: HBASE-7874-0.94.patch
>
>
> Usually, when regionserver cannot catch up with the put load given by the
> client, what happens is region server starts to block update requests from
> the client until required resource has been reclaimed (i.e. memstore has been
> flushed).
> But in more severe situations, the blocking time gets so long that the client
> begins to have SocketTimeoutException and then decides to retry, while in
> fact the updates are written into memstore later after they are unblocked.
> Even though the client has something like a binary rollback for retry
> intervals, this can still lead to a vicious circle, leaving the client to
> have very low throughput.
> Think we can enable an option to allow regionserver to check if the client
> has disconnected (just like what we do in scan) after coming back from
> "blocking", so that the regionserver has the same view as the client on
> whether updates are successfully committed.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7845) optimize hfile index size like leveldb's ByteWiseComparatorImpl::FindShortestSeparator() style
[ https://issues.apache.org/jira/browse/HBASE-7845?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13581060#comment-13581060 ] Liang Xie commented on HBASE-7845: -- Hi [~mcorgan], thanks for your comments, i haven't looked into HBASE-4676 yet, will try to do a simple benchmark if possible after maked an raw patch. IIRC, these two manners are not conflict, at least in leveldb, both prefix-compaction and above "faked" index key are exist:) > optimize hfile index size like leveldb's > ByteWiseComparatorImpl::FindShortestSeparator() style > -- > > Key: HBASE-7845 > URL: https://issues.apache.org/jira/browse/HBASE-7845 > Project: HBase > Issue Type: Improvement > Components: HFile >Affects Versions: 0.96.0 >Reporter: Liang Xie >Assignee: Liang Xie > > Leveldb uses ByteWiseComparatorImpl::FindShortestSeparator() & > FindShortSuccessor() to reduce index key size, it would be helpful under > special conditions. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7813) Bug in BulkDeleteEndpoint kills entire rows on COLUMN/VERSION Deletes
[ https://issues.apache.org/jira/browse/HBASE-7813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13581061#comment-13581061 ] Anoop Sam John commented on HBASE-7813: --- [~varunsharma]Any update on this? You found the actual issue? > Bug in BulkDeleteEndpoint kills entire rows on COLUMN/VERSION Deletes > - > > Key: HBASE-7813 > URL: https://issues.apache.org/jira/browse/HBASE-7813 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Affects Versions: 0.96.0, 0.94.4 >Reporter: Varun Sharma >Assignee: Varun Sharma > Fix For: 0.96.0, 0.94.6 > > Attachments: 7813-0.94.txt > > > Try a BulkDeleteEndpoint using DeleteType.COLUMN or DeleteType.VERSION and > give it a set of Columns which are not in the row. > A Delete is constructed using just the row alone. No family/column/version is > added and it is applied, thus deleting the entire row. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7813) Bug in BulkDeleteEndpoint kills entire rows on COLUMN/VERSION Deletes
[ https://issues.apache.org/jira/browse/HBASE-7813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13581063#comment-13581063 ] Varun Sharma commented on HBASE-7813: - Yeah, I don't think there was an issue here - we found a bug in the app logic instead, closing for now - may reopen if i see seomthing suspicious again > Bug in BulkDeleteEndpoint kills entire rows on COLUMN/VERSION Deletes > - > > Key: HBASE-7813 > URL: https://issues.apache.org/jira/browse/HBASE-7813 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Affects Versions: 0.96.0, 0.94.4 >Reporter: Varun Sharma >Assignee: Varun Sharma > Fix For: 0.96.0, 0.94.6 > > Attachments: 7813-0.94.txt > > > Try a BulkDeleteEndpoint using DeleteType.COLUMN or DeleteType.VERSION and > give it a set of Columns which are not in the row. > A Delete is constructed using just the row alone. No family/column/version is > added and it is applied, thus deleting the entire row. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-7813) Bug in BulkDeleteEndpoint kills entire rows on COLUMN/VERSION Deletes
[ https://issues.apache.org/jira/browse/HBASE-7813?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Varun Sharma updated HBASE-7813: Resolution: Invalid Status: Resolved (was: Patch Available) > Bug in BulkDeleteEndpoint kills entire rows on COLUMN/VERSION Deletes > - > > Key: HBASE-7813 > URL: https://issues.apache.org/jira/browse/HBASE-7813 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Affects Versions: 0.96.0, 0.94.4 >Reporter: Varun Sharma >Assignee: Varun Sharma > Fix For: 0.96.0, 0.94.6 > > Attachments: 7813-0.94.txt > > > Try a BulkDeleteEndpoint using DeleteType.COLUMN or DeleteType.VERSION and > give it a set of Columns which are not in the row. > A Delete is constructed using just the row alone. No family/column/version is > added and it is applied, thus deleting the entire row. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7495) parallel seek in StoreScanner
[
https://issues.apache.org/jira/browse/HBASE-7495?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13581094#comment-13581094
]
Lars Hofhansl commented on HBASE-7495:
--
+1 for trunk.
I'd like to have some more discussion before/if we consider this for 0.94.
Any thoughts about that?
> parallel seek in StoreScanner
> -
>
> Key: HBASE-7495
> URL: https://issues.apache.org/jira/browse/HBASE-7495
> Project: HBase
> Issue Type: Bug
> Components: Scanners
>Affects Versions: 0.94.3, 0.96.0
>Reporter: Liang Xie
>Assignee: Liang Xie
> Attachments: 7495-v12.txt, HBASE-7495-0.94.txt, HBASE-7495.txt,
> HBASE-7495.txt, HBASE-7495.txt, HBASE-7495-v10.txt, HBASE-7495-v11.txt,
> HBASE-7495-v2.txt, HBASE-7495-v3.txt, HBASE-7495-v4.txt, HBASE-7495-v4.txt,
> HBASE-7495-v5.txt, HBASE-7495-v6.txt, HBASE-7495-v7.txt, HBASE-7495-v8.txt,
> HBASE-7495-v9.txt
>
>
> seems there's a potential improvable space before doing scanner.next:
> {code:title=StoreScanner.java|borderStyle=solid}
> if (explicitColumnQuery && lazySeekEnabledGlobally) {
> for (KeyValueScanner scanner : scanners) {
> scanner.requestSeek(matcher.getStartKey(), false, true);
> }
> } else {
> for (KeyValueScanner scanner : scanners) {
> scanner.seek(matcher.getStartKey());
> }
> }
> {code}
> we can do scanner.requestSeek or scanner.seek in parallel, instead of current
> serialization, to reduce latency for special case.
> Any ideas on it ? I'll have a try if the comments/suggestions are positive：）
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
