[jira] [Updated] (HBASE-10885) Support visibility expressions on Deletes
[ https://issues.apache.org/jira/browse/HBASE-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10885: --- Attachment: HBASE-10885_new_tag_type_1.patch Support visibility expressions on Deletes - Key: HBASE-10885 URL: https://issues.apache.org/jira/browse/HBASE-10885 Project: HBase Issue Type: Improvement Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.99.0, 0.98.3 Attachments: HBASE-10885_1.patch, HBASE-10885_2.patch, HBASE-10885_new_tag_type_1.patch Accumulo can specify visibility expressions for delete markers. During compaction the cells covered by the tombstone are determined in part by matching the visibility expression. This is useful for the use case of data set coalescing, where entries from multiple data sets carrying different labels are combined into one common large table. Later, a subset of entries can be conveniently removed using visibility expressions. Currently doing the same in HBase would only be possible with a custom coprocessor. Otherwise, a Delete will affect all cells covered by the tombstone regardless of any visibility expression scoping. This is correct behavior in that no data spill is possible, but certainly could be surprising, and is only meant to be transitional. We decided not to support visibility expressions on Deletes to control the complexity of the initial implementation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10885) Support visibility expressions on Deletes
[ https://issues.apache.org/jira/browse/HBASE-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10885: --- Status: Open (was: Patch Available) Support visibility expressions on Deletes - Key: HBASE-10885 URL: https://issues.apache.org/jira/browse/HBASE-10885 Project: HBase Issue Type: Improvement Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.99.0, 0.98.3 Attachments: HBASE-10885_1.patch, HBASE-10885_2.patch, HBASE-10885_new_tag_type_1.patch Accumulo can specify visibility expressions for delete markers. During compaction the cells covered by the tombstone are determined in part by matching the visibility expression. This is useful for the use case of data set coalescing, where entries from multiple data sets carrying different labels are combined into one common large table. Later, a subset of entries can be conveniently removed using visibility expressions. Currently doing the same in HBase would only be possible with a custom coprocessor. Otherwise, a Delete will affect all cells covered by the tombstone regardless of any visibility expression scoping. This is correct behavior in that no data spill is possible, but certainly could be surprising, and is only meant to be transitional. We decided not to support visibility expressions on Deletes to control the complexity of the initial implementation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10885) Support visibility expressions on Deletes
[ https://issues.apache.org/jira/browse/HBASE-10885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-10885: --- Status: Patch Available (was: Open) Support visibility expressions on Deletes - Key: HBASE-10885 URL: https://issues.apache.org/jira/browse/HBASE-10885 Project: HBase Issue Type: Improvement Affects Versions: 0.98.1 Reporter: Andrew Purtell Assignee: ramkrishna.s.vasudevan Priority: Blocker Fix For: 0.99.0, 0.98.3 Attachments: HBASE-10885_1.patch, HBASE-10885_2.patch, HBASE-10885_new_tag_type_1.patch Accumulo can specify visibility expressions for delete markers. During compaction the cells covered by the tombstone are determined in part by matching the visibility expression. This is useful for the use case of data set coalescing, where entries from multiple data sets carrying different labels are combined into one common large table. Later, a subset of entries can be conveniently removed using visibility expressions. Currently doing the same in HBase would only be possible with a custom coprocessor. Otherwise, a Delete will affect all cells covered by the tombstone regardless of any visibility expression scoping. This is correct behavior in that no data spill is possible, but certainly could be surprising, and is only meant to be transitional. We decided not to support visibility expressions on Deletes to control the complexity of the initial implementation. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11119) Update ExportSnapShot to optionally not use a tmp file on external file system
[
https://issues.apache.org/jira/browse/HBASE-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13992524#comment-13992524
]
Hadoop QA commented on HBASE-9:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12643889/HBASE-9.patch
against trunk revision .
ATTACHMENT ID: 12643889
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces the following lines
longer than 100:
+System.err.println(Unable to remove existing snapshot tmp
directory: + snapshotTmpDir);
+ System.err.println(A snapshot with the same name '+ snapshotName
+' may be in-progress);
+ System.err.println(Please check + snapshotTmpDir + . If the
snapshot has completed, );
+ System.err.println(consider removing + snapshotTmpDir + by using
the -overwrite option);
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestMultiParallel
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9481//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9481//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9481//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9481//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9481//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9481//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9481//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9481//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9481//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9481//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9481//console
This message is automatically generated.
Update ExportSnapShot to optionally not use a tmp file on external file system
--
Key: HBASE-9
URL: https://issues.apache.org/jira/browse/HBASE-9
Project: HBase
Issue Type: New Feature
Reporter: Ted Malaska
Assignee: Ted Malaska
Priority: Minor
Attachments: HBASE-9.patch
There are FileSystem like S3 where renaming is extremely expensive. This
patch will add a parameter that says something like
use.tmp.folder
It will be defaulted to true. So default behavior is the same. If false is
set them the files will land in the final destination with no need for a
rename.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-8763) [BRAINSTORM] Combine MVCC and SeqId
[
https://issues.apache.org/jira/browse/HBASE-8763?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13992516#comment-13992516
]
stack commented on HBASE-8763:
--
This changes with hbase-11109:
{code}
+ flushSeqId = this.sequenceId.incrementAndGet();
+} else {
+ // use the provided sequence Id as WAL is not being used for this
flush.
+ flushSeqId = myseqid;
{code}
... but that is fine. Let 11109 worry about it.
I'm not sure I'm clear on what could be rolled back out of memstore around
flush. Or, can there be more doc on how mvcc and sequence id are interacting
here? For those who come after us?
This should be called sequenceId: +MutableLong seqNumber = new
MutableLong();
A left shift would be better? +return beginMemstoreInsert(curSeqNum +
10);
Did a quick skim. Early- vs late-binding would change this patch?
Is the best write up on how this is going to work going forward what is above
in this issue? Thanks [~jeffreyz]
[BRAINSTORM] Combine MVCC and SeqId
---
Key: HBASE-8763
URL: https://issues.apache.org/jira/browse/HBASE-8763
Project: HBase
Issue Type: Improvement
Components: regionserver
Reporter: Enis Soztutar
Assignee: Jeffrey Zhong
Priority: Critical
Attachments: hbase-8736-poc.patch, hbase-8763-poc-v1.patch,
hbase-8763-v1.patch, hbase-8763_wip1.patch
HBASE-8701 and a lot of recent issues include good discussions about mvcc +
seqId semantics. It seems that having mvcc and the seqId complicates the
comparator semantics a lot in regards to flush + WAL replay + compactions +
delete markers and out of order puts.
Thinking more about it I don't think we need a MVCC write number which is
different than the seqId. We can keep the MVCC semantics, read point and
smallest read points intact, but combine mvcc write number and seqId. This
will allow cleaner semantics + implementation + smaller data files.
We can do some brainstorming for 0.98. We still have to verify that this
would be semantically correct, it should be so by my current understanding.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-11143) ageOfLastShippedOp confusing
[ https://issues.apache.org/jira/browse/HBASE-11143?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993434#comment-13993434 ] Lars Hofhansl commented on HBASE-11143: --- Related to HBASE-9286, which added the code to always increase the metric even nothing is happening. It makes sense to keep that logic, though, and just reset the counter when there is nothing to replicate. ageOfLastShippedOp confusing Key: HBASE-11143 URL: https://issues.apache.org/jira/browse/HBASE-11143 Project: HBase Issue Type: Bug Components: Replication Reporter: Lars Hofhansl Fix For: 0.94.20 Attachments: 11143-0.94.txt We are trying to report on replication lag and find that there is no good single metric to do that. ageOfLastShippedOp is close, but unfortunately it is increased even when there is nothing to ship on a particular RegionServer. I would like discuss a few options here: Add a new metric: replicationQueueTime (or something) with the above meaning. I.e. if we have something to ship we set the age of that last shipped edit, if we fail we increment that last time (just like we do now). But if there is nothing to replicate we set it to current time (and hence that metric is reported to close to 0). Alternatively we could change the meaning of ageOfLastShippedOp to mean to do that. That might lead to surprises, but the current behavior is clearly weird when there is nothing to replicate. Comments? [~jdcryans], [~stack]. If approach sounds good, I'll make a patch for all branches. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11140) LocalHBaseCluster should create ConsensusProvider per each server
[
https://issues.apache.org/jira/browse/HBASE-11140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993428#comment-13993428
]
Hadoop QA commented on HBASE-11140:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644035/HBASE-11140.patch
against trunk revision .
ATTACHMENT ID: 12644035
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9488//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9488//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9488//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9488//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9488//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9488//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9488//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9488//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9488//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9488//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9488//console
This message is automatically generated.
LocalHBaseCluster should create ConsensusProvider per each server
-
Key: HBASE-11140
URL: https://issues.apache.org/jira/browse/HBASE-11140
Project: HBase
Issue Type: Sub-task
Components: Consensus
Affects Versions: 0.99.0
Reporter: Mikhail Antonov
Assignee: Mikhail Antonov
Fix For: 0.99.0
Attachments: HBASE-11140.patch
Right now there's a bug there when single ConsensusProvider instance is
shared across all threads running region servers and masters within
processes, which breaks certain tests in patches which used to pass
successfully before.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11143) ageOfLastShippedOp confusing
[ https://issues.apache.org/jira/browse/HBASE-11143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-11143: -- Attachment: 11143-0.94.txt Simple 0.94 patch. Sets the metric to current time when there's nothing to replicate (and it's not due to an error). When we're hanging somewhere because the slave cluster is down or replication takes a very long time, the metric is still incremented, though. I think that's OK, there might be a delay in that case, we just do not know. It's also nice as we can large values of this metric as an indicator that something is wrong. ageOfLastShippedOp confusing Key: HBASE-11143 URL: https://issues.apache.org/jira/browse/HBASE-11143 Project: HBase Issue Type: Bug Components: Replication Reporter: Lars Hofhansl Fix For: 0.94.20 Attachments: 11143-0.94.txt We are trying to report on replication lag and find that there is no good single metric to do that. ageOfLastShippedOp is close, but unfortunately it is increased even when there is nothing to ship on a particular RegionServer. I would like discuss a few options here: Add a new metric: replicationQueueTime (or something) with the above meaning. I.e. if we have something to ship we set the age of that last shipped edit, if we fail we increment that last time (just like we do now). But if there is nothing to replicate we set it to current time (and hence that metric is reported to close to 0). Alternatively we could change the meaning of ageOfLastShippedOp to mean to do that. That might lead to surprises, but the current behavior is clearly weird when there is nothing to replicate. Comments? [~jdcryans], [~stack]. If approach sounds good, I'll make a patch for all branches. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11140) LocalHBaseCluster should create ConsensusProvider per each server
[ https://issues.apache.org/jira/browse/HBASE-11140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993447#comment-13993447 ] Mikhail Antonov commented on HBASE-11140: - Since this is infrastracture patch, no tests included. LocalHBaseCluster should create ConsensusProvider per each server - Key: HBASE-11140 URL: https://issues.apache.org/jira/browse/HBASE-11140 Project: HBase Issue Type: Sub-task Components: Consensus Affects Versions: 0.99.0 Reporter: Mikhail Antonov Assignee: Mikhail Antonov Fix For: 0.99.0 Attachments: HBASE-11140.patch Right now there's a bug there when single ConsensusProvider instance is shared across all threads running region servers and masters within processes, which breaks certain tests in patches which used to pass successfully before. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11144) Filter to support scan multiple row key ranges
[ https://issues.apache.org/jira/browse/HBASE-11144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Jiajia updated HBASE-11144: -- Status: Patch Available (was: Reopened) Filter to support scan multiple row key ranges -- Key: HBASE-11144 URL: https://issues.apache.org/jira/browse/HBASE-11144 Project: HBase Issue Type: Improvement Components: Filters Reporter: Li Jiajia Attachments: MultiRowRangeFilter.patch Provide a filter feature to support scan multiple row key ranges. It can construct the row key ranges from the passed list which can be accessed by each region server. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10985) Decouple Split Transaction from Zookeeper
[
https://issues.apache.org/jira/browse/HBASE-10985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993453#comment-13993453
]
Hadoop QA commented on HBASE-10985:
---
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12644038/HBASE_10985-v4.patch
against trunk revision .
ATTACHMENT ID: 12644038
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 3 zombie test(s):
at
org.apache.hadoop.hbase.regionserver.wal.TestLogRolling.testLogRollOnDatanodeDeath(TestLogRolling.java:371)
at
org.apache.hadoop.hbase.mapreduce.TestImportExport.testImport94Table(TestImportExport.java:230)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/9489//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9489//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9489//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9489//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9489//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9489//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9489//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9489//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9489//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-thrift.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/9489//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/9489//console
This message is automatically generated.
Decouple Split Transaction from Zookeeper
-
Key: HBASE-10985
URL: https://issues.apache.org/jira/browse/HBASE-10985
Project: HBase
Issue Type: Sub-task
Components: Consensus, Zookeeper
Reporter: Sergey Soldatov
Attachments: HBASE-10985.patch, HBASE-10985.patch, HBASE-10985.patch,
HBASE_10985-v2.patch, HBASE_10985-v3.patch, HBASE_10985-v4.patch
As part of HBASE-10296 SplitTransaction should be decoupled from Zookeeper.
This is an initial patch for review. At the moment the consensus provider
placed directly to SplitTransaction to minimize affected code. In the ideal
world it should be done in HServer.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Comment Edited] (HBASE-11144) Filter to support scan multiple row key ranges
[ https://issues.apache.org/jira/browse/HBASE-11144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13994329#comment-13994329 ] Ted Yu edited comment on HBASE-11144 at 5/10/14 11:41 PM: -- Can you generate patch for trunk ? TestMultiRowRangeFilter doesn't have license header. Putting patch on reviewboard would help reviewers. was (Author: yuzhih...@gmail.com): Can you generate patch for trunk ? Please also add unit test for MultiRowRangeFilter. Putting patch on reviewboard would help reviewers. Filter to support scan multiple row key ranges -- Key: HBASE-11144 URL: https://issues.apache.org/jira/browse/HBASE-11144 Project: HBase Issue Type: Improvement Components: Filters Reporter: Li Jiajia Attachments: MultiRowRangeFilter.patch Provide a filter feature to support scan multiple row key ranges. It can construct the row key ranges from the passed list which can be accessed by each region server. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11108) Split ZKTable into interface and implementation
[ https://issues.apache.org/jira/browse/HBASE-11108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Antonov updated HBASE-11108: Attachment: HBASE-11108.patch fixed broken tests and long lines, excluded code pertaining to HBASE-11092, still keeping fix in LocalHBaseCluster from HBASE-11140. Split ZKTable into interface and implementation --- Key: HBASE-11108 URL: https://issues.apache.org/jira/browse/HBASE-11108 Project: HBase Issue Type: Sub-task Components: Consensus, Zookeeper Affects Versions: 0.99.0 Reporter: Konstantin Boudnik Assignee: Mikhail Antonov Attachments: HBASE-11108.patch, HBASE-11108.patch In HBASE-11071 we are trying to split admin handlers away from ZK. However, a ZKTable instance is being used in multiple places, hence it would be beneficial to hide its implementation behind a well defined interface. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11119) Update ExportSnapShot to optionally not use a tmp file on external file system
[ https://issues.apache.org/jira/browse/HBASE-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993379#comment-13993379 ] Hudson commented on HBASE-9: SUCCESS: Integrated in HBase-0.94 #1366 (See [https://builds.apache.org/job/HBase-0.94/1366/]) HBASE-9 Update ExportSnapShot to optionally not use a tmp file on external file system (Ted Malaska) (mbertozzi: rev 1593340) * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java Update ExportSnapShot to optionally not use a tmp file on external file system -- Key: HBASE-9 URL: https://issues.apache.org/jira/browse/HBASE-9 Project: HBase Issue Type: Improvement Components: snapshots Reporter: Ted Malaska Assignee: Ted Malaska Priority: Minor Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3 Attachments: HBASE-9.patch There are FileSystem like S3 where renaming is extremely expensive. This patch will add a parameter that says something like use.tmp.folder It will be defaulted to true. So default behavior is the same. If false is set them the files will land in the final destination with no need for a rename. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11144) Filter to support scan multiple row key ranges
[ https://issues.apache.org/jira/browse/HBASE-11144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Li Jiajia updated HBASE-11144: -- Description: Provide a filter feature to support scan multiple row key ranges. It can construct the row key ranges from the passed list which can be accessed by each region server. was: Filter to support scan multiple row key ranges. It can construct the row key ranges from the passed list which can be accessed by each region server. Filter to support scan multiple row key ranges -- Key: HBASE-11144 URL: https://issues.apache.org/jira/browse/HBASE-11144 Project: HBase Issue Type: Improvement Components: Filters Reporter: Li Jiajia Attachments: MultiRowRangeFilter.patch Provide a filter feature to support scan multiple row key ranges. It can construct the row key ranges from the passed list which can be accessed by each region server. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-11092) Server interface should have method getConsensusProvider()
[ https://issues.apache.org/jira/browse/HBASE-11092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11092: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk. Thank you for the patch [~mantonov]. Agree failure unrelated. Server interface should have method getConsensusProvider() -- Key: HBASE-11092 URL: https://issues.apache.org/jira/browse/HBASE-11092 Project: HBase Issue Type: Sub-task Components: Consensus Affects Versions: 0.99.0 Reporter: Mikhail Antonov Assignee: Mikhail Antonov Fix For: 0.99.0 Attachments: HBASE-11092.patch, HBASE-11092.patch, HBASE_11092.patch As discussed in comments to HBASE-10915, we need to have a proper way to retrieve instance of consensus provider, and Server interface seems the right one. Since Server interface lives in hbase-client maven module, the following approach is implemented in this patch: - hbase-client module has very basic (almost marker) interface ConsensusProvider to return instance of consensus provider from the Server - hbase-server module has BaseConsensusProvider which defines the consensus interfaces - Implementations shall subclass BaseConsensusProvider - whoever wants to get ConsensusProvider from raw Server interface on hbase-server side, has to typecast: (BaseConsensusProvider) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10900) FULL table backup and restore
[
https://issues.apache.org/jira/browse/HBASE-10900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Demai Ni updated HBASE-10900:
-
Description:
h2. Feature Description
This is a subtask of
[HBase-7912|https://issues.apache.org/jira/browse/HBASE-7912] to support FULL
backup/restore, and will complete the following function:
{code:title=Backup Restore example|borderStyle=solid}
/* backup from sourcecluster to targetcluster
*/
/* if no table name specified, all tables from source cluster will be backuped
*/
[sourcecluster]$ hbase backup create full
hdfs://hostname.targetcluster.org:9000/userid/backupdir t1_dn,t2_dn,t3_dn
/* restore on targetcluser, this is a local restore
*/
/* backup_1396650096738 - backup image name
*/
/* t1_dn,etc are the original table names. All tables will be restored if not
specified */
/* t1_dn_restore, etc. are the restored table. if not specified, orginal table
name will be used*/
[targetcluster]$ hbase restore /userid/backupdir backup_1396650096738
t1_dn,t2_dn,t3_dn t1_dn_restore,t2_dn_restore,t3_dn_restore
/* restore from targetcluster back to source cluster, this is a remote restore
[sourcecluster]$ hbase restore
hdfs://hostname.targetcluster.org:9000/userid/backupdir backup_1396650096738
t1_dn,t2_dn,t3_dn t1_dn_restore,t2_dn_restore,t3_dn_restore
{code}
h2. Detail layout and frame work for the next jiras
The patch is a wrapper of the existing snapshot and exportSnapshot, and will
use as the base framework for the over-all solution of
[HBase-7912|https://issues.apache.org/jira/browse/HBASE-7912] as described
below:
* *bin/hbase* : end-user command line interface to invoke BackupClient
and RestoreClient
* *BackupClient.java* : 'main' entry for backup operations. This patch will
only support 'full' backup. In future jiras, will support:
** *create* incremental backup
** *cancel* an ongoing backup
** *delete* an exisitng backup image
** *describe* the detailed informaiton of backup image
** show *history* of all successful backups
** show the *status* of the latest backup request
** *convert* incremental backup WAL files into HFiles. either on-the-fly
during create or after create
** *merge* backup image
** *stop* backup a table of existing backup image
** *show* tables of a backup image
* *BackupCommands.java* : a place to keep all the command usages and options
* *BackupManager.java* : handle backup requests on server-side, create BACKUP
ZOOKEEPER nodes to keep track backup. The timestamps kept in zookeeper will be
used for future incremental backup (not included in this jira). Create
BackupContext and DispatchRequest.
* *BackupHandler.java* : in this patch, it is a wrapper of snapshot and
exportsnapshot. In future jiras,
** *timestamps* info will be recorded in ZK
** carry on *incremental* backup.
** update backup *progress*
** set flags of *status*
** build up *backupManifest* file(in this jira only limited info for fullback.
later on, timestamps and dependency of multipl backup images are also recorded
here)
** clean up after *failed* backup
** clean up after *cancelled* backup
** allow on-the-fly *convert* during incremental backup
* *BackupContext.java* : encapsulate backup information like backup ID, table
names, directory info, phase, TimeStamps of backup progress, size of data,
ancestor info, etc.
* *BackupCopier.java* : the copying operation. Later on, to support progress
report and mapper estimation; and extends DisCp for progress updating to ZK
during backup.
* *BackupExcpetion.java*: to handle exception from backup/restore
* *BackupManifest.java* : encapsulate all the backup image information. The
manifest info will be bundled as manifest file together with data. So that each
backup image will contain all the info needed for restore.
* *BackupStatus.java* : encapsulate backup status at table level during
backup progress
* *BackupUtil.java* : utility methods during backup process
* *RestoreClient.java* : 'main' entry for restore operations. This patch will
only support 'full' backup.
* *RestoreUtil.java*: utility methods during restore process
* *ExportSnapshot.java* : remove 'final' so that another class
SnapshotCopy.java can extends from it
* *SnapshotCopy.java* : only a wrapper at this moment. But will be extended
to keep track progress(maybe should implemented in ExportSnapshot directly?)
* *BackupRestoreConstants.java* : add the constants used by backup/restore
code.
* *HBackupFilesystem.java* : the filesystem related api used by
BackupClient and RestoreClient.
h2. Global log roll
currently a customized one under *org.apache.hadoop.hbase.backup.master* and
*org.apache.hadoop.hbase.backup.regionserver*
A seperated jira will opened to provide a general 'global log roll', and
[jira] [Commented] (HBASE-10504) Define Replication Interface
[ https://issues.apache.org/jira/browse/HBASE-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13992995#comment-13992995 ] Enis Soztutar commented on HBASE-10504: --- I have a very basic prototype patch that we can consider as a direction for this. The idea is to break up ReplicationSource into two parts, the part that tails the WAL and listens to the replication queue, and the part that talks to the ReplicationSink in the peer cluster to ship the edits. The new interface ReplicationConsumer (we can change the names) is the plugin point. There is still one ReplicationSource per peer in every RS. The ReplicationSource tails the logs for the peer, and holds a ReplicationConsumer pointer. It calls ReplicationConsumer.shipEdits() method for actual shipping logic. A default HBaseClusterReplicator implements ReplicationConsumer and talks to the ReplicationSink on the other cluster etc. For each peer, there is now a class implementing ReplicationConsumer, and possibly some configuration. This allows one to implement a ReplicationConsumer for SOLR (or any other data store) and directly plug it in to the RS without having to do a proxy layer and mocking zookeeper / RS RPC etc. Let me know what you think. Early feedback will help shape up the patch. Define Replication Interface Key: HBASE-10504 URL: https://issues.apache.org/jira/browse/HBASE-10504 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Priority: Blocker Fix For: 0.99.0 Attachments: hbase-10504_wip1.patch HBase has replication. Fellas have been hijacking the replication apis to do all kinds of perverse stuff like indexing hbase content (hbase-indexer https://github.com/NGDATA/hbase-indexer) and our [~toffer] just showed up w/ overrides that replicate via an alternate channel (over a secure thrift channel between dcs over on HBASE-9360). This issue is about surfacing these APIs as public with guarantees to downstreamers similar to those we have on our public client-facing APIs (and so we don't break them for downstreamers). Any input [~phunt] or [~gabriel.reid] or [~toffer]? Thanks. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11119) Update ExportSnapShot to optionally not use a tmp file on external file system
[ https://issues.apache.org/jira/browse/HBASE-9?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13993450#comment-13993450 ] Hudson commented on HBASE-9: FAILURE: Integrated in hbase-0.96 #397 (See [https://builds.apache.org/job/hbase-0.96/397/]) HBASE-9 Update ExportSnapShot to optionally not use a tmp file on external file system (Ted Malaska) (mbertozzi: rev 1593339) * /hbase/branches/0.96/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/ExportSnapshot.java * /hbase/branches/0.96/hbase-server/src/test/java/org/apache/hadoop/hbase/snapshot/TestExportSnapshot.java Update ExportSnapShot to optionally not use a tmp file on external file system -- Key: HBASE-9 URL: https://issues.apache.org/jira/browse/HBASE-9 Project: HBase Issue Type: Improvement Components: snapshots Reporter: Ted Malaska Assignee: Ted Malaska Priority: Minor Fix For: 0.99.0, 0.96.3, 0.94.20, 0.98.3 Attachments: HBASE-9.patch There are FileSystem like S3 where renaming is extremely expensive. This patch will add a parameter that says something like use.tmp.folder It will be defaulted to true. So default behavior is the same. If false is set them the files will land in the final destination with no need for a rename. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-6990) Pretty print TTL
[
https://issues.apache.org/jira/browse/HBASE-6990?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Esteban Gutierrez updated HBASE-6990:
-
Attachment: HBASE-6990.v0.patch
Initial version. TTL now is printed in the following representations:
{code}
if (ttl 1 ttl 86400) = {TTL = X SECONDS}
if (ttl 86400) = {TTL = X SECONDS (Y DAYS)}
{code}
Pretty print TTL
Key: HBASE-6990
URL: https://issues.apache.org/jira/browse/HBASE-6990
Project: HBase
Issue Type: Improvement
Affects Versions: 0.94.6, 0.95.0
Reporter: Jean-Daniel Cryans
Assignee: Esteban Gutierrez
Priority: Minor
Attachments: HBASE-6990.v0.patch
I've seen a lot of users getting confused by the TTL configuration and I
think that if we just pretty printed it it would solve most of the issues.
For example, let's say a user wanted to set a TTL of 90 days. That would be
7776000. But let's say that it was typo'd to 7776 instead, it gives you
900 days!
So when we print the TTL we could do something like x days, x hours, x
minutes, x seconds (real_ttl_value). This would also help people when they
use ms instead of seconds as they would see really big values in there.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HBASE-7958) Statistics per-column family per-region
[ https://issues.apache.org/jira/browse/HBASE-7958?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13994412#comment-13994412 ] Andrew Purtell commented on HBASE-7958: --- Thinking about reviving this issue. [~jesse_yates], could you comment on why this fizzled? Statistics per-column family per-region --- Key: HBASE-7958 URL: https://issues.apache.org/jira/browse/HBASE-7958 Project: HBase Issue Type: New Feature Affects Versions: 0.95.2 Reporter: Jesse Yates Assignee: Jesse Yates Attachments: hbase-7958-v0-parent.patch, hbase-7958-v0.patch, hbase-7958_rough-cut-v0.patch Originating from this discussion on the dev list: http://search-hadoop.com/m/coDKU1urovS/Simple+stastics+per+region/v=plain Essentially, we should have built-in statistics gathering for HBase tables. This allows clients to have a better understanding of the distribution of keys within a table and a given region. We could also surface this information via the UI. There are a couple different proposals from the email, the overview is this: We add in something on compactions that gathers stats about the keys that are written and then we surface them to a table. The possible proposals include: *How to implement it?* # Coprocessors - ** advantage - it easily plugs in and people could pretty easily add their own statistics. ** disadvantage - UI elements would also require this, we get into dependent loading, which leads down the OSGi path. Also, these CPs need to be installed _after_ all the other CPs on compaction to ensure they see exactly what gets written (doable, but a pain) # Built into HBase as a custom scanner ** advantage - always goes in the right place and no need to muck about with loading CPs etc. ** disadvantage - less pluggable, at least for the initial cut *Where do we store data?* # .META. ** advantage - its an existing table, so we can jam it into another CF there ** disadvantage - this would make META much larger, possibly leading to splits AND will make it much harder for other processes to read the info # A new stats table ** advantage - cleanly separates out the information from META ** disadvantage - should use a 'system table' idea to prevent accidental deletion, manipulation by arbitrary clients, but still allow clients to read it. Once we have this framework, we can then move to an actual implementation of various statistics. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11107) Provide utility method equivalent to 0.92's Result.getBytes().getSize()
[
https://issues.apache.org/jira/browse/HBASE-11107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13994413#comment-13994413
]
Rekha Joshi commented on HBASE-11107:
-
[~gustavoanatoly] : got work to be done for now, so sure.please go ahead.
Provide utility method equivalent to 0.92's Result.getBytes().getSize()
---
Key: HBASE-11107
URL: https://issues.apache.org/jira/browse/HBASE-11107
Project: HBase
Issue Type: Task
Reporter: Ted Yu
Assignee: Rekha Joshi
Priority: Trivial
Currently user has to write code similar to the following for replacement of
Result.getBytes().getSize() :
{code}
+Cell[] cellValues = resultRow.rawCells();
+
+long size = 0L;
+if (null != cellValues) {
+ for (Cell cellValue : cellValues) {
+size += KeyValueUtil.ensureKeyValue(cellValue).heapSize();
+ }
+}
{code}
In ClientScanner, we have:
{code}
for (Cell kv : rs.rawCells()) {
// TODO make method in Cell or CellUtil
remainingResultSize -=
KeyValueUtil.ensureKeyValue(kv).heapSize();
}
{code}
A utility method should be provided which computes summation of Cell sizes in
a Result.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HBASE-11151) move tracing modules from hbase-server to hbase-common
Masatake Iwasaki created HBASE-11151: Summary: move tracing modules from hbase-server to hbase-common Key: HBASE-11151 URL: https://issues.apache.org/jira/browse/HBASE-11151 Project: HBase Issue Type: Improvement Components: documentation, util Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Minor Not only servers but also clients using tracing needs SpanReceiverHost. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HBASE-10561) Forward port: HBASE-10212 New rpc metric: number of active handler
[ https://issues.apache.org/jira/browse/HBASE-10561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Xie updated HBASE-10561: -- Attachment: HBASE-10561.txt Forward port: HBASE-10212 New rpc metric: number of active handler -- Key: HBASE-10561 URL: https://issues.apache.org/jira/browse/HBASE-10561 Project: HBase Issue Type: Sub-task Components: IPC/RPC Reporter: Lars Hofhansl Fix For: 0.99.0, 0.96.3, 0.98.3 Attachments: HBASE-10561.txt The metrics implementation has changed a lot in 0.96. Forward port HBASE-10212 to 0.96 and later. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-10985) Decouple Split Transaction from Zookeeper
[ https://issues.apache.org/jira/browse/HBASE-10985?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13992604#comment-13992604 ] Mikhail Antonov commented on HBASE-10985: - With HBASE-11092 committed to trunk yesterday the patch could be simplified a bit. Decouple Split Transaction from Zookeeper - Key: HBASE-10985 URL: https://issues.apache.org/jira/browse/HBASE-10985 Project: HBase Issue Type: Sub-task Components: Consensus, Zookeeper Reporter: Sergey Soldatov Attachments: HBASE-10985.patch, HBASE-10985.patch, HBASE-10985.patch, HBASE_10985-v2.patch, HBASE_10985-v3.patch As part of HBASE-10296 SplitTransaction should be decoupled from Zookeeper. This is an initial patch for review. At the moment the consensus provider placed directly to SplitTransaction to minimize affected code. In the ideal world it should be done in HServer. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11151) move tracing modules from hbase-server to hbase-common
[ https://issues.apache.org/jira/browse/HBASE-11151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13994430#comment-13994430 ] stack commented on HBASE-11151: --- +1 move tracing modules from hbase-server to hbase-common -- Key: HBASE-11151 URL: https://issues.apache.org/jira/browse/HBASE-11151 Project: HBase Issue Type: Improvement Components: documentation, util Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Minor Not only servers but also clients using tracing needs SpanReceiverHost. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11145) Issue with HLog sync
[
https://issues.apache.org/jira/browse/HBASE-11145?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13994431#comment-13994431
]
stack commented on HBASE-11145:
---
Thanks [~anoop.hbase]
Issue with HLog sync
Key: HBASE-11145
URL: https://issues.apache.org/jira/browse/HBASE-11145
Project: HBase
Issue Type: Bug
Reporter: Anoop Sam John
Assignee: stack
Priority: Critical
Fix For: 0.99.0
Got the below Exceptions Log in case of a write heavy test
{code}
2014-05-07 11:29:56,417 ERROR [main.append-pool1-t1]
wal.FSHLog$RingBufferEventHandler(1882): UNEXPECTED!!!
java.lang.IllegalStateException: Queue full
at java.util.AbstractQueue.add(Unknown Source)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$SyncRunner.offer(FSHLog.java:1227)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1878)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:133)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2014-05-07 11:29:56,418 ERROR [main.append-pool1-t1]
wal.FSHLog$RingBufferEventHandler(1882): UNEXPECTED!!!
java.lang.ArrayIndexOutOfBoundsException: 5
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1838)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:133)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2014-05-07 11:29:56,419 ERROR [main.append-pool1-t1]
wal.FSHLog$RingBufferEventHandler(1882): UNEXPECTED!!!
java.lang.ArrayIndexOutOfBoundsException: 6
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1838)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:133)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
2014-05-07 11:29:56,419 ERROR [main.append-pool1-t1]
wal.FSHLog$RingBufferEventHandler(1882): UNEXPECTED!!!
java.lang.ArrayIndexOutOfBoundsException: 7
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1838)
at
org.apache.hadoop.hbase.regionserver.wal.FSHLog$RingBufferEventHandler.onEvent(FSHLog.java:1)
at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:133)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
{code}
In FSHLog$SyncRunner.offer we do BlockingQueue.add() which throws Exception
as it is full. The problem is the below shown catch() we do not do any
cleanup.
{code}
this.syncRunners[index].offer(sequence, this.syncFutures,
this.syncFuturesCount);
attainSafePoint(sequence);
this.syncFuturesCount = 0;
} catch (Throwable t) {
LOG.error(UNEXPECTED!!!, t);
}
{code}
syncFuturesCount is not getting reset to 0 and so the subsequent onEvent()
handling throws ArrayIndexOutOfBoundsException.
I think we should do the below
1. Handle the Exception and call cleanupOutstandingSyncsOnException() as in
other cases of Exception handling
2. Instead of BlockingQueue.add() use offer() (?)
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HBASE-11135) Change region sequenceid generation so happens earlier in the append cycle rather than just before added to file
[ https://issues.apache.org/jira/browse/HBASE-11135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-11135: -- Attachment: 11135v5.txt Change region sequenceid generation so happens earlier in the append cycle rather than just before added to file Key: HBASE-11135 URL: https://issues.apache.org/jira/browse/HBASE-11135 Project: HBase Issue Type: Sub-task Components: wal Reporter: stack Assignee: stack Attachments: 11135.wip.txt, 11135v2.txt, 11135v5.txt, 11135v5.txt Currently we assign the region edit/sequence id just before we put it in the WAL. We do it in the single thread that feeds from the ring buffer. Doing it at this point, we can ensure order, that the edits will be in the file in accordance w/ the ordering of the region sequence id. But the point at which region sequence id is assigned an edit is deep down in the WAL system and there is a lag between our putting an edit into the WAL system and the edit actually getting its edit/sequence id. This lag -- late-binding -- complicates the unification of mvcc and region sequence id, especially around async WAL writes (and, related, for no-WAL writes) -- the parent for this issue (For async, how you get the edit id in our system when the threads have all gone home -- unless you make them wait?) Chatting w/ Jeffrey Zhong yesterday, we came up with a crazypants means of getting the region sequence id near-immediately. We'll run two ringbuffers. The first will mesh all handler threads and the consumer will generate ids (we will have order on other side of this first ring buffer), and then if async or no sync, we will just let the threads return ... updating mvcc just before we let them go. All other calls will go up on to the second ring buffer to be serviced as now (batching, distribution out among the sync'ing threads). The first rb will have no friction and should turn at fast rates compared to the second. There should not be noticeable slowdown nor do I foresee this refactor intefering w/ our multi-WAL plans. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HBASE-11144) Filter to support scan multiple row key ranges
[ https://issues.apache.org/jira/browse/HBASE-11144?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13994329#comment-13994329 ] Ted Yu commented on HBASE-11144: Can you generate patch for trunk ? Please also add unit test for MultiRowRangeFilter. Putting patch on reviewboard would help reviewers. Filter to support scan multiple row key ranges -- Key: HBASE-11144 URL: https://issues.apache.org/jira/browse/HBASE-11144 Project: HBase Issue Type: Improvement Components: Filters Reporter: Li Jiajia Attachments: MultiRowRangeFilter.patch Provide a filter feature to support scan multiple row key ranges. It can construct the row key ranges from the passed list which can be accessed by each region server. -- This message was sent by Atlassian JIRA (v6.2#6252)
