[jira] [Resolved] (HBASE-6200) KeyComparator.compareWithoutRow can be wrong when families have the same prefix
[
https://issues.apache.org/jira/browse/HBASE-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ramkrishna.s.vasudevan resolved HBASE-6200.
---
Resolution: Fixed
Hadoop Flags: Reviewed
Resolving as committed to all branches.
> KeyComparator.compareWithoutRow can be wrong when families have the same
> prefix
> ---
>
> Key: HBASE-6200
> URL: https://issues.apache.org/jira/browse/HBASE-6200
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.90.6, 0.92.1, 0.94.0
>Reporter: Jean-Daniel Cryans
>Assignee: Jieshan Bean
>Priority: Blocker
> Fix For: 0.90.7, 0.92.2, 0.96.0, 0.94.1
>
> Attachments: 6200-0.92.txt, 6200-0.94.txt, 6200-90.patch,
> 6200-trunk-v2.patch, 6200-trunk-v3.patch, 6200-trunk-v4.txt
>
>
> As reported by Desert Rose on IRC and on the ML, {{Result}} has a weird
> behavior when some families share the same prefix. He posted a link to his
> code to show how it fails, http://pastebin.com/7TBA1XGh
> Basically {{KeyComparator.compareWithoutRow}} doesn't differentiate families
> and qualifiers so "f:a" is said to be bigger than "f1:", which is false. Then
> what happens is that the KVs are returned in the right order from the RS but
> then doing {{Result.binarySearch}} it uses
> {{KeyComparator.compareWithoutRow}} which has a different sorting so the end
> result is undetermined.
> I added some debug and I can see that the data is returned in the right order
> but {{Arrays.binarySearch}} returned the wrong KV, which is then verified
> agains the passed family and qualifier which fails so null is returned.
> I don't know how frequent it is for users to have families with the same
> prefix, but those that do have that and that use those families at the same
> time will have big correctness issues. This is why I mark this as a blocker.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4920) We need a mascot, a totem
[ https://issues.apache.org/jira/browse/HBASE-4920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402010#comment-13402010 ] Esteban Gutierrez commented on HBASE-4920: -- Seems that I forgot to cast my vote: +1 Honey Badger > We need a mascot, a totem > - > > Key: HBASE-4920 > URL: https://issues.apache.org/jira/browse/HBASE-4920 > Project: HBase > Issue Type: Task >Reporter: stack > Attachments: HBase Orca Logo.jpg, Orca_479990801.jpg, Screen shot > 2011-11-30 at 4.06.17 PM.png, apache hbase orca logo_Proof 3.pdf, apache > logo_Proof 8.pdf, krake.zip, more_orcas.png, more_orcas2.png, photo (2).JPG, > plus_orca.png > > > We need a totem for our t-shirt that is yet to be printed. O'Reilly owns the > Clyesdale. We need something else. > We could have a fluffy little duck that quacks 'hbase!' when you squeeze it > and we could order boxes of them from some off-shore sweatshop that > subcontracts to a contractor who employs child labor only. > Or we could have an Orca (Big!, Fast!, Killer!, and in a poem that Marcy from > Salesforce showed me, that was a bit too spiritual for me to be seen quoting > here, it had the Orca as the 'Guardian of the Cosmic Memory': i.e. in > translation, bigdata). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6227) SSH and cluster startup causes data loss
[ https://issues.apache.org/jira/browse/HBASE-6227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402012#comment-13402012 ] Hadoop QA commented on HBASE-6227: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12533605/HBASE-6227v2-94.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2269//console This message is automatically generated. > SSH and cluster startup causes data loss > - > > Key: HBASE-6227 > URL: https://issues.apache.org/jira/browse/HBASE-6227 > Project: HBase > Issue Type: Bug > Components: master >Reporter: chunhui shen >Assignee: chunhui shen > Attachments: HBASE-6227.patch, HBASE-6227v2-94.patch, > HBASE-6227v2.patch > > > In AssignmentManager#processDeadServersAndRegionsInTransition, if > servershutdownhandler is processing and master consider it cluster startup, > master will assign all user regions, however, servershutdownhandler has not > completed splitting logs. > Let me describe it in more detail. > Suppose there are two regionservers A1 and B1, A1 carried META and ROOT > 1.master restart and completed assignRootAndMeta > 2.A1 and B1 are both restarted, new regionservers are A2 and B2 > 3.SSH which processed for A1 completed assigning ROOT and META > 4.master do rebuilding user regions and no regions added to master's region > list > 5.master consider it as a cluster startup, and assign all user regions > 6.SSH which processed for B1 start to split B1's logs > 7.All regions' data carried by B1 would loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6269) Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
[
https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ShiXing updated HBASE-6269:
---
Attachment: runAllTests.out.txt
I run
mvn test -P runAllTests
and the output is in the runAllTests.out.txt
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest
> KeyValue
> -
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.94.0
>Reporter: ShiXing
>Assignee: ShiXing
> Attachments: HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch,
> runAllTests.out.txt
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test
> case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp
> and different memstoreTS, after each Put/Increment, we do a memstore flush.
> So there will be two StoreFile with same KeyValue(except memstoreTS and
> SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 :
[jira] [Commented] (HBASE-6269) Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
[
https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402023#comment-13402023
]
Hadoop QA commented on HBASE-6269:
--
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12533611/runAllTests.out.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/2271//console
This message is automatically generated.
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest
> KeyValue
> -
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.94.0
>Reporter: ShiXing
>Assignee: ShiXing
> Attachments: HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch,
> runAllTests.out.txt
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test
> case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp
> and different memstoreTS, after each Put/Increment, we do a memstore flush.
> So there will be two StoreFile with same KeyValue(except memstoreTS and
> SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i <
[jira] [Commented] (HBASE-6274) Proto files should be in the same palce
[ https://issues.apache.org/jira/browse/HBASE-6274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402048#comment-13402048 ] Hadoop QA commented on HBASE-6274: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12533597/6274-hbase.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2270//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2270//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2270//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2270//console This message is automatically generated. > Proto files should be in the same palce > --- > > Key: HBASE-6274 > URL: https://issues.apache.org/jira/browse/HBASE-6274 > Project: HBase > Issue Type: Improvement >Affects Versions: 0.96.0 >Reporter: Jimmy Xiang >Assignee: Jimmy Xiang >Priority: Trivial > Fix For: 0.96.0 > > Attachments: 6274-hbase.patch > > > Currently, proto files are under hbase-server/src/main/protobuf and > hbase-server/src/protobuf. It's better to put them together. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6280) why using treeMap at default implement with class Batch.Callback
qiujaiwei created HBASE-6280:
Summary: why using treeMap at default implement with class
Batch.Callback
Key: HBASE-6280
URL: https://issues.apache.org/jira/browse/HBASE-6280
Project: HBase
Issue Type: Bug
Components: coprocessors
Affects Versions: 0.92.0
Environment: centos 6.2 hbase 0.92.0
Reporter: qiujaiwei
public Map coprocessorExec(
Class protocol, byte[] startKey, byte[] endKey,
Batch.Call callable)
throws IOException, Throwable {
final Map results = new TreeMap(
Bytes.BYTES_COMPARATOR);
coprocessorExec(protocol, startKey, endKey, callable,
new Batch.Callback(){
public void update(byte[] region, byte[] row, R value) {
results.put(region, value);
}
});
return results;
}
when mulit region call the Batch.Callback ,the treemap should lockup.
we meet this situation after we run 3 month.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6228) Fixup daughters twice cause daughter region assigned twice
[ https://issues.apache.org/jira/browse/HBASE-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] chunhui shen updated HBASE-6228: Attachment: HBASE-6228v3.patch Adding test case for this issue in the patch v3 > Fixup daughters twice cause daughter region assigned twice > --- > > Key: HBASE-6228 > URL: https://issues.apache.org/jira/browse/HBASE-6228 > Project: HBase > Issue Type: Bug > Components: master >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.96.0 > > Attachments: HBASE-6228.patch, HBASE-6228v2.patch, > HBASE-6228v2.patch, HBASE-6228v3.patch > > > First, how fixup daughters twice happen? > 1.we will fixupDaughters at the last of HMaster#finishInitialization > 2.ServerShutdownHandler will fixupDaughters when reassigning region through > ServerShutdownHandler#processDeadRegion > When fixupDaughters, we will added daughters to .META., but it coudn't > prevent the above case, because FindDaughterVisitor. > The detail is as the following: > Suppose region A is a splitted parent region, and its daughter region B is > missing > 1.First, ServerShutdownHander thread fixup daughter, so add daughter region B > to .META. with serverName=null, and assign the daughter. > 2.Then, Master's initialization thread will also find the daughter region B > is missing and assign it. It is because FindDaughterVisitor consider daughter > is missing if its serverName=null -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6273) HMasterInterface.isMasterRunning() requires clean up
[
https://issues.apache.org/jira/browse/HBASE-6273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402064#comment-13402064
]
nkeywal commented on HBASE-6273:
+1 as well. Errors can be:
1) Can't connect to ZK -> Hence you can't get the master address. This is
especially true in 0.96 as the connection to ZK is made on demand. I think we
need to have/keep this as an exception
2) There is no info on the master address in ZK. We should make this clear as
well. The best option I see is a specific exception such as
NoMasterAddressException or NoMasterAddressInZooKeeperException
And MasterNotRunning can be: the master is actually there, but marked
internally as non running (stopped). The server side method 'isMasterRunning'
will return false (you can contact the master but it's stopping). As such I
think is better not to have an exception named MasterNotRunning as it creates
confusion between the technical status and the functional one.
I think the pseudo code could be, on the client interface:
{noformat}
public boolean isMasterRunning() throws MasterConnectionException,
NoMasterAddressException, ZooKeeperConnectionException {
ZK zk = getZK(); // Can throw ZooKeeperConnectionException
Master m = getMaster(zk); // Can throw MasterConnectionException or
NoMasterAddressException
boolean isRunning = m.isMasterRunning(); // can throw
MasterConnectionException
return isRunning;
}
{noformat}
> HMasterInterface.isMasterRunning() requires clean up
>
>
> Key: HBASE-6273
> URL: https://issues.apache.org/jira/browse/HBASE-6273
> Project: HBase
> Issue Type: Bug
> Components: master
>Affects Versions: 0.94.0
>Reporter: ramkrishna.s.vasudevan
> Fix For: 0.96.0
>
>
> This JIRA is in reference to JD's comments regarding the clean up needed in
> isMasterRunning(). Refer to
> https://issues.apache.org/jira/browse/HBASE-6240?focusedCommentId=13400772&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13400772
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6280) why using treeMap at default implement with class Batch.Callback
[
https://issues.apache.org/jira/browse/HBASE-6280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402074#comment-13402074
]
Jieshan Bean commented on HBASE-6280:
-
Good finding, qiujaiwei. It seems a concurrency issue. We can see from the
method of HConnectionManager#processExecs:
{noformat}
Future future = pool.submit(
new Callable() {
public R call() throws Exception {
T instance = (T)Proxy.newProxyInstance(conf.getClassLoader(),
new Class[]{protocol},
invoker);
R result = callable.call(instance);
byte[] region = invoker.getRegionName();
if (callback != null) {
callback.update(region, r, result);
}
return result;
}
});
{noformat}
I think we should use a concurrent map instead of using TreeMap.
> why using treeMap at default implement with class Batch.Callback
> -
>
> Key: HBASE-6280
> URL: https://issues.apache.org/jira/browse/HBASE-6280
> Project: HBase
> Issue Type: Bug
> Components: coprocessors
>Affects Versions: 0.92.0
> Environment: centos 6.2 hbase 0.92.0
>Reporter: qiujaiwei
>
> public Map coprocessorExec(
> Class protocol, byte[] startKey, byte[] endKey,
> Batch.Call callable)
> throws IOException, Throwable {
> final Map results = new TreeMap(
> Bytes.BYTES_COMPARATOR);
> coprocessorExec(protocol, startKey, endKey, callable,
> new Batch.Callback(){
> public void update(byte[] region, byte[] row, R value) {
> results.put(region, value);
> }
> });
> return results;
> }
> when mulit region call the Batch.Callback ,the treemap should lockup.
> we meet this situation after we run 3 month.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6228) Fixup daughters twice cause daughter region assigned twice
[
https://issues.apache.org/jira/browse/HBASE-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402091#comment-13402091
]
ramkrishna.s.vasudevan commented on HBASE-6228:
---
@Chunhui
Thanks for the testcase. I ran the testcase without patch. It passes many
times even with out the fix.
{code}
RegionTransition rt = RegionTransition.parseFrom(data);
if (rt.getEventType() ==
EventHandler.EventType.RS_ZK_REGION_OPENED) {
// Make a new region plan
master.getAssignmentManager().getRegionPlan(daughterARegionState,
daughterARegionState.getServerName(), true);
return;
}
{code}
This part of code may not work right. Because the node state can be changed
before that Thread.sleep is done.
And so if we don't update the region plan we cannot assure that double
assignment can be done. Even i was trying to write a testcase and was caught
up in a similar state. Good on you Chunhui.
> Fixup daughters twice cause daughter region assigned twice
> ---
>
> Key: HBASE-6228
> URL: https://issues.apache.org/jira/browse/HBASE-6228
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: chunhui shen
>Assignee: chunhui shen
> Fix For: 0.96.0
>
> Attachments: HBASE-6228.patch, HBASE-6228v2.patch,
> HBASE-6228v2.patch, HBASE-6228v3.patch
>
>
> First, how fixup daughters twice happen?
> 1.we will fixupDaughters at the last of HMaster#finishInitialization
> 2.ServerShutdownHandler will fixupDaughters when reassigning region through
> ServerShutdownHandler#processDeadRegion
> When fixupDaughters, we will added daughters to .META., but it coudn't
> prevent the above case, because FindDaughterVisitor.
> The detail is as the following:
> Suppose region A is a splitted parent region, and its daughter region B is
> missing
> 1.First, ServerShutdownHander thread fixup daughter, so add daughter region B
> to .META. with serverName=null, and assign the daughter.
> 2.Then, Master's initialization thread will also find the daughter region B
> is missing and assign it. It is because FindDaughterVisitor consider daughter
> is missing if its serverName=null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6228) Fixup daughters twice cause daughter region assigned twice
[ https://issues.apache.org/jira/browse/HBASE-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402094#comment-13402094 ] Hadoop QA commented on HBASE-6228: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12533624/HBASE-6228v3.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.replication.TestReplication Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2272//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2272//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2272//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2272//console This message is automatically generated. > Fixup daughters twice cause daughter region assigned twice > --- > > Key: HBASE-6228 > URL: https://issues.apache.org/jira/browse/HBASE-6228 > Project: HBase > Issue Type: Bug > Components: master >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.96.0 > > Attachments: HBASE-6228.patch, HBASE-6228v2.patch, > HBASE-6228v2.patch, HBASE-6228v3.patch > > > First, how fixup daughters twice happen? > 1.we will fixupDaughters at the last of HMaster#finishInitialization > 2.ServerShutdownHandler will fixupDaughters when reassigning region through > ServerShutdownHandler#processDeadRegion > When fixupDaughters, we will added daughters to .META., but it coudn't > prevent the above case, because FindDaughterVisitor. > The detail is as the following: > Suppose region A is a splitted parent region, and its daughter region B is > missing > 1.First, ServerShutdownHander thread fixup daughter, so add daughter region B > to .META. with serverName=null, and assign the daughter. > 2.Then, Master's initialization thread will also find the daughter region B > is missing and assign it. It is because FindDaughterVisitor consider daughter > is missing if its serverName=null -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6228) Fixup daughters twice cause daughter region assigned twice
[ https://issues.apache.org/jira/browse/HBASE-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402106#comment-13402106 ] chunhui shen commented on HBASE-6228: - @ram Do you run the case in trunk? Maybe case is not good, but I run it and always failed without patch. bq.update the region plan we cannot assure that double assignment can be done If we restart master after fixup daughter, master won't contain any plans, so I just simulate it and ensure case could happen. Another thing, I think I must say: If we send open region RPC twice for one region, but not assigned twice because we have many protection through version. But we also have risk of data correctness. Why?: When region in opening, it will replay log edits and flush; if two regionservers do opening region together, could we ensure data correctness? > Fixup daughters twice cause daughter region assigned twice > --- > > Key: HBASE-6228 > URL: https://issues.apache.org/jira/browse/HBASE-6228 > Project: HBase > Issue Type: Bug > Components: master >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.96.0 > > Attachments: HBASE-6228.patch, HBASE-6228v2.patch, > HBASE-6228v2.patch, HBASE-6228v3.patch > > > First, how fixup daughters twice happen? > 1.we will fixupDaughters at the last of HMaster#finishInitialization > 2.ServerShutdownHandler will fixupDaughters when reassigning region through > ServerShutdownHandler#processDeadRegion > When fixupDaughters, we will added daughters to .META., but it coudn't > prevent the above case, because FindDaughterVisitor. > The detail is as the following: > Suppose region A is a splitted parent region, and its daughter region B is > missing > 1.First, ServerShutdownHander thread fixup daughter, so add daughter region B > to .META. with serverName=null, and assign the daughter. > 2.Then, Master's initialization thread will also find the daughter region B > is missing and assign it. It is because FindDaughterVisitor consider daughter > is missing if its serverName=null -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6228) Fixup daughters twice cause daughter region assigned twice
[ https://issues.apache.org/jira/browse/HBASE-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-6228: -- Attachment: HBASE-6228v4.patch > Fixup daughters twice cause daughter region assigned twice > --- > > Key: HBASE-6228 > URL: https://issues.apache.org/jira/browse/HBASE-6228 > Project: HBase > Issue Type: Bug > Components: master >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.96.0 > > Attachments: HBASE-6228.patch, HBASE-6228v2.patch, > HBASE-6228v2.patch, HBASE-6228v3.patch, HBASE-6228v4.patch > > > First, how fixup daughters twice happen? > 1.we will fixupDaughters at the last of HMaster#finishInitialization > 2.ServerShutdownHandler will fixupDaughters when reassigning region through > ServerShutdownHandler#processDeadRegion > When fixupDaughters, we will added daughters to .META., but it coudn't > prevent the above case, because FindDaughterVisitor. > The detail is as the following: > Suppose region A is a splitted parent region, and its daughter region B is > missing > 1.First, ServerShutdownHander thread fixup daughter, so add daughter region B > to .META. with serverName=null, and assign the daughter. > 2.Then, Master's initialization thread will also find the daughter region B > is missing and assign it. It is because FindDaughterVisitor consider daughter > is missing if its serverName=null -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6228) Fixup daughters twice cause daughter region assigned twice
[ https://issues.apache.org/jira/browse/HBASE-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-6228: -- Status: Patch Available (was: Open) > Fixup daughters twice cause daughter region assigned twice > --- > > Key: HBASE-6228 > URL: https://issues.apache.org/jira/browse/HBASE-6228 > Project: HBase > Issue Type: Bug > Components: master >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.96.0 > > Attachments: HBASE-6228.patch, HBASE-6228v2.patch, > HBASE-6228v2.patch, HBASE-6228v3.patch, HBASE-6228v4.patch > > > First, how fixup daughters twice happen? > 1.we will fixupDaughters at the last of HMaster#finishInitialization > 2.ServerShutdownHandler will fixupDaughters when reassigning region through > ServerShutdownHandler#processDeadRegion > When fixupDaughters, we will added daughters to .META., but it coudn't > prevent the above case, because FindDaughterVisitor. > The detail is as the following: > Suppose region A is a splitted parent region, and its daughter region B is > missing > 1.First, ServerShutdownHander thread fixup daughter, so add daughter region B > to .META. with serverName=null, and assign the daughter. > 2.Then, Master's initialization thread will also find the daughter region B > is missing and assign it. It is because FindDaughterVisitor consider daughter > is missing if its serverName=null -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6228) Fixup daughters twice cause daughter region assigned twice
[ https://issues.apache.org/jira/browse/HBASE-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-6228: -- Status: Open (was: Patch Available) > Fixup daughters twice cause daughter region assigned twice > --- > > Key: HBASE-6228 > URL: https://issues.apache.org/jira/browse/HBASE-6228 > Project: HBase > Issue Type: Bug > Components: master >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.96.0 > > Attachments: HBASE-6228.patch, HBASE-6228v2.patch, > HBASE-6228v2.patch, HBASE-6228v3.patch, HBASE-6228v4.patch > > > First, how fixup daughters twice happen? > 1.we will fixupDaughters at the last of HMaster#finishInitialization > 2.ServerShutdownHandler will fixupDaughters when reassigning region through > ServerShutdownHandler#processDeadRegion > When fixupDaughters, we will added daughters to .META., but it coudn't > prevent the above case, because FindDaughterVisitor. > The detail is as the following: > Suppose region A is a splitted parent region, and its daughter region B is > missing > 1.First, ServerShutdownHander thread fixup daughter, so add daughter region B > to .META. with serverName=null, and assign the daughter. > 2.Then, Master's initialization thread will also find the daughter region B > is missing and assign it. It is because FindDaughterVisitor consider daughter > is missing if its serverName=null -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6228) Fixup daughters twice cause daughter region assigned twice
[ https://issues.apache.org/jira/browse/HBASE-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402145#comment-13402145 ] ramkrishna.s.vasudevan commented on HBASE-6228: --- @Jon I have added a testcase to show the problem. bq.but my general impression is that I cannot easily tell if this patch (and several similar to it) are just pushing a race from one place to another. Could you let us know what type of issues are like that, if you have some list with you? @Chunhui What if the master initialization and SSH both are in fixUpDaughters? Then we are bound to get double assignment right? The scenario am telling is Master has just completed joinCluster(). By the time a RS went down while splitting . SSH starts and it comes to processDeadRegion. Later master also sees there are daughters to fixup. Now we may end up in same problem? I think here we need some sync mechanism. What do you feel Chunhui? > Fixup daughters twice cause daughter region assigned twice > --- > > Key: HBASE-6228 > URL: https://issues.apache.org/jira/browse/HBASE-6228 > Project: HBase > Issue Type: Bug > Components: master >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.96.0 > > Attachments: HBASE-6228.patch, HBASE-6228v2.patch, > HBASE-6228v2.patch, HBASE-6228v3.patch, HBASE-6228v4.patch > > > First, how fixup daughters twice happen? > 1.we will fixupDaughters at the last of HMaster#finishInitialization > 2.ServerShutdownHandler will fixupDaughters when reassigning region through > ServerShutdownHandler#processDeadRegion > When fixupDaughters, we will added daughters to .META., but it coudn't > prevent the above case, because FindDaughterVisitor. > The detail is as the following: > Suppose region A is a splitted parent region, and its daughter region B is > missing > 1.First, ServerShutdownHander thread fixup daughter, so add daughter region B > to .META. with serverName=null, and assign the daughter. > 2.Then, Master's initialization thread will also find the daughter region B > is missing and assign it. It is because FindDaughterVisitor consider daughter > is missing if its serverName=null -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6281) Assignment need not be called for disabling table regions during clean cluster start up.
[ https://issues.apache.org/jira/browse/HBASE-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] rajeshbabu updated HBASE-6281: -- Summary: Assignment need not be called for disabling table regions during clean cluster start up. (was: Assignment need not be called for disabling table regions during clean cluste start up.) > Assignment need not be called for disabling table regions during clean > cluster start up. > > > Key: HBASE-6281 > URL: https://issues.apache.org/jira/browse/HBASE-6281 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.1, 0.94.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > Fix For: 0.92.2, 0.96.0, 0.94.1 > > > Currently during clean cluster start up if there are tables in DISABLING > state, we do bulk assignment through assignAllUserRegions() and after region > is OPENED in RS, master checks if the table is in DISBALING/DISABLED state > (in Am.regionOnline) and again calls unassign. This roundtrip can be avoided > even before calling assignment. > This JIRA is to address the above scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6281) Assignment need not be called for disabling table regions during clean cluste start up.
rajeshbabu created HBASE-6281: - Summary: Assignment need not be called for disabling table regions during clean cluste start up. Key: HBASE-6281 URL: https://issues.apache.org/jira/browse/HBASE-6281 Project: HBase Issue Type: Bug Affects Versions: 0.94.0, 0.92.1 Reporter: rajeshbabu Assignee: rajeshbabu Fix For: 0.92.2, 0.96.0, 0.94.1 Currently during clean cluster start up if there are tables in DISABLING state, we do bulk assignment through assignAllUserRegions() and after region is OPENED in RS, master checks if the table is in DISBALING/DISABLED state (in Am.regionOnline) and again calls unassign. This roundtrip can be avoided even before calling assignment. This JIRA is to address the above scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6170) Timeouts for row lock and scan should be separate
[ https://issues.apache.org/jira/browse/HBASE-6170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402157#comment-13402157 ] Hudson commented on HBASE-6170: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #71 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/71/]) HBASE-6170 Timeouts for row lock and scan should be separate (Chris Trezzo) (Revision 1354325) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/Leases.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestScannerTimeout.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java * /hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java > Timeouts for row lock and scan should be separate > - > > Key: HBASE-6170 > URL: https://issues.apache.org/jira/browse/HBASE-6170 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.94.0 >Reporter: Otis Gospodnetic >Assignee: Chris Trezzo >Priority: Minor > Fix For: 0.96.0 > > Attachments: 6170v4.patch, HBASE-6170v1.patch, HBASE-6170v1.patch, > HBASE-6170v1.patch, HBASE-6170v2.patch, HBASE-6170v3.patch > > > Apparently the timeout used for row locking and for scanning is global. It > would be better to have two separate timeouts. > (opening the issue to make Lars George happy) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6228) Fixup daughters twice cause daughter region assigned twice
[ https://issues.apache.org/jira/browse/HBASE-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402167#comment-13402167 ] Hadoop QA commented on HBASE-6228: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12533642/HBASE-6228v4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2273//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2273//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2273//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2273//console This message is automatically generated. > Fixup daughters twice cause daughter region assigned twice > --- > > Key: HBASE-6228 > URL: https://issues.apache.org/jira/browse/HBASE-6228 > Project: HBase > Issue Type: Bug > Components: master >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.96.0 > > Attachments: HBASE-6228.patch, HBASE-6228v2.patch, > HBASE-6228v2.patch, HBASE-6228v3.patch, HBASE-6228v4.patch > > > First, how fixup daughters twice happen? > 1.we will fixupDaughters at the last of HMaster#finishInitialization > 2.ServerShutdownHandler will fixupDaughters when reassigning region through > ServerShutdownHandler#processDeadRegion > When fixupDaughters, we will added daughters to .META., but it coudn't > prevent the above case, because FindDaughterVisitor. > The detail is as the following: > Suppose region A is a splitted parent region, and its daughter region B is > missing > 1.First, ServerShutdownHander thread fixup daughter, so add daughter region B > to .META. with serverName=null, and assign the daughter. > 2.Then, Master's initialization thread will also find the daughter region B > is missing and assign it. It is because FindDaughterVisitor consider daughter > is missing if its serverName=null -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6228) Fixup daughters twice cause daughter region assigned twice
[
https://issues.apache.org/jira/browse/HBASE-6228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402168#comment-13402168
]
chunhui shen commented on HBASE-6228:
-
@ram
Could you talk about sync mechanism?
Based on the current patch, I think we won't assign twice. If need, we could
add a synchronized block for the following:
{code}isDaughterMissing(catalogTracker, daughter)) {
LOG.info("Fixup; missing daughter " + daughter.getRegionNameAsString());
MetaEditor.addDaughter(catalogTracker, daughter, null);
{code}
What do you think?
> Fixup daughters twice cause daughter region assigned twice
> ---
>
> Key: HBASE-6228
> URL: https://issues.apache.org/jira/browse/HBASE-6228
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: chunhui shen
>Assignee: chunhui shen
> Fix For: 0.96.0
>
> Attachments: HBASE-6228.patch, HBASE-6228v2.patch,
> HBASE-6228v2.patch, HBASE-6228v3.patch, HBASE-6228v4.patch
>
>
> First, how fixup daughters twice happen?
> 1.we will fixupDaughters at the last of HMaster#finishInitialization
> 2.ServerShutdownHandler will fixupDaughters when reassigning region through
> ServerShutdownHandler#processDeadRegion
> When fixupDaughters, we will added daughters to .META., but it coudn't
> prevent the above case, because FindDaughterVisitor.
> The detail is as the following:
> Suppose region A is a splitted parent region, and its daughter region B is
> missing
> 1.First, ServerShutdownHander thread fixup daughter, so add daughter region B
> to .META. with serverName=null, and assign the daughter.
> 2.Then, Master's initialization thread will also find the daughter region B
> is missing and assign it. It is because FindDaughterVisitor consider daughter
> is missing if its serverName=null
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2730) Expose RS work queue contents on web UI
[ https://issues.apache.org/jira/browse/HBASE-2730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402181#comment-13402181 ] Ted Yu commented on HBASE-2730: --- Can you attach screen shot of queue dump ? Thanks > Expose RS work queue contents on web UI > --- > > Key: HBASE-2730 > URL: https://issues.apache.org/jira/browse/HBASE-2730 > Project: HBase > Issue Type: New Feature > Components: monitoring, regionserver >Reporter: Todd Lipcon >Priority: Critical > Fix For: 0.96.0 > > Attachments: hbase-2730-0_94_0.patch > > > Would be nice to be able to see the contents of the various work queues - eg > to know what regions are pending compaction/split/flush/etc. This is handy > for debugging why a region might be blocked, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4364) Filters applied to columns not in the selected column list are ignored
[ https://issues.apache.org/jira/browse/HBASE-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-4364: - Attachment: (was: hbase-4364-0_94_0.patch) > Filters applied to columns not in the selected column list are ignored > -- > > Key: HBASE-4364 > URL: https://issues.apache.org/jira/browse/HBASE-4364 > Project: HBase > Issue Type: Bug > Components: filters >Affects Versions: 0.90.4, 0.92.0 >Reporter: Todd Lipcon >Priority: Critical > > For a scan, if you select some set of columns using addColumns(), and then > apply a SingleColumnValueFilter that restricts the results based on some > other columns which aren't selected, then those filter conditions are ignored. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-4364) Filters applied to columns not in the selected column list are ignored
[ https://issues.apache.org/jira/browse/HBASE-4364?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jie Huang updated HBASE-4364: - Attachment: hbase-4364-0_94_0.patch One possible solution for the feature: 1. Add sanityCheck() API in Filter. And call the check function in getScanner(). So the filter can check the scan and its criteria before hand. 2. In SingleColumnValueFilter, if the tested set doesn't include the criteria column, add it in sanityCheck(). Afterwards, remove the column in filterRow(kv), so that the user will get all required columns. I will add one corresponding testcase and do the verification later. Any comment, please let me know. thanks. > Filters applied to columns not in the selected column list are ignored > -- > > Key: HBASE-4364 > URL: https://issues.apache.org/jira/browse/HBASE-4364 > Project: HBase > Issue Type: Bug > Components: filters >Affects Versions: 0.90.4, 0.92.0 >Reporter: Todd Lipcon >Priority: Critical > Attachments: hbase-4364-0_94_0.patch > > > For a scan, if you select some set of columns using addColumns(), and then > apply a SingleColumnValueFilter that restricts the results based on some > other columns which aren't selected, then those filter conditions are ignored. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5904) is_enabled from shell returns differently from pre- and post- HBASE-5155
[ https://issues.apache.org/jira/browse/HBASE-5904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-5904: -- Resolution: Fixed Release Note: This reverts a patch included in 0.90.6 that broke compatibility. Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks for the patch Dave, Thanks for reviews Stack and Ram. > is_enabled from shell returns differently from pre- and post- HBASE-5155 > > > Key: HBASE-5904 > URL: https://issues.apache.org/jira/browse/HBASE-5904 > Project: HBase > Issue Type: Bug > Components: zookeeper >Affects Versions: 0.90.6 >Reporter: David S. Wang >Assignee: David S. Wang >Priority: Blocker > Fix For: 0.90.7 > > Attachments: HBASE-5904.patch > > > If I launch an hbase shell that uses HBase and ZooKeeper without HBASE-5155, > against HBase servers with HBASE-5155, then is_enabled for a table always > returns false even if the table is considered enabled by the servers from the > logs. If I then do the same thing but with an HBase shell and ZooKeeper with > HBASE-5155, then is_enabled returns as expected. > If I launch an HBase shell that uses HBase and ZooKeeper without HBASE-5155, > against HBase servers also without HBASE-5155, then is_enabled works as you'd > expect. But if I then do the same thing but with an HBase shell and > ZooKeeper with HBASE-5155, then is_enabled returns false even though the > table is considered enabled by the servers from the logs. > Additionally, if I then try to enable the table from the > HBASE-5155-containing shell, it hangs because the ZooKeeper code waits for > the ZNode to be updated with "ENABLED" in the data field, but what actually > happens is that the ZNode gets deleted since the servers are running without > HBASE-5155. > I think the culprit is that the indication of how a table is considered > enabled inside ZooKeeper has changed with HBASE-5155. Before HBASE-5155, a > table was considered enabled if the ZNode for it did not exist. After > HBASE-5155, a table is considered enabled if the ZNode for it exists and has > "ENABLED" in its data. I think the current code is incompatible when running > clients and servers where one side has HBASE-5155 and the other side does not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3855) Performance degradation of memstore because reseek is linear
[ https://issues.apache.org/jira/browse/HBASE-3855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402210#comment-13402210 ] Jonathan Hsieh commented on HBASE-3855: --- Hey [~dhruba], do you plan on taking on this issue on or shall we punt on this for 0.90.7 release? > Performance degradation of memstore because reseek is linear > > > Key: HBASE-3855 > URL: https://issues.apache.org/jira/browse/HBASE-3855 > Project: HBase > Issue Type: Improvement >Reporter: dhruba borthakur >Priority: Blocker > Fix For: 0.90.7 > > Attachments: memstoreReseek.txt, memstoreReseek2.txt > > > The scanner use reseek to find the next row (or next column) as part of a > scan. The reseek code iterates over a Set to position itself at the right > place. If there are many thousands of kvs that need to be skipped over, then > the time-cost is very high. In this case, a seek would be far lesser in cost > than a reseek. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6273) HMasterInterface.isMasterRunning() requires clean up
[ https://issues.apache.org/jira/browse/HBASE-6273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan reassigned HBASE-6273: - Assignee: ramkrishna.s.vasudevan > HMasterInterface.isMasterRunning() requires clean up > > > Key: HBASE-6273 > URL: https://issues.apache.org/jira/browse/HBASE-6273 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.94.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 0.96.0 > > > This JIRA is in reference to JD's comments regarding the clean up needed in > isMasterRunning(). Refer to > https://issues.apache.org/jira/browse/HBASE-6240?focusedCommentId=13400772&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13400772 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6170) Timeouts for row lock and scan should be separate
[ https://issues.apache.org/jira/browse/HBASE-6170?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Ted Yu updated HBASE-6170: -- Attachment: 6170.addendum Addendum that modifies hbase-default.xml > Timeouts for row lock and scan should be separate > - > > Key: HBASE-6170 > URL: https://issues.apache.org/jira/browse/HBASE-6170 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.94.0 >Reporter: Otis Gospodnetic >Assignee: Chris Trezzo >Priority: Minor > Fix For: 0.96.0 > > Attachments: 6170.addendum, 6170v4.patch, HBASE-6170v1.patch, > HBASE-6170v1.patch, HBASE-6170v1.patch, HBASE-6170v2.patch, HBASE-6170v3.patch > > > Apparently the timeout used for row locking and for scanning is global. It > would be better to have two separate timeouts. > (opening the issue to make Lars George happy) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6281) Assignment need not be called for disabling table regions during clean cluster start up.
[ https://issues.apache.org/jira/browse/HBASE-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] rajeshbabu updated HBASE-6281: -- Attachment: HBASE-6281_94.patch > Assignment need not be called for disabling table regions during clean > cluster start up. > > > Key: HBASE-6281 > URL: https://issues.apache.org/jira/browse/HBASE-6281 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.1, 0.94.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > Fix For: 0.92.2, 0.96.0, 0.94.1 > > Attachments: HBASE-6281_94.patch, HBASE-6281_trunk.patch > > > Currently during clean cluster start up if there are tables in DISABLING > state, we do bulk assignment through assignAllUserRegions() and after region > is OPENED in RS, master checks if the table is in DISBALING/DISABLED state > (in Am.regionOnline) and again calls unassign. This roundtrip can be avoided > even before calling assignment. > This JIRA is to address the above scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6281) Assignment need not be called for disabling table regions during clean cluster start up.
[ https://issues.apache.org/jira/browse/HBASE-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] rajeshbabu updated HBASE-6281: -- Attachment: HBASE-6281_trunk.patch > Assignment need not be called for disabling table regions during clean > cluster start up. > > > Key: HBASE-6281 > URL: https://issues.apache.org/jira/browse/HBASE-6281 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.1, 0.94.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > Fix For: 0.92.2, 0.96.0, 0.94.1 > > Attachments: HBASE-6281_94.patch, HBASE-6281_trunk.patch > > > Currently during clean cluster start up if there are tables in DISABLING > state, we do bulk assignment through assignAllUserRegions() and after region > is OPENED in RS, master checks if the table is in DISBALING/DISABLED state > (in Am.regionOnline) and again calls unassign. This roundtrip can be avoided > even before calling assignment. > This JIRA is to address the above scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6281) Assignment need not be called for disabling table regions during clean cluster start up.
[ https://issues.apache.org/jira/browse/HBASE-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] rajeshbabu updated HBASE-6281: -- Status: Patch Available (was: Open) Attached patch for 94 and trunk. Please review and provide comments/suggestions. running test suite for 94 and let you know result by tomorrow. > Assignment need not be called for disabling table regions during clean > cluster start up. > > > Key: HBASE-6281 > URL: https://issues.apache.org/jira/browse/HBASE-6281 > Project: HBase > Issue Type: Bug >Affects Versions: 0.94.0, 0.92.1 >Reporter: rajeshbabu >Assignee: rajeshbabu > Fix For: 0.92.2, 0.96.0, 0.94.1 > > Attachments: HBASE-6281_94.patch, HBASE-6281_trunk.patch > > > Currently during clean cluster start up if there are tables in DISABLING > state, we do bulk assignment through assignAllUserRegions() and after region > is OPENED in RS, master checks if the table is in DISBALING/DISABLED state > (in Am.regionOnline) and again calls unassign. This roundtrip can be avoided > even before calling assignment. > This JIRA is to address the above scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6192) Document ACL matrix in the book
[ https://issues.apache.org/jira/browse/HBASE-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Laxman updated HBASE-6192: -- Status: Patch Available (was: Open) > Document ACL matrix in the book > --- > > Key: HBASE-6192 > URL: https://issues.apache.org/jira/browse/HBASE-6192 > Project: HBase > Issue Type: Sub-task > Components: documentation, security >Affects Versions: 0.96.0, 0.94.1 >Reporter: Enis Soztutar >Assignee: Laxman > Labels: documentaion, security > Attachments: HBase Security-ACL Matrix.pdf, HBase Security-ACL > Matrix.pdf, HBase Security-ACL Matrix.pdf, HBase Security-ACL Matrix.xls, > HBase Security-ACL Matrix.xls, HBase Security-ACL Matrix.xls > > > We have an excellent matrix at > https://issues.apache.org/jira/secure/attachment/12531252/Security-ACL%20Matrix.pdf > for ACL. Once the changes are done, we can adapt that and put it in the > book, also add some more documentation about the new authorization features. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6192) Document ACL matrix in the book
[ https://issues.apache.org/jira/browse/HBASE-6192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402267#comment-13402267 ] Hadoop QA commented on HBASE-6192: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/1257/HBase%20Security-ACL%20Matrix.xls against trunk revision . +1 @author. The patch does not contain any @author tags. +0 tests included. The patch appears to be a documentation patch that doesn't require tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2275//console This message is automatically generated. > Document ACL matrix in the book > --- > > Key: HBASE-6192 > URL: https://issues.apache.org/jira/browse/HBASE-6192 > Project: HBase > Issue Type: Sub-task > Components: documentation, security >Affects Versions: 0.96.0, 0.94.1 >Reporter: Enis Soztutar >Assignee: Laxman > Labels: documentaion, security > Attachments: HBase Security-ACL Matrix.pdf, HBase Security-ACL > Matrix.pdf, HBase Security-ACL Matrix.pdf, HBase Security-ACL Matrix.xls, > HBase Security-ACL Matrix.xls, HBase Security-ACL Matrix.xls > > > We have an excellent matrix at > https://issues.apache.org/jira/secure/attachment/12531252/Security-ACL%20Matrix.pdf > for ACL. Once the changes are done, we can adapt that and put it in the > book, also add some more documentation about the new authorization features. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6170) Timeouts for row lock and scan should be separate
[ https://issues.apache.org/jira/browse/HBASE-6170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402272#comment-13402272 ] Hudson commented on HBASE-6170: --- Integrated in HBase-TRUNK #3080 (See [https://builds.apache.org/job/HBase-TRUNK/3080/]) HBASE-6170 add the two new config parameters in hbase-default.xml (Revision 1354500) Result = SUCCESS tedyu : Files : * /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml > Timeouts for row lock and scan should be separate > - > > Key: HBASE-6170 > URL: https://issues.apache.org/jira/browse/HBASE-6170 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.94.0 >Reporter: Otis Gospodnetic >Assignee: Chris Trezzo >Priority: Minor > Fix For: 0.96.0 > > Attachments: 6170.addendum, 6170v4.patch, HBASE-6170v1.patch, > HBASE-6170v1.patch, HBASE-6170v1.patch, HBASE-6170v2.patch, HBASE-6170v3.patch > > > Apparently the timeout used for row locking and for scanning is global. It > would be better to have two separate timeouts. > (opening the issue to make Lars George happy) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6241) HBaseCluster interface for interacting with the cluster from system tests
[
https://issues.apache.org/jira/browse/HBASE-6241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402293#comment-13402293
]
Zhihong Ted Yu commented on HBASE-6241:
---
{code}
+ private Class[] getIntegrationTestClasses() {
+//TODO: replace this with smt to find all classes in the classpath
+//having @Category(IntegrationTests.class) annotation
{code}
See TestCheckTestClasses.findTestClasses() for example on how the above can be
done.
> HBaseCluster interface for interacting with the cluster from system tests
> --
>
> Key: HBASE-6241
> URL: https://issues.apache.org/jira/browse/HBASE-6241
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: HBASE-6241_v0.2.patch
>
>
> We need to abstract away the cluster interactions for system tests running on
> actual clusters.
> MiniHBaseCluster and RealHBaseCluster should both implement this interface,
> and system tests should work with both.
> I'll split Devaraj's patch in HBASE-6053 for the initial version.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6281) Assignment need not be called for disabling table regions during clean cluster start up.
[ https://issues.apache.org/jira/browse/HBASE-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402295#comment-13402295 ] Hadoop QA commented on HBASE-6281: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12533661/HBASE-6281_trunk.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop2.0. The patch compiles against the hadoop 2.0 profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 6 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/2274//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2274//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/2274//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/2274//console This message is automatically generated. > Assignment need not be called for disabling table regions during clean > cluster start up. > > > Key: HBASE-6281 > URL: https://issues.apache.org/jira/browse/HBASE-6281 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.1, 0.94.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > Fix For: 0.92.2, 0.96.0, 0.94.1 > > Attachments: HBASE-6281_94.patch, HBASE-6281_trunk.patch > > > Currently during clean cluster start up if there are tables in DISABLING > state, we do bulk assignment through assignAllUserRegions() and after region > is OPENED in RS, master checks if the table is in DISBALING/DISABLED state > (in Am.regionOnline) and again calls unassign. This roundtrip can be avoided > even before calling assignment. > This JIRA is to address the above scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4920) We need a mascot, a totem
[ https://issues.apache.org/jira/browse/HBASE-4920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402306#comment-13402306 ] Jonathan Hsieh commented on HBASE-4920: --- Epic totem battle: whales vs squid. http://alaskarobotics.com/2009/12/01/whale-vs-squid/ > We need a mascot, a totem > - > > Key: HBASE-4920 > URL: https://issues.apache.org/jira/browse/HBASE-4920 > Project: HBase > Issue Type: Task >Reporter: stack > Attachments: HBase Orca Logo.jpg, Orca_479990801.jpg, Screen shot > 2011-11-30 at 4.06.17 PM.png, apache hbase orca logo_Proof 3.pdf, apache > logo_Proof 8.pdf, krake.zip, more_orcas.png, more_orcas2.png, photo (2).JPG, > plus_orca.png > > > We need a totem for our t-shirt that is yet to be printed. O'Reilly owns the > Clyesdale. We need something else. > We could have a fluffy little duck that quacks 'hbase!' when you squeeze it > and we could order boxes of them from some off-shore sweatshop that > subcontracts to a contractor who employs child labor only. > Or we could have an Orca (Big!, Fast!, Killer!, and in a poem that Marcy from > Salesforce showed me, that was a bit too spiritual for me to be seen quoting > here, it had the Orca as the 'Guardian of the Cosmic Memory': i.e. in > translation, bigdata). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6281) Assignment need not be called for disabling table regions during clean cluster start up.
[
https://issues.apache.org/jira/browse/HBASE-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402334#comment-13402334
]
Zhihong Ted Yu commented on HBASE-6281:
---
Patch looks good.
{code}
+// Skip assignment for regions of tables in DISABLING state also because
during clean cluster startup no
+// RS is alive and regions map also doesn't have any information about the
regions. See HBASE-6281.
{code}
Please wrap long lines above.
> Assignment need not be called for disabling table regions during clean
> cluster start up.
>
>
> Key: HBASE-6281
> URL: https://issues.apache.org/jira/browse/HBASE-6281
> Project: HBase
> Issue Type: Bug
>Affects Versions: 0.92.1, 0.94.0
>Reporter: rajeshbabu
>Assignee: rajeshbabu
> Fix For: 0.92.2, 0.96.0, 0.94.1
>
> Attachments: HBASE-6281_94.patch, HBASE-6281_trunk.patch
>
>
> Currently during clean cluster start up if there are tables in DISABLING
> state, we do bulk assignment through assignAllUserRegions() and after region
> is OPENED in RS, master checks if the table is in DISBALING/DISABLED state
> (in Am.regionOnline) and again calls unassign. This roundtrip can be avoided
> even before calling assignment.
> This JIRA is to address the above scenario.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6227) SSH and cluster startup causes data loss
[ https://issues.apache.org/jira/browse/HBASE-6227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402336#comment-13402336 ] ramkrishna.s.vasudevan commented on HBASE-6227: --- Planning to commit this now. > SSH and cluster startup causes data loss > - > > Key: HBASE-6227 > URL: https://issues.apache.org/jira/browse/HBASE-6227 > Project: HBase > Issue Type: Bug > Components: master >Reporter: chunhui shen >Assignee: chunhui shen > Attachments: HBASE-6227.patch, HBASE-6227v2-94.patch, > HBASE-6227v2.patch > > > In AssignmentManager#processDeadServersAndRegionsInTransition, if > servershutdownhandler is processing and master consider it cluster startup, > master will assign all user regions, however, servershutdownhandler has not > completed splitting logs. > Let me describe it in more detail. > Suppose there are two regionservers A1 and B1, A1 carried META and ROOT > 1.master restart and completed assignRootAndMeta > 2.A1 and B1 are both restarted, new regionservers are A2 and B2 > 3.SSH which processed for A1 completed assigning ROOT and META > 4.master do rebuilding user regions and no regions added to master's region > list > 5.master consider it as a cluster startup, and assign all user regions > 6.SSH which processed for B1 start to split B1's logs > 7.All regions' data carried by B1 would loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6091) Come up with strawman proposal for RC testing matrix
[ https://issues.apache.org/jira/browse/HBASE-6091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] David S. Wang resolved HBASE-6091. -- Resolution: Implemented > Come up with strawman proposal for RC testing matrix > > > Key: HBASE-6091 > URL: https://issues.apache.org/jira/browse/HBASE-6091 > Project: HBase > Issue Type: Bug > Components: documentation >Affects Versions: 0.96.0 >Reporter: David S. Wang >Assignee: David S. Wang > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6281) Assignment need not be called for disabling table regions during clean cluster start up.
[ https://issues.apache.org/jira/browse/HBASE-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402339#comment-13402339 ] Zhihong Ted Yu commented on HBASE-6281: --- For 0.92, there is no src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java It would be nice if you can create the file and put your test there. > Assignment need not be called for disabling table regions during clean > cluster start up. > > > Key: HBASE-6281 > URL: https://issues.apache.org/jira/browse/HBASE-6281 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.1, 0.94.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > Fix For: 0.92.2, 0.96.0, 0.94.1 > > Attachments: HBASE-6281_94.patch, HBASE-6281_trunk.patch > > > Currently during clean cluster start up if there are tables in DISABLING > state, we do bulk assignment through assignAllUserRegions() and after region > is OPENED in RS, master checks if the table is in DISBALING/DISABLED state > (in Am.regionOnline) and again calls unassign. This roundtrip can be avoided > even before calling assignment. > This JIRA is to address the above scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6281) Assignment need not be called for disabling table regions during clean cluster start up.
[ https://issues.apache.org/jira/browse/HBASE-6281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402348#comment-13402348 ] rajeshbabu commented on HBASE-6281: --- @Ted, Thanks for review. I will prepare patch for 92 and upload patches addressing above comment. > Assignment need not be called for disabling table regions during clean > cluster start up. > > > Key: HBASE-6281 > URL: https://issues.apache.org/jira/browse/HBASE-6281 > Project: HBase > Issue Type: Bug >Affects Versions: 0.92.1, 0.94.0 >Reporter: rajeshbabu >Assignee: rajeshbabu > Fix For: 0.92.2, 0.96.0, 0.94.1 > > Attachments: HBASE-6281_94.patch, HBASE-6281_trunk.patch > > > Currently during clean cluster start up if there are tables in DISABLING > state, we do bulk assignment through assignAllUserRegions() and after region > is OPENED in RS, master checks if the table is in DISBALING/DISABLED state > (in Am.regionOnline) and again calls unassign. This roundtrip can be avoided > even before calling assignment. > This JIRA is to address the above scenario. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6261) Better approximate high-percentile percentile latency metrics
[ https://issues.apache.org/jira/browse/HBASE-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402354#comment-13402354 ] Zhihong Ted Yu commented on HBASE-6261: --- So far I think the assumption is that the new algorithm would apply to the computation of all metrics. Is it possible to configure "Approximate Counts and Quantiles over Sliding Windows" for a few selected metrics (to be consumed by load balancer, e.g.) while the others get computed with light weight algorithm ? > Better approximate high-percentile percentile latency metrics > - > > Key: HBASE-6261 > URL: https://issues.apache.org/jira/browse/HBASE-6261 > Project: HBase > Issue Type: New Feature >Reporter: Andrew Wang > Labels: metrics > Attachments: Latencyestimation.pdf > > > The existing reservoir-sampling based latency metrics in HBase are not > well-suited for providing accurate estimates of high-percentile (e.g. 90th, > 95th, or 99th) latency. This is a well-studied problem in the literature (see > [1] and [2]), the question is determining which methods best suit our needs > and then implementing it. > Ideally, we should be able to estimate these high percentiles with minimal > memory and CPU usage as well as minimal error (e.g. 1% error on 90th, or .1% > on 99th). It's also desirable to provide this over different time-based > sliding windows, e.g. last 1 min, 5 mins, 15 mins, and 1 hour. > I'll note that this would also be useful in HDFS, or really anywhere latency > metrics are kept. > [1] http://www.cs.rutgers.edu/~muthu/bquant.pdf > [2] http://infolab.stanford.edu/~manku/papers/04pods-sliding.pdf -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6227) SSH and cluster startup causes data loss
[ https://issues.apache.org/jira/browse/HBASE-6227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-6227: -- Fix Version/s: 0.94.1 0.96.0 > SSH and cluster startup causes data loss > - > > Key: HBASE-6227 > URL: https://issues.apache.org/jira/browse/HBASE-6227 > Project: HBase > Issue Type: Bug > Components: master >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.96.0, 0.94.1 > > Attachments: HBASE-6227.patch, HBASE-6227v2-94.patch, > HBASE-6227v2.patch > > > In AssignmentManager#processDeadServersAndRegionsInTransition, if > servershutdownhandler is processing and master consider it cluster startup, > master will assign all user regions, however, servershutdownhandler has not > completed splitting logs. > Let me describe it in more detail. > Suppose there are two regionservers A1 and B1, A1 carried META and ROOT > 1.master restart and completed assignRootAndMeta > 2.A1 and B1 are both restarted, new regionservers are A2 and B2 > 3.SSH which processed for A1 completed assigning ROOT and META > 4.master do rebuilding user regions and no regions added to master's region > list > 5.master consider it as a cluster startup, and assign all user regions > 6.SSH which processed for B1 start to split B1's logs > 7.All regions' data carried by B1 would loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6227) SSH and cluster startup causes data loss
[ https://issues.apache.org/jira/browse/HBASE-6227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ramkrishna.s.vasudevan updated HBASE-6227: -- Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Commmitted to trunk and 0.94. Thanks for the patch Chunhui. Thanks for the review Ted. > SSH and cluster startup causes data loss > - > > Key: HBASE-6227 > URL: https://issues.apache.org/jira/browse/HBASE-6227 > Project: HBase > Issue Type: Bug > Components: master >Reporter: chunhui shen >Assignee: chunhui shen > Attachments: HBASE-6227.patch, HBASE-6227v2-94.patch, > HBASE-6227v2.patch > > > In AssignmentManager#processDeadServersAndRegionsInTransition, if > servershutdownhandler is processing and master consider it cluster startup, > master will assign all user regions, however, servershutdownhandler has not > completed splitting logs. > Let me describe it in more detail. > Suppose there are two regionservers A1 and B1, A1 carried META and ROOT > 1.master restart and completed assignRootAndMeta > 2.A1 and B1 are both restarted, new regionservers are A2 and B2 > 3.SSH which processed for A1 completed assigning ROOT and META > 4.master do rebuilding user regions and no regions added to master's region > list > 5.master consider it as a cluster startup, and assign all user regions > 6.SSH which processed for B1 start to split B1's logs > 7.All regions' data carried by B1 would loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4920) We need a mascot, a totem
[ https://issues.apache.org/jira/browse/HBASE-4920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402389#comment-13402389 ] Andrew Purtell commented on HBASE-4920: --- -1 on Honey Badger. I'm pretty ignorant of Internet memes, for full disclosure, but someone recently showed me the Youtube video behind this one and, frankly, it's one of the dumbest fucking things I've ever seen, to use a technical term. > We need a mascot, a totem > - > > Key: HBASE-4920 > URL: https://issues.apache.org/jira/browse/HBASE-4920 > Project: HBase > Issue Type: Task >Reporter: stack > Attachments: HBase Orca Logo.jpg, Orca_479990801.jpg, Screen shot > 2011-11-30 at 4.06.17 PM.png, apache hbase orca logo_Proof 3.pdf, apache > logo_Proof 8.pdf, krake.zip, more_orcas.png, more_orcas2.png, photo (2).JPG, > plus_orca.png > > > We need a totem for our t-shirt that is yet to be printed. O'Reilly owns the > Clyesdale. We need something else. > We could have a fluffy little duck that quacks 'hbase!' when you squeeze it > and we could order boxes of them from some off-shore sweatshop that > subcontracts to a contractor who employs child labor only. > Or we could have an Orca (Big!, Fast!, Killer!, and in a poem that Marcy from > Salesforce showed me, that was a bit too spiritual for me to be seen quoting > here, it had the Orca as the 'Guardian of the Cosmic Memory': i.e. in > translation, bigdata). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6170) Timeouts for row lock and scan should be separate
[ https://issues.apache.org/jira/browse/HBASE-6170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402397#comment-13402397 ] Jesse Yates commented on HBASE-6170: Are we sure that this didn't introduce any findbugs problems that might be a problem? I thought we had fixed the findbugs new-problem finding, so we need to make sure that no new bugs were actually introduced or if they were, that we consider them ok. > Timeouts for row lock and scan should be separate > - > > Key: HBASE-6170 > URL: https://issues.apache.org/jira/browse/HBASE-6170 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.94.0 >Reporter: Otis Gospodnetic >Assignee: Chris Trezzo >Priority: Minor > Fix For: 0.96.0 > > Attachments: 6170.addendum, 6170v4.patch, HBASE-6170v1.patch, > HBASE-6170v1.patch, HBASE-6170v1.patch, HBASE-6170v2.patch, HBASE-6170v3.patch > > > Apparently the timeout used for row locking and for scanning is global. It > would be better to have two separate timeouts. > (opening the issue to make Lars George happy) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6205) Support an option to keep data of dropped table for some time
[ https://issues.apache.org/jira/browse/HBASE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402404#comment-13402404 ] Jesse Yates commented on HBASE-6205: @Lars HBASE-5547 might not fully subsume this - conceptually we could have our own trash directory that just has a bunch of references files the original HFiles (like what I'm doing with snapshots in HBASE-6055). You would need to hack Reference.java a little bit to support full file references (or just use what I wrote: https://github.com/jyates/hbase/blob/timestamp-snapshots-r0/hbase-server/src/main/java/org/apache/hadoop/hbase/io/Reference.java). This way you don't end up doing any actual file copying, but still can keep track of the files. > Support an option to keep data of dropped table for some time > - > > Key: HBASE-6205 > URL: https://issues.apache.org/jira/browse/HBASE-6205 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.94.0, 0.96.0 >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.96.0 > > Attachments: HBASE-6205.patch, HBASE-6205v2.patch, > HBASE-6205v3.patch, HBASE-6205v4.patch, HBASE-6205v5.patch > > > User may drop table accidentally because of error code or other uncertain > reasons. > Unfortunately, it happens in our environment because one user make a mistake > between production cluster and testing cluster. > So, I just give a suggestion, do we need to support an option to keep data of > dropped table for some time, e.g. 1 day > In the patch: > We make a new dir named .trashtables in the rood dir. > In the DeleteTableHandler, we move files in dropped table's dir to trash > table dir instead of deleting them directly. > And Create new class TrashCleaner which will clean dropped tables if it is > time out with a period check. > Default keep time for dropped tables is 1 day, and check period is 1 hour. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6227) SSH and cluster startup causes data loss
[ https://issues.apache.org/jira/browse/HBASE-6227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402405#comment-13402405 ] Hudson commented on HBASE-6227: --- Integrated in HBase-0.94 #285 (See [https://builds.apache.org/job/HBase-0.94/285/]) HBASE-6227 SSH and cluster startup causes data loss Submitted by:Chunhui Reviewed by:Ted, Ram (Revision 1354635) Result = SUCCESS ramkrishna : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java > SSH and cluster startup causes data loss > - > > Key: HBASE-6227 > URL: https://issues.apache.org/jira/browse/HBASE-6227 > Project: HBase > Issue Type: Bug > Components: master >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.96.0, 0.94.1 > > Attachments: HBASE-6227.patch, HBASE-6227v2-94.patch, > HBASE-6227v2.patch > > > In AssignmentManager#processDeadServersAndRegionsInTransition, if > servershutdownhandler is processing and master consider it cluster startup, > master will assign all user regions, however, servershutdownhandler has not > completed splitting logs. > Let me describe it in more detail. > Suppose there are two regionservers A1 and B1, A1 carried META and ROOT > 1.master restart and completed assignRootAndMeta > 2.A1 and B1 are both restarted, new regionservers are A2 and B2 > 3.SSH which processed for A1 completed assigning ROOT and META > 4.master do rebuilding user regions and no regions added to master's region > list > 5.master consider it as a cluster startup, and assign all user regions > 6.SSH which processed for B1 start to split B1's logs > 7.All regions' data carried by B1 would loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6170) Timeouts for row lock and scan should be separate
[ https://issues.apache.org/jira/browse/HBASE-6170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402407#comment-13402407 ] Chris Trezzo commented on HBASE-6170: - I greped through the FindBugs report and I didn't really see anything that was related. It was complaining about synchronization in the leases class, but that was there before the change. > Timeouts for row lock and scan should be separate > - > > Key: HBASE-6170 > URL: https://issues.apache.org/jira/browse/HBASE-6170 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.94.0 >Reporter: Otis Gospodnetic >Assignee: Chris Trezzo >Priority: Minor > Fix For: 0.96.0 > > Attachments: 6170.addendum, 6170v4.patch, HBASE-6170v1.patch, > HBASE-6170v1.patch, HBASE-6170v1.patch, HBASE-6170v2.patch, HBASE-6170v3.patch > > > Apparently the timeout used for row locking and for scanning is global. It > would be better to have two separate timeouts. > (opening the issue to make Lars George happy) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5547) Don't delete HFiles when in "backup mode"
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402412#comment-13402412 ] Jesse Yates commented on HBASE-5547: Anyone else have any objections to Lar's proposal? Otherwise I'd like to get hacking on it this afternoon. > Don't delete HFiles when in "backup mode" > - > > Key: HBASE-5547 > URL: https://issues.apache.org/jira/browse/HBASE-5547 > Project: HBase > Issue Type: New Feature >Reporter: Lars Hofhansl >Assignee: Jesse Yates > Attachments: hbase-5447-v8.patch, hbase-5447-v8.patch, > java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch, > java_HBASE-5547_v7.patch > > > This came up in a discussion I had with Stack. > It would be nice if HBase could be notified that a backup is in progress (via > a znode for example) and in that case either: > 1. rename HFiles to be delete to .bck > 2. rename the HFiles into a special directory > 3. rename them to a general trash directory (which would not need to be tied > to backup mode). > That way it should be able to get a consistent backup based on HFiles (HDFS > snapshots or hard links would be better options here, but we do not have > those). > #1 makes cleanup a bit harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6279) Remove the RpcController argument from the PB methods' signatures
[ https://issues.apache.org/jira/browse/HBASE-6279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402421#comment-13402421 ] Devaraj Das commented on HBASE-6279: So in those cases where the interface extends proto-generated code, it seems like the RpcController cannot be avoided (at least I don't know if there is a way). For example, see AdminProtocol (in trunk) - it extends AdminService.BlockingInterface... But in the cases where the interface is written by hand (for example, HMasterInterface) the RpcController argument can be avoided. It makes sense to extend the proto-generated code to reduce maintenance burden. But yeah, then the RpcController will stick around.. [Meant to reply back yesterday, but forgot. Sorry] > Remove the RpcController argument from the PB methods' signatures > - > > Key: HBASE-6279 > URL: https://issues.apache.org/jira/browse/HBASE-6279 > Project: HBase > Issue Type: Bug >Reporter: Devaraj Das > > I don't see a reason for having the RPC methods (like HMasterInterface) > define methods with the RpcController argument (which is always passed as > null during invocation). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6261) Better approximate high-percentile percentile latency metrics
[ https://issues.apache.org/jira/browse/HBASE-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402429#comment-13402429 ] Andrew Wang commented on HBASE-6261: @Elliot: Moving averages can be cheaply computed on the existing reservoir sample, this is more about percentiles. I'm not sure how OpenTSDB factors into this, since you'd have to feed the latency stream to OpenTSDB to figure out percentiles, which seems expensive. Depending on how tight your speed and memory constraints are, I think we could do this in HBase at acceptably minimal cost, or make this configurable somehow. @Ted: The additional cost to do sliding windows is somewhat significant (I think 10s of MB more memory). Both the sliding and non-sliding methods allow for arbitrary percentiles. Anyway, I think reporting the 50th, 90th, 95th, and 99th should satisfy anyone. Mixing and matching algorithms is possible and probably even advised since it's only worth doing this for high-rate streams where accuracy is important. Implementations of the cheaper and less accurate algos are already available. > Better approximate high-percentile percentile latency metrics > - > > Key: HBASE-6261 > URL: https://issues.apache.org/jira/browse/HBASE-6261 > Project: HBase > Issue Type: New Feature >Reporter: Andrew Wang > Labels: metrics > Attachments: Latencyestimation.pdf > > > The existing reservoir-sampling based latency metrics in HBase are not > well-suited for providing accurate estimates of high-percentile (e.g. 90th, > 95th, or 99th) latency. This is a well-studied problem in the literature (see > [1] and [2]), the question is determining which methods best suit our needs > and then implementing it. > Ideally, we should be able to estimate these high percentiles with minimal > memory and CPU usage as well as minimal error (e.g. 1% error on 90th, or .1% > on 99th). It's also desirable to provide this over different time-based > sliding windows, e.g. last 1 min, 5 mins, 15 mins, and 1 hour. > I'll note that this would also be useful in HDFS, or really anywhere latency > metrics are kept. > [1] http://www.cs.rutgers.edu/~muthu/bquant.pdf > [2] http://infolab.stanford.edu/~manku/papers/04pods-sliding.pdf -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6205) Support an option to keep data of dropped table for some time
[ https://issues.apache.org/jira/browse/HBASE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402436#comment-13402436 ] Enis Soztutar commented on HBASE-6205: -- Considering this, HBASE-5547, and snapshots, it seems that we can decouple file management, and region-file association. We can build a very lightweight file manager, and remove all file deletion code from RS code. As in the bigtable design, we can keep the current hfile's (and WAL's) of the regions in META, and RS flushes, or rolls the log, adds the file reference at META. Then for a snapshot or a backup, we just need a point-in-time snapshot of the META table. A master thread can periodically scan the META, and META snapshots and the hdfs directories, and delete the files with 0 reference based on a policy. And deleting the table will just take a META snapshot for the table, and delete the META entries afterwards. This META snapshot will be kept for a while (similar to the normal snapshot retention). WDYT, how crazy is this? > Support an option to keep data of dropped table for some time > - > > Key: HBASE-6205 > URL: https://issues.apache.org/jira/browse/HBASE-6205 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.94.0, 0.96.0 >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.96.0 > > Attachments: HBASE-6205.patch, HBASE-6205v2.patch, > HBASE-6205v3.patch, HBASE-6205v4.patch, HBASE-6205v5.patch > > > User may drop table accidentally because of error code or other uncertain > reasons. > Unfortunately, it happens in our environment because one user make a mistake > between production cluster and testing cluster. > So, I just give a suggestion, do we need to support an option to keep data of > dropped table for some time, e.g. 1 day > In the patch: > We make a new dir named .trashtables in the rood dir. > In the DeleteTableHandler, we move files in dropped table's dir to trash > table dir instead of deleting them directly. > And Create new class TrashCleaner which will clean dropped tables if it is > time out with a period check. > Default keep time for dropped tables is 1 day, and check period is 1 hour. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4920) We need a mascot, a totem
[ https://issues.apache.org/jira/browse/HBASE-4920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402441#comment-13402441 ] Jonathan Hsieh commented on HBASE-4920: --- I'm also -1 for honey badger, for related technical reasons. :) > We need a mascot, a totem > - > > Key: HBASE-4920 > URL: https://issues.apache.org/jira/browse/HBASE-4920 > Project: HBase > Issue Type: Task >Reporter: stack > Attachments: HBase Orca Logo.jpg, Orca_479990801.jpg, Screen shot > 2011-11-30 at 4.06.17 PM.png, apache hbase orca logo_Proof 3.pdf, apache > logo_Proof 8.pdf, krake.zip, more_orcas.png, more_orcas2.png, photo (2).JPG, > plus_orca.png > > > We need a totem for our t-shirt that is yet to be printed. O'Reilly owns the > Clyesdale. We need something else. > We could have a fluffy little duck that quacks 'hbase!' when you squeeze it > and we could order boxes of them from some off-shore sweatshop that > subcontracts to a contractor who employs child labor only. > Or we could have an Orca (Big!, Fast!, Killer!, and in a poem that Marcy from > Salesforce showed me, that was a bit too spiritual for me to be seen quoting > here, it had the Orca as the 'Guardian of the Cosmic Memory': i.e. in > translation, bigdata). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6205) Support an option to keep data of dropped table for some time
[ https://issues.apache.org/jira/browse/HBASE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402442#comment-13402442 ] Zhihong Ted Yu commented on HBASE-6205: --- Interesting idea. .META. scanning is not atomic. How about storing this information in zookeeper ? > Support an option to keep data of dropped table for some time > - > > Key: HBASE-6205 > URL: https://issues.apache.org/jira/browse/HBASE-6205 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.94.0, 0.96.0 >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.96.0 > > Attachments: HBASE-6205.patch, HBASE-6205v2.patch, > HBASE-6205v3.patch, HBASE-6205v4.patch, HBASE-6205v5.patch > > > User may drop table accidentally because of error code or other uncertain > reasons. > Unfortunately, it happens in our environment because one user make a mistake > between production cluster and testing cluster. > So, I just give a suggestion, do we need to support an option to keep data of > dropped table for some time, e.g. 1 day > In the patch: > We make a new dir named .trashtables in the rood dir. > In the DeleteTableHandler, we move files in dropped table's dir to trash > table dir instead of deleting them directly. > And Create new class TrashCleaner which will clean dropped tables if it is > time out with a period check. > Default keep time for dropped tables is 1 day, and check period is 1 hour. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6261) Better approximate high-percentile percentile latency metrics
[ https://issues.apache.org/jira/browse/HBASE-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402443#comment-13402443 ] Elliott Clark commented on HBASE-6261: -- Basically I'm saying that I don't think that sliding windows are useful since most things that consume the metrics can do a moving average, which performs a very similar job as sliding windows. > Better approximate high-percentile percentile latency metrics > - > > Key: HBASE-6261 > URL: https://issues.apache.org/jira/browse/HBASE-6261 > Project: HBase > Issue Type: New Feature >Reporter: Andrew Wang > Labels: metrics > Attachments: Latencyestimation.pdf > > > The existing reservoir-sampling based latency metrics in HBase are not > well-suited for providing accurate estimates of high-percentile (e.g. 90th, > 95th, or 99th) latency. This is a well-studied problem in the literature (see > [1] and [2]), the question is determining which methods best suit our needs > and then implementing it. > Ideally, we should be able to estimate these high percentiles with minimal > memory and CPU usage as well as minimal error (e.g. 1% error on 90th, or .1% > on 99th). It's also desirable to provide this over different time-based > sliding windows, e.g. last 1 min, 5 mins, 15 mins, and 1 hour. > I'll note that this would also be useful in HDFS, or really anywhere latency > metrics are kept. > [1] http://www.cs.rutgers.edu/~muthu/bquant.pdf > [2] http://infolab.stanford.edu/~manku/papers/04pods-sliding.pdf -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6268) Can't enable a table on a 0.94 cluster from a 0.92 client
[ https://issues.apache.org/jira/browse/HBASE-6268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402449#comment-13402449 ] Jean-Daniel Cryans commented on HBASE-6268: --- This is why I'm not a big fan of backports that are not simple-ish fixes... So to me the most important would be to fix the 0.92 side as usually the infrastructure in place will rely on the older version first. In our case we started the migration of our clusters to 0.94 starting with the less important ones so our tools are still relying on 0.92 > Can't enable a table on a 0.94 cluster from a 0.92 client > - > > Key: HBASE-6268 > URL: https://issues.apache.org/jira/browse/HBASE-6268 > Project: HBase > Issue Type: Bug >Reporter: Jean-Daniel Cryans > Fix For: 0.92.2 > > > In 0.92 we know a table's enabled by doing this in HCM.isEnabledTable: > bq. return getTableState(zkw, tableName) == null; > In 0.94 we do: > bq. return getTableState(zkw, tableName) == TableState.ENABLED; > So what happens is that the the 0.92 client will hang forever since the znode > contains ENABLED instead of being absent. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6205) Support an option to keep data of dropped table for some time
[ https://issues.apache.org/jira/browse/HBASE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402453#comment-13402453 ] Andrew Purtell commented on HBASE-6205: --- bq. .META. scanning is not atomic. How about storing this information in zookeeper ? We could use ZK for coordination but would also need to store back after to .META., so we can repopulate ZK in cold boot scenarios. So... both? > Support an option to keep data of dropped table for some time > - > > Key: HBASE-6205 > URL: https://issues.apache.org/jira/browse/HBASE-6205 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.94.0, 0.96.0 >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.96.0 > > Attachments: HBASE-6205.patch, HBASE-6205v2.patch, > HBASE-6205v3.patch, HBASE-6205v4.patch, HBASE-6205v5.patch > > > User may drop table accidentally because of error code or other uncertain > reasons. > Unfortunately, it happens in our environment because one user make a mistake > between production cluster and testing cluster. > So, I just give a suggestion, do we need to support an option to keep data of > dropped table for some time, e.g. 1 day > In the patch: > We make a new dir named .trashtables in the rood dir. > In the DeleteTableHandler, we move files in dropped table's dir to trash > table dir instead of deleting them directly. > And Create new class TrashCleaner which will clean dropped tables if it is > time out with a period check. > Default keep time for dropped tables is 1 day, and check period is 1 hour. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6241) HBaseCluster interface for interacting with the cluster from system tests
[
https://issues.apache.org/jira/browse/HBASE-6241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402458#comment-13402458
]
Zhihong Ted Yu commented on HBASE-6241:
---
{code}
+ cluster.abortRegionServer(selected);
+ sleep(ONE_MIN);
+ cluster.startRegionServer(servers[selected].getHostname());
+ sleep(ONE_MIN);
{code}
Is there better way to detect the stop of region server has completed ?
{code}
+ public static enum ServiceType {
{code}
Should job/task tracker be in the enum as well ?
{code}
+ public abstract boolean signal(ServiceType service, String signal, String
hostname) throws IOException;
{code}
Line length should be 100 chars or shorter.
For HBaseCluster.java, remove year in license.
{code}
+ * @return Name of master that just went down.
+ */
+ public abstract String waitOnMaster(final int serverNumber);
{code}
If no enum argument is planned to be added, the method can be renamed as
waitOnMasterToStop().
{code}
+ public void restoreInitialState() throws IOException {
{code}
How about restoring to one of the ClusterStatus previously captured by
getClusterStatus() ? (future enhancement)
I think getInitialClusterStatus() and getClusterStatus() can be unified into
one method with enum indicating the type of status.
{code}
+ * @return Index into List of {@link
MiniHBaseCluster#getRegionServerThreads()}
+ * of HRS carrying .META.. Returns -1 if none found.
+ */
+ public abstract int getServerWith(byte[] regionName) throws IOException;
{code}
The return value isn't just for .META. server. Its javadoc, should be improved
- considering MiniHBaseCluster being a subclass of HBaseCluster.
More review comments to follow.
> HBaseCluster interface for interacting with the cluster from system tests
> --
>
> Key: HBASE-6241
> URL: https://issues.apache.org/jira/browse/HBASE-6241
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: HBASE-6241_v0.2.patch
>
>
> We need to abstract away the cluster interactions for system tests running on
> actual clusters.
> MiniHBaseCluster and RealHBaseCluster should both implement this interface,
> and system tests should work with both.
> I'll split Devaraj's patch in HBASE-6053 for the initial version.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2730) Expose RS work queue contents on web UI
[ https://issues.apache.org/jira/browse/HBASE-2730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402461#comment-13402461 ] Andrew Purtell commented on HBASE-2730: --- bq. By default, "hbase.regionserver.servlet.show.queuedump" equals false to disable this feature Why not add this to the debug dump? Why do we need a separate dump/report? Why should it be disabled by default? > Expose RS work queue contents on web UI > --- > > Key: HBASE-2730 > URL: https://issues.apache.org/jira/browse/HBASE-2730 > Project: HBase > Issue Type: New Feature > Components: monitoring, regionserver >Reporter: Todd Lipcon >Priority: Critical > Fix For: 0.96.0 > > Attachments: hbase-2730-0_94_0.patch > > > Would be nice to be able to see the contents of the various work queues - eg > to know what regions are pending compaction/split/flush/etc. This is handy > for debugging why a region might be blocked, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6227) SSH and cluster startup causes data loss
[ https://issues.apache.org/jira/browse/HBASE-6227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402466#comment-13402466 ] Hudson commented on HBASE-6227: --- Integrated in HBase-TRUNK #3082 (See [https://builds.apache.org/job/HBase-TRUNK/3082/]) HBASE-6227 SSH and cluster startup causes data loss Submitted by:Chunhui Reviewed by:Ram, Ted (Revision 1354634) Result = SUCCESS ramkrishna : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java > SSH and cluster startup causes data loss > - > > Key: HBASE-6227 > URL: https://issues.apache.org/jira/browse/HBASE-6227 > Project: HBase > Issue Type: Bug > Components: master >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.96.0, 0.94.1 > > Attachments: HBASE-6227.patch, HBASE-6227v2-94.patch, > HBASE-6227v2.patch > > > In AssignmentManager#processDeadServersAndRegionsInTransition, if > servershutdownhandler is processing and master consider it cluster startup, > master will assign all user regions, however, servershutdownhandler has not > completed splitting logs. > Let me describe it in more detail. > Suppose there are two regionservers A1 and B1, A1 carried META and ROOT > 1.master restart and completed assignRootAndMeta > 2.A1 and B1 are both restarted, new regionservers are A2 and B2 > 3.SSH which processed for A1 completed assigning ROOT and META > 4.master do rebuilding user regions and no regions added to master's region > list > 5.master consider it as a cluster startup, and assign all user regions > 6.SSH which processed for B1 start to split B1's logs > 7.All regions' data carried by B1 would loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6273) HMasterInterface.isMasterRunning() requires clean up
[ https://issues.apache.org/jira/browse/HBASE-6273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402480#comment-13402480 ] Jean-Daniel Cryans commented on HBASE-6273: --- +1 > HMasterInterface.isMasterRunning() requires clean up > > > Key: HBASE-6273 > URL: https://issues.apache.org/jira/browse/HBASE-6273 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.94.0 >Reporter: ramkrishna.s.vasudevan >Assignee: ramkrishna.s.vasudevan > Fix For: 0.96.0 > > > This JIRA is in reference to JD's comments regarding the clean up needed in > isMasterRunning(). Refer to > https://issues.apache.org/jira/browse/HBASE-6240?focusedCommentId=13400772&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13400772 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6279) Remove the RpcController argument from the PB methods' signatures
[
https://issues.apache.org/jira/browse/HBASE-6279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402482#comment-13402482
]
Andrew Purtell commented on HBASE-6279:
---
> It makes sense to extend the proto-generated code to reduce maintenance
> burden.
Maybe not. From https://developers.google.com/protocol-buffers/docs/cpptutorial:
{quote}
If you want to add richer behaviour to a generated class, the best way to do
this is to wrap the generated protocol buffer class in an application-specific
class. ... You should never add behaviour to the generated classes by
inheriting from them. This will break internal mechanisms and is not good
object-oriented practice anyway.
{quote}
Of course this refers to the message classes, but regards protobuf's "RPC
support" why not just use the message objects and leave out the RPC bits from
proto files? There is no actual RPC implementation provided by the protobuf
compiler or library, and there are classes such as RpcController that don't
make sense and are polluting the interfaces.
> Remove the RpcController argument from the PB methods' signatures
> -
>
> Key: HBASE-6279
> URL: https://issues.apache.org/jira/browse/HBASE-6279
> Project: HBase
> Issue Type: Bug
>Reporter: Devaraj Das
>
> I don't see a reason for having the RPC methods (like HMasterInterface)
> define methods with the RpcController argument (which is always passed as
> null during invocation).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6282) The introspection, etc. of objects in the RPC has to be handled for PB objects
Devaraj Das created HBASE-6282: -- Summary: The introspection, etc. of objects in the RPC has to be handled for PB objects Key: HBASE-6282 URL: https://issues.apache.org/jira/browse/HBASE-6282 Project: HBase Issue Type: Bug Components: ipc Reporter: Devaraj Das Fix For: 0.96.0 The places where the type of objects are inspected need to be updated to take into consideration PB types. I have noticed Objects.describeQuantity being used, and the private WritableRpcEngine.Server.logResponse method also needs updating (in the PB world, all information about operations/tablenames is contained in one PB argument). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4920) We need a mascot, a totem
[ https://issues.apache.org/jira/browse/HBASE-4920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402498#comment-13402498 ] Roman Shaposhnik commented on HBASE-4920: - @Jonathan "Obviously, You're Not a Golfer" (tm) ;-) > We need a mascot, a totem > - > > Key: HBASE-4920 > URL: https://issues.apache.org/jira/browse/HBASE-4920 > Project: HBase > Issue Type: Task >Reporter: stack > Attachments: HBase Orca Logo.jpg, Orca_479990801.jpg, Screen shot > 2011-11-30 at 4.06.17 PM.png, apache hbase orca logo_Proof 3.pdf, apache > logo_Proof 8.pdf, krake.zip, more_orcas.png, more_orcas2.png, photo (2).JPG, > plus_orca.png > > > We need a totem for our t-shirt that is yet to be printed. O'Reilly owns the > Clyesdale. We need something else. > We could have a fluffy little duck that quacks 'hbase!' when you squeeze it > and we could order boxes of them from some off-shore sweatshop that > subcontracts to a contractor who employs child labor only. > Or we could have an Orca (Big!, Fast!, Killer!, and in a poem that Marcy from > Salesforce showed me, that was a bit too spiritual for me to be seen quoting > here, it had the Orca as the 'Guardian of the Cosmic Memory': i.e. in > translation, bigdata). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6283) [region_mover.rb] Add option to exclude list of hosts on unload instead of just assuming the source node.
Jonathan Hsieh created HBASE-6283: - Summary: [region_mover.rb] Add option to exclude list of hosts on unload instead of just assuming the source node. Key: HBASE-6283 URL: https://issues.apache.org/jira/browse/HBASE-6283 Project: HBase Issue Type: Improvement Reporter: Jonathan Hsieh Currently, the region_mover.rb script excludes a single host, the host offloading data, as a region move target. This essentially limits the number of machines that can be shutdown at a time to one. For larger clusters, it is manageable to have several nodes down at a time and desirable to get this process done more quickly. The proposed patch adds an exclude file option, that allows multiple hosts to be excluded as targets. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HBASE-6283) [region_mover.rb] Add option to exclude list of hosts on unload instead of just assuming the source node.
[ https://issues.apache.org/jira/browse/HBASE-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh reassigned HBASE-6283: - Assignee: Jonathan Hsieh > [region_mover.rb] Add option to exclude list of hosts on unload instead of > just assuming the source node. > - > > Key: HBASE-6283 > URL: https://issues.apache.org/jira/browse/HBASE-6283 > Project: HBase > Issue Type: Improvement >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Labels: jruby > > Currently, the region_mover.rb script excludes a single host, the host > offloading data, as a region move target. This essentially limits the number > of machines that can be shutdown at a time to one. For larger clusters, it > is manageable to have several nodes down at a time and desirable to get this > process done more quickly. > The proposed patch adds an exclude file option, that allows multiple hosts to > be excluded as targets. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6269) Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
[
https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhihong Ted Yu updated HBASE-6269:
--
Status: Open (was: Patch Available)
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest
> KeyValue
> -
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.94.0
>Reporter: ShiXing
>Assignee: ShiXing
> Attachments: HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch,
> runAllTests.out.txt
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test
> case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp
> and different memstoreTS, after each Put/Increment, we do a memstore flush.
> So there will be two StoreFile with same KeyValue(except memstoreTS and
> SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value2 : value2
> get value after flush after put value2 : value0
> {code}
> I analyze the code for StoreFileScanner with lazy se
[jira] [Commented] (HBASE-6269) Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
[
https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402523#comment-13402523
]
Zhihong Ted Yu commented on HBASE-6269:
---
Integrated to trunk.
Thanks for the patch, Xing.
Thanks for the review, Ramkrishna and Annop.
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest
> KeyValue
> -
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.94.0
>Reporter: ShiXing
>Assignee: ShiXing
> Attachments: HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch,
> runAllTests.out.txt
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test
> case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp
> and different memstoreTS, after each Put/Increment, we do a memstore flush.
> So there will be two StoreFile with same KeyValue(except memstoreTS and
> SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> }
> {code}
> and the result print as followed:
> {code}
> get value before flush after put value0 : value0
> get value after flush after put value0 : value0
> get value before flush after put value1 : value1
> get value after flush after put value1 : value0
> get value before flush after put value
[jira] [Updated] (HBASE-6283) [region_mover.rb] Add option to exclude list of hosts on unload instead of just assuming the source node.
[ https://issues.apache.org/jira/browse/HBASE-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-6283: -- Attachment: region_mover.rb Patch for trunk. Was originally developed and tested against 0.90 (should apply to all active branches). Manually tested by having TestLoadAndVerify running and executing region_mover.rb unloads while running. > [region_mover.rb] Add option to exclude list of hosts on unload instead of > just assuming the source node. > - > > Key: HBASE-6283 > URL: https://issues.apache.org/jira/browse/HBASE-6283 > Project: HBase > Issue Type: Improvement >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Labels: jruby > Attachments: region_mover.rb > > > Currently, the region_mover.rb script excludes a single host, the host > offloading data, as a region move target. This essentially limits the number > of machines that can be shutdown at a time to one. For larger clusters, it > is manageable to have several nodes down at a time and desirable to get this > process done more quickly. > The proposed patch adds an exclude file option, that allows multiple hosts to > be excluded as targets. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6283) [region_mover.rb] Add option to exclude list of hosts on unload instead of just assuming the source node.
[ https://issues.apache.org/jira/browse/HBASE-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-6283: -- Attachment: hbase-6283.patch > [region_mover.rb] Add option to exclude list of hosts on unload instead of > just assuming the source node. > - > > Key: HBASE-6283 > URL: https://issues.apache.org/jira/browse/HBASE-6283 > Project: HBase > Issue Type: Improvement >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Labels: jruby > Attachments: hbase-6283.patch > > > Currently, the region_mover.rb script excludes a single host, the host > offloading data, as a region move target. This essentially limits the number > of machines that can be shutdown at a time to one. For larger clusters, it > is manageable to have several nodes down at a time and desirable to get this > process done more quickly. > The proposed patch adds an exclude file option, that allows multiple hosts to > be excluded as targets. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6283) [region_mover.rb] Add option to exclude list of hosts on unload instead of just assuming the source node.
[ https://issues.apache.org/jira/browse/HBASE-6283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Hsieh updated HBASE-6283: -- Attachment: (was: region_mover.rb) > [region_mover.rb] Add option to exclude list of hosts on unload instead of > just assuming the source node. > - > > Key: HBASE-6283 > URL: https://issues.apache.org/jira/browse/HBASE-6283 > Project: HBase > Issue Type: Improvement >Reporter: Jonathan Hsieh >Assignee: Jonathan Hsieh > Labels: jruby > Attachments: hbase-6283.patch > > > Currently, the region_mover.rb script excludes a single host, the host > offloading data, as a region move target. This essentially limits the number > of machines that can be shutdown at a time to one. For larger clusters, it > is manageable to have several nodes down at a time and desirable to get this > process done more quickly. > The proposed patch adds an exclude file option, that allows multiple hosts to > be excluded as targets. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6279) Remove the RpcController argument from the PB methods' signatures
[ https://issues.apache.org/jira/browse/HBASE-6279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402535#comment-13402535 ] Devaraj Das commented on HBASE-6279: Yeah, I agree that we don't need the RPC bits in the proto files. We could remove them, and instead define the RPCs with appropriate PB messages in the Java files... > Remove the RpcController argument from the PB methods' signatures > - > > Key: HBASE-6279 > URL: https://issues.apache.org/jira/browse/HBASE-6279 > Project: HBase > Issue Type: Bug >Reporter: Devaraj Das > > I don't see a reason for having the RPC methods (like HMasterInterface) > define methods with the RpcController argument (which is always passed as > null during invocation). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6241) HBaseCluster interface for interacting with the cluster from system tests
[
https://issues.apache.org/jira/browse/HBASE-6241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402544#comment-13402544
]
Zhihong Ted Yu commented on HBASE-6241:
---
{code}
+ * @return whether we are interacting with a real cluster as opposed to and
in-process mini
{code}
'and' -> 'an'
For HBaseClusterManager, I see enum STATUS but don't see method which issues
that command.
For HBaseTestingUtility.java:
{code}
+} catch (Exception ex) {
+ throw new RuntimeException(ex);
+}
{code}
The above doesn't seem necessary.
For RealHBaseCluster.java, please add license.
For waitForActiveAndReadyMaster():
{code}
+ } catch (MasterNotRunningException m) {
+LOG.warn("Master not started yet " + m);
{code}
Does it make sense to introduce timeout since loop may wait indefinitely.
> HBaseCluster interface for interacting with the cluster from system tests
> --
>
> Key: HBASE-6241
> URL: https://issues.apache.org/jira/browse/HBASE-6241
> Project: HBase
> Issue Type: Sub-task
>Reporter: Enis Soztutar
>Assignee: Enis Soztutar
> Attachments: HBASE-6241_v0.2.patch
>
>
> We need to abstract away the cluster interactions for system tests running on
> actual clusters.
> MiniHBaseCluster and RealHBaseCluster should both implement this interface,
> and system tests should work with both.
> I'll split Devaraj's patch in HBASE-6053 for the initial version.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6284) Introduce HRegion#doMiniBatchDelete()
Zhihong Ted Yu created HBASE-6284: - Summary: Introduce HRegion#doMiniBatchDelete() Key: HBASE-6284 URL: https://issues.apache.org/jira/browse/HBASE-6284 Project: HBase Issue Type: Bug Reporter: Zhihong Ted Yu >From Anoop under thread 'Can there be a doMiniBatchDelete in HRegion': The HTable#delete(List) groups the Deletes for the same RS and make one n/w call only. But within the RS, there will be N number of delete calls on the region one by one. This will include N number of HLog write and sync. If this also can be grouped can we get better performance for the multi row delete. I have made the new miniBatchDelete () and made the HTable#delete(List) to call this new batch delete. Just tested initially with the one node cluster. In that itself I am getting a performance boost which is very much promising. Only one CF and qualifier. 10K total rows delete with a batch of 100 deletes. Only deletes happening on the table from one thread. With the new way the net time taken is reduced by more than 1/10 Will test in a 4 node cluster also. I think it will worth doing this change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6285) HBase master should log INFO message when it attempts to assign a region
Aditya Kishore created HBASE-6285: - Summary: HBase master should log INFO message when it attempts to assign a region Key: HBASE-6285 URL: https://issues.apache.org/jira/browse/HBASE-6285 Project: HBase Issue Type: Bug Components: master Affects Versions: 0.94.0 Reporter: Aditya Kishore Assignee: Aditya Kishore Priority: Minor With the default logging level (INFO), it is very difficult to diagnose a large HBase cluster that is having problems assigning regions because the HBase master logs a DEBUG message when it instructs a region-server to assign a region. You actually have to crawl EVERY HBase region-server log to find out which node received the request for a particular region. Further, lets say the HBase master sends the request and something goes wrong, we might not even get a message in the region-server log. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6285) HBase master should log INFO message when it attempts to assign a region
[ https://issues.apache.org/jira/browse/HBASE-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aditya Kishore updated HBASE-6285: -- Attachment: HBASE-6285_trunk.patch Patch for trunk with existing DEBUG log message changed to INFO. > HBase master should log INFO message when it attempts to assign a region > > > Key: HBASE-6285 > URL: https://issues.apache.org/jira/browse/HBASE-6285 > Project: HBase > Issue Type: Bug > Components: master >Affects Versions: 0.94.0 >Reporter: Aditya Kishore >Assignee: Aditya Kishore >Priority: Minor > Attachments: HBASE-6285_trunk.patch > > > With the default logging level (INFO), it is very difficult to diagnose a > large HBase cluster that is having problems assigning regions because the > HBase master logs a DEBUG message when it instructs a region-server to assign > a region. > You actually have to crawl EVERY HBase region-server log to find out which > node received the request for a particular region. Further, lets say the > HBase master sends the request and something goes wrong, we might not even > get a message in the region-server log. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-6286) Upgrade maven-compiler-plugin to 2.5.1
[ https://issues.apache.org/jira/browse/HBASE-6286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Purtell updated HBASE-6286: -- Attachment: HBASE-6286.patch > Upgrade maven-compiler-plugin to 2.5.1 > -- > > Key: HBASE-6286 > URL: https://issues.apache.org/jira/browse/HBASE-6286 > Project: HBase > Issue Type: Improvement > Components: build >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Attachments: HBASE-6286.patch > > > time mvn -PlocalTests clean install -DskipTests > With 2.5.1: > |user|1m35.634s|1m31.178s|1m31.366s| > |sys|0m06.540s|0m05.376s|0m05.488s| > With 2.0.2 (current): > |user|2m01.168s|1m54.027s|1m57.799s| > |sys|0m05.896s|0m05.912s|0m06.032s| -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6286) Upgrade maven-compiler-plugin to 2.5.1
Andrew Purtell created HBASE-6286: - Summary: Upgrade maven-compiler-plugin to 2.5.1 Key: HBASE-6286 URL: https://issues.apache.org/jira/browse/HBASE-6286 Project: HBase Issue Type: Improvement Components: build Reporter: Andrew Purtell Assignee: Andrew Purtell Priority: Minor Attachments: HBASE-6286.patch time mvn -PlocalTests clean install -DskipTests With 2.5.1: |user|1m35.634s|1m31.178s|1m31.366s| |sys|0m06.540s|0m05.376s|0m05.488s| With 2.0.2 (current): |user|2m01.168s|1m54.027s|1m57.799s| |sys|0m05.896s|0m05.912s|0m06.032s| -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6269) Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
[
https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402580#comment-13402580
]
Hudson commented on HBASE-6269:
---
Integrated in HBase-TRUNK #3083 (See
[https://builds.apache.org/job/HBase-TRUNK/3083/])
HBASE-6269 Lazyseek should use the maxSequenseId StoreFile's KeyValue as
the latest KeyValue (Xing Shi) (Revision 1354703)
Result = SUCCESS
tedyu :
Files :
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest
> KeyValue
> -
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.94.0
>Reporter: ShiXing
>Assignee: ShiXing
> Attachments: HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch,
> runAllTests.out.txt
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test
> case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp
> and different memstoreTS, after each Put/Increment, we do a memstore flush.
> So there will be two StoreFile with same KeyValue(except memstoreTS and
> SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toS
[jira] [Commented] (HBASE-2611) Handle RS that fails while processing the failure of another one
[
https://issues.apache.org/jira/browse/HBASE-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402582#comment-13402582
]
Zhihong Ted Yu commented on HBASE-2611:
---
Putting patch on review board helps.
{code}
+ * @param opList: list of Op to be executed as one trx.
{code}
'trx' -> 'transaction'
{code}
+if(opList == null || opList.size() ==0)
{code}
Space between if and (.
{code}
+}catch (InterruptedException ie) {
+ LOG.warn("multi call interrupted; process failed!" + ie);
{code}
Restore interrupt status for the thread (same for doMultiAndWatch). Space
between } and catch.
{code}
+ LOG.warn("multi call failed! One of the passed ops has failed which
result in the rolled back.");
{code}
Line length beyond 100.
{code}
+ * @return
+ */
+ public SortedMap> copyDeadRSLogsWithMulti(
+ String deadRSZnode) {
{code}
javadoc for the return value.
{code}
+ LOG.warn("This is us! Skipping the processing as we might be closing
down.");
{code}
Add deadRSZnodePath to the log.
{code}
+RetryCounterFactory retryCounterFactory = new
RetryCounterFactory(Integer.MAX_VALUE, 3 * 1000);
{code}
I don't think MAX_VALUE is a good choice.
{code}
+SortedSet logQueue = new TreeSet();
{code}
Why is logQueue backed by a TreeSet ?
{code}
+LOG.warn("KeeperException occurred in multi; " +
+"seems some other regionserver took the logs before us.");
{code}
Add ke to the above message.
{code}
+Op deleteOpForLog = Op.delete(zNodeForCurrentLog, -1);
+znodesToWatch.add(logZnode);
+opsList.add(createOpForLog);
+opsList.add(deleteOpForLog);
{code}
Please reorder the above calls so that znodesToWatch.add() is after
opsList.add() calls. This would make code more readable.
> Handle RS that fails while processing the failure of another one
>
>
> Key: HBASE-2611
> URL: https://issues.apache.org/jira/browse/HBASE-2611
> Project: HBase
> Issue Type: Sub-task
> Components: replication
>Reporter: Jean-Daniel Cryans
>Assignee: Jean-Daniel Cryans
> Attachments: HBase-2611-upstream-v1.patch
>
>
> HBASE-2223 doesn't manage region servers that fail while doing the transfer
> of HLogs queues from other region servers that failed. Devise a reliable way
> to do it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6286) Upgrade maven-compiler-plugin to 2.5.1
[ https://issues.apache.org/jira/browse/HBASE-6286?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402584#comment-13402584 ] Zhihong Ted Yu commented on HBASE-6286: --- +1 on patch. > Upgrade maven-compiler-plugin to 2.5.1 > -- > > Key: HBASE-6286 > URL: https://issues.apache.org/jira/browse/HBASE-6286 > Project: HBase > Issue Type: Improvement > Components: build >Reporter: Andrew Purtell >Assignee: Andrew Purtell >Priority: Minor > Attachments: HBASE-6286.patch > > > time mvn -PlocalTests clean install -DskipTests > With 2.5.1: > |user|1m35.634s|1m31.178s|1m31.366s| > |sys|0m06.540s|0m05.376s|0m05.488s| > With 2.0.2 (current): > |user|2m01.168s|1m54.027s|1m57.799s| > |sys|0m05.896s|0m05.912s|0m06.032s| -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6205) Support an option to keep data of dropped table for some time
[ https://issues.apache.org/jira/browse/HBASE-6205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402586#comment-13402586 ] Lars Hofhansl commented on HBASE-6205: -- Sounds like an awful amount of work for this feature. How about this: When a table is dropped we just delay that action by configurable amount of time (say 5 mins), during that time the drop action can be canceled. Could just store tables to be dropped in a system table. > Support an option to keep data of dropped table for some time > - > > Key: HBASE-6205 > URL: https://issues.apache.org/jira/browse/HBASE-6205 > Project: HBase > Issue Type: New Feature >Affects Versions: 0.94.0, 0.96.0 >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.96.0 > > Attachments: HBASE-6205.patch, HBASE-6205v2.patch, > HBASE-6205v3.patch, HBASE-6205v4.patch, HBASE-6205v5.patch > > > User may drop table accidentally because of error code or other uncertain > reasons. > Unfortunately, it happens in our environment because one user make a mistake > between production cluster and testing cluster. > So, I just give a suggestion, do we need to support an option to keep data of > dropped table for some time, e.g. 1 day > In the patch: > We make a new dir named .trashtables in the rood dir. > In the DeleteTableHandler, we move files in dropped table's dir to trash > table dir instead of deleting them directly. > And Create new class TrashCleaner which will clean dropped tables if it is > time out with a period check. > Default keep time for dropped tables is 1 day, and check period is 1 hour. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2611) Handle RS that fails while processing the failure of another one
[ https://issues.apache.org/jira/browse/HBASE-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402594#comment-13402594 ] Zhihong Ted Yu commented on HBASE-2611: --- Suppose there are (relatively) large number of Op's in opsList, the chance of collision between active region servers is high. Some experiments should be performed so that we get idea of how long this procedure takes to succeed. > Handle RS that fails while processing the failure of another one > > > Key: HBASE-2611 > URL: https://issues.apache.org/jira/browse/HBASE-2611 > Project: HBase > Issue Type: Sub-task > Components: replication >Reporter: Jean-Daniel Cryans >Assignee: Jean-Daniel Cryans > Attachments: HBase-2611-upstream-v1.patch > > > HBASE-2223 doesn't manage region servers that fail while doing the transfer > of HLogs queues from other region servers that failed. Devise a reliable way > to do it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5941) improve multiDelete performance by grabbing locks ahead of time
[ https://issues.apache.org/jira/browse/HBASE-5941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Ted Yu updated HBASE-5941: -- Attachment: 5941.txt > improve multiDelete performance by grabbing locks ahead of time > --- > > Key: HBASE-5941 > URL: https://issues.apache.org/jira/browse/HBASE-5941 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.89.20100924, 0.89-fb, 0.94.1 >Reporter: Amitanand Aiyer >Assignee: Amitanand Aiyer >Priority: Minor > Fix For: 0.89-fb > > Attachments: 5941.txt > > > Ning reported that the performance of deletes is slower than the performance > of Puts. This should not be the case. > On digging up, it turns out that there is a difference between multiPut and > multiDelete in the way we grab locks. > multiPut grabs all the locks optimistically and processes the puts one by > one. multiDelete grabs locks and releases > them one at a time, for each delete operation, as if it were done > separately. This may be causing a performance > slow down for deletes. Trying to improve it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5705) Introduce Protocol Buffer RPC engine
[ https://issues.apache.org/jira/browse/HBASE-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-5705: --- Attachment: 5705-1.patch Early patch that at least builds. Still working on it. > Introduce Protocol Buffer RPC engine > > > Key: HBASE-5705 > URL: https://issues.apache.org/jira/browse/HBASE-5705 > Project: HBase > Issue Type: Sub-task > Components: ipc, master, migration, regionserver >Reporter: Devaraj Das >Assignee: Devaraj Das > Attachments: 5705-1.patch > > > Introduce Protocol Buffer RPC engine in the RPC core. Protocols that are PB > aware can be made to go through this RPC engine. The approach, in my current > thinking, would be similar to HADOOP-7773. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6284) Introduce HRegion#doMiniBatchDelete()
[ https://issues.apache.org/jira/browse/HBASE-6284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402656#comment-13402656 ] Jeff Whiting commented on HBASE-6284: - While adding this it is probably worth looking at HBASE-5941 and incorporating its performance improvements as well. > Introduce HRegion#doMiniBatchDelete() > - > > Key: HBASE-6284 > URL: https://issues.apache.org/jira/browse/HBASE-6284 > Project: HBase > Issue Type: Bug >Reporter: Zhihong Ted Yu > > From Anoop under thread 'Can there be a doMiniBatchDelete in HRegion': > The HTable#delete(List) groups the Deletes for the same RS and make > one n/w call only. But within the RS, there will be N number of delete calls > on the region one by one. This will include N number of HLog write and sync. > If this also can be grouped can we get better performance for the multi row > delete. > I have made the new miniBatchDelete () and made the > HTable#delete(List) to call this new batch delete. > Just tested initially with the one node cluster. In that itself I am getting > a performance boost which is very much promising. > Only one CF and qualifier. > 10K total rows delete with a batch of 100 deletes. Only deletes happening on > the table from one thread. > With the new way the net time taken is reduced by more than 1/10 > Will test in a 4 node cluster also. I think it will worth doing this change. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine
[ https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-5732: --- Resolution: Fixed Fix Version/s: 0.96.0 Status: Resolved (was: Patch Available) > Remove the SecureRPCEngine and merge the security-related logic in the core > engine > -- > > Key: HBASE-5732 > URL: https://issues.apache.org/jira/browse/HBASE-5732 > Project: HBase > Issue Type: Improvement >Reporter: Devaraj Das >Assignee: Devaraj Das > Fix For: 0.96.0 > > Attachments: 5732-rpcengine-merge.11.patch, > 5732-rpcengine-merge.12.patch, 5732-rpcengine-merge.7.patch, > rpcengine-merge.10.patch, rpcengine-merge.11.patch, rpcengine-merge.12.patch, > rpcengine-merge.3.patch, rpcengine-merge.4.patch, rpcengine-merge.9.patch, > rpcengine-merge.patch > > > Remove the SecureRPCEngine and merge the security-related logic in the core > engine. Follow up to HBASE-5727. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5612) Data types for HBase values
[ https://issues.apache.org/jira/browse/HBASE-5612?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402677#comment-13402677 ] Mikhail Bautin commented on HBASE-5612: --- Enis: yes, it makes sense to develop a more fine-grained data model for rows/qualifiers/values, not just for values. > Data types for HBase values > --- > > Key: HBASE-5612 > URL: https://issues.apache.org/jira/browse/HBASE-5612 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Assignee: Mikhail Bautin > > In many real-life applications all values in a certain column family are of a > certain data type, e.g. 64-bit integer. We could specify that in the column > descriptor and enable data type-specific compression such as variable-length > integer encoding. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6287) Port HBASE-5941 improve multiDelete performance by grabbing locks ahead of time
Zhihong Ted Yu created HBASE-6287: - Summary: Port HBASE-5941 improve multiDelete performance by grabbing locks ahead of time Key: HBASE-6287 URL: https://issues.apache.org/jira/browse/HBASE-6287 Project: HBase Issue Type: Bug Reporter: Zhihong Ted Yu HBASE-5941 has been integrated to 0.89-fb This JIRA ports it to HBase trunk -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6170) Timeouts for row lock and scan should be separate
[ https://issues.apache.org/jira/browse/HBASE-6170?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402682#comment-13402682 ] Hudson commented on HBASE-6170: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #72 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/72/]) HBASE-6170 add the two new config parameters in hbase-default.xml (Revision 1354500) Result = FAILURE tedyu : Files : * /hbase/trunk/hbase-server/src/main/resources/hbase-default.xml > Timeouts for row lock and scan should be separate > - > > Key: HBASE-6170 > URL: https://issues.apache.org/jira/browse/HBASE-6170 > Project: HBase > Issue Type: Improvement > Components: regionserver >Affects Versions: 0.94.0 >Reporter: Otis Gospodnetic >Assignee: Chris Trezzo >Priority: Minor > Fix For: 0.96.0 > > Attachments: 6170.addendum, 6170v4.patch, HBASE-6170v1.patch, > HBASE-6170v1.patch, HBASE-6170v1.patch, HBASE-6170v2.patch, HBASE-6170v3.patch > > > Apparently the timeout used for row locking and for scanning is global. It > would be better to have two separate timeouts. > (opening the issue to make Lars George happy) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6227) SSH and cluster startup causes data loss
[ https://issues.apache.org/jira/browse/HBASE-6227?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402683#comment-13402683 ] Hudson commented on HBASE-6227: --- Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #72 (See [https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/72/]) HBASE-6227 SSH and cluster startup causes data loss Submitted by:Chunhui Reviewed by:Ram, Ted (Revision 1354634) Result = FAILURE ramkrishna : Files : * /hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java > SSH and cluster startup causes data loss > - > > Key: HBASE-6227 > URL: https://issues.apache.org/jira/browse/HBASE-6227 > Project: HBase > Issue Type: Bug > Components: master >Reporter: chunhui shen >Assignee: chunhui shen > Fix For: 0.96.0, 0.94.1 > > Attachments: HBASE-6227.patch, HBASE-6227v2-94.patch, > HBASE-6227v2.patch > > > In AssignmentManager#processDeadServersAndRegionsInTransition, if > servershutdownhandler is processing and master consider it cluster startup, > master will assign all user regions, however, servershutdownhandler has not > completed splitting logs. > Let me describe it in more detail. > Suppose there are two regionservers A1 and B1, A1 carried META and ROOT > 1.master restart and completed assignRootAndMeta > 2.A1 and B1 are both restarted, new regionservers are A2 and B2 > 3.SSH which processed for A1 completed assigning ROOT and META > 4.master do rebuilding user regions and no regions added to master's region > list > 5.master consider it as a cluster startup, and assign all user regions > 6.SSH which processed for B1 start to split B1's logs > 7.All regions' data carried by B1 would loss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6269) Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest KeyValue
[
https://issues.apache.org/jira/browse/HBASE-6269?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402684#comment-13402684
]
Hudson commented on HBASE-6269:
---
Integrated in HBase-TRUNK-on-Hadoop-2.0.0 #72 (See
[https://builds.apache.org/job/HBase-TRUNK-on-Hadoop-2.0.0/72/])
HBASE-6269 Lazyseek should use the maxSequenseId StoreFile's KeyValue as
the latest KeyValue (Xing Shi) (Revision 1354703)
Result = FAILURE
tedyu :
Files :
*
/hbase/trunk/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/KeyValueHeap.java
*
/hbase/trunk/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
> Lazyseek should use the maxSequenseId StoreFile's KeyValue as the latest
> KeyValue
> -
>
> Key: HBASE-6269
> URL: https://issues.apache.org/jira/browse/HBASE-6269
> Project: HBase
> Issue Type: Bug
> Components: regionserver
>Affects Versions: 0.94.0
>Reporter: ShiXing
>Assignee: ShiXing
> Attachments: HBASE-6269-trunk-V1.patch, HBASE-6269-v1.patch,
> runAllTests.out.txt
>
>
> When I fix the bug HBASE-6195, there is happened to find sometimes the test
> case will fail, https://builds.apache.org/job/HBase-0.94/259/.
> If there are two Put/Increment with same row, family, qualifier, timestamp
> and different memstoreTS, after each Put/Increment, we do a memstore flush.
> So there will be two StoreFile with same KeyValue(except memstoreTS and
> SequenceId).
> When I got the row, I always got the old records, the test case like this:
> {code}
> public void testPutWithMemStoreFlush() throws Exception {
> Configuration conf = HBaseConfiguration.create();
> String method = "testPutWithMemStoreFlush";
> byte[] tableName = Bytes.toBytes(method);
> byte[] family = Bytes.toBytes("family");;
> byte[] qualifier = Bytes.toBytes("qualifier");
> byte[] row = Bytes.toBytes("putRow");
> byte[] value = null;
> this.region = initHRegion(tableName, method, conf, family);
> Put put = null;
> Get get = null;
> List kvs = null;
> Result res = null;
>
> put = new Put(row);
> value = Bytes.toBytes("value0");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
>
> System.out.print("get value after flush after put value0 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value1");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value1 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
>
> put = new Put(row);
> value = Bytes.toBytes("value2");
> put.add(family, qualifier, 1234567l, value);
> region.put(put);
> System.out.print("get value before flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
> System.out.println(Bytes.toString(kvs.get(i).getValue()));
> }
> region.flushcache();
> System.out.print("get value after flush after put value2 : ");
> get = new Get(row);
> get.addColumn(family, qualifier);
> get.setMaxVersions();
> res = this.region.get(get, null);
> kvs = res.getColumn(family, qualifier);
> for (int i = 0; i < kvs.size(); i++) {
>
[jira] [Commented] (HBASE-6287) Port HBASE-5941 improve multiDelete performance by grabbing locks ahead of time
[
https://issues.apache.org/jira/browse/HBASE-6287?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402687#comment-13402687
]
Zhihong Ted Yu commented on HBASE-6287:
---
0.89-fb doesn't have coprocessor.
We need to consider how the following should be handled:
{code}
private void doPrePutHook(BatchOperationInProgress>
batchOp)
...
if (coprocessorHost.prePut(put, walEdit, put.getWriteToWAL())) {
{code}
There is no coprocessorHost.preMutate() in trunk.
> Port HBASE-5941 improve multiDelete performance by grabbing locks ahead of
> time
> ---
>
> Key: HBASE-6287
> URL: https://issues.apache.org/jira/browse/HBASE-6287
> Project: HBase
> Issue Type: Bug
>Reporter: Zhihong Ted Yu
>
> HBASE-5941 has been integrated to 0.89-fb
> This JIRA ports it to HBase trunk
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-6251) [89-fb] Don't abort on flush failure in unit tests
[ https://issues.apache.org/jira/browse/HBASE-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mikhail Bautin resolved HBASE-6251. --- Resolution: Fixed > [89-fb] Don't abort on flush failure in unit tests > -- > > Key: HBASE-6251 > URL: https://issues.apache.org/jira/browse/HBASE-6251 > Project: HBase > Issue Type: Improvement >Reporter: Mikhail Bautin >Priority: Minor > > If HLog flush fails, we don't want to abort the whole process in unit tests. > Instead, we want to throw a RuntimeException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-6261) Better approximate high-percentile percentile latency metrics
[ https://issues.apache.org/jira/browse/HBASE-6261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402733#comment-13402733 ] Andrew Wang commented on HBASE-6261: I've got my Java implementation of the non-sliding biased quantiles algorithm (QuantileEstimationCKMS.java) up on github: https://github.com/umbrant/QuantileEstimation Benchmarking on my laptop, I pushed 1 million shuffled items [0, 10**9) through it in 1.2 seconds while asking it to track the 50th, 90th, 95th, and 99th percentiles with low error. It kept ~5500 samples to do this, which at ~36B per sample, is about 193KiB. Empirical error was basically 0. I also ran it for 10 million random longs, which took 19s and about 685KiB. I think this is pretty lightweight. If this sounds reasonable, I'll start working on a patch. > Better approximate high-percentile percentile latency metrics > - > > Key: HBASE-6261 > URL: https://issues.apache.org/jira/browse/HBASE-6261 > Project: HBase > Issue Type: New Feature >Reporter: Andrew Wang > Labels: metrics > Attachments: Latencyestimation.pdf > > > The existing reservoir-sampling based latency metrics in HBase are not > well-suited for providing accurate estimates of high-percentile (e.g. 90th, > 95th, or 99th) latency. This is a well-studied problem in the literature (see > [1] and [2]), the question is determining which methods best suit our needs > and then implementing it. > Ideally, we should be able to estimate these high percentiles with minimal > memory and CPU usage as well as minimal error (e.g. 1% error on 90th, or .1% > on 99th). It's also desirable to provide this over different time-based > sliding windows, e.g. last 1 min, 5 mins, 15 mins, and 1 hour. > I'll note that this would also be useful in HDFS, or really anywhere latency > metrics are kept. > [1] http://www.cs.rutgers.edu/~muthu/bquant.pdf > [2] http://infolab.stanford.edu/~manku/papers/04pods-sliding.pdf -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2611) Handle RS that fails while processing the failure of another one
[ https://issues.apache.org/jira/browse/HBASE-2611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13402775#comment-13402775 ] Himanshu Vashishtha commented on HBASE-2611: Thanks for the review Ted. I will upload a modified version on the rb. My initial idea of putting it here was to get some feedback on the approach. Yes, it is zk intensive as all other regionservers are competing to do the transaction. But, as soon as one is successful (the first one who create the list and issues the multi command), other regionservers which haven't had a chance to do a listChildern call on the dead regionserver znode will not see anything; and for other regionservers which have created the Ops, the very first Op will fail as the znode has already moved. Zookeeper#multi op is fail fast, it rolls back the transaction on first failure without retrying remaining Ops. I tested it on a 3 RS cluster with average load being 12-14 logs, and it usually is done within seconds after the regionserver failure is noticed. What sort of experiments you are thinking about. On an another note, TestReplication passes. > Handle RS that fails while processing the failure of another one > > > Key: HBASE-2611 > URL: https://issues.apache.org/jira/browse/HBASE-2611 > Project: HBase > Issue Type: Sub-task > Components: replication >Reporter: Jean-Daniel Cryans >Assignee: Jean-Daniel Cryans > Attachments: HBase-2611-upstream-v1.patch > > > HBASE-2223 doesn't manage region servers that fail while doing the transfer > of HLogs queues from other region servers that failed. Devise a reliable way > to do it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-6288) In hbase-daemons.sh, description of the default backup-master file path is wrong
Benjamin Kim created HBASE-6288:
---
Summary: In hbase-daemons.sh, description of the default
backup-master file path is wrong
Key: HBASE-6288
URL: https://issues.apache.org/jira/browse/HBASE-6288
Project: HBase
Issue Type: Task
Components: master, scripts, shell
Affects Versions: 0.94.0, 0.92.1, 0.92.0
Reporter: Benjamin Kim
In hbase-daemons.sh, description of the default backup-master file path is wrong
{code}
# HBASE_BACKUP_MASTERS File naming remote hosts.
# Default is $\{HADOOP_CONF_DIR\}/backup-masters
{code}
it says the default backup-masters file path is at a hadoop-conf-dir, but
shouldn't this be HBASE_CONF_DIR?
also adding following lines to conf/hbase-env.sh would be helpful
{code}
# File naming hosts on which backup HMaster will run.
$HBASE_HOME/conf/backup-masters by default.
export HBASE_BACKUP_MASTERS=$\{HBASE_HOME\}/conf/backup-masters
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
