[jira] [Updated] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb
[ https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5869: - Status: Open (was: Patch Available) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb - Key: HBASE-5869 URL: https://issues.apache.org/jira/browse/HBASE-5869 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Attachments: 5869v7.txt, 5869v8.txt, 5869v9.txt, firstcut.txt, secondcut.txt, v10.txt, v4.txt, v5.txt, v6.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb
[ https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5869: - Attachment: v10.txt File path to recovered edits prob. fixed Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb - Key: HBASE-5869 URL: https://issues.apache.org/jira/browse/HBASE-5869 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Attachments: 5869v7.txt, 5869v8.txt, 5869v9.txt, firstcut.txt, secondcut.txt, v10.txt, v4.txt, v5.txt, v6.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb
[ https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5869: - Status: Patch Available (was: Open) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb - Key: HBASE-5869 URL: https://issues.apache.org/jira/browse/HBASE-5869 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Attachments: 5869v7.txt, 5869v8.txt, 5869v9.txt, firstcut.txt, secondcut.txt, v10.txt, v4.txt, v5.txt, v6.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb
[ https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265693#comment-13265693 ] Hadoop QA commented on HBASE-5869: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525158/v10.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 47 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol org.apache.hadoop.hbase.client.TestShell org.apache.hadoop.hbase.master.TestMasterFailover Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1703//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1703//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1703//console This message is automatically generated. Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb - Key: HBASE-5869 URL: https://issues.apache.org/jira/browse/HBASE-5869 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Attachments: 5869v7.txt, 5869v8.txt, 5869v9.txt, firstcut.txt, secondcut.txt, v10.txt, v4.txt, v5.txt, v6.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-2214) Do HBASE-1996 -- setting size to return in scan rather than count of rows -- properly
[ https://issues.apache.org/jira/browse/HBASE-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ferdy Galema updated HBASE-2214: Attachment: HBASE-2214-v6.txt Do HBASE-1996 -- setting size to return in scan rather than count of rows -- properly - Key: HBASE-2214 URL: https://issues.apache.org/jira/browse/HBASE-2214 Project: HBase Issue Type: New Feature Reporter: stack Assignee: Ferdy Galema Attachments: HBASE-2214-0.94.txt, HBASE-2214-v4.txt, HBASE-2214-v5.txt, HBASE-2214-v6.txt, HBASE-2214_with_broken_TestShell.txt The notion that you set size rather than row count specifying how many rows a scanner should return in each cycle was raised over in hbase-1966. Its a good one making hbase regular though the data under it may vary. HBase-1966 was committed but the patch was constrained by the fact that it needed to not change RPC interface. This issue is about doing hbase-1966 for 0.21 in a clean, unconstrained way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2214) Do HBASE-1996 -- setting size to return in scan rather than count of rows -- properly
[
https://issues.apache.org/jira/browse/HBASE-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265702#comment-13265702
]
jirapos...@reviews.apache.org commented on HBASE-2214:
--
bq. On 2012-04-30 20:03:34, Michael Stack wrote:
bq. Where are we checking the size of the result made so far? I don't see
it in the below. I'd expect it inside in the RegionScanner. Any chance of a
test? Otherwise, patch looks great.
bq.
bq. ferdy wrote:
bq. Please see the testing method in the Testing Done field above. (Not
sure where to add a test in the project).
bq.
bq. Thanks for the feedback.
(One more thing, it works because of the previous work done in HBASE-1996. But
it's arguably not the best way to implement it.)
- ferdy
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4726/#review7383
---
On 2012-05-01 07:50:07, ferdy wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4726/
bq. ---
bq.
bq. (Updated 2012-05-01 07:50:07)
bq.
bq.
bq. Review request for hbase and Ted Yu.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. HBASE-2214 per scan max buffersize.
bq.
bq.
bq. This addresses bug HBASE-2214.
bq. https://issues.apache.org/jira/browse/HBASE-2214
bq.
bq.
bq. Diffs
bq. -
bq.
bq./src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java 1330680
bq./src/main/java/org/apache/hadoop/hbase/client/Scan.java 1330680
bq./src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
1330680
bq./src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java
1330680
bq.
/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java
1330680
bq.
/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java
1330680
bq.
/src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java
1330680
bq./src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java
1330680
bq.
/src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java
1330680
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1332544
bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
1332544
bq./src/main/java/org/apache/hadoop/hbase/regionserver/RegionScanner.java
1332544
bq./src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java
1332544
bq./src/main/protobuf/Client.proto 1330680
bq.
/src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java
1332544
bq.
bq. Diff: https://reviews.apache.org/r/4726/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. It works when running this test:
bq.
bq.
bq. new HBaseTestingUtility(conf).startMiniCluster();
bq.
bq. HBaseAdmin admin = new HBaseAdmin(conf);
bq. if (!admin.tableExists(test)) {
bq.HTableDescriptor tableDesc = new HTableDescriptor(test);
bq.tableDesc.addFamily(new HColumnDescriptor(fam));
bq.admin.createTable(tableDesc);
bq. }
bq.
bq.
bq. HTable table = new HTable(conf, test);
bq. Put put;
bq.
bq. put = new Put(Bytes.toBytes(row1));
bq.
put.add(Bytes.toBytes(fam),Bytes.toBytes(qual1),Bytes.toBytes(val1));
bq. table.put(put);
bq.
bq. put = new Put(Bytes.toBytes(row2));
bq.
put.add(Bytes.toBytes(fam),Bytes.toBytes(qual2),Bytes.toBytes(val2));
bq. table.put(put);
bq.
bq. put = new Put(Bytes.toBytes(row3));
bq.
put.add(Bytes.toBytes(fam),Bytes.toBytes(qual3),Bytes.toBytes(val3));
bq. table.put(put);
bq.
bq. table.flushCommits();
bq. //put a logging statement to ClientScanner#next() to see the effect.
bq. {
bq.System.out.println(returns all rows at once because of the
caching);
bq.Scan scan = new Scan();
bq.scan.setCaching(100);
bq.ResultScanner scanner = table.getScanner(scan);
bq.scanner.next(100);
bq. }
bq. {
bq.System.out.println(returns one row at a time because of the
maxResultSize);
bq.Scan scan = new Scan();
bq.scan.setCaching(100);
bq.scan.setMaxResultSize(1);
bq.ResultScanner scanner = table.getScanner(scan);
bq.scanner.next(100);
bq. }
bq.
bq.
bq. See output:
bq.
bq. returns all rows at once because of the caching
bq. 2012-04-25 22:18:47,494 DEBUG [main] client.ClientScanner(94): Creating
scanner over test starting at key ''
bq. 2012-04-25 22:18:47,494 DEBUG [main] client.ClientScanner(206): Advancing
[jira] [Updated] (HBASE-5548) Add ability to get a table in the shell
[ https://issues.apache.org/jira/browse/HBASE-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5548: --- Attachment: ruby_HBASE-5548-addendum.patch fixes for tests and slight bugs. Passes on locally on OSX. Should be good for testing (and submit?). Add ability to get a table in the shell --- Key: HBASE-5548 URL: https://issues.apache.org/jira/browse/HBASE-5548 Project: HBase Issue Type: Improvement Components: shell Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.96.0 Attachments: ruby_HBASE-5528-v0.patch, ruby_HBASE-5548-addendum.patch, ruby_HBASE-5548-v1.patch, ruby_HBASE-5548-v2.patch, ruby_HBASE-5548-v3.patch, ruby_HBASE-5548-v5.patch Currently, all the commands that operate on a table in the shell first have to take the table as name as input. There are two main considerations: * It is annoying to have to write the table name every time, when you should just be able to get a reference to a table * the current implementation is very wasteful - it creates a new HTable for each call (but reuses the connection since it uses the same configuration) We should be able to get a handle to a single HTable and then operate on that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5905) Protobuf interface for Admin: split between the internal and the external/customer interface
[
https://issues.apache.org/jira/browse/HBASE-5905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265710#comment-13265710
]
nkeywal commented on HBASE-5905:
I don't think so. But we could add a specific comment?
Protobuf interface for Admin: split between the internal and the
external/customer interface
Key: HBASE-5905
URL: https://issues.apache.org/jira/browse/HBASE-5905
Project: HBase
Issue Type: Improvement
Components: client, master, regionserver
Affects Versions: 0.96.0
Reporter: nkeywal
After a short discussion with Stack, I create a jira.
--
I'am a little bit confused by the protobuf interface for closeRegion.
We have two types of closeRegion today:
1) the external ones; available in client.HBaseAdmin. They take the server
and the region identifier as a parameter and nothing else.
2) The internal ones, called for example by the master. They have more
parameters (like versionOfClosingNode or transitionInZK).
When I look at protobuf.ProtobufUtil, I see:
public static void closeRegion(final AdminProtocol admin,
final byte[] regionName, final boolean transitionInZK) throws
IOException {
CloseRegionRequest closeRegionRequest =
RequestConverter.buildCloseRegionRequest(regionName, transitionInZK);
try {
admin.closeRegion(null, closeRegionRequest);
} catch (ServiceException se) {
throw getRemoteException(se);
}
}
In other words, it seems that we merged the two interfaces into a single one.
Is that the intend?
I checked, the internal fields in closeRegionRequest are all optional (that's
good). Still, it means that the end user could use them or at least would
need to distinguish between the optional for functional reasons and the
optional - do not use.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2214) Do HBASE-1996 -- setting size to return in scan rather than count of rows -- properly
[ https://issues.apache.org/jira/browse/HBASE-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265711#comment-13265711 ] Hadoop QA commented on HBASE-2214: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525160/HBASE-2214-v6.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.client.TestShell org.apache.hadoop.hbase.master.TestMasterNoCluster Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1704//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1704//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1704//console This message is automatically generated. Do HBASE-1996 -- setting size to return in scan rather than count of rows -- properly - Key: HBASE-2214 URL: https://issues.apache.org/jira/browse/HBASE-2214 Project: HBase Issue Type: New Feature Reporter: stack Assignee: Ferdy Galema Attachments: HBASE-2214-0.94.txt, HBASE-2214-v4.txt, HBASE-2214-v5.txt, HBASE-2214-v6.txt, HBASE-2214_with_broken_TestShell.txt The notion that you set size rather than row count specifying how many rows a scanner should return in each cycle was raised over in hbase-1966. Its a good one making hbase regular though the data under it may vary. HBase-1966 was committed but the patch was constrained by the fact that it needed to not change RPC interface. This issue is about doing hbase-1966 for 0.21 in a clean, unconstrained way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5547) Don't delete HFiles when in backup mode
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5547: --- Attachment: java_HBASE-5547_v4.patch Attaching patch rebased on trunk (but otherwise same as RB) as per request. Don't delete HFiles when in backup mode - Key: HBASE-5547 URL: https://issues.apache.org/jira/browse/HBASE-5547 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Assignee: Jesse Yates Attachments: java_HBASE-5547_v4.patch This came up in a discussion I had with Stack. It would be nice if HBase could be notified that a backup is in progress (via a znode for example) and in that case either: 1. rename HFiles to be delete to file.bck 2. rename the HFiles into a special directory 3. rename them to a general trash directory (which would not need to be tied to backup mode). That way it should be able to get a consistent backup based on HFiles (HDFS snapshots or hard links would be better options here, but we do not have those). #1 makes cleanup a bit harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5547) Don't delete HFiles when in backup mode
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-5547: -- Status: Patch Available (was: Open) Don't delete HFiles when in backup mode - Key: HBASE-5547 URL: https://issues.apache.org/jira/browse/HBASE-5547 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Assignee: Jesse Yates Attachments: java_HBASE-5547_v4.patch This came up in a discussion I had with Stack. It would be nice if HBase could be notified that a backup is in progress (via a znode for example) and in that case either: 1. rename HFiles to be delete to file.bck 2. rename the HFiles into a special directory 3. rename them to a general trash directory (which would not need to be tied to backup mode). That way it should be able to get a consistent backup based on HFiles (HDFS snapshots or hard links would be better options here, but we do not have those). #1 makes cleanup a bit harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265851#comment-13265851 ] Hadoop QA commented on HBASE-5547: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525163/java_HBASE-5547_v4.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 3 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.TestCheckTestClasses Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1705//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1705//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1705//console This message is automatically generated. Don't delete HFiles when in backup mode - Key: HBASE-5547 URL: https://issues.apache.org/jira/browse/HBASE-5547 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Assignee: Jesse Yates Attachments: java_HBASE-5547_v4.patch This came up in a discussion I had with Stack. It would be nice if HBase could be notified that a backup is in progress (via a znode for example) and in that case either: 1. rename HFiles to be delete to file.bck 2. rename the HFiles into a special directory 3. rename them to a general trash directory (which would not need to be tied to backup mode). That way it should be able to get a consistent backup based on HFiles (HDFS snapshots or hard links would be better options here, but we do not have those). #1 makes cleanup a bit harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3691) Add compressor support for 'snappy', google's compressor
[ https://issues.apache.org/jira/browse/HBASE-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Waterson updated HBASE-3691: -- Attachment: hbase-snappy-0.90.6.patch Yes, I have. I've applied hbase-snappy-0.90.5.patch and it seems to be working on a small (but heavily loaded) HBase 0.90.6 cluster. Add compressor support for 'snappy', google's compressor Key: HBASE-3691 URL: https://issues.apache.org/jira/browse/HBASE-3691 Project: HBase Issue Type: Task Reporter: stack Priority: Critical Fix For: 0.92.0 Attachments: hbase-snappy-0.90.6.patch, hbase-snappy-3691-trunk-002.patch, hbase-snappy-3691-trunk-003.patch, hbase-snappy-3691-trunk-004.patch, hbase-snappy-3691-trunk.patch http://code.google.com/p/snappy/ is apache licensed. bq. Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more. bq. Snappy is widely used inside Google, in everything from BigTable and MapReduce to our internal RPC systems. (Snappy has previously been referred to as Zippy in some presentations and the likes.) Lets get it in. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3691) Add compressor support for 'snappy', google's compressor
[ https://issues.apache.org/jira/browse/HBASE-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265877#comment-13265877 ] Chris Waterson commented on HBASE-3691: --- Urg, I've applied hbase-snappy-0.90.6.patch... Add compressor support for 'snappy', google's compressor Key: HBASE-3691 URL: https://issues.apache.org/jira/browse/HBASE-3691 Project: HBase Issue Type: Task Reporter: stack Priority: Critical Fix For: 0.92.0 Attachments: hbase-snappy-0.90.6.patch, hbase-snappy-3691-trunk-002.patch, hbase-snappy-3691-trunk-003.patch, hbase-snappy-3691-trunk-004.patch, hbase-snappy-3691-trunk.patch http://code.google.com/p/snappy/ is apache licensed. bq. Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more. bq. Snappy is widely used inside Google, in everything from BigTable and MapReduce to our internal RPC systems. (Snappy has previously been referred to as Zippy in some presentations and the likes.) Lets get it in. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5887) Make TestAcidGuarantees usable for system testing.
[ https://issues.apache.org/jira/browse/HBASE-5887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265892#comment-13265892 ] stack commented on HBASE-5887: -- +1 Make TestAcidGuarantees usable for system testing. -- Key: HBASE-5887 URL: https://issues.apache.org/jira/browse/HBASE-5887 Project: HBase Issue Type: Improvement Affects Versions: 0.92.0, 0.92.1, 0.94.0 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Attachments: hbase-5887-92.patch, hbase-5887.patch Currently, the TestAcidGuarantees run via main() will always abort with an NPE because it digs into a non-existant HBaseTestingUtility for a flusher thread. We should tool this up so that it works properly from the command line. This would be a very useful long running test when used in conjunction with fault injections to verify row acid properties. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5876) TestImportExport has been failing against hadoop 0.23 profile
[ https://issues.apache.org/jira/browse/HBASE-5876?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265893#comment-13265893 ] Jonathan Hsieh commented on HBASE-5876: --- The first problem is that when running under the hadoop23 some how the TestImportExport job uses the local mapreduce.framework.name which assumes and eventually tries to read from a LocalFileSystem. It should use the yarn mapreduce.framework.name which will read from the instantiated minimrcluster/minidfs cluster. This gets past the job submission step but test still fails later on. Work continues. TestImportExport has been failing against hadoop 0.23 profile - Key: HBASE-5876 URL: https://issues.apache.org/jira/browse/HBASE-5876 Project: HBase Issue Type: Bug Reporter: Zhihong Yu Assignee: Jonathan Hsieh TestImportExport has been failing against hadoop 0.23 profile -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5548) Add ability to get a table in the shell
[ https://issues.apache.org/jira/browse/HBASE-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5548: - Status: Patch Available (was: Reopened) Submitting addendum. Pardon my being zealous for this fancy new feature Add ability to get a table in the shell --- Key: HBASE-5548 URL: https://issues.apache.org/jira/browse/HBASE-5548 Project: HBase Issue Type: Improvement Components: shell Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.96.0 Attachments: ruby_HBASE-5528-v0.patch, ruby_HBASE-5548-addendum.patch, ruby_HBASE-5548-v1.patch, ruby_HBASE-5548-v2.patch, ruby_HBASE-5548-v3.patch, ruby_HBASE-5548-v5.patch Currently, all the commands that operate on a table in the shell first have to take the table as name as input. There are two main considerations: * It is annoying to have to write the table name every time, when you should just be able to get a reference to a table * the current implementation is very wasteful - it creates a new HTable for each call (but reuses the connection since it uses the same configuration) We should be able to get a handle to a single HTable and then operate on that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5897) prePut coprocessor hook causing substantial CPU usage
[ https://issues.apache.org/jira/browse/HBASE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265923#comment-13265923 ] Lars Hofhansl commented on HBASE-5897: -- I would like to commit v3 and roll another RC of 0.94.0 in the next few hours. Any objections? prePut coprocessor hook causing substantial CPU usage - Key: HBASE-5897 URL: https://issues.apache.org/jira/browse/HBASE-5897 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: 5897-simple.txt, 5897-v2.txt, 5897-v3.txt, hbase-5897.txt, testRegionServerCoprocessorExceptionWithRemove.stack I was running an insert workload against trunk under oprofile and saw that a significant portion of CPU usage was going to calling the prePut coprocessor hook inside doMiniBatchPut, even though I don't have any coprocessors installed. I ran a million-row insert and collected CPU time spent in the RS after commenting out the preput hook, and found CPU usage reduced by 33%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4298) Support to drain RS nodes through ZK
[ https://issues.apache.org/jira/browse/HBASE-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265924#comment-13265924 ] Dave Latham commented on HBASE-4298: Any reason this issue is still open? Support to drain RS nodes through ZK Key: HBASE-4298 URL: https://issues.apache.org/jira/browse/HBASE-4298 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.90.4 Environment: all Reporter: Aravind Gottipati Priority: Critical Labels: patch Fix For: 0.92.0 Attachments: 4298-trunk-v2.txt, 4298-trunk-v3.txt, 90_hbase.patch, drainingservertest-v2.txt, drainingservertest.txt, trunk_hbase.patch, trunk_with_test.txt HDFS currently has a way to exclude certain datanodes and prevent them from getting new blocks. HDFS goes one step further and even drains these nodes for you. This enhancement is a step in that direction. The idea is that we mark nodes in zookeeper as draining nodes. This means that they don't get any more new regions. These draining nodes look exactly the same as the corresponding nodes in /rs, except they live under /draining. Eventually, support for draining them can be added. I am submitting two patches for review - one for the 0.90 branch and one for trunk (in git). Here are the two patches 0.90 - https://github.com/aravind/hbase/commit/181041e72e7ffe6a4da6d82b431ef7f8c99e62d2 trunk - https://github.com/aravind/hbase/commit/e127b25ae3b4034103b185d8380f3b7267bc67d5 I have tested both these patches and they work as advertised. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5548) Add ability to get a table in the shell
[ https://issues.apache.org/jira/browse/HBASE-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265926#comment-13265926 ] Hadoop QA commented on HBASE-5548: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525162/ruby_HBASE-5548-addendum.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 12 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1706//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1706//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1706//console This message is automatically generated. Add ability to get a table in the shell --- Key: HBASE-5548 URL: https://issues.apache.org/jira/browse/HBASE-5548 Project: HBase Issue Type: Improvement Components: shell Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.96.0 Attachments: ruby_HBASE-5528-v0.patch, ruby_HBASE-5548-addendum.patch, ruby_HBASE-5548-v1.patch, ruby_HBASE-5548-v2.patch, ruby_HBASE-5548-v3.patch, ruby_HBASE-5548-v5.patch Currently, all the commands that operate on a table in the shell first have to take the table as name as input. There are two main considerations: * It is annoying to have to write the table name every time, when you should just be able to get a reference to a table * the current implementation is very wasteful - it creates a new HTable for each call (but reuses the connection since it uses the same configuration) We should be able to get a handle to a single HTable and then operate on that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5905) Protobuf interface for Admin: split between the internal and the external/customer interface
[
https://issues.apache.org/jira/browse/HBASE-5905?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265928#comment-13265928
]
Todd Lipcon commented on HBASE-5905:
Maybe we can split out an AdminProtocol from an InternalClusterProtocol, etc?
Protobuf interface for Admin: split between the internal and the
external/customer interface
Key: HBASE-5905
URL: https://issues.apache.org/jira/browse/HBASE-5905
Project: HBase
Issue Type: Improvement
Components: client, master, regionserver
Affects Versions: 0.96.0
Reporter: nkeywal
After a short discussion with Stack, I create a jira.
--
I'am a little bit confused by the protobuf interface for closeRegion.
We have two types of closeRegion today:
1) the external ones; available in client.HBaseAdmin. They take the server
and the region identifier as a parameter and nothing else.
2) The internal ones, called for example by the master. They have more
parameters (like versionOfClosingNode or transitionInZK).
When I look at protobuf.ProtobufUtil, I see:
public static void closeRegion(final AdminProtocol admin,
final byte[] regionName, final boolean transitionInZK) throws
IOException {
CloseRegionRequest closeRegionRequest =
RequestConverter.buildCloseRegionRequest(regionName, transitionInZK);
try {
admin.closeRegion(null, closeRegionRequest);
} catch (ServiceException se) {
throw getRemoteException(se);
}
}
In other words, it seems that we merged the two interfaces into a single one.
Is that the intend?
I checked, the internal fields in closeRegionRequest are all optional (that's
good). Still, it means that the end user could use them or at least would
need to distinguish between the optional for functional reasons and the
optional - do not use.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5548) Add ability to get a table in the shell
[ https://issues.apache.org/jira/browse/HBASE-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265930#comment-13265930 ] Jesse Yates commented on HBASE-5548: thanks stack! Add ability to get a table in the shell --- Key: HBASE-5548 URL: https://issues.apache.org/jira/browse/HBASE-5548 Project: HBase Issue Type: Improvement Components: shell Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.96.0 Attachments: ruby_HBASE-5528-v0.patch, ruby_HBASE-5548-addendum.patch, ruby_HBASE-5548-v1.patch, ruby_HBASE-5548-v2.patch, ruby_HBASE-5548-v3.patch, ruby_HBASE-5548-v5.patch Currently, all the commands that operate on a table in the shell first have to take the table as name as input. There are two main considerations: * It is annoying to have to write the table name every time, when you should just be able to get a reference to a table * the current implementation is very wasteful - it creates a new HTable for each call (but reuses the connection since it uses the same configuration) We should be able to get a handle to a single HTable and then operate on that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-4298) Support to drain RS nodes through ZK
[ https://issues.apache.org/jira/browse/HBASE-4298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack resolved HBASE-4298. -- Resolution: Fixed Committed to trunk and 0.92 long time ago. Resolving. Thanks for patch Aravind. Support to drain RS nodes through ZK Key: HBASE-4298 URL: https://issues.apache.org/jira/browse/HBASE-4298 Project: HBase Issue Type: Improvement Components: master Affects Versions: 0.90.4 Environment: all Reporter: Aravind Gottipati Priority: Critical Labels: patch Fix For: 0.92.0 Attachments: 4298-trunk-v2.txt, 4298-trunk-v3.txt, 90_hbase.patch, drainingservertest-v2.txt, drainingservertest.txt, trunk_hbase.patch, trunk_with_test.txt HDFS currently has a way to exclude certain datanodes and prevent them from getting new blocks. HDFS goes one step further and even drains these nodes for you. This enhancement is a step in that direction. The idea is that we mark nodes in zookeeper as draining nodes. This means that they don't get any more new regions. These draining nodes look exactly the same as the corresponding nodes in /rs, except they live under /draining. Eventually, support for draining them can be added. I am submitting two patches for review - one for the 0.90 branch and one for trunk (in git). Here are the two patches 0.90 - https://github.com/aravind/hbase/commit/181041e72e7ffe6a4da6d82b431ef7f8c99e62d2 trunk - https://github.com/aravind/hbase/commit/e127b25ae3b4034103b185d8380f3b7267bc67d5 I have tested both these patches and they work as advertised. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5548) Add ability to get a table in the shell
[ https://issues.apache.org/jira/browse/HBASE-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265945#comment-13265945 ] stack commented on HBASE-5548: -- Committed addendum. Thanks for following up Ted (and you Jesse) Add ability to get a table in the shell --- Key: HBASE-5548 URL: https://issues.apache.org/jira/browse/HBASE-5548 Project: HBase Issue Type: Improvement Components: shell Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.96.0 Attachments: ruby_HBASE-5528-v0.patch, ruby_HBASE-5548-addendum.patch, ruby_HBASE-5548-v1.patch, ruby_HBASE-5548-v2.patch, ruby_HBASE-5548-v3.patch, ruby_HBASE-5548-v5.patch Currently, all the commands that operate on a table in the shell first have to take the table as name as input. There are two main considerations: * It is annoying to have to write the table name every time, when you should just be able to get a reference to a table * the current implementation is very wasteful - it creates a new HTable for each call (but reuses the connection since it uses the same configuration) We should be able to get a handle to a single HTable and then operate on that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb
[ https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5869: - Attachment: v11.txt Hopefully got all the test failures with this version of patch Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb - Key: HBASE-5869 URL: https://issues.apache.org/jira/browse/HBASE-5869 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Attachments: 5869v7.txt, 5869v8.txt, 5869v9.txt, firstcut.txt, secondcut.txt, v10.txt, v11.txt, v4.txt, v5.txt, v6.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb
[ https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5869: - Status: Open (was: Patch Available) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb - Key: HBASE-5869 URL: https://issues.apache.org/jira/browse/HBASE-5869 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Attachments: 5869v7.txt, 5869v8.txt, 5869v9.txt, firstcut.txt, secondcut.txt, v10.txt, v11.txt, v4.txt, v5.txt, v6.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb
[ https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5869: - Status: Patch Available (was: Open) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb - Key: HBASE-5869 URL: https://issues.apache.org/jira/browse/HBASE-5869 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Attachments: 5869v7.txt, 5869v8.txt, 5869v9.txt, firstcut.txt, secondcut.txt, v10.txt, v11.txt, v4.txt, v5.txt, v6.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5548) Add ability to get a table in the shell
[ https://issues.apache.org/jira/browse/HBASE-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265961#comment-13265961 ] Hudson commented on HBASE-5548: --- Integrated in HBase-TRUNK #2832 (See [https://builds.apache.org/job/HBase-TRUNK/2832/]) HBASE-5548 Add ability to get a table in the shell; ADDENDUM (Revision 1332766) Result = FAILURE stack : Files : * /hbase/trunk/src/main/ruby/hbase/table.rb * /hbase/trunk/src/main/ruby/shell/commands/count.rb * /hbase/trunk/src/main/ruby/shell/commands/deleteall.rb * /hbase/trunk/src/main/ruby/shell/commands/get_counter.rb * /hbase/trunk/src/test/ruby/hbase/admin_test.rb * /hbase/trunk/src/test/ruby/hbase/table_test.rb * /hbase/trunk/src/test/ruby/shell/commands_test.rb * /hbase/trunk/src/test/ruby/test_helper.rb Add ability to get a table in the shell --- Key: HBASE-5548 URL: https://issues.apache.org/jira/browse/HBASE-5548 Project: HBase Issue Type: Improvement Components: shell Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.96.0 Attachments: ruby_HBASE-5528-v0.patch, ruby_HBASE-5548-addendum.patch, ruby_HBASE-5548-v1.patch, ruby_HBASE-5548-v2.patch, ruby_HBASE-5548-v3.patch, ruby_HBASE-5548-v5.patch Currently, all the commands that operate on a table in the shell first have to take the table as name as input. There are two main considerations: * It is annoying to have to write the table name every time, when you should just be able to get a reference to a table * the current implementation is very wasteful - it creates a new HTable for each call (but reuses the connection since it uses the same configuration) We should be able to get a handle to a single HTable and then operate on that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine
[ https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Devaraj Das updated HBASE-5732: --- Attachment: rpcengine-merge.4.patch Attaching an updated patch with the check for whether security is enabled before starting the secretmanager (in WritableRpcEngine.Server implementation). All the unit tests (except TestShell which fails without the patch too) pass with this patch. I'll test it manually with security/kerberos ON. In the meantime, it'd be nice to get feedback on the patch (and yes, I am yet to address the white-space comments). Thanks! Remove the SecureRPCEngine and merge the security-related logic in the core engine -- Key: HBASE-5732 URL: https://issues.apache.org/jira/browse/HBASE-5732 Project: HBase Issue Type: Improvement Reporter: Devaraj Das Assignee: Devaraj Das Attachments: rpcengine-merge.3.patch, rpcengine-merge.4.patch, rpcengine-merge.patch Remove the SecureRPCEngine and merge the security-related logic in the core engine. Follow up to HBASE-5727. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb
[ https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265971#comment-13265971 ] Hadoop QA commented on HBASE-5869: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525189/v11.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 47 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1707//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1707//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1707//console This message is automatically generated. Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb - Key: HBASE-5869 URL: https://issues.apache.org/jira/browse/HBASE-5869 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Attachments: 5869v7.txt, 5869v8.txt, 5869v9.txt, firstcut.txt, secondcut.txt, v10.txt, v11.txt, v4.txt, v5.txt, v6.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5547) Don't delete HFiles when in backup mode
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5547: --- Attachment: java_HBASE-5547_v5.patch Attaching version with fixed test and correction to the one real findbugs warning. Don't delete HFiles when in backup mode - Key: HBASE-5547 URL: https://issues.apache.org/jira/browse/HBASE-5547 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Assignee: Jesse Yates Attachments: java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch This came up in a discussion I had with Stack. It would be nice if HBase could be notified that a backup is in progress (via a znode for example) and in that case either: 1. rename HFiles to be delete to file.bck 2. rename the HFiles into a special directory 3. rename them to a general trash directory (which would not need to be tied to backup mode). That way it should be able to get a consistent backup based on HFiles (HDFS snapshots or hard links would be better options here, but we do not have those). #1 makes cleanup a bit harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5897) prePut coprocessor hook causing substantial CPU usage
[ https://issues.apache.org/jira/browse/HBASE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5897: - Attachment: 5897-v3-0.94.txt Patch for 0.94 prePut coprocessor hook causing substantial CPU usage - Key: HBASE-5897 URL: https://issues.apache.org/jira/browse/HBASE-5897 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: 5897-simple.txt, 5897-v2.txt, 5897-v3-0.94.txt, 5897-v3.txt, hbase-5897.txt, testRegionServerCoprocessorExceptionWithRemove.stack I was running an insert workload against trunk under oprofile and saw that a significant portion of CPU usage was going to calling the prePut coprocessor hook inside doMiniBatchPut, even though I don't have any coprocessors installed. I ran a million-row insert and collected CPU time spent in the RS after commenting out the preput hook, and found CPU usage reduced by 33%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5897) prePut coprocessor hook causing substantial CPU usage
[ https://issues.apache.org/jira/browse/HBASE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265982#comment-13265982 ] stack commented on HBASE-5897: -- None from me. prePut coprocessor hook causing substantial CPU usage - Key: HBASE-5897 URL: https://issues.apache.org/jira/browse/HBASE-5897 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: 5897-simple.txt, 5897-v2.txt, 5897-v3-0.94.txt, 5897-v3.txt, hbase-5897.txt, testRegionServerCoprocessorExceptionWithRemove.stack I was running an insert workload against trunk under oprofile and saw that a significant portion of CPU usage was going to calling the prePut coprocessor hook inside doMiniBatchPut, even though I don't have any coprocessors installed. I ran a million-row insert and collected CPU time spent in the RS after commenting out the preput hook, and found CPU usage reduced by 33%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5897) prePut coprocessor hook causing substantial CPU usage
[ https://issues.apache.org/jira/browse/HBASE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265985#comment-13265985 ] Hadoop QA commented on HBASE-5897: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525198/5897-v3-0.94.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1708//console This message is automatically generated. prePut coprocessor hook causing substantial CPU usage - Key: HBASE-5897 URL: https://issues.apache.org/jira/browse/HBASE-5897 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: 5897-simple.txt, 5897-v2.txt, 5897-v3-0.94.txt, 5897-v3.txt, hbase-5897.txt, testRegionServerCoprocessorExceptionWithRemove.stack I was running an insert workload against trunk under oprofile and saw that a significant portion of CPU usage was going to calling the prePut coprocessor hook inside doMiniBatchPut, even though I don't have any coprocessors installed. I ran a million-row insert and collected CPU time spent in the RS after commenting out the preput hook, and found CPU usage reduced by 33%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3691) Add compressor support for 'snappy', google's compressor
[ https://issues.apache.org/jira/browse/HBASE-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-3691: - Fix Version/s: 0.90.7 Applied to 0.90 branch. Add compressor support for 'snappy', google's compressor Key: HBASE-3691 URL: https://issues.apache.org/jira/browse/HBASE-3691 Project: HBase Issue Type: Task Reporter: stack Priority: Critical Fix For: 0.90.7, 0.92.0 Attachments: hbase-snappy-0.90.6.patch, hbase-snappy-3691-trunk-002.patch, hbase-snappy-3691-trunk-003.patch, hbase-snappy-3691-trunk-004.patch, hbase-snappy-3691-trunk.patch http://code.google.com/p/snappy/ is apache licensed. bq. Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more. bq. Snappy is widely used inside Google, in everything from BigTable and MapReduce to our internal RPC systems. (Snappy has previously been referred to as Zippy in some presentations and the likes.) Lets get it in. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5886) Add new metric for possible data loss due to puts without WAL
[ https://issues.apache.org/jira/browse/HBASE-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13265995#comment-13265995 ] Matteo Bertozzi commented on HBASE-5886: Any other feedback on the patch? Add a config option hbase.warn.nowal to write the warn message only if the option is set? Add new metric for possible data loss due to puts without WAL -- Key: HBASE-5886 URL: https://issues.apache.org/jira/browse/HBASE-5886 Project: HBase Issue Type: New Feature Components: metrics, regionserver Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Labels: metrics Attachments: HBASE-5886-v0.patch, HBASE-5886-v1.patch, HBASE-5886-v2.patch Add a metrics to keep track of puts without WAL and possible data loss size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb
[ https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5869: - Status: Open (was: Patch Available) These three tests hung. If I run them local, they are fine. Retrying. Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb - Key: HBASE-5869 URL: https://issues.apache.org/jira/browse/HBASE-5869 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Attachments: 5869v7.txt, 5869v8.txt, 5869v9.txt, firstcut.txt, secondcut.txt, v10.txt, v11.txt, v12.txt, v4.txt, v5.txt, v6.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb
[ https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5869: - Status: Patch Available (was: Open) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb - Key: HBASE-5869 URL: https://issues.apache.org/jira/browse/HBASE-5869 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Attachments: 5869v7.txt, 5869v8.txt, 5869v9.txt, firstcut.txt, secondcut.txt, v10.txt, v11.txt, v12.txt, v4.txt, v5.txt, v6.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb
[ https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5869: - Attachment: v12.txt Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb - Key: HBASE-5869 URL: https://issues.apache.org/jira/browse/HBASE-5869 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Attachments: 5869v7.txt, 5869v8.txt, 5869v9.txt, firstcut.txt, secondcut.txt, v10.txt, v11.txt, v12.txt, v4.txt, v5.txt, v6.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5444) Add PB-based calls to HMasterRegionInterface
[ https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266009#comment-13266009 ] jirapos...@reviews.apache.org commented on HBASE-5444: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4463/ --- (Updated 2012-05-01 19:53:51.307399) Review request for hbase and Michael Stack. Changes --- Update against newest trunk and followed Ted's suggestion about breaking out totalRequestsCount computation into own function. Summary --- Adds PB-based calls replacing HMasterRegionInterface. There are some temporary hacks, e.g. converting PB-based ServerLoad to existing HServerLoad so I didn't need to convert ClusterStatus (which brings in a lot of other changes). That will be cleaned up in HBASE-5445. This addresses bug HBASE-5444. https://issues.apache.org/jira/browse/HBASE-5444 Diffs (updated) - src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 0db2760 src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 973c7cb src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java fd97830 src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java bb6ab3b src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java f56127d src/main/java/org/apache/hadoop/hbase/master/HMaster.java 81e9023 src/main/java/org/apache/hadoop/hbase/master/MXBean.java 7f44dc2 src/main/java/org/apache/hadoop/hbase/master/MXBeanImpl.java 45b8fe7 src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java be63838 src/main/java/org/apache/hadoop/hbase/master/RegionServerStatusProtocol.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 80271b1 src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 994cb76 src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java efcf74d src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 5d7f07b src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 69434f7 src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 3c7c091 src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java ebffad6 src/main/protobuf/RegionServerStatus.proto PRE-CREATION src/main/protobuf/hbase.proto 12e6053 src/main/resources/hbase-webapps/master/table.jsp 3ef1190 src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 72554cb src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java d039be3 src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 36046f8 src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java bd5fa90 src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java f8029ba src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java e99d251 Diff: https://reviews.apache.org/r/4463/diff Testing --- Ran jenkins job, all unit tests passed. Thanks, Gregory Add PB-based calls to HMasterRegionInterface Key: HBASE-5444 URL: https://issues.apache.org/jira/browse/HBASE-5444 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: Gregory Chanan Attachments: HBASE-5444-v6-trunk.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266013#comment-13266013 ] Hadoop QA commented on HBASE-5547: -- -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525196/java_HBASE-5547_v5.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 15 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hbase.regionserver.TestRegionHFileArchiving org.apache.hadoop.hbase.regionserver.TestServerCustomProtocol Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1709//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1709//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1709//console This message is automatically generated. Don't delete HFiles when in backup mode - Key: HBASE-5547 URL: https://issues.apache.org/jira/browse/HBASE-5547 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Assignee: Jesse Yates Attachments: java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch This came up in a discussion I had with Stack. It would be nice if HBase could be notified that a backup is in progress (via a znode for example) and in that case either: 1. rename HFiles to be delete to file.bck 2. rename the HFiles into a special directory 3. rename them to a general trash directory (which would not need to be tied to backup mode). That way it should be able to get a consistent backup based on HFiles (HDFS snapshots or hard links would be better options here, but we do not have those). #1 makes cleanup a bit harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5444) Add PB-based calls to HMasterRegionInterface
[ https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266028#comment-13266028 ] jirapos...@reviews.apache.org commented on HBASE-5444: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4463/#review7441 --- src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java https://reviews.apache.org/r/4463/#comment16369 Insert a space between if and (. src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java https://reviews.apache.org/r/4463/#comment16372 You can return list.toArray() directly, similar to line 1179. - Ted On 2012-05-01 19:53:51, Gregory Chanan wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4463/ bq. --- bq. bq. (Updated 2012-05-01 19:53:51) bq. bq. bq. Review request for hbase and Michael Stack. bq. bq. bq. Summary bq. --- bq. bq. Adds PB-based calls replacing HMasterRegionInterface. bq. bq. There are some temporary hacks, e.g. converting PB-based ServerLoad to existing HServerLoad so I didn't need to convert ClusterStatus (which brings in a lot of other changes). That will be cleaned up in HBASE-5445. bq. bq. bq. This addresses bug HBASE-5444. bq. https://issues.apache.org/jira/browse/HBASE-5444 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 0db2760 bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 973c7cb bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java fd97830 bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java bb6ab3b bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java f56127d bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 81e9023 bq.src/main/java/org/apache/hadoop/hbase/master/MXBean.java 7f44dc2 bq.src/main/java/org/apache/hadoop/hbase/master/MXBeanImpl.java 45b8fe7 bq.src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java be63838 bq. src/main/java/org/apache/hadoop/hbase/master/RegionServerStatusProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 80271b1 bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 994cb76 bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java efcf74d bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 5d7f07b bq. src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 69434f7 bq. src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 3c7c091 bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java ebffad6 bq.src/main/protobuf/RegionServerStatus.proto PRE-CREATION bq.src/main/protobuf/hbase.proto 12e6053 bq.src/main/resources/hbase-webapps/master/table.jsp 3ef1190 bq.src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 72554cb bq.src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java d039be3 bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 36046f8 bq.src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java bd5fa90 bq.src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java f8029ba bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java e99d251 bq. bq. Diff: https://reviews.apache.org/r/4463/diff bq. bq. bq. Testing bq. --- bq. bq. Ran jenkins job, all unit tests passed. bq. bq. bq. Thanks, bq. bq. Gregory bq. bq. Add PB-based calls to HMasterRegionInterface Key: HBASE-5444 URL: https://issues.apache.org/jira/browse/HBASE-5444 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: Gregory Chanan Attachments: HBASE-5444-v6-trunk.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5444) Add PB-based calls to HMasterRegionInterface
[ https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266033#comment-13266033 ] jirapos...@reviews.apache.org commented on HBASE-5444: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4463/#review7442 --- A few comments below the thrust of which are about encapsulating pb if possible rather than have it spread around in classes.See what you think. It would not be hard to get me commit this as is. src/main/java/org/apache/hadoop/hbase/HConstants.java https://reviews.apache.org/r/4463/#comment16370 Were you going to move this down to where its used G? Does it need to be up here? src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java https://reviews.apache.org/r/4463/#comment16371 Hurray! src/main/java/org/apache/hadoop/hbase/master/MXBean.java https://reviews.apache.org/r/4463/#comment16373 All of these classes are importing generated pb classes. Would it be better to have a high-level ServerLoad class that hid inside it the pb stuff instead? Less pb generated class pollution. src/main/java/org/apache/hadoop/hbase/master/ServerManager.java https://reviews.apache.org/r/4463/#comment16374 Yeah, its kinda ugly having this package reach over into pb package. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java https://reviews.apache.org/r/4463/#comment16375 We should not be reaching over into the master package. Put this protocol class at the top level since shared by master and regionserver? - Michael On 2012-05-01 19:53:51, Gregory Chanan wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4463/ bq. --- bq. bq. (Updated 2012-05-01 19:53:51) bq. bq. bq. Review request for hbase and Michael Stack. bq. bq. bq. Summary bq. --- bq. bq. Adds PB-based calls replacing HMasterRegionInterface. bq. bq. There are some temporary hacks, e.g. converting PB-based ServerLoad to existing HServerLoad so I didn't need to convert ClusterStatus (which brings in a lot of other changes). That will be cleaned up in HBASE-5445. bq. bq. bq. This addresses bug HBASE-5444. bq. https://issues.apache.org/jira/browse/HBASE-5444 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 0db2760 bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 973c7cb bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java fd97830 bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java bb6ab3b bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java f56127d bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 81e9023 bq.src/main/java/org/apache/hadoop/hbase/master/MXBean.java 7f44dc2 bq.src/main/java/org/apache/hadoop/hbase/master/MXBeanImpl.java 45b8fe7 bq.src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java be63838 bq. src/main/java/org/apache/hadoop/hbase/master/RegionServerStatusProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 80271b1 bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 994cb76 bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java efcf74d bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 5d7f07b bq. src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 69434f7 bq. src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 3c7c091 bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java ebffad6 bq.src/main/protobuf/RegionServerStatus.proto PRE-CREATION bq.src/main/protobuf/hbase.proto 12e6053 bq.src/main/resources/hbase-webapps/master/table.jsp 3ef1190 bq.src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 72554cb bq.src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java d039be3 bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 36046f8 bq.src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java bd5fa90 bq.src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java f8029ba bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java e99d251 bq. bq. Diff:
[jira] [Updated] (HBASE-5898) Consider double-checked locking for block cache lock
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhihong Yu updated HBASE-5898: -- Attachment: 5898-TestBlocksRead.txt Some changes to TestBlocksRead are needed to make it pass. See if the increase in blocks read is acceptable. Consider double-checked locking for block cache lock Key: HBASE-5898 URL: https://issues.apache.org/jira/browse/HBASE-5898 Project: HBase Issue Type: Improvement Components: performance Affects Versions: 0.94.1 Reporter: Todd Lipcon Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5732) Remove the SecureRPCEngine and merge the security-related logic in the core engine
[ https://issues.apache.org/jira/browse/HBASE-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266042#comment-13266042 ] jirapos...@reviews.apache.org commented on HBASE-5732: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4953/ --- Review request for Ted Yu, Michael Stack and Andrew Purtell. Summary --- Reviewboard request for HBASE-5732 This addresses bug HBASE-5732. https://issues.apache.org/jira/browse/HBASE-5732 Diffs - http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1332383 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/ConnectionHeader.java 1332383 http://svn.apache.org/repos/asf/hbase/trunk/pom.xml 1332383 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 1332383 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/Status.java 1332383 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 1332383 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCProtos.java 1332383 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/AccessDeniedException.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBasePolicyProvider.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcClient.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/HBaseSaslRpcServer.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java 1332383 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlFilter.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControlLists.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessController.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/AccessControllerProtocol.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/Permission.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/TableAuthManager.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/TablePermission.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/UserPermission.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/access/ZKPermissionWatcher.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationKey.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationProtocol.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenIdentifier.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSecretManager.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/AuthenticationTokenSelector.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/TokenProvider.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/TokenUtil.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/token/ZKSecretWatcher.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPC.proto 1332383 http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/security/access/SecureTestUtil.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/test/java/org/apache/hadoop/hbase/security/access/TestAccessControlFilter.java PRE-CREATION
[jira] [Commented] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb
[ https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266041#comment-13266041 ] Hadoop QA commented on HBASE-5869: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525202/v12.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 50 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1710//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1710//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1710//console This message is automatically generated. Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb - Key: HBASE-5869 URL: https://issues.apache.org/jira/browse/HBASE-5869 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Attachments: 5869v7.txt, 5869v8.txt, 5869v9.txt, firstcut.txt, secondcut.txt, v10.txt, v11.txt, v12.txt, v4.txt, v5.txt, v6.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for block cache lock
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266045#comment-13266045 ] Jean-Daniel Cryans commented on HBASE-5898: --- Once this is fixed I think we can close HBASE-5000. Consider double-checked locking for block cache lock Key: HBASE-5898 URL: https://issues.apache.org/jira/browse/HBASE-5898 Project: HBase Issue Type: Improvement Components: performance Affects Versions: 0.94.1 Reporter: Todd Lipcon Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for block cache lock
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266054#comment-13266054 ] Todd Lipcon commented on HBASE-5898: Really? I don't think this solves the same problem as HBASE-5000. This addresses the case where there's a really high cache hit ratio, whereas that one addresses the case where there's a 0% cache hit ratio. Consider double-checked locking for block cache lock Key: HBASE-5898 URL: https://issues.apache.org/jira/browse/HBASE-5898 Project: HBase Issue Type: Improvement Components: performance Affects Versions: 0.94.1 Reporter: Todd Lipcon Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for block cache lock
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266059#comment-13266059 ] stack commented on HBASE-5898: -- @Todd Double-checked locking should ...should usually be avoided. [1]. Looking at block cache, it looks like a block should be fully initialized before its added to the cache so we should avoid the horror stories detailed in the article. Let me try take it for a run... 1. http://en.wikipedia.org/wiki/Double-checked_locking#Usage_in_Java Consider double-checked locking for block cache lock Key: HBASE-5898 URL: https://issues.apache.org/jira/browse/HBASE-5898 Project: HBase Issue Type: Improvement Components: performance Affects Versions: 0.94.1 Reporter: Todd Lipcon Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb
[ https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266061#comment-13266061 ] stack commented on HBASE-5869: -- Whoopdeee! Let me post latest up on rb in case anyone wants take a look. Will apply tomorrow unless objection. Let me do another test run too to be sure. Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb - Key: HBASE-5869 URL: https://issues.apache.org/jira/browse/HBASE-5869 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Attachments: 5869v7.txt, 5869v8.txt, 5869v9.txt, firstcut.txt, secondcut.txt, v10.txt, v11.txt, v12.txt, v4.txt, v5.txt, v6.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for block cache lock
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266064#comment-13266064 ] Todd Lipcon commented on HBASE-5898: bq. @Todd Double-checked locking should ...should usually be avoided. [1]. I am above the law, Stack, didn't you know that? :) Seriously, though, I think the use of the idiom here is safe. Consider double-checked locking for block cache lock Key: HBASE-5898 URL: https://issues.apache.org/jira/browse/HBASE-5898 Project: HBase Issue Type: Improvement Components: performance Affects Versions: 0.94.1 Reporter: Todd Lipcon Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb
[ https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266068#comment-13266068 ] jirapos...@reviews.apache.org commented on HBASE-5869: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4926/ --- (Updated 2012-05-01 20:42:36.337375) Review request for hbase and Jimmy Xiang. Changes --- Same as original w/ a few fixes for tests that failed: 1. In distributed log tests, was failing to pick up the recovered.edits file because string passed included state of the split log task when what was wanted was servername only Summary --- Convert two zk users to pb: distributed log splitting and regions in transition. Refactored distributed log splitting so we only serialize/deserialize in one location. Less changes needed to do same for regions in transition. Moves serialization/deserialization out of the ZKAssign, ZKSplit and into the classes themselves so can encapsulate how serialization is done into one place (try to make the ZK* classes just deal in bytes -- about 90% done). Moved classes used by various packages up to top level to minimize imports that are across package (zookeeper into protobuf and/or into regionserver and/or master packages, etc). A src/main/java/org/apache/hadoop/hbase/DeserializationException.java New generic deserialization exception. A src/main/java/org/apache/hadoop/hbase/zookeeper/EmptyWatcher.java D src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java Moved under zookeeper package. A src/main/java/org/apache/hadoop/hbase/HBaseException.java New base hbase exception as suggested by hbase-5796. New DeserializationException inherits from this. A src/main/java/org/apache/hadoop/hbase/RegionTransition.java State of a region in transition. Top-level because used by a few top-level packages. Encapsulates pb serialization/deserialization. M src/main/java/org/apache/hadoop/hbase/ServerName.java Add method to deserialize a ServeName, etc. Encapsulates pb'ing. M src/main/java/org/apache/hadoop/hbase/SplitLogCounters.java Counters used by distributed log splitting. A SplitLogTask Class that encapsulates log splitting state. Also encapsulates pb'ing. M src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java Implement code for state. Added functions to go from code to state and vice versa. Used serializing. M src/main/java/org/apache/hadoop/hbase/executor/ExecutorService.java Remove unused imports. D src/main/java/org/apache/hadoop/hbase/executor/RegionTransitionData.java Removed. Replaced by RegionTransition moved to package top-level. M src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java M src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java Use new DeserializationException. Move to using new RegionTransition from RegionTransitionData class. Pass deserialized class rather than byte array. Remove duplicated code. M src/main/java/org/apache/hadoop/hbase/master/HMaster.java Use new ServerName parse method rather than ZKUtil one. M src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java M src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java Redo to use new SplitLogTask and SplitLogCounter classes. M src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java expectPBMagicPrefix added M src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java Use new RegionTransition in place of RegionTransitionData. M src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java Define moved from ZKSplitLog to SplitLogManager. M src/main/java/org/apache/hadoop/hbase/zookeeper/MasterAddressTracker.java M src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java Changed method name from getZNodeData to toByteArray to match how we've named it elsewhere. Use new DeserializationException M src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java Use new RegionTransion class M src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java Moved stuff that was in here up into SplitLogManager where better belongs. Also moved serialization/deserialization up into the class itself: SplitLogTask. Moved counters out to SplitLogCounter class. M src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java Moved deserialization of ServerName out of here and up into ServerName. M src/main/protobuf/ZooKeeper.proto Add two new classes, RegionTransition and SplitLogTask. This addresses bug HBASE-5869. https://issues.apache.org/jira/browse/HBASE-5869 Diffs (updated) - src/main/java/org/apache/hadoop/hbase/DeserializationException.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java
[jira] [Updated] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb
[ https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5869: - Status: Open (was: Patch Available) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb - Key: HBASE-5869 URL: https://issues.apache.org/jira/browse/HBASE-5869 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Attachments: 5869v7.txt, 5869v8.txt, 5869v9.txt, firstcut.txt, secondcut.txt, v10.txt, v11.txt, v12.txt, v13.txt, v4.txt, v5.txt, v6.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb
[ https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5869: - Status: Patch Available (was: Open) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb - Key: HBASE-5869 URL: https://issues.apache.org/jira/browse/HBASE-5869 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Attachments: 5869v7.txt, 5869v8.txt, 5869v9.txt, firstcut.txt, secondcut.txt, v10.txt, v11.txt, v12.txt, v13.txt, v4.txt, v5.txt, v6.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb
[ https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5869: - Attachment: v13.txt What I applied to rb and what I want to commit. Same as v12 only removes needless region creation over in testwalobserver tests... was causing testwalobserver take time going down. Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb - Key: HBASE-5869 URL: https://issues.apache.org/jira/browse/HBASE-5869 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Attachments: 5869v7.txt, 5869v8.txt, 5869v9.txt, firstcut.txt, secondcut.txt, v10.txt, v11.txt, v12.txt, v13.txt, v4.txt, v5.txt, v6.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5897) prePut coprocessor hook causing substantial CPU usage
[ https://issues.apache.org/jira/browse/HBASE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5897: - Comment: was deleted (was: -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525198/5897-v3-0.94.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1708//console This message is automatically generated.) prePut coprocessor hook causing substantial CPU usage - Key: HBASE-5897 URL: https://issues.apache.org/jira/browse/HBASE-5897 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: 5897-simple.txt, 5897-v2.txt, 5897-v3-0.94.txt, 5897-v3.txt, hbase-5897.txt, testRegionServerCoprocessorExceptionWithRemove.stack I was running an insert workload against trunk under oprofile and saw that a significant portion of CPU usage was going to calling the prePut coprocessor hook inside doMiniBatchPut, even though I don't have any coprocessors installed. I ran a million-row insert and collected CPU time spent in the RS after commenting out the preput hook, and found CPU usage reduced by 33%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3691) Add compressor support for 'snappy', google's compressor
[
https://issues.apache.org/jira/browse/HBASE-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266074#comment-13266074
]
Zhihong Yu commented on HBASE-3691:
---
This test failure might be related:
{code}
Running org.apache.hadoop.hbase.util.TestCompressionTest
Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 0.028 sec
FAILURE!
{code}
Add compressor support for 'snappy', google's compressor
Key: HBASE-3691
URL: https://issues.apache.org/jira/browse/HBASE-3691
Project: HBase
Issue Type: Task
Reporter: stack
Priority: Critical
Fix For: 0.90.7, 0.92.0
Attachments: hbase-snappy-0.90.6.patch,
hbase-snappy-3691-trunk-002.patch, hbase-snappy-3691-trunk-003.patch,
hbase-snappy-3691-trunk-004.patch, hbase-snappy-3691-trunk.patch
http://code.google.com/p/snappy/ is apache licensed.
bq. Snappy is a compression/decompression library. It does not aim for
maximum compression, or compatibility with any other compression library;
instead, it aims for very high speeds and reasonable compression. For
instance, compared to the fastest mode of zlib, Snappy is an order of
magnitude faster for most inputs, but the resulting compressed files are
anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in
64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses
at about 500 MB/sec or more.
bq. Snappy is widely used inside Google, in everything from BigTable and
MapReduce to our internal RPC systems. (Snappy has previously been referred
to as Zippy in some presentations and the likes.)
Lets get it in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for block cache lock
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5898: - Assignee: Todd Lipcon Assigning Todd. Consider double-checked locking for block cache lock Key: HBASE-5898 URL: https://issues.apache.org/jira/browse/HBASE-5898 Project: HBase Issue Type: Improvement Components: performance Affects Versions: 0.94.1 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5898) Consider double-checked locking for block cache lock
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5898: - Priority: Critical (was: Major) Marking this critical. Seems like small change w/ big win. All that is missing is a bit of exercise while under heavy read load. Consider double-checked locking for block cache lock Key: HBASE-5898 URL: https://issues.apache.org/jira/browse/HBASE-5898 Project: HBase Issue Type: Improvement Components: performance Affects Versions: 0.94.1 Reporter: Todd Lipcon Priority: Critical Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266081#comment-13266081 ] Jesse Yates commented on HBASE-5547: arg! Thought I had resolved the concurrency issue in TestRegionHFileArchiving. Looking into it. Don't delete HFiles when in backup mode - Key: HBASE-5547 URL: https://issues.apache.org/jira/browse/HBASE-5547 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Assignee: Jesse Yates Attachments: java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch This came up in a discussion I had with Stack. It would be nice if HBase could be notified that a backup is in progress (via a znode for example) and in that case either: 1. rename HFiles to be delete to file.bck 2. rename the HFiles into a special directory 3. rename them to a general trash directory (which would not need to be tied to backup mode). That way it should be able to get a consistent backup based on HFiles (HDFS snapshots or hard links would be better options here, but we do not have those). #1 makes cleanup a bit harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5907) enhance HLog pretty printer to print additional useful stats
[ https://issues.apache.org/jira/browse/HBASE-5907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266085#comment-13266085 ] stack commented on HBASE-5907: -- +1 Have you tried it against a few hlogs Kannan? enhance HLog pretty printer to print additional useful stats Key: HBASE-5907 URL: https://issues.apache.org/jira/browse/HBASE-5907 Project: HBase Issue Type: Improvement Reporter: Kannan Muthukkaruppan Priority: Minor Attachments: D2979.1.patch, D2979.2.patch It would be useful for analysis purposes to enhance the HLog pretty printer to optionally print a bunch of additional stats such as: 1) # of txns 2) # of KVs updated 3) avg size of txns 4) avg size of KVs 5) avg # of KVs written per txn 5) unique CF signatures involved in put/delete operatons; and breakdown of some of the above metrics by these signatures, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for block cache lock
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266090#comment-13266090 ] Todd Lipcon commented on HBASE-5898: The downside is that for a workload with a lower cache hit ratio, we'll ask the cache twice for every cache miss instead of just once. So we should see what the performance difference is for something like a 10MB cache with 1GB dataset, where the dataset fits in the OS buffer cache. Consider double-checked locking for block cache lock Key: HBASE-5898 URL: https://issues.apache.org/jira/browse/HBASE-5898 Project: HBase Issue Type: Improvement Components: performance Affects Versions: 0.94.1 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5897) prePut coprocessor hook causing substantial CPU usage
[ https://issues.apache.org/jira/browse/HBASE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5897: - Status: Open (was: Patch Available) prePut coprocessor hook causing substantial CPU usage - Key: HBASE-5897 URL: https://issues.apache.org/jira/browse/HBASE-5897 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: 5897-simple.txt, 5897-v2.txt, 5897-v3-0.92.txt, 5897-v3-0.94.txt, 5897-v3.txt, hbase-5897.txt, testRegionServerCoprocessorExceptionWithRemove.stack I was running an insert workload against trunk under oprofile and saw that a significant portion of CPU usage was going to calling the prePut coprocessor hook inside doMiniBatchPut, even though I don't have any coprocessors installed. I ran a million-row insert and collected CPU time spent in the RS after commenting out the preput hook, and found CPU usage reduced by 33%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HBASE-5897) prePut coprocessor hook causing substantial CPU usage
[ https://issues.apache.org/jira/browse/HBASE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl resolved HBASE-5897. -- Resolution: Fixed Hadoop Flags: Reviewed Committed to 0.92, 0.94, and 0.96. prePut coprocessor hook causing substantial CPU usage - Key: HBASE-5897 URL: https://issues.apache.org/jira/browse/HBASE-5897 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: 5897-simple.txt, 5897-v2.txt, 5897-v3-0.92.txt, 5897-v3-0.94.txt, 5897-v3.txt, hbase-5897.txt, testRegionServerCoprocessorExceptionWithRemove.stack I was running an insert workload against trunk under oprofile and saw that a significant portion of CPU usage was going to calling the prePut coprocessor hook inside doMiniBatchPut, even though I don't have any coprocessors installed. I ran a million-row insert and collected CPU time spent in the RS after commenting out the preput hook, and found CPU usage reduced by 33%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5897) prePut coprocessor hook causing substantial CPU usage
[ https://issues.apache.org/jira/browse/HBASE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lars Hofhansl updated HBASE-5897: - Attachment: 5897-v3-0.92.txt 0.92 patch prePut coprocessor hook causing substantial CPU usage - Key: HBASE-5897 URL: https://issues.apache.org/jira/browse/HBASE-5897 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: 5897-simple.txt, 5897-v2.txt, 5897-v3-0.92.txt, 5897-v3-0.94.txt, 5897-v3.txt, hbase-5897.txt, testRegionServerCoprocessorExceptionWithRemove.stack I was running an insert workload against trunk under oprofile and saw that a significant portion of CPU usage was going to calling the prePut coprocessor hook inside doMiniBatchPut, even though I don't have any coprocessors installed. I ran a million-row insert and collected CPU time spent in the RS after commenting out the preput hook, and found CPU usage reduced by 33%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-3691) Add compressor support for 'snappy', google's compressor
[ https://issues.apache.org/jira/browse/HBASE-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266094#comment-13266094 ] stack commented on HBASE-3691: -- Applied addendum to 0.90 branch. Add compressor support for 'snappy', google's compressor Key: HBASE-3691 URL: https://issues.apache.org/jira/browse/HBASE-3691 Project: HBase Issue Type: Task Reporter: stack Priority: Critical Fix For: 0.90.7, 0.92.0 Attachments: 3691-addendum.txt, hbase-snappy-0.90.6.patch, hbase-snappy-3691-trunk-002.patch, hbase-snappy-3691-trunk-003.patch, hbase-snappy-3691-trunk-004.patch, hbase-snappy-3691-trunk.patch http://code.google.com/p/snappy/ is apache licensed. bq. Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more. bq. Snappy is widely used inside Google, in everything from BigTable and MapReduce to our internal RPC systems. (Snappy has previously been referred to as Zippy in some presentations and the likes.) Lets get it in. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-3691) Add compressor support for 'snappy', google's compressor
[ https://issues.apache.org/jira/browse/HBASE-3691?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-3691: - Attachment: 3691-addendum.txt Removed snappy test; the plumbing is not usually in place (Thanks for noticing Ted) Add compressor support for 'snappy', google's compressor Key: HBASE-3691 URL: https://issues.apache.org/jira/browse/HBASE-3691 Project: HBase Issue Type: Task Reporter: stack Priority: Critical Fix For: 0.90.7, 0.92.0 Attachments: 3691-addendum.txt, hbase-snappy-0.90.6.patch, hbase-snappy-3691-trunk-002.patch, hbase-snappy-3691-trunk-003.patch, hbase-snappy-3691-trunk-004.patch, hbase-snappy-3691-trunk.patch http://code.google.com/p/snappy/ is apache licensed. bq. Snappy is a compression/decompression library. It does not aim for maximum compression, or compatibility with any other compression library; instead, it aims for very high speeds and reasonable compression. For instance, compared to the fastest mode of zlib, Snappy is an order of magnitude faster for most inputs, but the resulting compressed files are anywhere from 20% to 100% bigger. On a single core of a Core i7 processor in 64-bit mode, Snappy compresses at about 250 MB/sec or more and decompresses at about 500 MB/sec or more. bq. Snappy is widely used inside Google, in everything from BigTable and MapReduce to our internal RPC systems. (Snappy has previously been referred to as Zippy in some presentations and the likes.) Lets get it in. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5785) Adding unit tests for protbuf utils introduced for HRegionInterface pb conversion
[ https://issues.apache.org/jira/browse/HBASE-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5785: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk. Good one Jimmy. Thanks. Adding unit tests for protbuf utils introduced for HRegionInterface pb conversion - Key: HBASE-5785 URL: https://issues.apache.org/jira/browse/HBASE-5785 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Affects Versions: 0.96.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Critical Labels: noob Fix For: 0.96.0 Attachments: hbase-5785.patch We need to add some unit tests for the probuf utilities to catch issues earlier. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5155) ServerShutDownHandler And Disable/Delete should not happen parallely leading to recreation of regions that were deleted
[
https://issues.apache.org/jira/browse/HBASE-5155?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266099#comment-13266099
]
stack commented on HBASE-5155:
--
Should we revert and roll a 0.90.7?
ServerShutDownHandler And Disable/Delete should not happen parallely leading
to recreation of regions that were deleted
---
Key: HBASE-5155
URL: https://issues.apache.org/jira/browse/HBASE-5155
Project: HBase
Issue Type: Bug
Components: master
Affects Versions: 0.90.4
Reporter: ramkrishna.s.vasudevan
Assignee: ramkrishna.s.vasudevan
Priority: Blocker
Fix For: 0.90.6
Attachments: HBASE-5155_1.patch, HBASE-5155_2.patch,
HBASE-5155_3.patch, HBASE-5155_latest.patch, hbase-5155_6.patch
ServerShutDownHandler and disable/delete table handler races. This is not an
issue due to TM.
- A regionserver goes down. In our cluster the regionserver holds lot of
regions.
- A region R1 has two daughters D1 and D2.
- The ServerShutdownHandler gets called and scans the META and gets all the
user regions
- Parallely a table is disabled. (No problem in this step).
- Delete table is done.
- The tables and its regions are deleted including R1, D1 and D2.. (So META
is cleaned)
- Now ServerShutdownhandler starts to processTheDeadRegion
{code}
if (hri.isOffline() hri.isSplit()) {
LOG.debug(Offlined and split region + hri.getRegionNameAsString() +
; checking daughter presence);
fixupDaughters(result, assignmentManager, catalogTracker);
{code}
As part of fixUpDaughters as the daughers D1 and D2 is missing for R1
{code}
if (isDaughterMissing(catalogTracker, daughter)) {
LOG.info(Fixup; missing daughter + daughter.getRegionNameAsString());
MetaEditor.addDaughter(catalogTracker, daughter, null);
// TODO: Log WARN if the regiondir does not exist in the fs. If its not
// there then something wonky about the split -- things will keep going
// but could be missing references to parent region.
// And assign it.
assignmentManager.assign(daughter, true);
{code}
we call assign of the daughers.
Now after this we again start with the below code.
{code}
if (processDeadRegion(e.getKey(), e.getValue(),
this.services.getAssignmentManager(),
this.server.getCatalogTracker())) {
this.services.getAssignmentManager().assign(e.getKey(), true);
{code}
Now when the SSH scanned the META it had R1, D1 and D2.
So as part of the above code D1 and D2 which where assigned by fixUpDaughters
is again assigned by
{code}
this.services.getAssignmentManager().assign(e.getKey(), true);
{code}
Thus leading to a zookeeper issue due to bad version and killing the master.
The important part here is the regions that were deleted are recreated which
i think is more critical.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5886) Add new metric for possible data loss due to puts without WAL
[ https://issues.apache.org/jira/browse/HBASE-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266102#comment-13266102 ] Todd Lipcon commented on HBASE-5886: Or maybe drop it to INFO level? Add new metric for possible data loss due to puts without WAL -- Key: HBASE-5886 URL: https://issues.apache.org/jira/browse/HBASE-5886 Project: HBase Issue Type: New Feature Components: metrics, regionserver Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Labels: metrics Attachments: HBASE-5886-v0.patch, HBASE-5886-v1.patch, HBASE-5886-v2.patch Add a metrics to keep track of puts without WAL and possible data loss size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for block cache lock
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266101#comment-13266101 ] stack commented on HBASE-5898: -- Thanks for the prescription Todd. Consider double-checked locking for block cache lock Key: HBASE-5898 URL: https://issues.apache.org/jira/browse/HBASE-5898 Project: HBase Issue Type: Improvement Components: performance Affects Versions: 0.94.1 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5888) Clover profile in build
[ https://issues.apache.org/jira/browse/HBASE-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] stack updated HBASE-5888: - Resolution: Fixed Fix Version/s: 0.96.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk. Thanks for the patch Enis. Clover profile in build --- Key: HBASE-5888 URL: https://issues.apache.org/jira/browse/HBASE-5888 Project: HBase Issue Type: Improvement Components: build, test Affects Versions: 0.92.2, 0.96.0, 0.94.1 Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.96.0 Attachments: hbase-clover_v1.patch, hbase-clover_v2.patch Clover is disabled right now. I would like to add a profile that enables clover reports. We can also backport this to 0.92, and 0.94, since we are also interested in test coverage for those branches. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5847) Support createTable(splitKeys) in Thrift
[ https://issues.apache.org/jira/browse/HBASE-5847?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266108#comment-13266108 ] stack commented on HBASE-5847: -- @Nicolas Up to you. Not necessary. Just thought if it was there already, you could take it from thrift2. One day we'll have to reconcile the two and go for one or the other. Until then, they will both get the fly-by-treatment w/ fellas adding what they need just now. Support createTable(splitKeys) in Thrift Key: HBASE-5847 URL: https://issues.apache.org/jira/browse/HBASE-5847 Project: HBase Issue Type: Improvement Reporter: Nicolas Spiegelberg Assignee: Nicolas Spiegelberg Priority: Trivial The Thrift API does not allow a user to create a table with multiple split keys. This is needed for a handful of new internal projects that are written in PHP/C++. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5886) Add new metric for possible data loss due to puts without WAL
[ https://issues.apache.org/jira/browse/HBASE-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266117#comment-13266117 ] stack commented on HBASE-5886: -- I love the war story. Any more where that came from for our entertainment? Patch looks good to me. I can drop the log level on commit if thats what you want Matteo. Add new metric for possible data loss due to puts without WAL -- Key: HBASE-5886 URL: https://issues.apache.org/jira/browse/HBASE-5886 Project: HBase Issue Type: New Feature Components: metrics, regionserver Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Labels: metrics Attachments: HBASE-5886-v0.patch, HBASE-5886-v1.patch, HBASE-5886-v2.patch Add a metrics to keep track of puts without WAL and possible data loss size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5904) is_enabled from shell returns differently from pre- and post- HBASE-5155
[ https://issues.apache.org/jira/browse/HBASE-5904?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266120#comment-13266120 ] stack commented on HBASE-5904: -- @David Ignore my query over in hbase-5155. I should probably run a 0.90.7 once it goes in... is_enabled from shell returns differently from pre- and post- HBASE-5155 Key: HBASE-5904 URL: https://issues.apache.org/jira/browse/HBASE-5904 Project: HBase Issue Type: Bug Components: zookeeper Affects Versions: 0.90.6 Reporter: David S. Wang If I launch an hbase shell that uses HBase and ZooKeeper without HBASE-5155, against HBase servers with HBASE-5155, then is_enabled for a table always returns false even if the table is considered enabled by the servers from the logs. If I then do the same thing but with an HBase shell and ZooKeeper with HBASE-5155, then is_enabled returns as expected. If I launch an HBase shell that uses HBase and ZooKeeper without HBASE-5155, against HBase servers also without HBASE-5155, then is_enabled works as you'd expect. But if I then do the same thing but with an HBase shell and ZooKeeper with HBASE-5155, then is_enabled returns false even though the table is considered enabled by the servers from the logs. Additionally, if I then try to enable the table from the HBASE-5155-containing shell, it hangs because the ZooKeeper code waits for the ZNode to be updated with ENABLED in the data field, but what actually happens is that the ZNode gets deleted since the servers are running without HBASE-5155. I think the culprit is that the indication of how a table is considered enabled inside ZooKeeper has changed with HBASE-5155. Before HBASE-5155, a table was considered enabled if the ZNode for it did not exist. After HBASE-5155, a table is considered enabled if the ZNode for it exists and has ENABLED in its data. I think the current code is incompatible when running clients and servers where one side has HBASE-5155 and the other side does not. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5886) Add new metric for possible data loss due to puts without WAL
[ https://issues.apache.org/jira/browse/HBASE-5886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266124#comment-13266124 ] Matteo Bertozzi commented on HBASE-5886: INFO level is good for me. The ideas is having something to grep. Add new metric for possible data loss due to puts without WAL -- Key: HBASE-5886 URL: https://issues.apache.org/jira/browse/HBASE-5886 Project: HBase Issue Type: New Feature Components: metrics, regionserver Reporter: Matteo Bertozzi Assignee: Matteo Bertozzi Priority: Minor Labels: metrics Attachments: HBASE-5886-v0.patch, HBASE-5886-v1.patch, HBASE-5886-v2.patch Add a metrics to keep track of puts without WAL and possible data loss size. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for block cache lock
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266123#comment-13266123 ] Jean-Daniel Cryans commented on HBASE-5898: --- bq. This addresses the case where there's a really high cache hit ratio, whereas that one addresses the case where there's a 0% cache hit ratio. Now that I actually read the patch I see how I was wrong, sorry for the noise. Consider double-checked locking for block cache lock Key: HBASE-5898 URL: https://issues.apache.org/jira/browse/HBASE-5898 Project: HBase Issue Type: Improvement Components: performance Affects Versions: 0.94.1 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5908) TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use append to corrupt the HLog
[ https://issues.apache.org/jira/browse/HBASE-5908?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266127#comment-13266127 ] Hudson commented on HBASE-5908: --- Integrated in HBase-0.94-security #25 (See [https://builds.apache.org/job/HBase-0.94-security/25/]) HBASE-5908. TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use append to corrupt the HLog. Contributed by Gregory Chanan. (Revision 1332494) Result = SUCCESS todd : Files : * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/regionserver/wal/TestHLogSplit.java TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses should not use append to corrupt the HLog Key: HBASE-5908 URL: https://issues.apache.org/jira/browse/HBASE-5908 Project: HBase Issue Type: Bug Components: test, wal Affects Versions: 0.96.0 Reporter: Gregory Chanan Assignee: Gregory Chanan Priority: Minor Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0 Attachments: HBASE-5908-trunk.patch TestHLogSplit.testTralingGarbageCorruptionFileSkipErrorsPasses fails against a version of hadoop with https://issues.apache.org/jira/browse/HADOOP-8230 The failure: java.io.IOException: Append is not supported. Please see the dfs.support.append configuration parameter. Instead of using append, we can probably just: - copy over the contents to a new file - append the garbage to the new file - copy back to the old file -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5897) prePut coprocessor hook causing substantial CPU usage
[ https://issues.apache.org/jira/browse/HBASE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266129#comment-13266129 ] Hudson commented on HBASE-5897: --- Integrated in HBase-0.94-security #25 (See [https://builds.apache.org/job/HBase-0.94-security/25/]) HBASE-5897 prePut coprocessor hook causing substantial CPU usage (Revision 1332810) Result = SUCCESS larsh : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java prePut coprocessor hook causing substantial CPU usage - Key: HBASE-5897 URL: https://issues.apache.org/jira/browse/HBASE-5897 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: 5897-simple.txt, 5897-v2.txt, 5897-v3-0.92.txt, 5897-v3-0.94.txt, 5897-v3.txt, hbase-5897.txt, testRegionServerCoprocessorExceptionWithRemove.stack I was running an insert workload against trunk under oprofile and saw that a significant portion of CPU usage was going to calling the prePut coprocessor hook inside doMiniBatchPut, even though I don't have any coprocessors installed. I ran a million-row insert and collected CPU time spent in the RS after commenting out the preput hook, and found CPU usage reduced by 33%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5906) TestChangingEncoding failing sporadically in 0.94 build
[ https://issues.apache.org/jira/browse/HBASE-5906?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266130#comment-13266130 ] Hudson commented on HBASE-5906: --- Integrated in HBase-0.94-security #25 (See [https://builds.apache.org/job/HBase-0.94-security/25/]) HBASE-5906 TestChangingEncoding failing sporadically in 0.94 build (Revision 1332319) Result = SUCCESS stack : Files : * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/io/encoding/TestChangingEncoding.java TestChangingEncoding failing sporadically in 0.94 build --- Key: HBASE-5906 URL: https://issues.apache.org/jira/browse/HBASE-5906 Project: HBase Issue Type: Bug Reporter: stack Attachments: 5906.txt The test passes locally for me and Elliott but takes a long time to run. Timeout is only two minutes for the test though. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5611) Replayed edits from regions that failed to open during recovery aren't removed from the global MemStore size
[
https://issues.apache.org/jira/browse/HBASE-5611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266126#comment-13266126
]
Hudson commented on HBASE-5611:
---
Integrated in HBase-0.94-security #25 (See
[https://builds.apache.org/job/HBase-0.94-security/25/])
HBASE-5611 Replayed edits from regions that failed to open during recovery
aren't removed from the global MemStore size - v2 (Jieshan) (Revision 1332344)
Result = SUCCESS
tedyu :
Files :
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerAccounting.java
*
/hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/handler/OpenRegionHandler.java
Replayed edits from regions that failed to open during recovery aren't
removed from the global MemStore size
Key: HBASE-5611
URL: https://issues.apache.org/jira/browse/HBASE-5611
Project: HBase
Issue Type: Bug
Affects Versions: 0.90.6
Reporter: Jean-Daniel Cryans
Assignee: Jieshan Bean
Priority: Critical
Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0
Attachments: 5611-94-v2.txt, 5611-94.addendum, HBASE-5611-92.patch,
HBASE-5611-94-minorchange.patch, HBASE-5611-trunk-v2-minorchange.patch
This bug is rather easy to get if the {{TimeoutMonitor}} is on, else I think
it's still possible to hit it if a region fails to open for more obscure
reasons like HDFS errors.
Consider a region that just went through distributed splitting and that's now
being opened by a new RS. The first thing it does is to read the recovery
files and put the edits in the {{MemStores}}. If this process takes a long
time, the master will move that region away. At that point the edits are
still accounted for in the global {{MemStore}} size but they are dropped when
the {{HRegion}} gets cleaned up. It's completely invisible until the
{{MemStoreFlusher}} needs to force flush a region and that none of them have
edits:
{noformat}
2012-03-21 00:33:39,303 DEBUG
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Flush thread woke up
because memory above low water=5.9g
2012-03-21 00:33:39,303 ERROR
org.apache.hadoop.hbase.regionserver.MemStoreFlusher: Cache flusher failed
for entry null
java.lang.IllegalStateException
at
com.google.common.base.Preconditions.checkState(Preconditions.java:129)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushOneForGlobalPressure(MemStoreFlusher.java:199)
at
org.apache.hadoop.hbase.regionserver.MemStoreFlusher.run(MemStoreFlusher.java:223)
at java.lang.Thread.run(Thread.java:662)
{noformat}
The {{null}} here is a region. In my case I had so many edits in the
{{MemStore}} during recovery that I'm over the low barrier although in fact
I'm at 0. It happened yesterday and it still printing this out.
To fix this we need to be able to decrease the global {{MemStore}} size when
the region can't open.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5712) Parallelize load of .regioninfo files in diagnostic/repair portion of hbck.
[ https://issues.apache.org/jira/browse/HBASE-5712?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266128#comment-13266128 ] Hudson commented on HBASE-5712: --- Integrated in HBase-0.94-security #25 (See [https://builds.apache.org/job/HBase-0.94-security/25/]) HBASE-5712 Parallelize load of .regioninfo files in diagnostic/repair portion of hbck (Revision 1332071) Result = SUCCESS jmhsieh : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java * /hbase/branches/0.94/src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java Parallelize load of .regioninfo files in diagnostic/repair portion of hbck. --- Key: HBASE-5712 URL: https://issues.apache.org/jira/browse/HBASE-5712 Project: HBase Issue Type: Improvement Components: hbck Affects Versions: 0.90.7, 0.92.2, 0.94.0, 0.96.0 Reporter: Jonathan Hsieh Assignee: Jonathan Hsieh Fix For: 0.90.7, 0.92.2, 0.94.0, 0.96.0 Attachments: hbase-5712-90-v2.patch, hbase-5712-90.patch, hbase-5712-v2.patch, hbase-5712.patch On heavily loaded hdfs's some dfs nodes may not respond quickly and backs off for 60s before attempting to read data from another datanode. Portions of the information gathered from hdfs (.regioninfo files) are loaded serially. With HBase with clusters with 100's, or 1000's, or 1's regions encountering these 60s delay blocks progress and can be very painful. There is already some parallelization of portions of the hdfs information load operations and the goal here is move the reading of .regioninfos into the parallelized sections.. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5901) Use union type protobufs instead of class/byte pairs for multi requests
[ https://issues.apache.org/jira/browse/HBASE-5901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266133#comment-13266133 ] Jimmy Xiang commented on HBASE-5901: +1 Use union type protobufs instead of class/byte pairs for multi requests --- Key: HBASE-5901 URL: https://issues.apache.org/jira/browse/HBASE-5901 Project: HBase Issue Type: Improvement Components: ipc, performance Affects Versions: 0.96.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical Attachments: hbase-5901.txt The current implementation of multi actions uses repeated NameBytesPairs for the contents of multi actions. Instead, we should introduce a union type protobuf for the valid actions. This makes the RPCs smaller since they don't need to carry class names, and makes deserialization faster since it can avoid some copying and reflection. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5884) MapReduce package info has broken link to bulk-loads
[ https://issues.apache.org/jira/browse/HBASE-5884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266131#comment-13266131 ] Hudson commented on HBASE-5884: --- Integrated in HBase-0.94-security #25 (See [https://builds.apache.org/job/HBase-0.94-security/25/]) HBASE-5884 MapReduce package info has broken link to bulk-loads (Revision 1332441) Result = SUCCESS stack : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/mapreduce/package-info.java MapReduce package info has broken link to bulk-loads Key: HBASE-5884 URL: https://issues.apache.org/jira/browse/HBASE-5884 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 0.96.0 Reporter: Jesse Yates Assignee: Jesse Yates Priority: Trivial Fix For: 0.94.0, 0.96.0 Attachments: doc_HBASE-5884.patch Bulk Loads link goes to an old link, which we have dropped recently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HBASE-5901) Use union type protobufs instead of class/byte pairs for multi requests
[ https://issues.apache.org/jira/browse/HBASE-5901?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HBASE-5901: --- Attachment: hbase-5901.txt Patch rebased against trunk (something else conflicted with it) Use union type protobufs instead of class/byte pairs for multi requests --- Key: HBASE-5901 URL: https://issues.apache.org/jira/browse/HBASE-5901 Project: HBase Issue Type: Improvement Components: ipc, performance Affects Versions: 0.96.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical Attachments: hbase-5901.txt, hbase-5901.txt The current implementation of multi actions uses repeated NameBytesPairs for the contents of multi actions. Instead, we should introduce a union type protobuf for the valid actions. This makes the RPCs smaller since they don't need to carry class names, and makes deserialization faster since it can avoid some copying and reflection. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb
[ https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266144#comment-13266144 ] Hadoop QA commented on HBASE-5869: -- +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12525216/v13.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 50 new or modified tests. +1 hadoop23. The patch compiles against the hadoop 0.23.x profile. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . Test results: https://builds.apache.org/job/PreCommit-HBASE-Build/1711//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HBASE-Build/1711//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Console output: https://builds.apache.org/job/PreCommit-HBASE-Build/1711//console This message is automatically generated. Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb - Key: HBASE-5869 URL: https://issues.apache.org/jira/browse/HBASE-5869 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Attachments: 5869v7.txt, 5869v8.txt, 5869v9.txt, firstcut.txt, secondcut.txt, v10.txt, v11.txt, v12.txt, v13.txt, v4.txt, v5.txt, v6.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5897) prePut coprocessor hook causing substantial CPU usage
[ https://issues.apache.org/jira/browse/HBASE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266147#comment-13266147 ] Hudson commented on HBASE-5897: --- Integrated in HBase-0.94 #165 (See [https://builds.apache.org/job/HBase-0.94/165/]) HBASE-5897 prePut coprocessor hook causing substantial CPU usage (Revision 1332810) Result = FAILURE larsh : Files : * /hbase/branches/0.94/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java prePut coprocessor hook causing substantial CPU usage - Key: HBASE-5897 URL: https://issues.apache.org/jira/browse/HBASE-5897 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: 5897-simple.txt, 5897-v2.txt, 5897-v3-0.92.txt, 5897-v3-0.94.txt, 5897-v3.txt, hbase-5897.txt, testRegionServerCoprocessorExceptionWithRemove.stack I was running an insert workload against trunk under oprofile and saw that a significant portion of CPU usage was going to calling the prePut coprocessor hook inside doMiniBatchPut, even though I don't have any coprocessors installed. I ran a million-row insert and collected CPU time spent in the RS after commenting out the preput hook, and found CPU usage reduced by 33%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5898) Consider double-checked locking for block cache lock
[ https://issues.apache.org/jira/browse/HBASE-5898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266149#comment-13266149 ] Jean-Daniel Cryans commented on HBASE-5898: --- bq. The downside is that for a workload with a lower cache hit ratio, we'll ask the cache twice for every cache miss instead of just once. Won't this also skew the cache stats themselves? Consider double-checked locking for block cache lock Key: HBASE-5898 URL: https://issues.apache.org/jira/browse/HBASE-5898 Project: HBase Issue Type: Improvement Components: performance Affects Versions: 0.94.1 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical Attachments: 5898-TestBlocksRead.txt, hbase-5898.txt Running a workload with a high query rate against a dataset that fits in cache, I saw a lot of CPU being used in IdLock.getLockEntry, being called by HFileReaderV2.readBlock. Even though it was all cache hits, it was wasting a lot of CPU doing lock management here. I wrote a quick patch to switch to a double-checked locking and it improved throughput substantially for this workload. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HBASE-5910) Move incrementing/mutating of Dynamic and Regular Metrics into one place
Elliott Clark created HBASE-5910: Summary: Move incrementing/mutating of Dynamic and Regular Metrics into one place Key: HBASE-5910 URL: https://issues.apache.org/jira/browse/HBASE-5910 Project: HBase Issue Type: Improvement Reporter: Elliott Clark Assignee: Elliott Clark Currently there are several places that were are timing the length of a request. * In HRegion * In HRegionServer * And in HFile. Some of these calls to get the system time can be merged into single or at most two different times. Also these timing calls should be changed to use NanoTime to get elapsed time that is drift resistant. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5888) Clover profile in build
[ https://issues.apache.org/jira/browse/HBASE-5888?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266165#comment-13266165 ] Hudson commented on HBASE-5888: --- Integrated in HBase-TRUNK #2834 (See [https://builds.apache.org/job/HBase-TRUNK/2834/]) HBASE-5888 Clover profile in build (Revision 1332829) Result = FAILURE stack : Files : * /hbase/trunk/pom.xml Clover profile in build --- Key: HBASE-5888 URL: https://issues.apache.org/jira/browse/HBASE-5888 Project: HBase Issue Type: Improvement Components: build, test Affects Versions: 0.92.2, 0.96.0, 0.94.1 Reporter: Enis Soztutar Assignee: Enis Soztutar Fix For: 0.96.0 Attachments: hbase-clover_v1.patch, hbase-clover_v2.patch Clover is disabled right now. I would like to add a profile that enables clover reports. We can also backport this to 0.92, and 0.94, since we are also interested in test coverage for those branches. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5897) prePut coprocessor hook causing substantial CPU usage
[ https://issues.apache.org/jira/browse/HBASE-5897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266164#comment-13266164 ] Hudson commented on HBASE-5897: --- Integrated in HBase-TRUNK #2834 (See [https://builds.apache.org/job/HBase-TRUNK/2834/]) HBASE-5897 prePut coprocessor hook causing substantial CPU usage (Revision 1332811) Result = FAILURE larsh : Files : * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java prePut coprocessor hook causing substantial CPU usage - Key: HBASE-5897 URL: https://issues.apache.org/jira/browse/HBASE-5897 Project: HBase Issue Type: Bug Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical Fix For: 0.92.2, 0.94.0, 0.96.0 Attachments: 5897-simple.txt, 5897-v2.txt, 5897-v3-0.92.txt, 5897-v3-0.94.txt, 5897-v3.txt, hbase-5897.txt, testRegionServerCoprocessorExceptionWithRemove.stack I was running an insert workload against trunk under oprofile and saw that a significant portion of CPU usage was going to calling the prePut coprocessor hook inside doMiniBatchPut, even though I don't have any coprocessors installed. I ran a million-row insert and collected CPU time spent in the RS after commenting out the preput hook, and found CPU usage reduced by 33%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5785) Adding unit tests for protbuf utils introduced for HRegionInterface pb conversion
[ https://issues.apache.org/jira/browse/HBASE-5785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266163#comment-13266163 ] Hudson commented on HBASE-5785: --- Integrated in HBase-TRUNK #2834 (See [https://builds.apache.org/job/HBase-TRUNK/2834/]) HBASE-5785 Adding unit tests for protbuf utils introduced for HRegionInterface pb conversion (Revision 1332824) Result = FAILURE stack : Files : * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/protobuf * /hbase/trunk/src/test/java/org/apache/hadoop/hbase/protobuf/TestProtobufUtil.java Adding unit tests for protbuf utils introduced for HRegionInterface pb conversion - Key: HBASE-5785 URL: https://issues.apache.org/jira/browse/HBASE-5785 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Affects Versions: 0.96.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Priority: Critical Labels: noob Fix For: 0.96.0 Attachments: hbase-5785.patch We need to add some unit tests for the probuf utilities to catch issues earlier. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5444) Add PB-based calls to HMasterRegionInterface
[ https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266168#comment-13266168 ] jirapos...@reviews.apache.org commented on HBASE-5444: -- bq. On 2012-05-01 20:16:06, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java, line 1128 bq. https://reviews.apache.org/r/4463/diff/4/?file=105814#file105814line1128 bq. bq. Insert a space between if and (. Will do. bq. On 2012-05-01 20:16:06, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java, line 1145 bq. https://reviews.apache.org/r/4463/diff/4/?file=105814#file105814line1145 bq. bq. You can return list.toArray() directly, similar to line 1179. I still need to convert from Coprocessor to String (via getName) if I call toArray(). Am I missing something? - Gregory --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4463/#review7441 --- On 2012-05-01 19:53:51, Gregory Chanan wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4463/ bq. --- bq. bq. (Updated 2012-05-01 19:53:51) bq. bq. bq. Review request for hbase and Michael Stack. bq. bq. bq. Summary bq. --- bq. bq. Adds PB-based calls replacing HMasterRegionInterface. bq. bq. There are some temporary hacks, e.g. converting PB-based ServerLoad to existing HServerLoad so I didn't need to convert ClusterStatus (which brings in a lot of other changes). That will be cleaned up in HBASE-5445. bq. bq. bq. This addresses bug HBASE-5444. bq. https://issues.apache.org/jira/browse/HBASE-5444 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 0db2760 bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 973c7cb bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java fd97830 bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java bb6ab3b bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java f56127d bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 81e9023 bq.src/main/java/org/apache/hadoop/hbase/master/MXBean.java 7f44dc2 bq.src/main/java/org/apache/hadoop/hbase/master/MXBeanImpl.java 45b8fe7 bq.src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java be63838 bq. src/main/java/org/apache/hadoop/hbase/master/RegionServerStatusProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 80271b1 bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 994cb76 bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java efcf74d bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 5d7f07b bq. src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 69434f7 bq. src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 3c7c091 bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java ebffad6 bq.src/main/protobuf/RegionServerStatus.proto PRE-CREATION bq.src/main/protobuf/hbase.proto 12e6053 bq.src/main/resources/hbase-webapps/master/table.jsp 3ef1190 bq.src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 72554cb bq.src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java d039be3 bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 36046f8 bq.src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java bd5fa90 bq.src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java f8029ba bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java e99d251 bq. bq. Diff: https://reviews.apache.org/r/4463/diff bq. bq. bq. Testing bq. --- bq. bq. Ran jenkins job, all unit tests passed. bq. bq. bq. Thanks, bq. bq. Gregory bq. bq. Add PB-based calls to HMasterRegionInterface Key: HBASE-5444 URL: https://issues.apache.org/jira/browse/HBASE-5444 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: Gregory Chanan Attachments: HBASE-5444-v6-trunk.patch
[jira] [Commented] (HBASE-5444) Add PB-based calls to HMasterRegionInterface
[ https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266167#comment-13266167 ] jirapos...@reviews.apache.org commented on HBASE-5444: -- bq. On 2012-05-01 20:20:09, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/HConstants.java, line 679 bq. https://reviews.apache.org/r/4463/diff/4/?file=105802#file105802line679 bq. bq. Were you going to move this down to where its used G? Does it need to be up here? I can just get rid of this and use null everywhere. That's what it looks like Jimmy did. bq. On 2012-05-01 20:20:09, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java, line 173 bq. https://reviews.apache.org/r/4463/diff/4/?file=105817#file105817line173 bq. bq. We should not be reaching over into the master package. Put this protocol class at the top level since shared by master and regionserver? Will do. bq. On 2012-05-01 20:20:09, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/master/MXBean.java, line 23 bq. https://reviews.apache.org/r/4463/diff/4/?file=105809#file105809line23 bq. bq. All of these classes are importing generated pb classes. Would it be better to have a high-level ServerLoad class that hid inside it the pb stuff instead? Less pb generated class pollution. I think that's a good idea. I was considering doing that, but wasn't sure if I should do it for all the generated pb types. Since ServerLoad is sprinkled everywhere, it seems like that is at least a good place to start. - Gregory --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4463/#review7442 --- On 2012-05-01 19:53:51, Gregory Chanan wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4463/ bq. --- bq. bq. (Updated 2012-05-01 19:53:51) bq. bq. bq. Review request for hbase and Michael Stack. bq. bq. bq. Summary bq. --- bq. bq. Adds PB-based calls replacing HMasterRegionInterface. bq. bq. There are some temporary hacks, e.g. converting PB-based ServerLoad to existing HServerLoad so I didn't need to convert ClusterStatus (which brings in a lot of other changes). That will be cleaned up in HBASE-5445. bq. bq. bq. This addresses bug HBASE-5444. bq. https://issues.apache.org/jira/browse/HBASE-5444 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/HConstants.java a9d80a0 bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 0db2760 bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 973c7cb bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java fd97830 bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java bb6ab3b bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java f56127d bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 81e9023 bq.src/main/java/org/apache/hadoop/hbase/master/MXBean.java 7f44dc2 bq.src/main/java/org/apache/hadoop/hbase/master/MXBeanImpl.java 45b8fe7 bq.src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java be63838 bq. src/main/java/org/apache/hadoop/hbase/master/RegionServerStatusProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 80271b1 bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 994cb76 bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java efcf74d bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 5d7f07b bq. src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 69434f7 bq. src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon 3c7c091 bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionServerStatusProtos.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java ebffad6 bq.src/main/protobuf/RegionServerStatus.proto PRE-CREATION bq.src/main/protobuf/hbase.proto 12e6053 bq.src/main/resources/hbase-webapps/master/table.jsp 3ef1190 bq.src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 72554cb bq.src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java d039be3 bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 36046f8 bq.src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java bd5fa90 bq.src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java f8029ba bq.
[jira] [Commented] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb
[ https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266179#comment-13266179 ] jirapos...@reviews.apache.org commented on HBASE-5869: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4926/#review7446 --- src/main/java/org/apache/hadoop/hbase/SplitLogCounters.java https://reviews.apache.org/r/4926/#comment16378 Usually this line is not the first in a file. src/main/java/org/apache/hadoop/hbase/SplitLogCounters.java https://reviews.apache.org/r/4926/#comment16379 'log' seems redundant here. src/main/java/org/apache/hadoop/hbase/SplitLogCounters.java https://reviews.apache.org/r/4926/#comment16377 What if an AtomicInteger counter is added in the future ? src/main/java/org/apache/hadoop/hbase/SplitLogTask.java https://reviews.apache.org/r/4926/#comment16381 'date' - 'data' src/main/java/org/apache/hadoop/hbase/SplitLogTask.java https://reviews.apache.org/r/4926/#comment16382 'An' - 'A' src/main/java/org/apache/hadoop/hbase/SplitLogTask.java https://reviews.apache.org/r/4926/#comment16383 What would the first 64 bytes of data represent ? Do we know that data.length = 64 ? src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java https://reviews.apache.org/r/4926/#comment16408 Should e1 be included in the log ? 're-' before 'resubmit' is not necessary. - Ted On 2012-05-01 20:42:36, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4926/ bq. --- bq. bq. (Updated 2012-05-01 20:42:36) bq. bq. bq. Review request for hbase and Jimmy Xiang. bq. bq. bq. Summary bq. --- bq. bq. Convert two zk users to pb: distributed log splitting and regions in transition. bq. bq. Refactored distributed log splitting so we only serialize/deserialize in one location. bq. Less changes needed to do same for regions in transition. bq. bq. Moves serialization/deserialization out of the ZKAssign, ZKSplit and into bq. the classes themselves so can encapsulate how serialization is done into one place bq. (try to make the ZK* classes just deal in bytes -- about 90% done). bq. bq. Moved classes used by various packages up to top level to minimize imports bq. that are across package (zookeeper into protobuf and/or into regionserver and/or bq. master packages, etc). bq. bq. A src/main/java/org/apache/hadoop/hbase/DeserializationException.java bq.New generic deserialization exception. bq. A src/main/java/org/apache/hadoop/hbase/zookeeper/EmptyWatcher.java bq. D src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java bq.Moved under zookeeper package. bq. A src/main/java/org/apache/hadoop/hbase/HBaseException.java bq.New base hbase exception as suggested by hbase-5796. New DeserializationException bq.inherits from this. bq. A src/main/java/org/apache/hadoop/hbase/RegionTransition.java bq.State of a region in transition. Top-level because used by a bq.few top-level packages. Encapsulates pb serialization/deserialization. bq. M src/main/java/org/apache/hadoop/hbase/ServerName.java bq.Add method to deserialize a ServeName, etc. Encapsulates pb'ing. bq. M src/main/java/org/apache/hadoop/hbase/SplitLogCounters.java bq.Counters used by distributed log splitting. bq. A SplitLogTask bq. Class that encapsulates log splitting state. Also encapsulates pb'ing. bq. M src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java bq.Implement code for state. Added functions to go from code to state and vice bq.versa. Used serializing. bq. M src/main/java/org/apache/hadoop/hbase/executor/ExecutorService.java bq.Remove unused imports. bq. D src/main/java/org/apache/hadoop/hbase/executor/RegionTransitionData.java bq.Removed. Replaced by RegionTransition moved to package top-level. bq. M src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java bq. M src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java bq.Use new DeserializationException. Move to using new RegionTransition bq.from RegionTransitionData class. Pass deserialized class rather than bq.byte array. Remove duplicated code. bq. M src/main/java/org/apache/hadoop/hbase/master/HMaster.java bq.Use new ServerName parse method rather than ZKUtil one. bq. M src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java bq. M src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java bq.Redo to use new SplitLogTask and SplitLogCounter classes. bq. M
[jira] [Updated] (HBASE-5547) Don't delete HFiles when in backup mode
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jesse Yates updated HBASE-5547: --- Attachment: java_HBASE-5547_v6.patch Attaching new patch. Just ran the test 10 times locally, no failures. Don't delete HFiles when in backup mode - Key: HBASE-5547 URL: https://issues.apache.org/jira/browse/HBASE-5547 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Assignee: Jesse Yates Attachments: java_HBASE-5547_v4.patch, java_HBASE-5547_v5.patch, java_HBASE-5547_v6.patch This came up in a discussion I had with Stack. It would be nice if HBase could be notified that a backup is in progress (via a znode for example) and in that case either: 1. rename HFiles to be delete to file.bck 2. rename the HFiles into a special directory 3. rename them to a general trash directory (which would not need to be tied to backup mode). That way it should be able to get a consistent backup based on HFiles (HDFS snapshots or hard links would be better options here, but we do not have those). #1 makes cleanup a bit harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266190#comment-13266190 ] jirapos...@reviews.apache.org commented on HBASE-5547: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/#review7448 --- Great work. High-level, do we have to have backup/archiving touch all parts of the code base? Can we not hide it behind more general Interfaces at least when we get down into Store and Region classes src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java https://reviews.apache.org/r/4633/#comment16389 Tables or hfiles? src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveMonitor.java https://reviews.apache.org/r/4633/#comment16384 Should this interface be at top level? Its used out of the master and regionserver packages The implementations could be stay under backup long as we only use the Interface? src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java https://reviews.apache.org/r/4633/#comment16390 hfiles or tables? Looks like this class would select hfiles by selected tables src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java https://reviews.apache.org/r/4633/#comment16387 Tracker in hbase usually means tracking znodes up in zookeeper. This tracks something else. Could confuse. Use different action? Manager src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java https://reviews.apache.org/r/4633/#comment16385 Good. I like that you are taking a 'Server' here rather than a Master or Regionserver or even a ServerName. Does this have to be public? Seems like its utility used by the later HFileArchiveTracker. src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java https://reviews.apache.org/r/4633/#comment16386 Do we have to have a set? Does this Set change during live of this tracker? Should javadoc say that previous members of Set are cleared out? src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTableTracker.java https://reviews.apache.org/r/4633/#comment16388 good src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java https://reviews.apache.org/r/4633/#comment16392 How does this tracker relate to the containing class? They are different types of trackers? Could confuse? src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java https://reviews.apache.org/r/4633/#comment16391 Why pass it in? Why not just create one and keep it running internally? The class HFileArchiveTableTracker is a little confusing src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java https://reviews.apache.org/r/4633/#comment16395 Want to mention in class comment that user needs to call start, etc., to start up the tracking? src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java https://reviews.apache.org/r/4633/#comment16399 I think it kinda bad that zkw does this... you'd think each tracker would look after its own stuff (this is not you.. you are folllowing the established pattern as you should -- I'm just moaning) src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java https://reviews.apache.org/r/4633/#comment16400 Is this what ZKUtil.getNodeName does? src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java https://reviews.apache.org/r/4633/#comment16401 nit style. if you did if (!path.startsWith...) return; ... you could save an indent of the whole method. Next time. src/main/java/org/apache/hadoop/hbase/backup/HFileArchiveTracker.java https://reviews.apache.org/r/4633/#comment16403 These znodes have nothing in them... just the tablename of the znode is used? Nothing to pb then? src/main/java/org/apache/hadoop/hbase/backup/ServerHFileTableArchiveTracker.java https://reviews.apache.org/r/4633/#comment16405 This is under each table znode? Why we inherit from HFileArchiveTableTracker? src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4633/#comment16407 Is this doing backup? Or making it so the hfiles are not deleted on this table? src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4633/#comment16409 ditto... this just throws a switch on the don't delete operation? Something else does the backup? src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java https://reviews.apache.org/r/4633/#comment16410 table hfiles? src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java
[jira] [Commented] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb
[ https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266195#comment-13266195 ] jirapos...@reviews.apache.org commented on HBASE-5869: -- bq. On 2012-05-01 21:59:01, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/SplitLogCounters.java, line 99 bq. https://reviews.apache.org/r/4926/diff/3/?file=105878#file105878line99 bq. bq. What if an AtomicInteger counter is added in the future ? Open new JIRA. This is just a move of existing code. bq. On 2012-05-01 21:59:01, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/SplitLogTask.java, line 155 bq. https://reviews.apache.org/r/4926/diff/3/?file=105879#file105879line155 bq. bq. What would the first 64 bytes of data represent ? bq. Do we know that data.length = 64 ? Let me fix toStringBinary so it deals w/ case where data is 64 bytes. Regards what this represents, it could be anything. Just saving our logs from being filled w/ binary. bq. On 2012-05-01 21:59:01, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java, line 567 bq. https://reviews.apache.org/r/4926/diff/3/?file=105887#file105887line567 bq. bq. Should e1 be included in the log ? bq. 're-' before 'resubmit' is not necessary. Will add it. re- makes sense because this is retry inside exception handling. - Michael --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4926/#review7446 --- On 2012-05-01 20:42:36, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4926/ bq. --- bq. bq. (Updated 2012-05-01 20:42:36) bq. bq. bq. Review request for hbase and Jimmy Xiang. bq. bq. bq. Summary bq. --- bq. bq. Convert two zk users to pb: distributed log splitting and regions in transition. bq. bq. Refactored distributed log splitting so we only serialize/deserialize in one location. bq. Less changes needed to do same for regions in transition. bq. bq. Moves serialization/deserialization out of the ZKAssign, ZKSplit and into bq. the classes themselves so can encapsulate how serialization is done into one place bq. (try to make the ZK* classes just deal in bytes -- about 90% done). bq. bq. Moved classes used by various packages up to top level to minimize imports bq. that are across package (zookeeper into protobuf and/or into regionserver and/or bq. master packages, etc). bq. bq. A src/main/java/org/apache/hadoop/hbase/DeserializationException.java bq.New generic deserialization exception. bq. A src/main/java/org/apache/hadoop/hbase/zookeeper/EmptyWatcher.java bq. D src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java bq.Moved under zookeeper package. bq. A src/main/java/org/apache/hadoop/hbase/HBaseException.java bq.New base hbase exception as suggested by hbase-5796. New DeserializationException bq.inherits from this. bq. A src/main/java/org/apache/hadoop/hbase/RegionTransition.java bq.State of a region in transition. Top-level because used by a bq.few top-level packages. Encapsulates pb serialization/deserialization. bq. M src/main/java/org/apache/hadoop/hbase/ServerName.java bq.Add method to deserialize a ServeName, etc. Encapsulates pb'ing. bq. M src/main/java/org/apache/hadoop/hbase/SplitLogCounters.java bq.Counters used by distributed log splitting. bq. A SplitLogTask bq. Class that encapsulates log splitting state. Also encapsulates pb'ing. bq. M src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java bq.Implement code for state. Added functions to go from code to state and vice bq.versa. Used serializing. bq. M src/main/java/org/apache/hadoop/hbase/executor/ExecutorService.java bq.Remove unused imports. bq. D src/main/java/org/apache/hadoop/hbase/executor/RegionTransitionData.java bq.Removed. Replaced by RegionTransition moved to package top-level. bq. M src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java bq. M src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java bq.Use new DeserializationException. Move to using new RegionTransition bq.from RegionTransitionData class. Pass deserialized class rather than bq.byte array. Remove duplicated code. bq. M src/main/java/org/apache/hadoop/hbase/master/HMaster.java bq.Use new ServerName parse method rather than ZKUtil one. bq. M src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java bq. M src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java bq.Redo to
[jira] [Commented] (HBASE-5869) Move SplitLogManager splitlog taskstate and AssignmentManager RegionTransitionData znode datas to pb
[ https://issues.apache.org/jira/browse/HBASE-5869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266198#comment-13266198 ] jirapos...@reviews.apache.org commented on HBASE-5869: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4926/#review7452 --- src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java https://reviews.apache.org/r/4926/#comment16434 The port numbers don't match. - Ted On 2012-05-01 20:42:36, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4926/ bq. --- bq. bq. (Updated 2012-05-01 20:42:36) bq. bq. bq. Review request for hbase and Jimmy Xiang. bq. bq. bq. Summary bq. --- bq. bq. Convert two zk users to pb: distributed log splitting and regions in transition. bq. bq. Refactored distributed log splitting so we only serialize/deserialize in one location. bq. Less changes needed to do same for regions in transition. bq. bq. Moves serialization/deserialization out of the ZKAssign, ZKSplit and into bq. the classes themselves so can encapsulate how serialization is done into one place bq. (try to make the ZK* classes just deal in bytes -- about 90% done). bq. bq. Moved classes used by various packages up to top level to minimize imports bq. that are across package (zookeeper into protobuf and/or into regionserver and/or bq. master packages, etc). bq. bq. A src/main/java/org/apache/hadoop/hbase/DeserializationException.java bq.New generic deserialization exception. bq. A src/main/java/org/apache/hadoop/hbase/zookeeper/EmptyWatcher.java bq. D src/main/java/org/apache/hadoop/hbase/EmptyWatcher.java bq.Moved under zookeeper package. bq. A src/main/java/org/apache/hadoop/hbase/HBaseException.java bq.New base hbase exception as suggested by hbase-5796. New DeserializationException bq.inherits from this. bq. A src/main/java/org/apache/hadoop/hbase/RegionTransition.java bq.State of a region in transition. Top-level because used by a bq.few top-level packages. Encapsulates pb serialization/deserialization. bq. M src/main/java/org/apache/hadoop/hbase/ServerName.java bq.Add method to deserialize a ServeName, etc. Encapsulates pb'ing. bq. M src/main/java/org/apache/hadoop/hbase/SplitLogCounters.java bq.Counters used by distributed log splitting. bq. A SplitLogTask bq. Class that encapsulates log splitting state. Also encapsulates pb'ing. bq. M src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java bq.Implement code for state. Added functions to go from code to state and vice bq.versa. Used serializing. bq. M src/main/java/org/apache/hadoop/hbase/executor/ExecutorService.java bq.Remove unused imports. bq. D src/main/java/org/apache/hadoop/hbase/executor/RegionTransitionData.java bq.Removed. Replaced by RegionTransition moved to package top-level. bq. M src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java bq. M src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java bq.Use new DeserializationException. Move to using new RegionTransition bq.from RegionTransitionData class. Pass deserialized class rather than bq.byte array. Remove duplicated code. bq. M src/main/java/org/apache/hadoop/hbase/master/HMaster.java bq.Use new ServerName parse method rather than ZKUtil one. bq. M src/main/java/org/apache/hadoop/hbase/master/SplitLogManager.java bq. M src/main/java/org/apache/hadoop/hbase/regionserver/SplitLogWorker.java bq.Redo to use new SplitLogTask and SplitLogCounter classes. bq. M src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java bq.expectPBMagicPrefix added bq. M src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java bq.Use new RegionTransition in place of RegionTransitionData. bq. M src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLogSplitter.java bq.Define moved from ZKSplitLog to SplitLogManager. bq. M src/main/java/org/apache/hadoop/hbase/zookeeper/MasterAddressTracker.java bq. M src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java bq.Changed method name from getZNodeData to toByteArray to match how we've bq.named it elsewhere. Use new DeserializationException bq. M src/main/java/org/apache/hadoop/hbase/zookeeper/ZKAssign.java bq.Use new RegionTransion class bq. M src/main/java/org/apache/hadoop/hbase/zookeeper/ZKSplitLog.java bq.Moved stuff that was in here up into SplitLogManager where better bq.belongs. Also moved serialization/deserialization up into the
[jira] [Commented] (HBASE-5901) Use union type protobufs instead of class/byte pairs for multi requests
[ https://issues.apache.org/jira/browse/HBASE-5901?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13266197#comment-13266197 ] stack commented on HBASE-5901: -- @Todd you going to commit? I can otherwise. Use union type protobufs instead of class/byte pairs for multi requests --- Key: HBASE-5901 URL: https://issues.apache.org/jira/browse/HBASE-5901 Project: HBase Issue Type: Improvement Components: ipc, performance Affects Versions: 0.96.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Critical Attachments: hbase-5901.txt, hbase-5901.txt The current implementation of multi actions uses repeated NameBytesPairs for the contents of multi actions. Instead, we should introduce a union type protobuf for the valid actions. This makes the RPCs smaller since they don't need to carry class names, and makes deserialization faster since it can avoid some copying and reflection. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
