[jira] [Commented] (HBASE-5621) Convert admin protocol of HRegionInterface to PB
[ https://issues.apache.org/jira/browse/HBASE-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13257629#comment-13257629 ] jirapos...@reviews.apache.org commented on HBASE-5621: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4714/ --- (Updated 2012-04-19 17:46:17.775148) Review request for hbase. Changes --- Rebased to the latest of the trunk. Summary --- This is the admin part of HBase-5443. AdminProtocol part. This addresses bug HBASE-5621. https://issues.apache.org/jira/browse/HBASE-5621 Diffs (updated) - src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java 408db79 src/main/java/org/apache/hadoop/hbase/client/AdminProtocol.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/client/ClientProtocol.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java ee16e72 src/main/java/org/apache/hadoop/hbase/client/HConnection.java 23f8e5a src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 820e2a9 src/main/java/org/apache/hadoop/hbase/client/HTable.java 2c87d50 src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java cd4cccb src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java 2fc4a15 src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java 57c9443 src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 52d179d src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 09601b8 src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java d0570b9 src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 7239c5a src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 70901fe src/main/java/org/apache/hadoop/hbase/protobuf/AdminProtocol.java 422e865 src/main/java/org/apache/hadoop/hbase/protobuf/ClientProtocol.java 3d6a23a src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java b056830 src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java a912cc3 src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java ecaf9fe src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java e78e56d src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 61a5988 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java 759633d src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java 7c59995 src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java 04fe8b6 src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 66156c2 src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java 83a165c src/main/protobuf/Admin.proto 132c5dd src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java d6ae0e2 src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditorNoCluster.java 3cfc02b src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java 8af0f91 src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java aa7f51b src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java 7dd60de src/test/java/org/apache/hadoop/hbase/client/TestHTableUtil.java 24f878e src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java 301ee27 src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java a59e152 src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 7ffd6bd src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java cedf31e src/test/java/org/apache/hadoop/hbase/master/TestMaster.java 3122b15 src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java c0ac12c src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java d0cad45 src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java 6b64f10 Diff: https://reviews.apache.org/r/4714/diff Testing --- All unit tests passed. Thanks, Jimmy Convert admin protocol of HRegionInterface to PB Key: HBASE-5621 URL: https://issues.apache.org/jira/browse/HBASE-5621 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5621) Convert admin protocol of HRegionInterface to PB
[ https://issues.apache.org/jira/browse/HBASE-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13257796#comment-13257796 ] jirapos...@reviews.apache.org commented on HBASE-5621: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4714/#review7046 --- security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java https://reviews.apache.org/r/4714/#comment15632 There are a bunch of import changes here. Are they all needed? security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java https://reviews.apache.org/r/4714/#comment15631 Why can we get away w/ removing the try/catch? Because the caller handles it? src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java https://reviews.apache.org/r/4714/#comment15633 good src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java https://reviews.apache.org/r/4714/#comment15634 Does this belong in this patch? Is it part of another patch? src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java https://reviews.apache.org/r/4714/#comment15635 Yeah, this stuff is from another patch? Why you adding it? src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java https://reviews.apache.org/r/4714/#comment15636 Are these from Elliotts' patch? Maybe its reviewboard that is messing up? I'm only looking at diff between your v2 and v3 patch. - Michael On 2012-04-19 17:46:17, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4714/ bq. --- bq. bq. (Updated 2012-04-19 17:46:17) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This is the admin part of HBase-5443. AdminProtocol part. bq. bq. bq. This addresses bug HBASE-5621. bq. https://issues.apache.org/jira/browse/HBASE-5621 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java 408db79 bq.src/main/java/org/apache/hadoop/hbase/client/AdminProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/client/ClientProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java ee16e72 bq.src/main/java/org/apache/hadoop/hbase/client/HConnection.java 23f8e5a bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 820e2a9 bq.src/main/java/org/apache/hadoop/hbase/client/HTable.java 2c87d50 bq.src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java cd4cccb bq.src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java 2fc4a15 bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java 57c9443 bq.src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 52d179d bq.src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 09601b8 bq. src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java d0570b9 bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 7239c5a bq.src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 70901fe bq.src/main/java/org/apache/hadoop/hbase/protobuf/AdminProtocol.java 422e865 bq.src/main/java/org/apache/hadoop/hbase/protobuf/ClientProtocol.java 3d6a23a bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java b056830 bq.src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java a912cc3 bq.src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java ecaf9fe bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java e78e56d bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 61a5988 bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java 759633d bq.src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java 7c59995 bq. src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java 04fe8b6 bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 66156c2 bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java 83a165c bq.src/main/protobuf/Admin.proto 132c5dd bq.src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java d6ae0e2 bq. src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditorNoCluster.java 3cfc02b bq. src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java 8af0f91 bq.src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java aa7f51b bq.
[jira] [Commented] (HBASE-5621) Convert admin protocol of HRegionInterface to PB
[ https://issues.apache.org/jira/browse/HBASE-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13257837#comment-13257837 ] jirapos...@reviews.apache.org commented on HBASE-5621: -- bq. On 2012-04-19 20:43:31, Michael Stack wrote: bq. I did a rebase. So it picked up some other changes if you diff 2 from 3. bq. On 2012-04-19 20:43:31, Michael Stack wrote: bq. security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java, line 50 bq. https://reviews.apache.org/r/4714/diff/2/?file=102179#file102179line50 bq. bq. There are a bunch of import changes here. Are they all needed? It must be from other patch. This patch doesn't touch this file. bq. On 2012-04-19 20:43:31, Michael Stack wrote: bq. security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java, line 187 bq. https://reviews.apache.org/r/4714/diff/2/?file=102179#file102179line187 bq. bq. Why can we get away w/ removing the try/catch? Because the caller handles it? That change in diff2 is already checked in as part of 5443 addendum. That's why it shows some change here. This patch doesn't change this file. bq. On 2012-04-19 20:43:31, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java, line 411 bq. https://reviews.apache.org/r/4714/diff/2-3/?file=102195#file102195line411 bq. bq. Does this belong in this patch? Is it part of another patch? bq. From another patch due to rebasing. bq. On 2012-04-19 20:43:31, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java, line 2220 bq. https://reviews.apache.org/r/4714/diff/2-3/?file=102195#file102195line2220 bq. bq. Yeah, this stuff is from another patch? Why you adding it? bq. From another patch due to rebasing. bq. On 2012-04-19 20:43:31, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java, line 2214 bq. https://reviews.apache.org/r/4714/diff/2-3/?file=102203#file102203line2214 bq. bq. Are these from Elliotts' patch? bq. bq. Maybe its reviewboard that is messing up? I'm only looking at diff between your v2 and v3 patch. bq. From another patch due to rebasing. - Jimmy --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4714/#review7046 --- On 2012-04-19 17:46:17, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4714/ bq. --- bq. bq. (Updated 2012-04-19 17:46:17) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This is the admin part of HBase-5443. AdminProtocol part. bq. bq. bq. This addresses bug HBASE-5621. bq. https://issues.apache.org/jira/browse/HBASE-5621 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java 408db79 bq.src/main/java/org/apache/hadoop/hbase/client/AdminProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/client/ClientProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java ee16e72 bq.src/main/java/org/apache/hadoop/hbase/client/HConnection.java 23f8e5a bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 820e2a9 bq.src/main/java/org/apache/hadoop/hbase/client/HTable.java 2c87d50 bq.src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java cd4cccb bq.src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java 2fc4a15 bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java 57c9443 bq.src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 52d179d bq.src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 09601b8 bq. src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java d0570b9 bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 7239c5a bq.src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 70901fe bq.src/main/java/org/apache/hadoop/hbase/protobuf/AdminProtocol.java 422e865 bq.src/main/java/org/apache/hadoop/hbase/protobuf/ClientProtocol.java 3d6a23a bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java b056830 bq.src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java a912cc3 bq.src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java ecaf9fe bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java e78e56d bq.
[jira] [Commented] (HBASE-5621) Convert admin protocol of HRegionInterface to PB
[ https://issues.apache.org/jira/browse/HBASE-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13257912#comment-13257912 ] jirapos...@reviews.apache.org commented on HBASE-5621: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4714/#review7051 --- Ship it! I'm good w/ this patch. Good stuff Jimmy src/main/java/org/apache/hadoop/hbase/client/HConnection.java https://reviews.apache.org/r/4714/#comment15645 Please file an issue to address this recursion (an AdminProtocol 'has' a HConnection but you get the AdminProtocol from an HConnection). src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java https://reviews.apache.org/r/4714/#comment15646 This is a noop - Michael On 2012-04-19 17:46:17, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4714/ bq. --- bq. bq. (Updated 2012-04-19 17:46:17) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This is the admin part of HBase-5443. AdminProtocol part. bq. bq. bq. This addresses bug HBASE-5621. bq. https://issues.apache.org/jira/browse/HBASE-5621 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java 408db79 bq.src/main/java/org/apache/hadoop/hbase/client/AdminProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/client/ClientProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java ee16e72 bq.src/main/java/org/apache/hadoop/hbase/client/HConnection.java 23f8e5a bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 820e2a9 bq.src/main/java/org/apache/hadoop/hbase/client/HTable.java 2c87d50 bq.src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java cd4cccb bq.src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java 2fc4a15 bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java 57c9443 bq.src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java 52d179d bq.src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 09601b8 bq. src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java d0570b9 bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 7239c5a bq.src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 70901fe bq.src/main/java/org/apache/hadoop/hbase/protobuf/AdminProtocol.java 422e865 bq.src/main/java/org/apache/hadoop/hbase/protobuf/ClientProtocol.java 3d6a23a bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java b056830 bq.src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java a912cc3 bq.src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java ecaf9fe bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java e78e56d bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 61a5988 bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java 759633d bq.src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java 7c59995 bq. src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java 04fe8b6 bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 66156c2 bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java 83a165c bq.src/main/protobuf/Admin.proto 132c5dd bq.src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java d6ae0e2 bq. src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditorNoCluster.java 3cfc02b bq. src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java 8af0f91 bq.src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java aa7f51b bq.src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java 7dd60de bq.src/test/java/org/apache/hadoop/hbase/client/TestHTableUtil.java 24f878e bq. src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java 301ee27 bq.src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java a59e152 bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 7ffd6bd bq.src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java cedf31e bq.src/test/java/org/apache/hadoop/hbase/master/TestMaster.java 3122b15 bq.src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java c0ac12c bq.
[jira] [Commented] (HBASE-2214) Do HBASE-1996 -- setting size to return in scan rather than count of rows -- properly
[ https://issues.apache.org/jira/browse/HBASE-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255866#comment-13255866 ] jirapos...@reviews.apache.org commented on HBASE-2214: -- bq. On 2012-04-17 05:46:43, Michael Stack wrote: bq. I think this an important addition. Any chance of a test Ferdy to demonstrate it works. Its kinda crazy you only have to do this much to make it work. It was indeed to good to be true. It does not yet work. There are still some other classes that need modification. (In order to make it work by piggybacking on the existing maxScannerResultSize mechanism). Will get back at this with new version of patch. Thanks. bq. On 2012-04-17 05:46:43, Michael Stack wrote: bq. /src/main/java/org/apache/hadoop/hbase/regionserver/RegionScanner.java, line 26 bq. https://reviews.apache.org/r/4726/diff/2/?file=101902#file101902line26 bq. bq. This is not used? It is, namely in the javadoc comment. Removing it makes the link unclickable. bq. On 2012-04-17 05:46:43, Michael Stack wrote: bq. /src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java, line 3376 bq. https://reviews.apache.org/r/4726/diff/2/?file=101900#file101900line3376 bq. bq. Needs to be public? I cannot reduce the visibilty because it needs to in the interface. (Yeah it's a bit of a hack I guess) bq. On 2012-04-17 05:46:43, Michael Stack wrote: bq. /src/main/java/org/apache/hadoop/hbase/regionserver/RegionScanner.java, line 59 bq. https://reviews.apache.org/r/4726/diff/2/?file=101902#file101902line59 bq. bq. Is this a good name for this attribute? Its the result size per next? getMaxResultSize? Or something like that? maxResultSize seems better indeed. I will rename this next patch. - ferdy --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4726/#review6968 --- On 2012-04-15 13:38:31, ferdy wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4726/ bq. --- bq. bq. (Updated 2012-04-15 13:38:31) bq. bq. bq. Review request for hbase and Ted Yu. bq. bq. bq. Summary bq. --- bq. bq. HBASE-2214 per scan max buffersize. bq. bq. bq. This addresses bug HBASE-2214. bq. https://issues.apache.org/jira/browse/HBASE-2214 bq. bq. bq. Diffs bq. - bq. bq./src/main/java/org/apache/hadoop/hbase/client/Scan.java 1326084 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1326084 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1326084 bq./src/main/java/org/apache/hadoop/hbase/regionserver/RegionScanner.java 1326084 bq. /src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java 1326084 bq. bq. Diff: https://reviews.apache.org/r/4726/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. ferdy bq. bq. Do HBASE-1996 -- setting size to return in scan rather than count of rows -- properly - Key: HBASE-2214 URL: https://issues.apache.org/jira/browse/HBASE-2214 Project: HBase Issue Type: New Feature Reporter: stack Assignee: Ferdy Galema Fix For: 0.94.1 Attachments: HBASE-2214-0.94.txt, HBASE-2214_with_broken_TestShell.txt The notion that you set size rather than row count specifying how many rows a scanner should return in each cycle was raised over in hbase-1966. Its a good one making hbase regular though the data under it may vary. HBase-1966 was committed but the patch was constrained by the fact that it needed to not change RPC interface. This issue is about doing hbase-1966 for 0.21 in a clean, unconstrained way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5621) Convert admin protocol of HRegionInterface to PB
[ https://issues.apache.org/jira/browse/HBASE-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255036#comment-13255036 ] jirapos...@reviews.apache.org commented on HBASE-5621: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4714/ --- (Updated 2012-04-16 20:49:59.450802) Review request for hbase. Changes --- Addressed some of Stack's comments: 1. moved Client/AdminProtcol.java to client package, 2. moved some sharable pb calls to ProtobufUtil to avoid scattering pb everywhere. As to move some calls from HConnection to HCM, it is not as straightforward as I think. The main reason is becuase HConnection is not just a network connection to a server. Probably it is better to handle it in a separate jira. Summary --- This is the admin part of HBase-5443. AdminProtocol part. This addresses bug HBASE-5621. https://issues.apache.org/jira/browse/HBASE-5621 Diffs (updated) - security/src/main/java/org/apache/hadoop/hbase/ipc/SecureRpcEngine.java 8219bea src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java 408db79 src/main/java/org/apache/hadoop/hbase/client/AdminProtocol.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/client/ClientProtocol.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java ee16e72 src/main/java/org/apache/hadoop/hbase/client/HConnection.java 23f8e5a src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 820e2a9 src/main/java/org/apache/hadoop/hbase/client/HTable.java 2c87d50 src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java fe80fcf src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java 2a9d86e src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java d71e97e src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 3a3a79f src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java b7afa58 src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java dd0a1bc src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 9f159f2 src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java d0570b9 src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java ab33ac7 src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 70901fe src/main/java/org/apache/hadoop/hbase/protobuf/AdminProtocol.java 422e865 src/main/java/org/apache/hadoop/hbase/protobuf/ClientProtocol.java 3d6a23a src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java b056830 src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java a912cc3 src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java ecaf9fe src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java e78e56d src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1c0541f src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java 759633d src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java 9487a1c src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java 04fe8b6 src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 61f8128 src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java 83a165c src/main/protobuf/Admin.proto 132c5dd src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java d6ae0e2 src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditorNoCluster.java 3cfc02b src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java 8af0f91 src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java aa7f51b src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java 7dd60de src/test/java/org/apache/hadoop/hbase/client/TestHTableUtil.java 24f878e src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java 301ee27 src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java a59e152 src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java b84a115 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java cedf31e src/test/java/org/apache/hadoop/hbase/master/TestMaster.java 3122b15 src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java c0ac12c src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java d0cad45 src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java 6b64f10 Diff: https://reviews.apache.org/r/4714/diff Testing --- All unit tests passed. Thanks, Jimmy Convert admin protocol of HRegionInterface to PB Key: HBASE-5621 URL:
[jira] [Commented] (HBASE-2214) Do HBASE-1996 -- setting size to return in scan rather than count of rows -- properly
[ https://issues.apache.org/jira/browse/HBASE-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255331#comment-13255331 ] jirapos...@reviews.apache.org commented on HBASE-2214: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4726/#review6968 --- I think this an important addition. Any chance of a test Ferdy to demonstrate it works. Its kinda crazy you only have to do this much to make it work. /src/main/java/org/apache/hadoop/hbase/client/Scan.java https://reviews.apache.org/r/4726/#comment15518 Good /src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java https://reviews.apache.org/r/4726/#comment15519 Needs to be public? /src/main/java/org/apache/hadoop/hbase/regionserver/RegionScanner.java https://reviews.apache.org/r/4726/#comment15520 This is not used? /src/main/java/org/apache/hadoop/hbase/regionserver/RegionScanner.java https://reviews.apache.org/r/4726/#comment15521 Is this a good name for this attribute? Its the result size per next? getMaxResultSize? Or something like that? - Michael On 2012-04-15 13:38:31, ferdy wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4726/ bq. --- bq. bq. (Updated 2012-04-15 13:38:31) bq. bq. bq. Review request for hbase and Ted Yu. bq. bq. bq. Summary bq. --- bq. bq. HBASE-2214 per scan max buffersize. bq. bq. bq. This addresses bug HBASE-2214. bq. https://issues.apache.org/jira/browse/HBASE-2214 bq. bq. bq. Diffs bq. - bq. bq./src/main/java/org/apache/hadoop/hbase/client/Scan.java 1326084 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1326084 bq./src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1326084 bq./src/main/java/org/apache/hadoop/hbase/regionserver/RegionScanner.java 1326084 bq. /src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java 1326084 bq. bq. Diff: https://reviews.apache.org/r/4726/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. ferdy bq. bq. Do HBASE-1996 -- setting size to return in scan rather than count of rows -- properly - Key: HBASE-2214 URL: https://issues.apache.org/jira/browse/HBASE-2214 Project: HBase Issue Type: New Feature Reporter: stack Assignee: Ferdy Galema Fix For: 0.94.1 Attachments: HBASE-2214-0.94.txt, HBASE-2214_with_broken_TestShell.txt The notion that you set size rather than row count specifying how many rows a scanner should return in each cycle was raised over in hbase-1966. Its a good one making hbase regular though the data under it may vary. HBase-1966 was committed but the patch was constrained by the fact that it needed to not change RPC interface. This issue is about doing hbase-1966 for 0.21 in a clean, unconstrained way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-2214) Do HBASE-1996 -- setting size to return in scan rather than count of rows -- properly
[ https://issues.apache.org/jira/browse/HBASE-2214?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254329#comment-13254329 ] jirapos...@reviews.apache.org commented on HBASE-2214: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4726/ --- (Updated 2012-04-15 13:38:31.680141) Review request for hbase and Ted Yu. Summary --- HBASE-2214 per scan max buffersize. This addresses bug HBASE-2214. https://issues.apache.org/jira/browse/HBASE-2214 Diffs - /src/main/java/org/apache/hadoop/hbase/client/Scan.java 1326084 /src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1326084 /src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 1326084 /src/main/java/org/apache/hadoop/hbase/regionserver/RegionScanner.java 1326084 /src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java 1326084 Diff: https://reviews.apache.org/r/4726/diff Testing --- Thanks, ferdy Do HBASE-1996 -- setting size to return in scan rather than count of rows -- properly - Key: HBASE-2214 URL: https://issues.apache.org/jira/browse/HBASE-2214 Project: HBase Issue Type: New Feature Reporter: stack Assignee: Ferdy Galema Fix For: 0.94.1 Attachments: HBASE-2214-0.94.txt, HBASE-2214_with_broken_TestShell.txt The notion that you set size rather than row count specifying how many rows a scanner should return in each cycle was raised over in hbase-1966. Its a good one making hbase regular though the data under it may vary. HBase-1966 was committed but the patch was constrained by the fact that it needed to not change RPC interface. This issue is about doing hbase-1966 for 0.21 in a clean, unconstrained way. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5621) Convert admin protocol of HRegionInterface to PB
[ https://issues.apache.org/jira/browse/HBASE-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254364#comment-13254364 ] jirapos...@reviews.apache.org commented on HBASE-5621: -- bq. On 2012-04-13 23:59:40, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java, line 2814 bq. https://reviews.apache.org/r/4714/diff/1/?file=101799#file101799line2814 bq. bq. Where has allt his code gone? bq. bq. Jimmy Xiang wrote: bq. Some are moved to RegionServer as I did for 5620 bq. bq. Michael Stack wrote: bq. But I did not see it in the patch? Its already moved? It should be in the patch, probably in the second page of touched files. bq. On 2012-04-13 23:59:40, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/ClientProtocol.java, line 29 bq. https://reviews.apache.org/r/4714/diff/1/?file=101781#file101781line29 bq. bq. Ditto bq. bq. Jimmy Xiang wrote: bq. Will move to client. bq. bq. Michael Stack wrote: bq. Is this only used out of the client package? If so, yeah, move it there I'd say. It is used out of the client package, and the server side implements it. bq. On 2012-04-13 23:59:40, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/AdminProtocol.java, line 28 bq. https://reviews.apache.org/r/4714/diff/1/?file=101780#file101780line28 bq. bq. Is this right? It does more than talk to a regionserver? You have to have really nice comments on your class now that you are at the top level Jimmy (smile) bq. bq. This class is only used by HBaseAdmin? Is that right? Or do other classes use it? If only HBaseAdmin, maybe it should not be top level? bq. bq. Jimmy Xiang wrote: bq. Let me move it to client. bq. bq. Michael Stack wrote: bq. And it should not be public I'd say. Yes, it is private. bq. On 2012-04-13 23:59:40, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java, line 632 bq. https://reviews.apache.org/r/4714/diff/1/?file=101782#file101782line632 bq. bq. Do we have to have protobuf stuff sprinkled all over the code base? Its not too bad but just wondering. Maybe we do but just wondering. I suppose there is nothing to do about it. If we did same operation over and over w/ some pb stuff and the output was a non-pb object, then yes, we could hide the pb stuff over in a class but what we do here is particular to this method Can't generalize. bq. bq. I do see that you have added some static methods into protobuf utils. This one is not generic so doesn't deserve to go there? bq. bq. Jimmy Xiang wrote: bq. For getRegionInfo, I used to put it in the protobuf util since it is used lots of places. I will move it to protobuf util. bq. bq. Michael Stack wrote: bq. Agree. If used more than once, move it over to protobuf util otherwise I don't see a way around not importing protobuf classes when its a particular usage. Cool, will fix. bq. On 2012-04-13 23:59:40, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/client/HConnection.java, line 32 bq. https://reviews.apache.org/r/4714/diff/1/?file=101784#file101784line32 bq. bq. Did you say you were going to remove these from here? bq. bq. Jimmy Xiang wrote: bq. I tried but it is hard. Let me think about it again. bq. bq. Michael Stack wrote: bq. Even if this stuff just moved to HCM, it'd give us a clean HConnection. I was thinking to have a ProtocolFactory/Manager to manage the protocols, and let the connection manager to many connections only. Let me move them to HCM for now. - Jimmy --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4714/#review6922 --- On 2012-04-13 23:03:35, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4714/ bq. --- bq. bq. (Updated 2012-04-13 23:03:35) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This is the admin part of HBase-5443. AdminProtocol part. bq. bq. bq. This addresses bug HBASE-5621. bq. https://issues.apache.org/jira/browse/HBASE-5621 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/AdminProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/ClientProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java 408db79 bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java ee16e72 bq.
[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253567#comment-13253567 ] jirapos...@reviews.apache.org commented on HBASE-5547: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/#review6914 --- src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4633/#comment15366 Can the checks in the while loop be reversed, so that we check the counter first and then call the allRegionsDoneBackup(). Will this help in removing the call to disableHFileBackup, or am i missing anything here. src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java https://reviews.apache.org/r/4633/#comment15363 yeah, instantiation of this call is done more than once in a single code path (call to HBaseAdmin-enableHFileBackup()). Good to have some closing behavior. src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java https://reviews.apache.org/r/4633/#comment15364 How costly is this op; the reason I ask is it will be useful only once; once the znode is there, this call has no value. (Still learning zk stuff, kindly ignore if you think so) src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java https://reviews.apache.org/r/4633/#comment15375 minor nit: is enabled? src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java https://reviews.apache.org/r/4633/#comment15376 you made the deleteRegion to rawDeleteRegion? lol, but that sounds a bit secondary :) Why not keep delete or sth deleteWithNoArchive? src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java https://reviews.apache.org/r/4633/#comment15369 Yes, please add some doc to it. It is instantiated in HMaster, so can't be abstract? Tracker and Monitor doesn't seem to give a clearer picture, somehow, especially, when tracker implements the monitor? - Himanshu On 2012-04-07 19:51:11, Jesse Yates wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4633/ bq. --- bq. bq. (Updated 2012-04-07 19:51:11) bq. bq. bq. Review request for hbase, Michael Stack and Lars Hofhansl. bq. bq. bq. Summary bq. --- bq. bq. Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap. bq. bq. Implementation wise: bq. Updated the HMasterInterface to pass the calls onto the zookeeper. bq. Added a zk listener to handle updates from the master to the RS to backup. bq. Added a utility for removing files and finding archive directories bq. Added tests for the regionserver and catalogjanitor approaches. bq. Added creation of manager in regionserver. bq. bq. bq. This addresses bug HBASE-5547. bq. https://issues.apache.org/jira/browse/HBASE-5547 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/HConstants.java a4b989e bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 bq.src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb21bdd bq. src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java c3df319 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d bq. src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53 bq.src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 509a467 bq.src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f bq.src/main/resources/hbase-default.xml 44ee689 bq.src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java
[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253606#comment-13253606 ] jirapos...@reviews.apache.org commented on HBASE-5547: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/#review6917 --- src/main/java/org/apache/hadoop/hbase/regionserver/Store.java https://reviews.apache.org/r/4633/#comment15382 You wouldn't have chance to do that if it fails as there will be an IOException? src/main/java/org/apache/hadoop/hbase/regionserver/Store.java https://reviews.apache.org/r/4633/#comment15385 I see; you trying to archive files, and just delete in case there is any failure for some hfiles. src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java https://reviews.apache.org/r/4633/#comment15387 I miss this; who is calling this addTable? It would be great to have the usecase. (sorry for reviewing it piece meal, its a huge patch as per my standards :)) src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java https://reviews.apache.org/r/4633/#comment15389 And, what are the better alternatives (some subclass holding all these constants, or something more aesthetic). - Himanshu On 2012-04-07 19:51:11, Jesse Yates wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4633/ bq. --- bq. bq. (Updated 2012-04-07 19:51:11) bq. bq. bq. Review request for hbase, Michael Stack and Lars Hofhansl. bq. bq. bq. Summary bq. --- bq. bq. Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap. bq. bq. Implementation wise: bq. Updated the HMasterInterface to pass the calls onto the zookeeper. bq. Added a zk listener to handle updates from the master to the RS to backup. bq. Added a utility for removing files and finding archive directories bq. Added tests for the regionserver and catalogjanitor approaches. bq. Added creation of manager in regionserver. bq. bq. bq. This addresses bug HBASE-5547. bq. https://issues.apache.org/jira/browse/HBASE-5547 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/HConstants.java a4b989e bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 bq.src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb21bdd bq. src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java c3df319 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d bq. src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53 bq.src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 509a467 bq.src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f bq.src/main/resources/hbase-default.xml 44ee689 bq.src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 41616c8 bq.src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java b4dcb83 bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759 bq. bq. Diff: https://reviews.apache.org/r/4633/diff bq. bq. bq. Testing bq. --- bq. bq. Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper. bq. bq. bq. Thanks, bq. bq.
[jira] [Commented] (HBASE-5621) Convert admin protocol of HRegionInterface to PB
[ https://issues.apache.org/jira/browse/HBASE-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253846#comment-13253846 ] jirapos...@reviews.apache.org commented on HBASE-5621: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4714/ --- Review request for hbase. Summary --- This is the admin part of HBase-5443. AdminProtocol part. This addresses bug HBASE-5621. https://issues.apache.org/jira/browse/HBASE-5621 Diffs - src/main/java/org/apache/hadoop/hbase/AdminProtocol.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ClientProtocol.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java 408db79 src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java ee16e72 src/main/java/org/apache/hadoop/hbase/client/HConnection.java 23f8e5a src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 820e2a9 src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java fe80fcf src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java 2a9d86e src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 3a3a79f src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java b7afa58 src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java dd0a1bc src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 9f159f2 src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java ab33ac7 src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 70901fe src/main/java/org/apache/hadoop/hbase/protobuf/AdminProtocol.java 422e865 src/main/java/org/apache/hadoop/hbase/protobuf/ClientProtocol.java 3d6a23a src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java b056830 src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java a912cc3 src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java e78e56d src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 0d22c0e src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java 9487a1c src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java 04fe8b6 src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 61f8128 src/main/java/org/apache/hadoop/hbase/util/HBaseFsckRepair.java ee4f556 src/main/protobuf/Admin.proto 132c5dd src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java d6ae0e2 src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditorNoCluster.java 3cfc02b src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java 8af0f91 src/test/java/org/apache/hadoop/hbase/client/TestAdmin.java aa7f51b src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java 7dd60de src/test/java/org/apache/hadoop/hbase/client/TestHTableUtil.java 24f878e src/test/java/org/apache/hadoop/hbase/mapreduce/TestLoadIncrementalHFilesSplitRecovery.java 301ee27 src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java a59e152 src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java b84a115 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java cedf31e src/test/java/org/apache/hadoop/hbase/master/TestMaster.java 3122b15 src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java c0ac12c src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java d0cad45 src/test/java/org/apache/hadoop/hbase/util/TestHBaseFsck.java a06942d Diff: https://reviews.apache.org/r/4714/diff Testing --- All unit tests passed. Thanks, Jimmy Convert admin protocol of HRegionInterface to PB Key: HBASE-5621 URL: https://issues.apache.org/jira/browse/HBASE-5621 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5621) Convert admin protocol of HRegionInterface to PB
[ https://issues.apache.org/jira/browse/HBASE-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253893#comment-13253893 ] jirapos...@reviews.apache.org commented on HBASE-5621: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4714/#review6922 --- A few questions below... Looks great. src/main/java/org/apache/hadoop/hbase/AdminProtocol.java https://reviews.apache.org/r/4714/#comment15405 Is this right? It does more than talk to a regionserver? You have to have really nice comments on your class now that you are at the top level Jimmy (smile) This class is only used by HBaseAdmin? Is that right? Or do other classes use it? If only HBaseAdmin, maybe it should not be top level? src/main/java/org/apache/hadoop/hbase/ClientProtocol.java https://reviews.apache.org/r/4714/#comment15406 Ditto src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java https://reviews.apache.org/r/4714/#comment15407 Do we have to have protobuf stuff sprinkled all over the code base? Its not too bad but just wondering. Maybe we do but just wondering. I suppose there is nothing to do about it. If we did same operation over and over w/ some pb stuff and the output was a non-pb object, then yes, we could hide the pb stuff over in a class but what we do here is particular to this method Can't generalize. I do see that you have added some static methods into protobuf utils. This one is not generic so doesn't deserve to go there? src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4714/#comment15408 Great stuff. src/main/java/org/apache/hadoop/hbase/client/HConnection.java https://reviews.apache.org/r/4714/#comment15409 Did you say you were going to remove these from here? src/main/java/org/apache/hadoop/hbase/client/HConnection.java https://reviews.apache.org/r/4714/#comment15410 Yeah, does this belong in here? src/main/java/org/apache/hadoop/hbase/client/HConnection.java https://reviews.apache.org/r/4714/#comment15411 ditto An AdminProtocol should have an HConnection? Even if you moved this up into HConnectionManager for now... that'd be better. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java https://reviews.apache.org/r/4714/#comment15412 Yeah, I wonder if this getClient and getAdmin should be @Overrides? Shoudl they be methods of HCM altogether? Would that be better? OR should it happen elsewhere? In HBaseAdmin or in HTable where we put together protocol and connection? src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java https://reviews.apache.org/r/4714/#comment15413 Hurray! src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java https://reviews.apache.org/r/4714/#comment15414 Where has allt his code gone? - Michael On 2012-04-13 23:03:35, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4714/ bq. --- bq. bq. (Updated 2012-04-13 23:03:35) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This is the admin part of HBase-5443. AdminProtocol part. bq. bq. bq. This addresses bug HBASE-5621. bq. https://issues.apache.org/jira/browse/HBASE-5621 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/AdminProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/ClientProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java 408db79 bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java ee16e72 bq.src/main/java/org/apache/hadoop/hbase/client/HConnection.java 23f8e5a bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 820e2a9 bq.src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java fe80fcf bq.src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java 2a9d86e bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseRPC.java 3a3a79f bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java b7afa58 bq.src/main/java/org/apache/hadoop/hbase/ipc/RpcEngine.java dd0a1bc bq.src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 9f159f2 bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java ab33ac7 bq.src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 70901fe bq.src/main/java/org/apache/hadoop/hbase/protobuf/AdminProtocol.java 422e865 bq.
[jira] [Commented] (HBASE-5621) Convert admin protocol of HRegionInterface to PB
[ https://issues.apache.org/jira/browse/HBASE-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253909#comment-13253909 ] jirapos...@reviews.apache.org commented on HBASE-5621: -- bq. On 2012-04-13 23:59:40, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/AdminProtocol.java, line 28 bq. https://reviews.apache.org/r/4714/diff/1/?file=101780#file101780line28 bq. bq. Is this right? It does more than talk to a regionserver? You have to have really nice comments on your class now that you are at the top level Jimmy (smile) bq. bq. This class is only used by HBaseAdmin? Is that right? Or do other classes use it? If only HBaseAdmin, maybe it should not be top level? Let me move it to client. bq. On 2012-04-13 23:59:40, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/ClientProtocol.java, line 29 bq. https://reviews.apache.org/r/4714/diff/1/?file=101781#file101781line29 bq. bq. Ditto Will move to client. bq. On 2012-04-13 23:59:40, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java, line 632 bq. https://reviews.apache.org/r/4714/diff/1/?file=101782#file101782line632 bq. bq. Do we have to have protobuf stuff sprinkled all over the code base? Its not too bad but just wondering. Maybe we do but just wondering. I suppose there is nothing to do about it. If we did same operation over and over w/ some pb stuff and the output was a non-pb object, then yes, we could hide the pb stuff over in a class but what we do here is particular to this method Can't generalize. bq. bq. I do see that you have added some static methods into protobuf utils. This one is not generic so doesn't deserve to go there? For getRegionInfo, I used to put it in the protobuf util since it is used lots of places. I will move it to protobuf util. bq. On 2012-04-13 23:59:40, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/client/HConnection.java, line 32 bq. https://reviews.apache.org/r/4714/diff/1/?file=101784#file101784line32 bq. bq. Did you say you were going to remove these from here? I tried but it is hard. Let me think about it again. bq. On 2012-04-13 23:59:40, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/client/HConnection.java, line 223 bq. https://reviews.apache.org/r/4714/diff/1/?file=101784#file101784line223 bq. bq. Yeah, does this belong in here? Will fix it. bq. On 2012-04-13 23:59:40, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/client/HConnection.java, line 260 bq. https://reviews.apache.org/r/4714/diff/1/?file=101784#file101784line260 bq. bq. ditto bq. bq. An AdminProtocol should have an HConnection? bq. bq. Even if you moved this up into HConnectionManager for now... that'd be better. Let me think about it and fix it. bq. On 2012-04-13 23:59:40, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java, line 1376 bq. https://reviews.apache.org/r/4714/diff/1/?file=101785#file101785line1376 bq. bq. Yeah, I wonder if this getClient and getAdmin should be @Overrides? Shoudl they be methods of HCM altogether? Would that be better? OR should it happen elsewhere? In HBaseAdmin or in HTable where we put together protocol and connection? Let me think about it and fix it. bq. On 2012-04-13 23:59:40, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java, line 2814 bq. https://reviews.apache.org/r/4714/diff/1/?file=101799#file101799line2814 bq. bq. Where has allt his code gone? Some are moved to RegionServer as I did for 5620 - Jimmy --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4714/#review6922 --- On 2012-04-13 23:03:35, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4714/ bq. --- bq. bq. (Updated 2012-04-13 23:03:35) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This is the admin part of HBase-5443. AdminProtocol part. bq. bq. bq. This addresses bug HBASE-5621. bq. https://issues.apache.org/jira/browse/HBASE-5621 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/AdminProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/ClientProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java 408db79 bq.
[jira] [Commented] (HBASE-5621) Convert admin protocol of HRegionInterface to PB
[ https://issues.apache.org/jira/browse/HBASE-5621?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13253982#comment-13253982 ] jirapos...@reviews.apache.org commented on HBASE-5621: -- bq. On 2012-04-13 23:59:40, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/AdminProtocol.java, line 28 bq. https://reviews.apache.org/r/4714/diff/1/?file=101780#file101780line28 bq. bq. Is this right? It does more than talk to a regionserver? You have to have really nice comments on your class now that you are at the top level Jimmy (smile) bq. bq. This class is only used by HBaseAdmin? Is that right? Or do other classes use it? If only HBaseAdmin, maybe it should not be top level? bq. bq. Jimmy Xiang wrote: bq. Let me move it to client. And it should not be public I'd say. bq. On 2012-04-13 23:59:40, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/ClientProtocol.java, line 29 bq. https://reviews.apache.org/r/4714/diff/1/?file=101781#file101781line29 bq. bq. Ditto bq. bq. Jimmy Xiang wrote: bq. Will move to client. Is this only used out of the client package? If so, yeah, move it there I'd say. bq. On 2012-04-13 23:59:40, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java, line 632 bq. https://reviews.apache.org/r/4714/diff/1/?file=101782#file101782line632 bq. bq. Do we have to have protobuf stuff sprinkled all over the code base? Its not too bad but just wondering. Maybe we do but just wondering. I suppose there is nothing to do about it. If we did same operation over and over w/ some pb stuff and the output was a non-pb object, then yes, we could hide the pb stuff over in a class but what we do here is particular to this method Can't generalize. bq. bq. I do see that you have added some static methods into protobuf utils. This one is not generic so doesn't deserve to go there? bq. bq. Jimmy Xiang wrote: bq. For getRegionInfo, I used to put it in the protobuf util since it is used lots of places. I will move it to protobuf util. Agree. If used more than once, move it over to protobuf util otherwise I don't see a way around not importing protobuf classes when its a particular usage. bq. On 2012-04-13 23:59:40, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/client/HConnection.java, line 32 bq. https://reviews.apache.org/r/4714/diff/1/?file=101784#file101784line32 bq. bq. Did you say you were going to remove these from here? bq. bq. Jimmy Xiang wrote: bq. I tried but it is hard. Let me think about it again. Even if this stuff just moved to HCM, it'd give us a clean HConnection. bq. On 2012-04-13 23:59:40, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/client/HConnection.java, line 260 bq. https://reviews.apache.org/r/4714/diff/1/?file=101784#file101784line260 bq. bq. ditto bq. bq. An AdminProtocol should have an HConnection? bq. bq. Even if you moved this up into HConnectionManager for now... that'd be better. bq. bq. Jimmy Xiang wrote: bq. Let me think about it and fix it. Thanks. bq. On 2012-04-13 23:59:40, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java, line 2814 bq. https://reviews.apache.org/r/4714/diff/1/?file=101799#file101799line2814 bq. bq. Where has allt his code gone? bq. bq. Jimmy Xiang wrote: bq. Some are moved to RegionServer as I did for 5620 But I did not see it in the patch? Its already moved? - Michael --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4714/#review6922 --- On 2012-04-13 23:03:35, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4714/ bq. --- bq. bq. (Updated 2012-04-13 23:03:35) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This is the admin part of HBase-5443. AdminProtocol part. bq. bq. bq. This addresses bug HBASE-5621. bq. https://issues.apache.org/jira/browse/HBASE-5621 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/AdminProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/ClientProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/catalog/CatalogTracker.java 408db79 bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java ee16e72 bq.src/main/java/org/apache/hadoop/hbase/client/HConnection.java 23f8e5a bq.
[jira] [Commented] (HBASE-5620) Convert the client protocol of HRegionInterface to PB
[ https://issues.apache.org/jira/browse/HBASE-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13252613#comment-13252613 ] jirapos...@reviews.apache.org commented on HBASE-5620: -- bq. On 2012-04-12 00:03:44, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java, line 187 bq. https://reviews.apache.org/r/4629/diff/2/?file=100889#file100889line187 bq. bq. Do these have unit tests? Looks like it would be easy enough to add and could find some nasty bugs early. Not for now, I can add it in a separate patch if it is ok with you. bq. On 2012-04-12 00:03:44, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/protobuf/AdminProtocol.java, line 28 bq. https://reviews.apache.org/r/4629/diff/2/?file=100885#file100885line28 bq. bq. This is the right comment? Fixed. bq. On 2012-04-12 00:03:44, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java, line 57 bq. https://reviews.apache.org/r/4629/diff/2/?file=100882#file100882line57 bq. bq. Fix this text Fixed. bq. On 2012-04-12 00:03:44, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java, line 190 bq. https://reviews.apache.org/r/4629/diff/2/?file=100876#file100876line190 bq. bq. Yeah, we were going to try and get the data up from rpc layer Yes, after changed pb Result to use key value bytes, we should be able to get the data now. bq. On 2012-04-12 00:03:44, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/client/HTable.java, line 672 bq. https://reviews.apache.org/r/4629/diff/2/?file=100875#file100875line672 bq. bq. Is there an EOFE story here too? You are dealing w/ that elsewhere? EOFE is also an IOE. It should be covered as before. - Jimmy --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4629/#review6863 --- On 2012-04-07 21:30:32, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4629/ bq. --- bq. bq. (Updated 2012-04-07 21:30:32) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This is the client protocol part of region interface. The admin protocol part will be done in a different jira. bq. bq. The HRegionInterface is still there since the admin part is not done yet. The other reason is that in case some people still wants the old interface bq. bq. Filters, comparators and coprocessor parameters are still Writable. They will be addressed in different jiras. bq. bq. The existing client interface is not changed so that we don't break existing clients. bq. bq. bq. This addresses bug HBASE-5620. bq. https://issues.apache.org/jira/browse/HBASE-5620 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java 0129ee9 bq.src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java 97293aa bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 bq.src/main/java/org/apache/hadoop/hbase/client/HConnection.java 5d43086 bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 0b783ce bq.src/main/java/org/apache/hadoop/hbase/client/HTable.java aa7652f bq.src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 9903df3 bq.src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java ddcf9ad bq.src/main/java/org/apache/hadoop/hbase/filter/ParseConstants.java 1acbdab bq.src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java cbfa489 bq.src/main/java/org/apache/hadoop/hbase/io/TimeRange.java d135393 bq.src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java 05ae717 bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java f1f06b0 bq.src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 1316457 bq. src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 19ae18c bq.src/main/java/org/apache/hadoop/hbase/protobuf/AdminProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/protobuf/ClientProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 2eb57de bq.src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java PRE-CREATION bq.
[jira] [Commented] (HBASE-5620) Convert the client protocol of HRegionInterface to PB
[ https://issues.apache.org/jira/browse/HBASE-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13252617#comment-13252617 ] jirapos...@reviews.apache.org commented on HBASE-5620: -- bq. On 2012-04-12 04:50:15, Michael Stack wrote: bq. What you want to do Jimmy? On next iteration, if all tests pass, should we check it in? I think its missing unit tests for the new protobuf utility. We should add those. I'd be fine adding them in a separate patch if that would suit you. This thing is already massive. We could commit to a branch too but that probably doesn't get us far; you have your own setup where you are to which you can commit, right? Could you please check it in if all tests pass? It can save me some rebasing/merging efforts. :) I can add unit tests for the new protobuf utility in a separate patch. I do have my own setup to commit my changes. I am also working on HBASE-5621. I can resolve other issues in HBASE-5621, or separate patches. bq. On 2012-04-12 04:50:15, Michael Stack wrote: bq. src/main/protobuf/Client.proto, line 56 bq. https://reviews.apache.org/r/4629/diff/2/?file=100900#file100900line56 bq. bq. Call this keyValueBytes then so method names give more of a clue as to what you'll be getting; save on the surprises? Sure, will fix. bq. On 2012-04-12 04:50:15, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java, line 143 bq. https://reviews.apache.org/r/4629/diff/2/?file=100898#file100898line143 bq. bq. Should this class implement Closeable, Stoppable and Abortable? RegionServerServices defined Stoppable, Abortable. Now RegionServer implements RegionServerServices. Not sure about Closeable. bq. On 2012-04-12 04:50:15, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java, line 98 bq. https://reviews.apache.org/r/4629/diff/2/?file=100898#file100898line98 bq. bq. Shouldn't this implement Server and RegionServerServices? bq. bq. I would love to take the time to break this class up into smaller pieces but thats not for this JIRA. Can do that after this goes in. RegionServer now implements RegionServerServices which extends Server. It will be great to break this class up. bq. On 2012-04-12 04:50:15, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java, line 94 bq. https://reviews.apache.org/r/4629/diff/2/?file=100898#file100898line94 bq. bq. Fix comment. bq. bq. Should you explain your intent for this class too? That HRegionServer will wither away? I think it woud be useful since it was not plain to me at first but after you explained it, I was on your side. Sure, added comments. Thanks. - Jimmy --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4629/#review6868 --- On 2012-04-07 21:30:32, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4629/ bq. --- bq. bq. (Updated 2012-04-07 21:30:32) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This is the client protocol part of region interface. The admin protocol part will be done in a different jira. bq. bq. The HRegionInterface is still there since the admin part is not done yet. The other reason is that in case some people still wants the old interface bq. bq. Filters, comparators and coprocessor parameters are still Writable. They will be addressed in different jiras. bq. bq. The existing client interface is not changed so that we don't break existing clients. bq. bq. bq. This addresses bug HBASE-5620. bq. https://issues.apache.org/jira/browse/HBASE-5620 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java 0129ee9 bq.src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java 97293aa bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 bq.src/main/java/org/apache/hadoop/hbase/client/HConnection.java 5d43086 bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 0b783ce bq.src/main/java/org/apache/hadoop/hbase/client/HTable.java aa7652f bq.src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 9903df3 bq.src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java ddcf9ad bq.src/main/java/org/apache/hadoop/hbase/filter/ParseConstants.java 1acbdab bq.src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java cbfa489 bq.
[jira] [Commented] (HBASE-5620) Convert the client protocol of HRegionInterface to PB
[ https://issues.apache.org/jira/browse/HBASE-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13252619#comment-13252619 ] jirapos...@reviews.apache.org commented on HBASE-5620: -- bq. On 2012-04-12 00:03:44, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java, line 187 bq. https://reviews.apache.org/r/4629/diff/2/?file=100889#file100889line187 bq. bq. Do these have unit tests? Looks like it would be easy enough to add and could find some nasty bugs early. bq. bq. Jimmy Xiang wrote: bq. Not for now, I can add it in a separate patch if it is ok with you. Yes. Make it a blocker though. I see unit tests finding bugs in this stuff. - Michael --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4629/#review6863 --- On 2012-04-07 21:30:32, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4629/ bq. --- bq. bq. (Updated 2012-04-07 21:30:32) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This is the client protocol part of region interface. The admin protocol part will be done in a different jira. bq. bq. The HRegionInterface is still there since the admin part is not done yet. The other reason is that in case some people still wants the old interface bq. bq. Filters, comparators and coprocessor parameters are still Writable. They will be addressed in different jiras. bq. bq. The existing client interface is not changed so that we don't break existing clients. bq. bq. bq. This addresses bug HBASE-5620. bq. https://issues.apache.org/jira/browse/HBASE-5620 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java 0129ee9 bq.src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java 97293aa bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 bq.src/main/java/org/apache/hadoop/hbase/client/HConnection.java 5d43086 bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 0b783ce bq.src/main/java/org/apache/hadoop/hbase/client/HTable.java aa7652f bq.src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 9903df3 bq.src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java ddcf9ad bq.src/main/java/org/apache/hadoop/hbase/filter/ParseConstants.java 1acbdab bq.src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java cbfa489 bq.src/main/java/org/apache/hadoop/hbase/io/TimeRange.java d135393 bq.src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java 05ae717 bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java f1f06b0 bq.src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 1316457 bq. src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 19ae18c bq.src/main/java/org/apache/hadoop/hbase/protobuf/AdminProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/protobuf/ClientProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 2eb57de bq.src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 4026da0 bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionAdminProtos.java 2169310 bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionClientProtos.java b36a9c0 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java 703e73d bq.src/main/java/org/apache/hadoop/hbase/regionserver/Leases.java b520f3f bq.src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java PRE-CREATION bq.src/main/protobuf/Admin.proto PRE-CREATION bq.src/main/protobuf/Client.proto PRE-CREATION bq.src/main/protobuf/RegionAdmin.proto c64d68b bq.src/main/protobuf/RegionClient.proto 358382b bq.src/main/protobuf/hbase.proto da78788 bq.src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java c7284dc bq.
[jira] [Commented] (HBASE-5620) Convert the client protocol of HRegionInterface to PB
[ https://issues.apache.org/jira/browse/HBASE-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13252622#comment-13252622 ] jirapos...@reviews.apache.org commented on HBASE-5620: -- bq. On 2012-04-12 04:50:15, Michael Stack wrote: bq. What you want to do Jimmy? On next iteration, if all tests pass, should we check it in? I think its missing unit tests for the new protobuf utility. We should add those. I'd be fine adding them in a separate patch if that would suit you. This thing is already massive. We could commit to a branch too but that probably doesn't get us far; you have your own setup where you are to which you can commit, right? bq. bq. Jimmy Xiang wrote: bq. Could you please check it in if all tests pass? It can save me some rebasing/merging efforts. :) bq. bq. I can add unit tests for the new protobuf utility in a separate patch. I do have my own setup to commit my changes. I am also working on HBASE-5621. bq. I can resolve other issues in HBASE-5621, or separate patches. bq. I can. You going to do another version to address above feedback? bq. On 2012-04-12 04:50:15, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java, line 143 bq. https://reviews.apache.org/r/4629/diff/2/?file=100898#file100898line143 bq. bq. Should this class implement Closeable, Stoppable and Abortable? bq. bq. Jimmy Xiang wrote: bq. RegionServerServices defined Stoppable, Abortable. Now RegionServer implements RegionServerServices. bq. Not sure about Closeable. Good - Michael --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4629/#review6868 --- On 2012-04-07 21:30:32, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4629/ bq. --- bq. bq. (Updated 2012-04-07 21:30:32) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This is the client protocol part of region interface. The admin protocol part will be done in a different jira. bq. bq. The HRegionInterface is still there since the admin part is not done yet. The other reason is that in case some people still wants the old interface bq. bq. Filters, comparators and coprocessor parameters are still Writable. They will be addressed in different jiras. bq. bq. The existing client interface is not changed so that we don't break existing clients. bq. bq. bq. This addresses bug HBASE-5620. bq. https://issues.apache.org/jira/browse/HBASE-5620 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java 0129ee9 bq.src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java 97293aa bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 bq.src/main/java/org/apache/hadoop/hbase/client/HConnection.java 5d43086 bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 0b783ce bq.src/main/java/org/apache/hadoop/hbase/client/HTable.java aa7652f bq.src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 9903df3 bq.src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java ddcf9ad bq.src/main/java/org/apache/hadoop/hbase/filter/ParseConstants.java 1acbdab bq.src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java cbfa489 bq.src/main/java/org/apache/hadoop/hbase/io/TimeRange.java d135393 bq.src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java 05ae717 bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java f1f06b0 bq.src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 1316457 bq. src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 19ae18c bq.src/main/java/org/apache/hadoop/hbase/protobuf/AdminProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/protobuf/ClientProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 2eb57de bq.src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 4026da0 bq.
[jira] [Commented] (HBASE-5620) Convert the client protocol of HRegionInterface to PB
[ https://issues.apache.org/jira/browse/HBASE-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13252623#comment-13252623 ] jirapos...@reviews.apache.org commented on HBASE-5620: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4629/ --- (Updated 2012-04-12 17:25:51.153610) Review request for hbase. Changes --- Addressed Stack's comments. Summary --- This is the client protocol part of region interface. The admin protocol part will be done in a different jira. The HRegionInterface is still there since the admin part is not done yet. The other reason is that in case some people still wants the old interface Filters, comparators and coprocessor parameters are still Writable. They will be addressed in different jiras. The existing client interface is not changed so that we don't break existing clients. This addresses bug HBASE-5620. https://issues.apache.org/jira/browse/HBASE-5620 Diffs (updated) - src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java 0129ee9 src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java 3167f23 src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 src/main/java/org/apache/hadoop/hbase/client/HConnection.java 5d43086 src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 0b783ce src/main/java/org/apache/hadoop/hbase/client/HTable.java aa7652f src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 9903df3 src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java ddcf9ad src/main/java/org/apache/hadoop/hbase/filter/ParseConstants.java 1acbdab src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java cbfa489 src/main/java/org/apache/hadoop/hbase/io/TimeRange.java d135393 src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java 05ae717 src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java f1f06b0 src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 0573c68 src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java b71ae66 src/main/java/org/apache/hadoop/hbase/protobuf/AdminProtocol.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/protobuf/ClientProtocol.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 2eb57de src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 4026da0 src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionAdminProtos.java 2169310 src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionClientProtos.java b36a9c0 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 9b34e61 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java 703e73d src/main/java/org/apache/hadoop/hbase/regionserver/Leases.java b520f3f src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java PRE-CREATION src/main/protobuf/Admin.proto PRE-CREATION src/main/protobuf/Client.proto PRE-CREATION src/main/protobuf/RegionAdmin.proto c64d68b src/main/protobuf/RegionClient.proto 358382b src/main/protobuf/hbase.proto da78788 src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java c7284dc src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditorNoCluster.java 0042468 src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java e34d8bc src/test/java/org/apache/hadoop/hbase/io/TestHbaseObjectWritable.java f2f8ee3 src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 41616c8 src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 6ed4ba2 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java b4dcb83 src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java ceca6f5 src/test/java/org/apache/hadoop/hbase/regionserver/OOMERegionServer.java cac2989 src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java a1bf73b Diff: https://reviews.apache.org/r/4629/diff Testing --- mvn -PrunAllTests clean test is green, except some flaky tests which need to run again. Also tested it on a real cluster with ycsb and bigtop. Thanks, Jimmy Convert the client protocol of HRegionInterface to PB - Key: HBASE-5620
[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13252963#comment-13252963 ] jirapos...@reviews.apache.org commented on HBASE-5547: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/#review6902 --- Patch looks good. Some concerns in the below. src/main/java/org/apache/hadoop/hbase/HConstants.java https://reviews.apache.org/r/4633/#comment15341 Do these defines need to be global? Could they be package scoped? src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4633/#comment15342 What is one of these? Might say so in here. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4633/#comment15343 Good doc. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4633/#comment15344 Are you missing a version that takes a String? src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java https://reviews.apache.org/r/4633/#comment15345 Does it have to be public? Can it be only of this package? src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java https://reviews.apache.org/r/4633/#comment15346 Is there a close on this class so we can cleanup the zkw? src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java https://reviews.apache.org/r/4633/#comment15347 Should this stuff be generally accessible? src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java https://reviews.apache.org/r/4633/#comment15348 We are in master package. Why reach over into regionserver package? src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java https://reviews.apache.org/r/4633/#comment15349 Can hfilemanager be an interface so that when standing up a region to test, we don't have to know much about this type?' hmm... I see it is an Interface. Good. Thanks. src/main/java/org/apache/hadoop/hbase/master/HMaster.java https://reviews.apache.org/r/4633/#comment15350 This thing is watching a per table config? Other per table configs are whether a table is enabled or not. Should we coalesce? This is also a configuration maintained via another system, one that persists to zk rather than to an hbase-site.xml or into an HTableDescriptor. Why not go into an htabledescriptor? Nicolas just made it dynamically configurable though the table is rolled src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java https://reviews.apache.org/r/4633/#comment15351 Does it need to be public? I see master reaches over to get this class so for that its needed but maybe it coudl be packaged elsewhere? src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java https://reviews.apache.org/r/4633/#comment15352 Do you have to stop this on way out? src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java https://reviews.apache.org/r/4633/#comment15354 Why a ref into regionserver package? src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java https://reviews.apache.org/r/4633/#comment15353 Whats this class do? Should it be abstract since others subclass it? Does it need to be public? src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java https://reviews.apache.org/r/4633/#comment15355 Why these references? Should this tracker be in regionserver package? (Though seems like we are collecting trackers here) src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java https://reviews.apache.org/r/4633/#comment15356 I think its bad that we do this. Not your fault. - Michael On 2012-04-07 19:51:11, Jesse Yates wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4633/ bq. --- bq. bq. (Updated 2012-04-07 19:51:11) bq. bq. bq. Review request for hbase, Michael Stack and Lars Hofhansl. bq. bq. bq. Summary bq. --- bq. bq. Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap. bq. bq. Implementation wise: bq. Updated
[jira] [Commented] (HBASE-5719) Enhance hbck to sideline overlapped mega regions
[ https://issues.apache.org/jira/browse/HBASE-5719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13251683#comment-13251683 ] jirapos...@reviews.apache.org commented on HBASE-5719: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4649/#review6851 --- Ship it! Looks good to me Jimmy. Mind checking 0.90/0.92/0.94 and doing ports if necessary? Should be trivial. I have one nit that you can address or ignore. :) src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitCalculator.java https://reviews.apache.org/r/4649/#comment15241 nit: would be easier to read if assertTrue((r1.equals(ac) r2.equals(ae)) || (r1.equals(ae) r2.equals(ac))); - jmhsieh On 2012-04-09 19:27:53, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4649/ bq. --- bq. bq. (Updated 2012-04-09 19:27:53) bq. bq. bq. Review request for hbase and jmhsieh. bq. bq. bq. Summary bq. --- bq. bq. Make it configurable to sideline some regions in big overlapped groups which hbck doesn't handle currently. bq. bq. The regions chose to sideline are those which overlap with most other regions. bq. bq. bq. This addresses bug HBASE-5719. bq. https://issues.apache.org/jira/browse/HBASE-5719 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 54f9b21 bq.src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java 17678dd bq. src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitCalculator.java ac3b225 bq. bq. Diff: https://reviews.apache.org/r/4649/diff bq. bq. bq. Testing bq. --- bq. bq. mvn -PlocalTests -Dtest=TestHBaseFsck* clean test bq. bq. Also tested in real system to fix inconsistencies. bq. bq. bq. Thanks, bq. bq. Jimmy bq. bq. Enhance hbck to sideline overlapped mega regions Key: HBASE-5719 URL: https://issues.apache.org/jira/browse/HBASE-5719 Project: HBase Issue Type: New Feature Components: hbck Affects Versions: 0.94.0, 0.96.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: hbase-5719.patch If there are too many regions in one overlapped group (by default, more than 10), hbck currently doesn't merge them since it takes time. In this case, we can sideline some regions in the group and break the overlapping to fix the inconsistency. Later on, sidelined regions can be bulk loaded manually. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5620) Convert the client protocol of HRegionInterface to PB
[ https://issues.apache.org/jira/browse/HBASE-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13251719#comment-13251719 ] jirapos...@reviews.apache.org commented on HBASE-5620: -- bq. On 2012-04-11 06:22:10, Michael Stack wrote: bq. I got as far as HTable. Still have more to go. This is some pretty great work Jimmy. This stuff is hard. One thought I was having that is a little related (but it can be safely ignored) is what if there were only one method on an HRegionServer and you gave it an array of Gets, Puts, Delete, Increment, pb messages and you just let the regionserver sort it out. is that even posssible? I'll do rest of review tomorrow. Thanks for reviewing. As to one method to handle all actions, it is supported by the multi call. It is too complex to match the response. bq. On 2012-04-11 06:22:10, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java, line 691 bq. https://reviews.apache.org/r/4629/diff/2/?file=100870#file100870line691 bq. bq. I said it before but I like this code removal It is not used so I removed it. bq. On 2012-04-11 06:22:10, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java, line 533 bq. https://reviews.apache.org/r/4629/diff/2/?file=100872#file100872line533 bq. bq. What about the EOFE you added today? Does that need to go in here anywhere? EOFE is an IOException. Good catch. I addressed it in HBASE-5621, which is in testing now. bq. On 2012-04-11 06:22:10, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java, line 574 bq. https://reviews.apache.org/r/4629/diff/2/?file=100872#file100872line574 bq. bq. So, to be clear, if a client does not close a scanner now, how does it get cleaned up on the server? Does it live forever? In this case, the Leases daemon will expire the scanner and clean it up. bq. On 2012-04-11 06:22:10, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java, line 184 bq. https://reviews.apache.org/r/4629/diff/2/?file=100874#file100874line184 bq. bq. Rename to hbase.clientprotocol.class and CLIENT_PROTOCOL_CLASS. Seems more consistent? Will fix. bq. On 2012-04-11 06:22:10, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/client/HConnection.java, line 235 bq. https://reviews.apache.org/r/4629/diff/2/?file=100873#file100873line235 bq. bq. Does this feel right Jimmy? Seems odd asking an HConnection for a ClientProtocol? It feels like a ClientProtocol should have an HConnection? Or that we'd put together a ClientProtocol + an HConnection for it to use in a different object altogether. What you think? bq. bq. Maybe this is a question for another client implementation that we do later? But would be interested if you had any ideas on this. I agree a ClientProtocol should have an HConnection. We can hide the HConnection. Given a region location, we can create a ClientProtocol which manages its own HConnection. bq. On 2012-04-11 06:22:10, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java, line 187 bq. https://reviews.apache.org/r/4629/diff/2/?file=100874#file100874line187 bq. bq. Rename too accordingly. Will fix. bq. On 2012-04-11 06:22:10, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java, line 565 bq. https://reviews.apache.org/r/4629/diff/2/?file=100874#file100874line565 bq. bq. This is pb VersionedProtocol? So far, it can be any VersionedProtocol: ClientProtocol or HRegionInterface. bq. On 2012-04-11 06:22:10, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java, line 621 bq. https://reviews.apache.org/r/4629/diff/2/?file=100874#file100874line621 bq. bq. Want to adjust this message? Fixed. bq. On 2012-04-11 06:22:10, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java, line 1440 bq. https://reviews.apache.org/r/4629/diff/2/?file=100874#file100874line1440 bq. bq. This set of protocols is covered by the synchronize held above? i.e. all access on protocols are under a block like this? Good point. Will fix. bq. On 2012-04-11 06:22:10, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java, line 1449 bq. https://reviews.apache.org/r/4629/diff/2/?file=100874#file100874line1449 bq. bq. Is there a difference here? We create an ISA only when we need it previously? Now we create it always? Is there a diff here? HServerAddress is deprecated. We are not going to get an ISA passed in. The protocol is cached so we don't need to create it many times. - Jimmy
[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13251727#comment-13251727 ] jirapos...@reviews.apache.org commented on HBASE-5547: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/#review6820 --- src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4633/#comment15202 done. Originally modeled after the catalog tracker in naming. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4633/#comment15203 done. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4633/#comment15204 done. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4633/#comment15205 done src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4633/#comment15206 done. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4633/#comment15207 Would rather not, since in that case we would keep around a persistent connection to ZK, which is against what the community has been doing with cutting the client/zk connection (though admittedly that not directly applicable here). src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4633/#comment15208 changed to confirmedArchiving. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4633/#comment15184 Yeah, it would. But then you should not attempt to expect the archive as valid and fail your backup process. Was going to leave it to the user to determine if they wanted to do the cleanup. However, I would be ok if we added a method that cleaned up the filesystem for files created in the backup after the archiving started. Unfortunately, this means add an extra method on the master which I was trying to avoid, but I guess its not the end of the world. We also have a two-phase commit issue here as we also need to then write a 'consistent-confirmed' file into the archive when it completes to ensure that failed clients don't break the archiving. thoughts? src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4633/#comment15209 done. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4633/#comment15183 Method name isn't enough? src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java https://reviews.apache.org/r/4633/#comment15210 done. src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java https://reviews.apache.org/r/4633/#comment15211 done. src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java https://reviews.apache.org/r/4633/#comment15212 done - regionsBeingArchived it is. src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java https://reviews.apache.org/r/4633/#comment15213 done. src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java https://reviews.apache.org/r/4633/#comment15185 Cruft from the earlier version. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java https://reviews.apache.org/r/4633/#comment15186 yeah, guess it should be. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java https://reviews.apache.org/r/4633/#comment15214 done. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java https://reviews.apache.org/r/4633/#comment15187 This is a super rare case where the name of the regiondir to be backed up is the same as the one in the backup already. Just being extra careful that the random storefile name doesn't match, though it is _very_ unlikely to happen, it still could (which could lead to bad circumstances). Currently, the storefile check doesn't check the archive directory and I think it would be pretty big overhead to expect it to do that as well (lots of added dependencies, calls, etc). We can just do the check and renaming here - its not super complicated and seems to work. src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java https://reviews.apache.org/r/4633/#comment15215 done. src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java https://reviews.apache.org/r/4633/#comment15216 done. - Jesse On 2012-04-07 19:51:11, Jesse Yates wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4633/ bq.
[jira] [Commented] (HBASE-5719) Enhance hbck to sideline overlapped mega regions
[ https://issues.apache.org/jira/browse/HBASE-5719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13251743#comment-13251743 ] jirapos...@reviews.apache.org commented on HBASE-5719: -- bq. On 2012-04-11 15:44:08, jmhsieh wrote: bq. src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitCalculator.java, lines 384-392 bq. https://reviews.apache.org/r/4649/diff/2-3/?file=100674#file100674line384 bq. bq. nit: would be easier to read if bq. assertTrue((r1.equals(ac) r2.equals(ae)) || (r1.equals(ae) r2.equals(ac))); That's the original version actually. However, the equals method somehow doesn't work. I didn't try to implement the equals method in the SimpleRange class. - Jimmy --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4649/#review6851 --- On 2012-04-09 19:27:53, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4649/ bq. --- bq. bq. (Updated 2012-04-09 19:27:53) bq. bq. bq. Review request for hbase and jmhsieh. bq. bq. bq. Summary bq. --- bq. bq. Make it configurable to sideline some regions in big overlapped groups which hbck doesn't handle currently. bq. bq. The regions chose to sideline are those which overlap with most other regions. bq. bq. bq. This addresses bug HBASE-5719. bq. https://issues.apache.org/jira/browse/HBASE-5719 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 54f9b21 bq.src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java 17678dd bq. src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitCalculator.java ac3b225 bq. bq. Diff: https://reviews.apache.org/r/4649/diff bq. bq. bq. Testing bq. --- bq. bq. mvn -PlocalTests -Dtest=TestHBaseFsck* clean test bq. bq. Also tested in real system to fix inconsistencies. bq. bq. bq. Thanks, bq. bq. Jimmy bq. bq. Enhance hbck to sideline overlapped mega regions Key: HBASE-5719 URL: https://issues.apache.org/jira/browse/HBASE-5719 Project: HBase Issue Type: New Feature Components: hbck Affects Versions: 0.94.0, 0.96.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: hbase-5719.patch If there are too many regions in one overlapped group (by default, more than 10), hbck currently doesn't merge them since it takes time. In this case, we can sideline some regions in the group and break the overlapping to fix the inconsistency. Later on, sidelined regions can be bulk loaded manually. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5719) Enhance hbck to sideline overlapped mega regions
[ https://issues.apache.org/jira/browse/HBASE-5719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13251768#comment-13251768 ] jirapos...@reviews.apache.org commented on HBASE-5719: -- bq. On 2012-04-11 15:44:08, jmhsieh wrote: bq. src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitCalculator.java, lines 384-392 bq. https://reviews.apache.org/r/4649/diff/2-3/?file=100674#file100674line384 bq. bq. nit: would be easier to read if bq. assertTrue((r1.equals(ac) r2.equals(ae)) || (r1.equals(ae) r2.equals(ac))); bq. bq. Jimmy Xiang wrote: bq. That's the original version actually. However, the equals method somehow doesn't work. I didn't try to implement the equals method in the SimpleRange class. I see. Either is fine by me -- as is or you can make the change.:) - jmhsieh --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4649/#review6851 --- On 2012-04-09 19:27:53, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4649/ bq. --- bq. bq. (Updated 2012-04-09 19:27:53) bq. bq. bq. Review request for hbase and jmhsieh. bq. bq. bq. Summary bq. --- bq. bq. Make it configurable to sideline some regions in big overlapped groups which hbck doesn't handle currently. bq. bq. The regions chose to sideline are those which overlap with most other regions. bq. bq. bq. This addresses bug HBASE-5719. bq. https://issues.apache.org/jira/browse/HBASE-5719 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 54f9b21 bq.src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java 17678dd bq. src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitCalculator.java ac3b225 bq. bq. Diff: https://reviews.apache.org/r/4649/diff bq. bq. bq. Testing bq. --- bq. bq. mvn -PlocalTests -Dtest=TestHBaseFsck* clean test bq. bq. Also tested in real system to fix inconsistencies. bq. bq. bq. Thanks, bq. bq. Jimmy bq. bq. Enhance hbck to sideline overlapped mega regions Key: HBASE-5719 URL: https://issues.apache.org/jira/browse/HBASE-5719 Project: HBase Issue Type: New Feature Components: hbck Affects Versions: 0.94.0, 0.96.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: hbase-5719.patch If there are too many regions in one overlapped group (by default, more than 10), hbck currently doesn't merge them since it takes time. In this case, we can sideline some regions in the group and break the overlapping to fix the inconsistency. Later on, sidelined regions can be bulk loaded manually. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5741) ImportTsv does not check for table existence
[ https://issues.apache.org/jira/browse/HBASE-5741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13251851#comment-13251851 ] jirapos...@reviews.apache.org commented on HBASE-5741: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4700/ --- (Updated 2012-04-11 19:01:39.660787) Review request for hbase. Changes --- Changes done as per Ted. Summary --- There is a bulk output option in the importtsv workload. It outputs the HFiles in a user defined directory. The current code assumes that a table with its name equal to the given output directory exists, and throws an exception otherwise. Here is a patch for creating a table in case it doesn't exist. This addresses bug HBase-5741. https://issues.apache.org/jira/browse/HBase-5741 Diffs (updated) - src/main/java/org/apache/hadoop/hbase/mapreduce/ImportTsv.java ab22fc4 src/test/java/org/apache/hadoop/hbase/mapreduce/TestImportTsv.java ac30a62 Diff: https://reviews.apache.org/r/4700/diff Testing --- Added a new test for bulkoutput; All importtsv tests pass. Thanks, Himanshu ImportTsv does not check for table existence - Key: HBASE-5741 URL: https://issues.apache.org/jira/browse/HBASE-5741 Project: HBase Issue Type: Bug Components: mapreduce Affects Versions: 0.90.4 Reporter: Clint Heath Assignee: Himanshu Vashishtha Attachments: HBase-5741.patch The usage statement for the importtsv command to hbase claims this: Note: if you do not use this option, then the target table must already exist in HBase (in reference to the importtsv.bulk.output command-line option) The truth is, the table must exist no matter what, importtsv cannot and will not create it for you. This is the case because the createSubmittableJob method of ImportTsv does not even attempt to check if the table exists already, much less create it: (From org.apache.hadoop.hbase.mapreduce.ImportTsv.java) 305 HTable table = new HTable(conf, tableName); The HTable method signature in use there assumes the table exists and runs a meta scan on it: (From org.apache.hadoop.hbase.client.HTable.java) 142 * Creates an object to access a HBase table. ... 151 public HTable(Configuration conf, final String tableName) What we should do inside of createSubmittableJob is something similar to what the completebulkloads command would do: (Taken from org.apache.hadoop.hbase.mapreduce.LoadIncrementalHFiles.java) 690 boolean tableExists = this.doesTableExist(tableName); 691 if (!tableExists) this.createTable(tableName,dirPath); Currently the docs are misleading, the table in fact must exist prior to running importtsv. We should check if it exists rather than assume it's already there and throw the below exception: 12/03/14 17:15:42 WARN client.HConnectionManager$HConnectionImplementation: Encountered problems when prefetch META table: org.apache.hadoop.hbase.TableNotFoundException: Cannot find row in .META. for table: myTable2, row=myTable2,,99 at org.apache.hadoop.hbase.client.MetaScanner.metaScan(MetaScanner.java:150) ... -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5620) Convert the client protocol of HRegionInterface to PB
[ https://issues.apache.org/jira/browse/HBASE-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13252005#comment-13252005 ] jirapos...@reviews.apache.org commented on HBASE-5620: -- bq. On 2012-04-11 06:22:10, Michael Stack wrote: bq. I got as far as HTable. Still have more to go. This is some pretty great work Jimmy. This stuff is hard. One thought I was having that is a little related (but it can be safely ignored) is what if there were only one method on an HRegionServer and you gave it an array of Gets, Puts, Delete, Increment, pb messages and you just let the regionserver sort it out. is that even posssible? I'll do rest of review tomorrow. bq. bq. Jimmy Xiang wrote: bq. Thanks for reviewing. As to one method to handle all actions, it is supported by the multi call. It is too complex to match the response. Sorry, I don't get the last part? The response is too hard to make when the query is lots of types? bq. On 2012-04-11 06:22:10, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java, line 691 bq. https://reviews.apache.org/r/4629/diff/2/?file=100870#file100870line691 bq. bq. I said it before but I like this code removal bq. bq. Jimmy Xiang wrote: bq. It is not used so I removed it. Great bq. On 2012-04-11 06:22:10, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/client/HConnection.java, line 235 bq. https://reviews.apache.org/r/4629/diff/2/?file=100873#file100873line235 bq. bq. Does this feel right Jimmy? Seems odd asking an HConnection for a ClientProtocol? It feels like a ClientProtocol should have an HConnection? Or that we'd put together a ClientProtocol + an HConnection for it to use in a different object altogether. What you think? bq. bq. Maybe this is a question for another client implementation that we do later? But would be interested if you had any ideas on this. bq. bq. Jimmy Xiang wrote: bq. I agree a ClientProtocol should have an HConnection. We can hide the HConnection. Given a region location, we can create a ClientProtocol which manages its own HConnection. Sounds better - Michael --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4629/#review6833 --- On 2012-04-07 21:30:32, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4629/ bq. --- bq. bq. (Updated 2012-04-07 21:30:32) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This is the client protocol part of region interface. The admin protocol part will be done in a different jira. bq. bq. The HRegionInterface is still there since the admin part is not done yet. The other reason is that in case some people still wants the old interface bq. bq. Filters, comparators and coprocessor parameters are still Writable. They will be addressed in different jiras. bq. bq. The existing client interface is not changed so that we don't break existing clients. bq. bq. bq. This addresses bug HBASE-5620. bq. https://issues.apache.org/jira/browse/HBASE-5620 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java 0129ee9 bq.src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java 97293aa bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 bq.src/main/java/org/apache/hadoop/hbase/client/HConnection.java 5d43086 bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 0b783ce bq.src/main/java/org/apache/hadoop/hbase/client/HTable.java aa7652f bq.src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 9903df3 bq.src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java ddcf9ad bq.src/main/java/org/apache/hadoop/hbase/filter/ParseConstants.java 1acbdab bq.src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java cbfa489 bq.src/main/java/org/apache/hadoop/hbase/io/TimeRange.java d135393 bq.src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java 05ae717 bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java f1f06b0 bq.src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 1316457 bq. src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 19ae18c bq.src/main/java/org/apache/hadoop/hbase/protobuf/AdminProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/protobuf/ClientProtocol.java PRE-CREATION bq.
[jira] [Commented] (HBASE-5620) Convert the client protocol of HRegionInterface to PB
[ https://issues.apache.org/jira/browse/HBASE-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13252035#comment-13252035 ] jirapos...@reviews.apache.org commented on HBASE-5620: -- bq. On 2012-04-11 06:22:10, Michael Stack wrote: bq. I got as far as HTable. Still have more to go. This is some pretty great work Jimmy. This stuff is hard. One thought I was having that is a little related (but it can be safely ignored) is what if there were only one method on an HRegionServer and you gave it an array of Gets, Puts, Delete, Increment, pb messages and you just let the regionserver sort it out. is that even posssible? I'll do rest of review tomorrow. bq. bq. Jimmy Xiang wrote: bq. Thanks for reviewing. As to one method to handle all actions, it is supported by the multi call. It is too complex to match the response. bq. bq. Michael Stack wrote: bq. Sorry, I don't get the last part? The response is too hard to make when the query is lots of types? Right, basically the code is not efficient. - Jimmy --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4629/#review6833 --- On 2012-04-07 21:30:32, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4629/ bq. --- bq. bq. (Updated 2012-04-07 21:30:32) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This is the client protocol part of region interface. The admin protocol part will be done in a different jira. bq. bq. The HRegionInterface is still there since the admin part is not done yet. The other reason is that in case some people still wants the old interface bq. bq. Filters, comparators and coprocessor parameters are still Writable. They will be addressed in different jiras. bq. bq. The existing client interface is not changed so that we don't break existing clients. bq. bq. bq. This addresses bug HBASE-5620. bq. https://issues.apache.org/jira/browse/HBASE-5620 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java 0129ee9 bq.src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java 97293aa bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 bq.src/main/java/org/apache/hadoop/hbase/client/HConnection.java 5d43086 bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 0b783ce bq.src/main/java/org/apache/hadoop/hbase/client/HTable.java aa7652f bq.src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 9903df3 bq.src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java ddcf9ad bq.src/main/java/org/apache/hadoop/hbase/filter/ParseConstants.java 1acbdab bq.src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java cbfa489 bq.src/main/java/org/apache/hadoop/hbase/io/TimeRange.java d135393 bq.src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java 05ae717 bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java f1f06b0 bq.src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 1316457 bq. src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 19ae18c bq.src/main/java/org/apache/hadoop/hbase/protobuf/AdminProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/protobuf/ClientProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 2eb57de bq.src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 4026da0 bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionAdminProtos.java 2169310 bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionClientProtos.java b36a9c0 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java 703e73d bq.src/main/java/org/apache/hadoop/hbase/regionserver/Leases.java b520f3f bq.src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java PRE-CREATION bq.src/main/protobuf/Admin.proto PRE-CREATION bq.
[jira] [Commented] (HBASE-5620) Convert the client protocol of HRegionInterface to PB
[ https://issues.apache.org/jira/browse/HBASE-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13252072#comment-13252072 ] jirapos...@reviews.apache.org commented on HBASE-5620: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4629/#review6863 --- I'm halfway down on second page src/main/java/org/apache/hadoop/hbase/client/HTable.java https://reviews.apache.org/r/4629/#comment15282 Is there an EOFE story here too? You are dealing w/ that elsewhere? src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java https://reviews.apache.org/r/4629/#comment15283 Yeah, we were going to try and get the data up from rpc layer src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java https://reviews.apache.org/r/4629/#comment15284 Does this call need to return a Result or do we need to add a new method to support being able to get results as part of the call to open a scanner? src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java https://reviews.apache.org/r/4629/#comment15285 Fix this text src/main/java/org/apache/hadoop/hbase/protobuf/AdminProtocol.java https://reviews.apache.org/r/4629/#comment15286 This is the right comment? src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java https://reviews.apache.org/r/4629/#comment15287 Do these have unit tests? Looks like it would be easy enough to add and could find some nasty bugs early. - Michael On 2012-04-07 21:30:32, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4629/ bq. --- bq. bq. (Updated 2012-04-07 21:30:32) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This is the client protocol part of region interface. The admin protocol part will be done in a different jira. bq. bq. The HRegionInterface is still there since the admin part is not done yet. The other reason is that in case some people still wants the old interface bq. bq. Filters, comparators and coprocessor parameters are still Writable. They will be addressed in different jiras. bq. bq. The existing client interface is not changed so that we don't break existing clients. bq. bq. bq. This addresses bug HBASE-5620. bq. https://issues.apache.org/jira/browse/HBASE-5620 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java 0129ee9 bq.src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java 97293aa bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 bq.src/main/java/org/apache/hadoop/hbase/client/HConnection.java 5d43086 bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java 0b783ce bq.src/main/java/org/apache/hadoop/hbase/client/HTable.java aa7652f bq.src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 9903df3 bq.src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java ddcf9ad bq.src/main/java/org/apache/hadoop/hbase/filter/ParseConstants.java 1acbdab bq.src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java cbfa489 bq.src/main/java/org/apache/hadoop/hbase/io/TimeRange.java d135393 bq.src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java 05ae717 bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java f1f06b0 bq.src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 1316457 bq. src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 19ae18c bq.src/main/java/org/apache/hadoop/hbase/protobuf/AdminProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/protobuf/ClientProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 2eb57de bq.src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/ClientProtos.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 4026da0 bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionAdminProtos.java 2169310 bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionClientProtos.java b36a9c0 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d bq.
[jira] [Commented] (HBASE-5645) [findbugs] Fix correctness warnings
[ https://issues.apache.org/jira/browse/HBASE-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250969#comment-13250969 ] jirapos...@reviews.apache.org commented on HBASE-5645: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4612/ --- (Updated 2012-04-10 19:11:48.808176) Review request for hbase. Changes --- Updated patch to work with trunk. Fixed hanging test. Ran test-patch.sh and -PrunAllTests. Summary --- Fixes for issues reported in findbugs for Correctness category. This addresses bug HBASE-5645. https://issues.apache.org/jira/browse/HBASE-5645 Diffs (updated) - dev-support/findHangingTest.sh 4518c68 dev-support/test-patch.properties 203df69 src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java c4a8a9e src/main/java/org/apache/hadoop/hbase/mapreduce/CellCounter.java 63b2ed5 src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java de4b1e6 src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java 1676832 src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb21bdd src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 1c253a0 src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreLAB.java 2694897 src/main/java/org/apache/hadoop/hbase/regionserver/ShutdownHook.java 00f8d99 src/main/java/org/apache/hadoop/hbase/regionserver/Store.java bf1618e src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 1636bfb src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 698bb3d src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java 545bd02 src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java 62cf6ac src/main/java/org/apache/hadoop/hbase/util/FSUtils.java aebe5b0 src/main/java/org/apache/hadoop/hbase/util/JVMClusterUtil.java c21377c src/main/java/org/apache/hadoop/hbase/util/PoolMap.java 8e2a856 Diff: https://reviews.apache.org/r/4612/diff Testing --- Ran test-patch.sh and unit tests. Thanks, David [findbugs] Fix correctness warnings --- Key: HBASE-5645 URL: https://issues.apache.org/jira/browse/HBASE-5645 Project: HBase Issue Type: Sub-task Components: scripts Reporter: Jonathan Hsieh Assignee: David S. Wang Attachments: HBASE-5645-2.patch, HBASE-5645.patch, HBASE-5645.patch See https://builds.apache.org/job/PreCommit-HBASE-Build/1313//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Fix the warnings in the correctness section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5760) Unit tests should write only under /target
[ https://issues.apache.org/jira/browse/HBASE-5760?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13251040#comment-13251040 ] jirapos...@reviews.apache.org commented on HBASE-5760: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4692/ --- (Updated 2012-04-10 20:34:07.296907) Review request for hbase. Summary --- Some of the unit test runs result in files under $hbase_home/test, $hbase_home/build, or $hbase_home/. We should ensure that all tests use target as their data location. This addresses bug HBASE-5760. https://issues.apache.org/jira/browse/HBASE-5760 Diffs - src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java 3f239e0 src/test/java/org/apache/hadoop/hbase/coprocessor/TestCoprocessorInterface.java 32f0cc3 src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverInterface.java 60f21c0 src/test/java/org/apache/hadoop/hbase/coprocessor/TestRegionObserverStacking.java 002d611 src/test/java/org/apache/hadoop/hbase/mapreduce/TestHFileOutputFormat.java a6bcbac Diff: https://reviews.apache.org/r/4692/diff Testing --- Results : Failed tests: testKeyUpdate(org.apache.hadoop.hbase.security.token.TestZKSecretWatcher): expected:AuthenticationKey[ id=2, expiration=9223372036854775807 ] but was:AuthenticationKey[ id=1, expiration=1334028188780 ] Tests in error: testDelayedRpcImmediateReturnValue(org.apache.hadoop.hbase.ipc.TestDelayedRpc): Call to /127.0.0.1:65022 failed on socket timeout exception: java.net.SocketTimeoutException: 1000 millis timeout while waiting for channel to be ready for read. ch : java.nio.channels.SocketChannel[connected local=/127.0.0.1:65039 remote=/127.0.0.1:65022] testTralingGarbageCorruptionFileSkipErrorsPasses(org.apache.hadoop.hbase.regionserver.wal.TestHLogSplit): All datanodes 127.0.0.1:52012 are bad. Aborting... org.apache.hadoop.hbase.security.access.TestAccessControlFilter: Shutting down org.apache.hadoop.hbase.security.access.TestAccessController: Shutting down org.apache.hadoop.hbase.security.access.TestTablePermissions: Shutting down org.apache.hadoop.hbase.security.access.TestZKPermissionsWatcher: Shutting down org.apache.hadoop.hbase.security.token.TestTokenAuthentication: Shutting down Tests run: 900, Failures: 1, Errors: 7, Skipped: 9 I think the ZKSecretWatcher is unrelated. After the tests, there is no residual test files lying around. Thanks, enis Unit tests should write only under /target -- Key: HBASE-5760 URL: https://issues.apache.org/jira/browse/HBASE-5760 Project: HBase Issue Type: Improvement Components: test Affects Versions: 0.96.0 Reporter: Enis Soztutar Assignee: Enis Soztutar Priority: Minor Some of the unit test runs result in files under $hbase_home/test, $hbase_home/build, or $hbase_home/. We should ensure that all tests use target as their data location. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5719) Enhance hbck to sideline overlapped mega regions
[ https://issues.apache.org/jira/browse/HBASE-5719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250024#comment-13250024 ] jirapos...@reviews.apache.org commented on HBASE-5719: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4649/#review6764 --- Jimmy, Looks good to me. Mostly some questions and a little bit of of follow up to make it easier to read and to provide user more info. src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java https://reviews.apache.org/r/4649/#comment14916 nice catch. src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java https://reviews.apache.org/r/4649/#comment15130 Maybe this one is at info level? src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java https://reviews.apache.org/r/4649/#comment14917 nit: typo (old was correct) src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java https://reviews.apache.org/r/4649/#comment14918 Is this n per group or globally? please make it clear in usage src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java https://reviews.apache.org/r/4649/#comment14919 nit: should be like previous -maxMerge's comments. src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java https://reviews.apache.org/r/4649/#comment15129 To find? Would returning a sort make sense? if there are no overlaps will this return the specified number (or less?) Could this return several regions that now leave holes? nit: It might make testing clearer to return the overlapCount - regions map and check that. src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java https://reviews.apache.org/r/4649/#comment15132 Add comment about structure contents. something like: Integer key is overlap count and List is regions that have that many overlaps. maybe rename to overlapRangeMap? src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java https://reviews.apache.org/r/4649/#comment15135 add comment: calculates the # of overlaps for each region and populates rangeOverlapMap src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java https://reviews.apache.org/r/4649/#comment15133 add comment? We always overlap with ourselves -- which is why 1 is the condition below.. src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java https://reviews.apache.org/r/4649/#comment15136 what is the intuition here? src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitCalculator.java https://reviews.apache.org/r/4649/#comment15140 nit: it would easer to read if this was called ai, the next ae, and the one after that ac. Also would help if you put the expected overlap count in comments! - jmhsieh On 2012-04-06 18:27:34, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4649/ bq. --- bq. bq. (Updated 2012-04-06 18:27:34) bq. bq. bq. Review request for hbase and jmhsieh. bq. bq. bq. Summary bq. --- bq. bq. Make it configurable to sideline some regions in big overlapped groups which hbck doesn't handle currently. bq. bq. The regions chose to sideline are those which overlap with most other regions. bq. bq. bq. This addresses bug HBASE-5719. bq. https://issues.apache.org/jira/browse/HBASE-5719 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 54f9b21 bq.src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java 17678dd bq. src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitCalculator.java ac3b225 bq. bq. Diff: https://reviews.apache.org/r/4649/diff bq. bq. bq. Testing bq. --- bq. bq. mvn -PlocalTests -Dtest=TestHBaseFsck* clean test bq. bq. Also tested in real system to fix inconsistencies. bq. bq. bq. Thanks, bq. bq. Jimmy bq. bq. Enhance hbck to sideline overlapped mega regions Key: HBASE-5719 URL: https://issues.apache.org/jira/browse/HBASE-5719 Project: HBase Issue Type: New Feature Components: hbck Affects Versions: 0.94.0, 0.96.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: hbase-5719.patch If there are too many regions in one overlapped group (by default, more than 10), hbck currently doesn't merge them since it takes time. In this case, we can sideline some
[jira] [Commented] (HBASE-5719) Enhance hbck to sideline overlapped mega regions
[ https://issues.apache.org/jira/browse/HBASE-5719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250087#comment-13250087 ] jirapos...@reviews.apache.org commented on HBASE-5719: -- bq. On 2012-04-09 18:20:11, jmhsieh wrote: bq. src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 1869 bq. https://reviews.apache.org/r/4649/diff/2/?file=100672#file100672line1869 bq. bq. Maybe this one is at info level? Will change to info level. bq. On 2012-04-09 18:20:11, jmhsieh wrote: bq. src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 2828 bq. https://reviews.apache.org/r/4649/diff/2/?file=100672#file100672line2828 bq. bq. nit: typo (old was correct) Changed it back. It must be because the mouse moved when I was typing something. I didn't remember I changed it. :) bq. On 2012-04-09 18:20:11, jmhsieh wrote: bq. src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 2895 bq. https://reviews.apache.org/r/4649/diff/2/?file=100672#file100672line2895 bq. bq. Is this n per group or globally? please make it clear in usage It is per group. Will fix it. bq. On 2012-04-09 18:20:11, jmhsieh wrote: bq. src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java, line 2981 bq. https://reviews.apache.org/r/4649/diff/2/?file=100672#file100672line2981 bq. bq. nit: should be like previous -maxMerge's comments. Will fix it. bq. On 2012-04-09 18:20:11, jmhsieh wrote: bq. src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitCalculator.java, line 360 bq. https://reviews.apache.org/r/4649/diff/2/?file=100674#file100674line360 bq. bq. nit: it would easer to read if this was called ai, the next ae, and the one after that ac. bq. bq. Also would help if you put the expected overlap count in comments! Sure. bq. On 2012-04-09 18:20:11, jmhsieh wrote: bq. src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java, line 212 bq. https://reviews.apache.org/r/4649/diff/2/?file=100673#file100673line212 bq. bq. what is the intuition here? Added comment bq. On 2012-04-09 18:20:11, jmhsieh wrote: bq. src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java, line 189 bq. https://reviews.apache.org/r/4649/diff/2/?file=100673#file100673line189 bq. bq. add comment? bq. bq. We always overlap with ourselves -- which is why 1 is the condition below.. Will do. bq. On 2012-04-09 18:20:11, jmhsieh wrote: bq. src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java, line 178 bq. https://reviews.apache.org/r/4649/diff/2/?file=100673#file100673line178 bq. bq. add comment: calculates the # of overlaps for each region and populates rangeOverlapMap Sure bq. On 2012-04-09 18:20:11, jmhsieh wrote: bq. src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java, line 177 bq. https://reviews.apache.org/r/4649/diff/2/?file=100673#file100673line177 bq. bq. Add comment about structure contents. something like: Integer key is overlap count and List is regions that have that many overlaps. bq. maybe rename to overlapRangeMap? Sure. bq. On 2012-04-09 18:20:11, jmhsieh wrote: bq. src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java, line 171 bq. https://reviews.apache.org/r/4649/diff/2/?file=100673#file100673line171 bq. bq. To find? Would returning a sort make sense? if there are no overlaps will this return the specified number (or less?) bq. bq. Could this return several regions that now leave holes? bq. bq. nit: It might make testing clearer to return the overlapCount - regions map and check that. Added comment. - Jimmy --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4649/#review6764 --- On 2012-04-06 18:27:34, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4649/ bq. --- bq. bq. (Updated 2012-04-06 18:27:34) bq. bq. bq. Review request for hbase and jmhsieh. bq. bq. bq. Summary bq. --- bq. bq. Make it configurable to sideline some regions in big overlapped groups which hbck doesn't handle currently. bq. bq. The regions chose to sideline are those which overlap with most other regions. bq. bq. bq. This addresses bug HBASE-5719. bq. https://issues.apache.org/jira/browse/HBASE-5719 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 54f9b21 bq.
[jira] [Commented] (HBASE-5719) Enhance hbck to sideline overlapped mega regions
[ https://issues.apache.org/jira/browse/HBASE-5719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13250092#comment-13250092 ] jirapos...@reviews.apache.org commented on HBASE-5719: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4649/ --- (Updated 2012-04-09 19:27:53.361210) Review request for hbase and jmhsieh. Changes --- Addressed Jon's second review comments. Summary --- Make it configurable to sideline some regions in big overlapped groups which hbck doesn't handle currently. The regions chose to sideline are those which overlap with most other regions. This addresses bug HBASE-5719. https://issues.apache.org/jira/browse/HBASE-5719 Diffs (updated) - src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 54f9b21 src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java 17678dd src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitCalculator.java ac3b225 Diff: https://reviews.apache.org/r/4649/diff Testing --- mvn -PlocalTests -Dtest=TestHBaseFsck* clean test Also tested in real system to fix inconsistencies. Thanks, Jimmy Enhance hbck to sideline overlapped mega regions Key: HBASE-5719 URL: https://issues.apache.org/jira/browse/HBASE-5719 Project: HBase Issue Type: New Feature Components: hbck Affects Versions: 0.94.0, 0.96.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: hbase-5719.patch If there are too many regions in one overlapped group (by default, more than 10), hbck currently doesn't merge them since it takes time. In this case, we can sideline some regions in the group and break the overlapping to fix the inconsistency. Later on, sidelined regions can be bulk loaded manually. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249319#comment-13249319 ] jirapos...@reviews.apache.org commented on HBASE-5547: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/ --- (Updated 2012-04-07 19:51:11.515830) Review request for hbase, Michael Stack and Lars Hofhansl. Changes --- A bit rearchitecting in the new patch: 1) Separated the client and server archiving into distinct classes (and saved changes to HMasterInferface) 2) A lot of the functions in the ArchiveUtil were moved to their respective classes and move a couple smaller methods in 3) Added a synchronous enable as the default for the client 4) Merged the tracker and the manger into the tracker and pulled out the important methods into a monitor interface. 5) Addressed Ted's comments from the last patch (and hopefully didn't add them back in the change). 6) Added a bunch of testing Summary --- Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap. Implementation wise: Updated the HMasterInterface to pass the calls onto the zookeeper. Added a zk listener to handle updates from the master to the RS to backup. Added a utility for removing files and finding archive directories Added tests for the regionserver and catalogjanitor approaches. Added creation of manager in regionserver. This addresses bug HBASE-5547. https://issues.apache.org/jira/browse/HBASE-5547 Diffs (updated) - src/main/java/org/apache/hadoop/hbase/HConstants.java a4b989e src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb21bdd src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java c3df319 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53 src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 509a467 src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f src/main/resources/hbase-default.xml 44ee689 src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 41616c8 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java b4dcb83 src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759 Diff: https://reviews.apache.org/r/4633/diff Testing --- Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper. Thanks, Jesse Don't delete HFiles when in backup mode - Key: HBASE-5547 URL: https://issues.apache.org/jira/browse/HBASE-5547 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Assignee: Jesse Yates This came up in a discussion I had with Stack. It would be nice if HBase could be notified that a backup is in progress (via a znode for example) and in that case either: 1. rename HFiles to be delete to file.bck 2. rename the HFiles into a special directory 3. rename them to a general trash directory (which would not need to be tied to backup mode). That way it should be able to get a consistent backup based on HFiles (HDFS snapshots or hard links would be better options here, but we do not have those). #1 makes cleanup a bit harder. -- This message is automatically generated by JIRA. If you
[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249332#comment-13249332 ] jirapos...@reviews.apache.org commented on HBASE-5547: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/#review6777 --- In the javadoc, 'backups' were used. I think backup should be good enough. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4633/#comment14940 How about naming this method createHFileArchiveManager or buildHFileArchiveManager ? src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4633/#comment14941 The directory is specified by HConstants.HFILE_ARCHIVE_DIRECTORY. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4633/#comment14943 Should read 'all online region servers hosting the' src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4633/#comment14944 Typo: received. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4633/#comment14942 Please add javadoc for archive. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4633/#comment14945 This would create a manager. Can we cache the archive manager ? src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4633/#comment14952 Parameter for allRegionsDoneBackup() is called confirmedDone This seems to be inconsistent with the return from regionsArchivingTable() src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4633/#comment14947 This call would leave the backup for tableName in an inconsistent state, right ? src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4633/#comment14946 tableName should be included. src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4633/#comment14948 Please mention this call is asynchronous. src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java https://reviews.apache.org/r/4633/#comment14954 License. src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java https://reviews.apache.org/r/4633/#comment14949 Should read 'archiving for the table' src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java https://reviews.apache.org/r/4633/#comment14950 How about naming this method regionsBeingArchived or regionsBeingArchivedForTable ? I think the first should be enough since table is a parameter. src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java https://reviews.apache.org/r/4633/#comment14955 No year, please. src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java https://reviews.apache.org/r/4633/#comment14956 This doesn't sound consistent with the class being a monitor. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java https://reviews.apache.org/r/4633/#comment14957 No year. src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java https://reviews.apache.org/r/4633/#comment14959 Year is not needed. src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java https://reviews.apache.org/r/4633/#comment14960 License. - Ted On 2012-04-07 19:51:11, Jesse Yates wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4633/ bq. --- bq. bq. (Updated 2012-04-07 19:51:11) bq. bq. bq. Review request for hbase, Michael Stack and Lars Hofhansl. bq. bq. bq. Summary bq. --- bq. bq. Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap. bq. bq. Implementation wise: bq. Updated the HMasterInterface to pass the calls onto the zookeeper. bq. Added a zk listener to handle updates from the master to the RS to backup. bq. Added a utility for removing files and finding archive directories bq. Added tests for the regionserver and catalogjanitor approaches. bq.
[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249333#comment-13249333 ] jirapos...@reviews.apache.org commented on HBASE-5547: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/#review6778 --- This is pretty nice! Did a first pass. The only part I a bit hard to comprehend is the archiveDir vs. the timestamped archiveDir. Somebody with more knowledge about ZK should look through he ZK code. For example do we need to call sync() in some cases to ensure we get the latest ZK view? src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java https://reviews.apache.org/r/4633/#comment14951 Is this a warning? src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java https://reviews.apache.org/r/4633/#comment14953 How can that be the case? Can we get a new file with the same name as a previously deleted one? This might make it hard to piece together the set of files required for backup. src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java https://reviews.apache.org/r/4633/#comment14958 License missing - Lars On 2012-04-07 19:51:11, Jesse Yates wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4633/ bq. --- bq. bq. (Updated 2012-04-07 19:51:11) bq. bq. bq. Review request for hbase, Michael Stack and Lars Hofhansl. bq. bq. bq. Summary bq. --- bq. bq. Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap. bq. bq. Implementation wise: bq. Updated the HMasterInterface to pass the calls onto the zookeeper. bq. Added a zk listener to handle updates from the master to the RS to backup. bq. Added a utility for removing files and finding archive directories bq. Added tests for the regionserver and catalogjanitor approaches. bq. Added creation of manager in regionserver. bq. bq. bq. This addresses bug HBASE-5547. bq. https://issues.apache.org/jira/browse/HBASE-5547 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/HConstants.java a4b989e bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 bq.src/main/java/org/apache/hadoop/hbase/client/HFileArchiveManager.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java fb21bdd bq. src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveMonitor.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java c3df319 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d bq. src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53 bq.src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 509a467 bq.src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/zookeeper/RegionServerHFileTracker.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 4fc105f bq.src/main/resources/hbase-default.xml 44ee689 bq.src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java 41616c8 bq.src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java b4dcb83 bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/util/HFileArchiveTestingUtil.java PRE-CREATION bq.src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java 7d02759 bq. bq. Diff: https://reviews.apache.org/r/4633/diff bq. bq. bq. Testing bq. --- bq. bq. Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper. bq. bq. bq. Thanks, bq. bq. Jesse bq. bq. Don't delete HFiles when in backup
[jira] [Commented] (HBASE-5719) Enhance hbck to sideline overlapped mega regions
[ https://issues.apache.org/jira/browse/HBASE-5719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13248574#comment-13248574 ] jirapos...@reviews.apache.org commented on HBASE-5719: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4649/ --- (Updated 2012-04-06 18:27:34.615205) Review request for hbase and jmhsieh. Changes --- Addressed Jon's comments. Summary --- Make it configurable to sideline some regions in big overlapped groups which hbck doesn't handle currently. The regions chose to sideline are those which overlap with most other regions. This addresses bug HBASE-5719. https://issues.apache.org/jira/browse/HBASE-5719 Diffs (updated) - src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 54f9b21 src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java 17678dd src/test/java/org/apache/hadoop/hbase/util/TestRegionSplitCalculator.java ac3b225 Diff: https://reviews.apache.org/r/4649/diff Testing --- mvn -PlocalTests -Dtest=TestHBaseFsck* clean test Also tested in real system to fix inconsistencies. Thanks, Jimmy Enhance hbck to sideline overlapped mega regions Key: HBASE-5719 URL: https://issues.apache.org/jira/browse/HBASE-5719 Project: HBase Issue Type: New Feature Components: hbck Affects Versions: 0.94.0, 0.96.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: hbase-5719.patch If there are too many regions in one overlapped group (by default, more than 10), hbck currently doesn't merge them since it takes time. In this case, we can sideline some regions in the group and break the overlapping to fix the inconsistency. Later on, sidelined regions can be bulk loaded manually. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5707) Move clusterid and clusterup (shutdown) znodes over to pb
[
https://issues.apache.org/jira/browse/HBASE-5707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13248625#comment-13248625
]
jirapos...@reviews.apache.org commented on HBASE-5707:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4627/#review6750
---
Looks good to me. I prefer to call it startDate instead of startdate. It is up
to you.
- Jimmy
On 2012-04-03 21:04:54, Michael Stack wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4627/
bq. ---
bq.
bq. (Updated 2012-04-03 21:04:54)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Move two more znodes to serialize their content as pb.
bq.
bq. Here are the changes:
bq.
bq. {code}
bq. M src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
bq.Get the clusterid via utility in the ClusterId class rather than
bq.get it directly.
bq. M src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
bq. M src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterStatusTracker.java
bq.Serialize and deserialize using new ClusterId pb message.
bq.Include the PBUF preamble on znode content.
bq. M src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java
bq.Rename getRootRegionZNodeContent to getZNodeData so it aligns in name
bq.with the other cases where we compose the znode content up in ClusterId
bq.and in ClusterStatusTracker.
bq. M src/main/protobuf/ZooKeeper.proto
bq.Add new messages for ClusterId and ClusterUp
bq. {code}
bq.
bq.
bq. This addresses bug hbase-5707.
bq. https://issues.apache.org/jira/browse/hbase-5707
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
aa30969
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java
8ff87fe
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java 3fa83e6
bq.
src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterStatusTracker.java
61e7367
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java
6b2ea57
bq.src/main/protobuf/ZooKeeper.proto 20f8eb0
bq.
bq. Diff: https://reviews.apache.org/r/4627/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq.
bq. Thanks,
bq.
bq. Michael
bq.
bq.
Move clusterid and clusterup (shutdown) znodes over to pb
-
Key: HBASE-5707
URL: https://issues.apache.org/jira/browse/HBASE-5707
Project: HBase
Issue Type: Task
Reporter: stack
Attachments: 5707.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5620) Convert the client protocol of HRegionInterface to PB
[ https://issues.apache.org/jira/browse/HBASE-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13248772#comment-13248772 ] jirapos...@reviews.apache.org commented on HBASE-5620: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4629/#review6751 --- Good stuff Jimmy. A few comments below. Sorry I ramble at times. Read to the end before reacting because I change my mind as I work through the patch. You forgot to add RequestConverter? src/main/java/org/apache/hadoop/hbase/HConstants.java https://reviews.apache.org/r/4629/#comment14787 Should these be in here? Can they be elsewhere, as defines in RegionClassProtocol? src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java https://reviews.apache.org/r/4629/#comment14788 I don't know how you get away with these removes but I'm glad to see them go -- anything that shrinks this catalog package is good by me src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4629/#comment14789 RegionClientProtocol is a bad name, no? Should it be just ClientProtocol and AdminProtocol? src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4629/#comment14790 this should be getClient? We're going to get a client for a table? Maybe should be getTableClient? src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4629/#comment14791 Why we need a converter? src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4629/#comment14792 Whats the NULL_CONTROLLER about? src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java https://reviews.apache.org/r/4629/#comment14793 So we don't close scanners anymore? src/main/java/org/apache/hadoop/hbase/client/HConnection.java https://reviews.apache.org/r/4629/#comment14794 Why do we need to pollute HConnection w/ higher level notions such as RegionClientProtocol (or ClientProtocol?) hmmm ClientProtocol would be ok but not RegionClientProtocol... src/main/java/org/apache/hadoop/hbase/client/HConnection.java https://reviews.apache.org/r/4629/#comment14795 I wonder if this needs to be public and in the Interface? Could it be an internal thing? How uses this thing that comes out? Maybe it needs to be here but sure seems like an internal thing: i.e. we pass hostname and port of a particular regionserver but then internally as we go about the cluster the client will go to new regionservers and set up connections Looking around though, it seems that while most uses of this method should be shut down: e.g. over in catalog package and replaced by calls that go via HTable, there are a few places we need this still probably .. as in the master sendRegionClose, etc. calls. So you can't get rid of it. But this looks like it should be called RegionServerProtocol or ClientProtocol and not RegionClientProtocol. It connects to a RegionServer not a Region. Or what you think Jimmy?When I look at the RegionClient proto file, is it true to say that in each message, we MUST pass a region since we are going against a particular region... which would mean this is indeed a RegionClientProtocol (you just have to specify coordinates in two steps; 1. pass servername setting up connection, then 2. for any call on the connection, need to specify the region.) src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java https://reviews.apache.org/r/4629/#comment14796 Move these defines into this class rather than up in HConstants? src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java https://reviews.apache.org/r/4629/#comment14797 Whats happening here? We want to drop the methods that took hostname and port only and move to those that take servername? src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java https://reviews.apache.org/r/4629/#comment14798 Is this good? Should we have servername? What if server restarts in between? Would we want to notice that we don't have a connection to new instantiation and go create one? src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java https://reviews.apache.org/r/4629/#comment14799 Does this need to be synchronized at all? src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java https://reviews.apache.org/r/4629/#comment14800 I suppose HBaseClient down in rpc knows it because it has to deserialize the pb? src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java https://reviews.apache.org/r/4629/#comment14801 scannerid is internal or
[jira] [Commented] (HBASE-5717) Scanner metrics are only reported if you get to the end of a scanner
[ https://issues.apache.org/jira/browse/HBASE-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13248883#comment-13248883 ] jirapos...@reviews.apache.org commented on HBASE-5717: -- bq. On 2012-04-04 22:03:32, Ian Varley wrote: bq. Good catch. There is another scenario that could be useful when you have long running scan operation. While a scan is running, e.g. before it finishes or closed, if somehow the ScanMetrics can be updated for time to time without perf impact, that can provide more real-time number for mapreduce HBase Ccounters which uses ScanMetrics. But that could be a separate nice-to-have improvement. - Ming --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4640/#review6697 --- On 2012-04-05 18:52:50, Ian Varley wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4640/ bq. --- bq. bq. (Updated 2012-04-05 18:52:50) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Fix for persistence of scan metrics when the scanner doesn't run to exhaustion. bq. bq. bq. This addresses bug HBASE-5717. bq. https://issues.apache.org/jira/browse/HBASE-5717 bq. bq. bq. Diffs bq. - bq. bq./src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1309585 bq./src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java 1309585 bq. bq. Diff: https://reviews.apache.org/r/4640/diff bq. bq. bq. Testing bq. --- bq. bq. Altered the scan metrics unit test to show this problem (now fails without changes to ClientScanner.java). bq. bq. bq. Thanks, bq. bq. Ian bq. bq. Scanner metrics are only reported if you get to the end of a scanner Key: HBASE-5717 URL: https://issues.apache.org/jira/browse/HBASE-5717 Project: HBase Issue Type: Bug Components: client, metrics Reporter: Ian Varley Priority: Minor Attachments: ClientScanner_HBASE_5717-v2.patch, ClientScanner_HBASE_5717-v3.patch, ClientScanner_HBASE_5717.patch Original Estimate: 4h Remaining Estimate: 4h When you turn on Scanner Metrics, the metrics are currently only made available if you run over all records available in the scanner. If you stop iterating before the end, the values are never flushed into the metrics object (in the Scan attribute). Will supply a patch with fix and test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5620) Convert the client protocol of HRegionInterface to PB
[ https://issues.apache.org/jira/browse/HBASE-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249006#comment-13249006 ] jirapos...@reviews.apache.org commented on HBASE-5620: -- bq. On 2012-04-06 21:02:20, Michael Stack wrote: bq. Good stuff Jimmy. A few comments below. Sorry I ramble at times. Read to the end before reacting because I change my mind as I work through the patch. You forgot to add RequestConverter? Thanks for reviewing. RequestConverter is the 21st file. There are more files on page 2. Please review them too. Thanks. bq. On 2012-04-06 21:02:20, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java, line 766 bq. https://reviews.apache.org/r/4629/diff/1/?file=98832#file98832line766 bq. bq. white space Will remove. bq. On 2012-04-06 21:02:20, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/protobuf/RegionAdminProtocol.java, line 37 bq. https://reviews.apache.org/r/4629/diff/1/?file=98833#file98833line37 bq. bq. Is this just regionadmin protocol? Or is it general admin? Should I change it to AdminProtocol? There is no other admin. bq. On 2012-04-06 21:02:20, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java, line 510 bq. https://reviews.apache.org/r/4629/diff/1/?file=98832#file98832line510 bq. bq. Its as though we should give clients a way to pass in the raw pbs rather than have them go via our versions of Get, Delete, etc. I agree. We can add some new methods later. For now, I'd like to make the existing functions work at first. bq. On 2012-04-06 21:02:20, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java, line 160 bq. https://reviews.apache.org/r/4629/diff/1/?file=98832#file98832line160 bq. bq. Ugh. KeyValue needs to be an Interface that both pb and our current KV implements. bq. bq. This will be a performance killer? Should Result let out the raw pb KV? I was thinking to has some new methods which use raw pbs, which old clients should migrate to. Let me change the current KeyValue class to an interface. bq. On 2012-04-06 21:02:20, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java, line 149 bq. https://reviews.apache.org/r/4629/diff/1/?file=98832#file98832line149 bq. bq. Why not throw ServiceException? I like to throw ServiceException too. Let me give it a try. If the existing interface is not affected, I will do it. bq. On 2012-04-06 21:02:20, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java, line 418 bq. https://reviews.apache.org/r/4629/diff/1/?file=98830#file98830line418 bq. bq. Is ServiceException a pb exception? It is from pb. But it is not a pb message as I know. bq. On 2012-04-06 21:02:20, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/ipc/RegionProtocols.java, line 36 bq. https://reviews.apache.org/r/4629/diff/1/?file=98829#file98829line36 bq. bq. What is this for? Region server could implement multiple protocols such as RegionClientProtocol and RegionAdminProtocol. bq. On 2012-04-06 21:02:20, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/ipc/RegionProtocols.java, line 27 bq. https://reviews.apache.org/r/4629/diff/1/?file=98829#file98829line27 bq. bq. Fix Fixed. bq. On 2012-04-06 21:02:20, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java, line 363 bq. https://reviews.apache.org/r/4629/diff/1/?file=98825#file98825line363 bq. bq. Do you have to? This stuff is nasty enough w/o lettting it out of this package. It is to support Filters and comparators which are still Writable. We can change it back once Filters and comparators are moved to pb too. bq. On 2012-04-06 21:02:20, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java, line 230 bq. https://reviews.apache.org/r/4629/diff/1/?file=98822#file98822line230 bq. bq. scannerid is internal or something now? I think it is still public as it is defined in the pb messages. For ScannerCallable, it is internal though. bq. On 2012-04-06 21:02:20, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java, line 192 bq. https://reviews.apache.org/r/4629/diff/1/?file=98822#file98822line192 bq. bq. I suppose HBaseClient down in rpc knows it because it has to deserialize the pb? You are right. We need to come up a way to pop this info up. bq. On 2012-04-06 21:02:20, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java, line 1441 bq. https://reviews.apache.org/r/4629/diff/1/?file=98820#file98820line1441 bq. bq.
[jira] [Commented] (HBASE-5620) Convert the client protocol of HRegionInterface to PB
[ https://issues.apache.org/jira/browse/HBASE-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249054#comment-13249054 ] jirapos...@reviews.apache.org commented on HBASE-5620: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4629/#review6759 --- src/main/java/org/apache/hadoop/hbase/ipc/RegionProtocols.java https://reviews.apache.org/r/4629/#comment14879 Where is RegionProtocols used? We put the two protocols together? Do we need to? - Michael On 2012-04-03 23:32:10, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4629/ bq. --- bq. bq. (Updated 2012-04-03 23:32:10) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This is the client protocol part of region interface. The admin protocol part will be done in a different jira. bq. bq. The HRegionInterface is still there since the admin part is not done yet. The other reason is that in case some people still wants the old interface bq. bq. Filters, comparators and coprocessor parameters are still Writable. They will be addressed in different jiras. bq. bq. The existing client interface is not changed so that we don't break existing clients. bq. bq. bq. This addresses bug HBASE-5620. bq. https://issues.apache.org/jira/browse/HBASE-5620 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/HConstants.java 21ac4ba bq.src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java 0129ee9 bq.src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java 97293aa bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 bq.src/main/java/org/apache/hadoop/hbase/client/HConnection.java 5d43086 bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa30969 bq.src/main/java/org/apache/hadoop/hbase/client/HTable.java 3db0295 bq.src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 9903df3 bq.src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java ddcf9ad bq.src/main/java/org/apache/hadoop/hbase/filter/ParseConstants.java 1acbdab bq.src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java cbfa489 bq.src/main/java/org/apache/hadoop/hbase/io/TimeRange.java d135393 bq.src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java 05ae717 bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java f1f06b0 bq.src/main/java/org/apache/hadoop/hbase/ipc/RegionProtocols.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 1316457 bq. src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 19ae18c bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 2eb57de bq.src/main/java/org/apache/hadoop/hbase/protobuf/RegionAdminProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/protobuf/RegionClientProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 4026da0 bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionClientProtos.java b36a9c0 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 4f80999 bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java 703e73d bq.src/main/java/org/apache/hadoop/hbase/regionserver/Leases.java b520f3f bq.src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java PRE-CREATION bq.src/main/protobuf/RegionClient.proto 358382b bq.src/main/protobuf/hbase.proto da78788 bq.src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java c7284dc bq. src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditorNoCluster.java 0042468 bq. src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java e34d8bc bq.src/test/java/org/apache/hadoop/hbase/io/TestHbaseObjectWritable.java f2f8ee3 bq.src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java d2b3060 bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 91dce36 bq.src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 227c5f2 bq.src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java ceca6f5 bq.
[jira] [Commented] (HBASE-5620) Convert the client protocol of HRegionInterface to PB
[ https://issues.apache.org/jira/browse/HBASE-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249075#comment-13249075 ] jirapos...@reviews.apache.org commented on HBASE-5620: -- bq. On 2012-04-06 23:41:33, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/ipc/RegionProtocols.java, line 36 bq. https://reviews.apache.org/r/4629/diff/1/?file=98829#file98829line36 bq. bq. Where is RegionProtocols used? We put the two protocols together? Do we need to? Ok, let me remove it. - Jimmy --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4629/#review6759 --- On 2012-04-03 23:32:10, Jimmy Xiang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4629/ bq. --- bq. bq. (Updated 2012-04-03 23:32:10) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This is the client protocol part of region interface. The admin protocol part will be done in a different jira. bq. bq. The HRegionInterface is still there since the admin part is not done yet. The other reason is that in case some people still wants the old interface bq. bq. Filters, comparators and coprocessor parameters are still Writable. They will be addressed in different jiras. bq. bq. The existing client interface is not changed so that we don't break existing clients. bq. bq. bq. This addresses bug HBASE-5620. bq. https://issues.apache.org/jira/browse/HBASE-5620 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/HConstants.java 21ac4ba bq.src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java 0129ee9 bq.src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java 97293aa bq.src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 bq.src/main/java/org/apache/hadoop/hbase/client/HConnection.java 5d43086 bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa30969 bq.src/main/java/org/apache/hadoop/hbase/client/HTable.java 3db0295 bq.src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 9903df3 bq.src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java ddcf9ad bq.src/main/java/org/apache/hadoop/hbase/filter/ParseConstants.java 1acbdab bq.src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java cbfa489 bq.src/main/java/org/apache/hadoop/hbase/io/TimeRange.java d135393 bq.src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java 05ae717 bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java f1f06b0 bq.src/main/java/org/apache/hadoop/hbase/ipc/RegionProtocols.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 1316457 bq. src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 19ae18c bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 2eb57de bq.src/main/java/org/apache/hadoop/hbase/protobuf/RegionAdminProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/protobuf/RegionClientProtocol.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java PRE-CREATION bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 4026da0 bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionClientProtos.java b36a9c0 bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 4f80999 bq. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java 703e73d bq.src/main/java/org/apache/hadoop/hbase/regionserver/Leases.java b520f3f bq.src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java PRE-CREATION bq.src/main/protobuf/RegionClient.proto 358382b bq.src/main/protobuf/hbase.proto da78788 bq.src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java c7284dc bq. src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditorNoCluster.java 0042468 bq. src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java e34d8bc bq.src/test/java/org/apache/hadoop/hbase/io/TestHbaseObjectWritable.java f2f8ee3 bq.src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java d2b3060 bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 91dce36 bq.src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 227c5f2 bq.
[jira] [Commented] (HBASE-5601) Add per-column-family data block cache hit ratios
[ https://issues.apache.org/jira/browse/HBASE-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249147#comment-13249147 ] jirapos...@reviews.apache.org commented on HBASE-5601: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4675/ --- Review request for hbase. Summary --- HBASE-5601. Add per-column-family data block cache hit ratios. First shot at the patch. Changes are as follows (some are backwards incompatible): - Renamed hbase.RegionServerDynamicStatistics.XXX - hbase.regionserver.dyn.XXX - Renamed the metrics name generation from hbase.RegionServerDynamicStatistics.tbl.usertable.cf.ycsb.bt.Data.fsBlockReadCacheHitCnt - tbl=usertable.cf=ycsb.bt=Data.at=scan.blockCacheHitCount Notice that tbl.TableName is changed to tbl=TableName, cf.Family is changed to cf=Family, the fs or compaction prefix is changed to at=scan or at=compaction. at stands for access type. - Through using CacheAccessStats/CacheSizeStats, metric names across per-cf and global are made consistent. - For compaction, per-block type metrics are disabled. For compaction we do not need to distinguish between block types. - Added a configuration hbase.metrics.dynamic.enabled for disabling dynamic metrics. It is enabled by default. - Changed bloom filter metrics from keyMaybeInBloomCnt - bloomHitCount - Removed some code for keeping BC related to META block types Some changes in HFileReaders are due to automatic trailing whitespace settings in my eclipse. I can regenerate those parts if it is hard to review. This addresses bug HBASE-5601. https://issues.apache.org/jira/browse/HBASE-5601 Diffs - src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java e3c4fe4 src/main/java/org/apache/hadoop/hbase/io/hfile/CacheAccessMetrics.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheAccessStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheAccessStatsImpl.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2ba3ab2 src/main/java/org/apache/hadoop/hbase/io/hfile/CacheSizeMetrics.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheSizeStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheSizeStatsImpl.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java ef03075 src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStatsImpl.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java e952ef9 src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 79ffee0 src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java aa99add src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 91562f7 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 87634ba src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 1795229 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java 251ccd4 src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java f1a68e0 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicMetrics.java 3b46dd8 src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java da08532 src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java 5ad7406 src/main/resources/hbase-default.xml 44ee689 src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java f7430ee src/test/java/org/apache/hadoop/hbase/io/hfile/TestForceCacheImportantBlocks.java 5f8214e src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java d980948 src/test/java/org/apache/hadoop/hbase/io/hfile/TestScannerSelectionUsingTTL.java 17284a3 src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java bd83a71 src/test/java/org/apache/hadoop/hbase/regionserver/metrics/TestSchemaMetrics.java dbe425f Diff: https://reviews.apache.org/r/4675/diff Testing --- Tested with existing unit tests, and manually using FileContext Thanks, enis Add per-column-family data block cache hit ratios - Key: HBASE-5601 URL: https://issues.apache.org/jira/browse/HBASE-5601 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Enis Soztutar In addition to the overall block cache hit ratio it would be extremely useful to have
[jira] [Commented] (HBASE-5601) Add per-column-family data block cache hit ratios
[ https://issues.apache.org/jira/browse/HBASE-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13249148#comment-13249148 ] jirapos...@reviews.apache.org commented on HBASE-5601: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4675/ --- (Updated 2012-04-07 03:40:41.839514) Review request for hbase. Changes --- Fixed a failing test case. Summary --- HBASE-5601. Add per-column-family data block cache hit ratios. First shot at the patch. Changes are as follows (some are backwards incompatible): - Renamed hbase.RegionServerDynamicStatistics.XXX - hbase.regionserver.dyn.XXX - Renamed the metrics name generation from hbase.RegionServerDynamicStatistics.tbl.usertable.cf.ycsb.bt.Data.fsBlockReadCacheHitCnt - tbl=usertable.cf=ycsb.bt=Data.at=scan.blockCacheHitCount Notice that tbl.TableName is changed to tbl=TableName, cf.Family is changed to cf=Family, the fs or compaction prefix is changed to at=scan or at=compaction. at stands for access type. - Through using CacheAccessStats/CacheSizeStats, metric names across per-cf and global are made consistent. - For compaction, per-block type metrics are disabled. For compaction we do not need to distinguish between block types. - Added a configuration hbase.metrics.dynamic.enabled for disabling dynamic metrics. It is enabled by default. - Changed bloom filter metrics from keyMaybeInBloomCnt - bloomHitCount - Removed some code for keeping BC related to META block types Some changes in HFileReaders are due to automatic trailing whitespace settings in my eclipse. I can regenerate those parts if it is hard to review. This addresses bug HBASE-5601. https://issues.apache.org/jira/browse/HBASE-5601 Diffs (updated) - src/main/java/org/apache/hadoop/hbase/io/hfile/BlockCache.java 2ba3ab2 src/main/java/org/apache/hadoop/hbase/io/hfile/BlockType.java e3c4fe4 src/main/java/org/apache/hadoop/hbase/io/hfile/CacheAccessMetrics.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheAccessStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheAccessStatsImpl.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheSizeMetrics.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheSizeStats.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheSizeStatsImpl.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStats.java ef03075 src/main/java/org/apache/hadoop/hbase/io/hfile/CacheStatsImpl.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/io/hfile/DoubleBlockCache.java e952ef9 src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV1.java 79ffee0 src/main/java/org/apache/hadoop/hbase/io/hfile/HFileReaderV2.java aa99add src/main/java/org/apache/hadoop/hbase/io/hfile/LruBlockCache.java 91562f7 src/main/java/org/apache/hadoop/hbase/io/hfile/SimpleBlockCache.java 87634ba src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SingleSizeCache.java 1795229 src/main/java/org/apache/hadoop/hbase/io/hfile/slab/SlabCache.java 251ccd4 src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java f1a68e0 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 8a61f7d src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerDynamicMetrics.java 3b46dd8 src/main/java/org/apache/hadoop/hbase/regionserver/metrics/RegionServerMetrics.java da08532 src/main/java/org/apache/hadoop/hbase/regionserver/metrics/SchemaMetrics.java 5ad7406 src/main/resources/hbase-default.xml 44ee689 src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java f7430ee src/test/java/org/apache/hadoop/hbase/io/hfile/TestForceCacheImportantBlocks.java 5f8214e src/test/java/org/apache/hadoop/hbase/io/hfile/TestLruBlockCache.java d980948 src/test/java/org/apache/hadoop/hbase/io/hfile/TestScannerSelectionUsingTTL.java 17284a3 src/test/java/org/apache/hadoop/hbase/regionserver/TestBlocksRead.java bd83a71 src/test/java/org/apache/hadoop/hbase/regionserver/metrics/TestSchemaMetrics.java dbe425f Diff: https://reviews.apache.org/r/4675/diff Testing --- Tested with existing unit tests, and manually using FileContext Thanks, enis Add per-column-family data block cache hit ratios - Key: HBASE-5601 URL: https://issues.apache.org/jira/browse/HBASE-5601 Project: HBase Issue Type: Improvement Reporter: Mikhail Bautin Assignee: Enis Soztutar
[jira] [Commented] (HBASE-5717) Scanner metrics are only reported if you get to the end of a scanner
[ https://issues.apache.org/jira/browse/HBASE-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247457#comment-13247457 ] jirapos...@reviews.apache.org commented on HBASE-5717: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4640/ --- (Updated 2012-04-05 18:15:41.419224) Review request for hbase. Changes --- Added finally block (with try/catch inside). I'd prefer to also add logging rather than swallowing the exceptions, but that seems like it should be a different Jira (and maybe should cover all cases that swallow exceptions). Also: sorry for a) sending a review on the board for something that's probably too small, and b) writing comments on my own review (intended to annotate, didn't realize it would appear as if I were a separate reviewer). Summary --- Fix for persistence of scan metrics when the scanner doesn't run to exhaustion. This addresses bug HBASE-5717. https://issues.apache.org/jira/browse/HBASE-5717 Diffs (updated) - /src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java 1309585 /src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1309585 Diff: https://reviews.apache.org/r/4640/diff Testing --- Altered the scan metrics unit test to show this problem (now fails without changes to ClientScanner.java). Thanks, Ian Scanner metrics are only reported if you get to the end of a scanner Key: HBASE-5717 URL: https://issues.apache.org/jira/browse/HBASE-5717 Project: HBase Issue Type: Bug Components: client, metrics Reporter: Ian Varley Priority: Minor Attachments: ClientScanner_HBASE_5717-v2.patch, ClientScanner_HBASE_5717.patch Original Estimate: 4h Remaining Estimate: 4h When you turn on Scanner Metrics, the metrics are currently only made available if you run over all records available in the scanner. If you stop iterating before the end, the values are never flushed into the metrics object (in the Scan attribute). Will supply a patch with fix and test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5717) Scanner metrics are only reported if you get to the end of a scanner
[ https://issues.apache.org/jira/browse/HBASE-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247466#comment-13247466 ] jirapos...@reviews.apache.org commented on HBASE-5717: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4640/#review6715 --- Ship it! The only comment I have is that there is now no test that verifies that metrics are collected when we scanned all the way to the end, but did not close the scanner. - Lars On 2012-04-05 18:15:41, Ian Varley wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4640/ bq. --- bq. bq. (Updated 2012-04-05 18:15:41) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Fix for persistence of scan metrics when the scanner doesn't run to exhaustion. bq. bq. bq. This addresses bug HBASE-5717. bq. https://issues.apache.org/jira/browse/HBASE-5717 bq. bq. bq. Diffs bq. - bq. bq./src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java 1309585 bq./src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1309585 bq. bq. Diff: https://reviews.apache.org/r/4640/diff bq. bq. bq. Testing bq. --- bq. bq. Altered the scan metrics unit test to show this problem (now fails without changes to ClientScanner.java). bq. bq. bq. Thanks, bq. bq. Ian bq. bq. Scanner metrics are only reported if you get to the end of a scanner Key: HBASE-5717 URL: https://issues.apache.org/jira/browse/HBASE-5717 Project: HBase Issue Type: Bug Components: client, metrics Reporter: Ian Varley Priority: Minor Attachments: ClientScanner_HBASE_5717-v2.patch, ClientScanner_HBASE_5717.patch Original Estimate: 4h Remaining Estimate: 4h When you turn on Scanner Metrics, the metrics are currently only made available if you run over all records available in the scanner. If you stop iterating before the end, the values are never flushed into the metrics object (in the Scan attribute). Will supply a patch with fix and test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247503#comment-13247503 ] jirapos...@reviews.apache.org commented on HBASE-5547: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/#review6714 --- Thanks for the review Ted. Any thoughts on combining the Tracker and Manager (see inline comment below on HFileArchiveManager). src/main/java/org/apache/hadoop/hbase/HConstants.java https://reviews.apache.org/r/4633/#comment14522 fair enough. done. src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java https://reviews.apache.org/r/4633/#comment14517 Disable right now just wipes out all the tables currently in backup mode (not saving any info), but I guess it could be reasonable to want to start backing up all the tables - though in real usage you probably only want to do one table at a time to minimize impact on a cluster. I'd rather not give people that power directly, but make them jump through a couple hoops (listing out all the tables and then iteratively starting backups on those tables) due: 1) how much data that can be, and 2) in a real 'backup' context this means they are cloning all the tables (doing a full table scan on all tables) which would be highly disruptive. src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java https://reviews.apache.org/r/4633/#comment14518 done. src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java https://reviews.apache.org/r/4633/#comment14519 done src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java https://reviews.apache.org/r/4633/#comment14520 I've been debating whether or not to roll this implementation into the HFileArchiveTracker and just move this into an interface. It doesn't actually do very much - essentially a prettier front to a map and could easily be moved into the tracker. Thoughts? src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java https://reviews.apache.org/r/4633/#comment14521 done. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java https://reviews.apache.org/r/4633/#comment14526 done. src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java https://reviews.apache.org/r/4633/#comment14523 That case should only come up in testing, so logging seems excessive. src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java https://reviews.apache.org/r/4633/#comment14524 used store because in code that's what we are really dealing with, but everyone just refers to them as hfiles. going with your way for consistency. src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java https://reviews.apache.org/r/4633/#comment14525 done. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java https://reviews.apache.org/r/4633/#comment14527 done. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java https://reviews.apache.org/r/4633/#comment14528 yup, nice catch (no pun intended). src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java https://reviews.apache.org/r/4633/#comment14529 true. earlier iteration comment that got confused with manager.keepFiles src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java https://reviews.apache.org/r/4633/#comment14530 how so? renames an the src path to the dest path. Seems to work in tests. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java https://reviews.apache.org/r/4633/#comment14531 then its something we don't care about - only care about the hfiles, which are stored in one directory level below the region. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java https://reviews.apache.org/r/4633/#comment14532 done. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java https://reviews.apache.org/r/4633/#comment14534 Definitely need to add another unit test covering the 'resolve' part of this method - lot of gymnastics/corner cases going on. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java https://reviews.apache.org/r/4633/#comment14536 reworked a little bit in the new version. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java https://reviews.apache.org/r/4633/#comment14533 going to bail out earlier if we can't delete the old file. src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java https://reviews.apache.org/r/4633/#comment14535 done. src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java https://reviews.apache.org/r/4633/#comment14537 done.
[jira] [Commented] (HBASE-5719) Enhance hbck to sideline overlapped mega regions
[
https://issues.apache.org/jira/browse/HBASE-5719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247976#comment-13247976
]
jirapos...@reviews.apache.org commented on HBASE-5719:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4649/#review6713
---
Nice feature. Can we call it something other than breakXxx? Maybe
-sidelineLargeOverlap
src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4649/#comment14511
why commented?
src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4649/#comment14512
why commented?
src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4649/#comment14513
break? maybe a different word?
src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4649/#comment14514
instead of nesting, maybe set hfiles to an empty array?
if (hfiles==null) {
hfiles = new FileStatus[0];
}
for (FileStatus hfile : hfiles)
...
src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4649/#comment14565
Instead of None/Multiple just say how many htds?
src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4649/#comment14516
We lost info here.
src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4649/#comment14608
Pretty important to note that we only break groups that exceed the max
number of regions to merge!
src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4649/#comment14569
Comment about what you are trying to accomplish with this method?
src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4649/#comment14607
Is this really a failure?
src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4649/#comment14570
I think we should bail out in this case.
src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4649/#comment14571
LOG.debug/trace?
src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4649/#comment14572
LOG.debug/trace?
src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4649/#comment14562
wrong comment for this method
src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
https://reviews.apache.org/r/4649/#comment14563
Please update usage with new options.
src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java
https://reviews.apache.org/r/4649/#comment14568
Why this change?
src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java
https://reviews.apache.org/r/4649/#comment14567
This is a really important function.
Please java doc (maybe an example?) and at least add a unit tests that
cover edge cases here for this.
What is the definition of a megaOverlap?
What are the properties of the argument and the return values?
What is the intuition behind how the regions selected are selected and
returned?
- jmhsieh
On 2012-04-04 23:52:50, Jimmy Xiang wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4649/
bq. ---
bq.
bq. (Updated 2012-04-04 23:52:50)
bq.
bq.
bq. Review request for hbase and jmhsieh.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Make it configurable to sideline some regions in big overlapped groups
which hbck doesn't handle currently.
bq.
bq. The regions chose to sideline are those which overlap with most other
regions.
bq.
bq.
bq. This addresses bug HBASE-5719.
bq. https://issues.apache.org/jira/browse/HBASE-5719
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 54f9b21
bq.src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java
17678dd
bq.
bq. Diff: https://reviews.apache.org/r/4649/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. mvn -PlocalTests -Dtest=TestHBaseFsck* clean test
bq.
bq. Also tested in real system to fix inconsistencies.
bq.
bq.
bq. Thanks,
bq.
bq. Jimmy
bq.
bq.
Enhance hbck to sideline overlapped mega regions
Key: HBASE-5719
URL: https://issues.apache.org/jira/browse/HBASE-5719
Project: HBase
Issue Type: New Feature
[jira] [Commented] (HBASE-5548) Add ability to get a table in the shell
[ https://issues.apache.org/jira/browse/HBASE-5548?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13248070#comment-13248070 ] jirapos...@reviews.apache.org commented on HBASE-5548: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4668/ --- Review request for hbase and Michael Stack. Summary --- Adding the ability to get a reference to a table in the shell. Previously, all commands that acted on a table would need to take the name of the table as a string, which is annoying in an OO REPL. This patch introduces the ability to get and hold a reference to a table both on creation (via create(...)) and at will (via get_table(...)). Further, to actually make the table useful, modifications to table specific class were made so you can have a reference and just do things like put, scan, get, etc. on that table reference. To accommodate new table functionality, table specific methods are easily added (one line) in a dynamic fashion via class methods in the Table. See examples in get, put, scan, etc.. There is also a lot of admin functionality tied to a table - things like disabling, dropping, describing, etc - that were added to the table class. Now you can do things like 'table.disable' and 'table.describe'. Again these were dynamically added, so new admin functionality for a table is as simple as adding the method name to one line in the Table class. This addresses bug HBASE-5548. https://issues.apache.org/jira/browse/HBASE-5548 Diffs - src/main/ruby/hbase/hbase.rb 2c37840 src/main/ruby/hbase/table.rb 41dcf21 src/main/ruby/shell.rb 53f3de8 src/main/ruby/shell/commands.rb af6df33 src/main/ruby/shell/commands/count.rb 6596441 src/main/ruby/shell/commands/create.rb 14c1b0f src/main/ruby/shell/commands/delete.rb 12bc405 src/main/ruby/shell/commands/deleteall.rb 5731b60 src/main/ruby/shell/commands/get.rb 754c3d6 src/main/ruby/shell/commands/get_counter.rb 3cbe226 src/main/ruby/shell/commands/get_table.rb PRE-CREATION src/main/ruby/shell/commands/incr.rb 38a2fc5 src/main/ruby/shell/commands/put.rb dde0433 src/main/ruby/shell/commands/scan.rb e58aaac src/main/ruby/shell/commands/table_help.rb PRE-CREATION src/test/ruby/hbase/admin_test.rb 0c2672b Diff: https://reviews.apache.org/r/4668/diff Testing --- Manual and basic unit tests. Thanks, Jesse Add ability to get a table in the shell --- Key: HBASE-5548 URL: https://issues.apache.org/jira/browse/HBASE-5548 Project: HBase Issue Type: Improvement Components: shell Reporter: Jesse Yates Assignee: Jesse Yates Fix For: 0.96.0, 0.94.1 Attachments: ruby_HBASE-5528-v0.patch, ruby_HBASE-5548-v1.patch, ruby_HBASE-5548-v2.patch, ruby_HBASE-5548-v3.patch Currently, all the commands that operate on a table in the shell first have to take the table as name as input. There are two main considerations: * It is annoying to have to write the table name every time, when you should just be able to get a reference to a table * the current implementation is very wasteful - it creates a new HTable for each call (but reuses the connection since it uses the same configuration) We should be able to get a handle to a single HTable and then operate on that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5709) Move active master and backup master znodes to use pbs
[ https://issues.apache.org/jira/browse/HBASE-5709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246358#comment-13246358 ] jirapos...@reviews.apache.org commented on HBASE-5709: -- bq. On 2012-04-04 00:21:57, Jimmy Xiang wrote: bq. For ServerName, port and startcode are unassigned in pb, how do you handle those -1 default values? For example, hbck uses -1 port. Am I not creating a ServerName passing port and startcode? This code just does what the old code does. It did not intercept -1s, etc., so neither does this patch. I think I'm not understanding. Help me out. Give it another go Jimmy. - Michael --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4630/#review6667 --- On 2012-04-03 23:38:15, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4630/ bq. --- bq. bq. (Updated 2012-04-03 23:38:15) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Serialize active master using pb. Do same for backup masters. bq. bq. D src/main/java/org/apache/hadoop/hbase/MasterAddressTracker.java bq.Moved under zookeeper. All other Trackers are under zookeeper but this one. bq. M src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java bq.Use new utility method for getting current master that is over in MasterAddressTracker. bq. M src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java bq.Move dup code out into a new handle function. bq.watcher.masterAddressZNode is no longer public. Use accessor. bq.Use new setMasterAddress,getMasterAddress function on bq.MasterAddressTracker for setting znode and backup master znode. bq. M src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java bq.Generated code. bq. M src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java bq.Fix up import because MasterAddressTracker moved. bq. A src/main/java/org/apache/hadoop/hbase/zookeeper/MasterAddressTracker.java bq.Add back MasterAddressTracker here. Add some utility functions for setting bq.and getting the master address. bq. M src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java bq.Move dataToServerName into ZKUtil. Its useful beyond this class only. bq. M src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bq.Go via accessor to get master znode path and use new utility method bq.to get current master address. dataToServerName moved here from bq.RootRegionTracker. bq. M src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java bq.Shutdown access to masterAddressZNode. Add an accessor method. bq. M src/main/protobuf/ZooKeeper.proto bq.Add new master message class. bq. M src/test/java/org/apache/hadoop/hbase/master/TestActiveMasterManager.java bq. M src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java bq.Go via new accessor and use utility over in MasterAddressTracker. bq. M src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java bq.Fix import to point at new location. bq. bq. bq. This addresses bug hbase-5709. bq. https://issues.apache.org/jira/browse/hbase-5709 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/MasterAddressTracker.java a246abe bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa30969 bq.src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java ef19cb9 bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java 8ff87fe bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 4f80999 bq. src/main/java/org/apache/hadoop/hbase/zookeeper/MasterAddressTracker.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java 6b2ea57 bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 037e354 bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 0f83655 bq.src/main/protobuf/ZooKeeper.proto 20f8eb0 bq. src/test/java/org/apache/hadoop/hbase/master/TestActiveMasterManager.java 05f6b1a bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java e91d83c bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java 8478260 bq. bq. Diff: https://reviews.apache.org/r/4630/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Michael bq. bq. Move active master
[jira] [Commented] (HBASE-5709) Move active master and backup master znodes to use pbs
[ https://issues.apache.org/jira/browse/HBASE-5709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246427#comment-13246427 ] jirapos...@reviews.apache.org commented on HBASE-5709: -- bq. On 2012-04-04 00:21:57, Jimmy Xiang wrote: bq. For ServerName, port and startcode are unassigned in pb, how do you handle those -1 default values? For example, hbck uses -1 port. bq. bq. Michael Stack wrote: bq. Am I not creating a ServerName passing port and startcode? This code just does what the old code does. It did not intercept -1s, etc., so neither does this patch. bq. bq. I think I'm not understanding. Help me out. Give it another go Jimmy. This patch is good to me. What I was thinking about is the region transition data. Sorry for the confusion. - Jimmy --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4630/#review6667 --- On 2012-04-03 23:38:15, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4630/ bq. --- bq. bq. (Updated 2012-04-03 23:38:15) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Serialize active master using pb. Do same for backup masters. bq. bq. D src/main/java/org/apache/hadoop/hbase/MasterAddressTracker.java bq.Moved under zookeeper. All other Trackers are under zookeeper but this one. bq. M src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java bq.Use new utility method for getting current master that is over in MasterAddressTracker. bq. M src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java bq.Move dup code out into a new handle function. bq.watcher.masterAddressZNode is no longer public. Use accessor. bq.Use new setMasterAddress,getMasterAddress function on bq.MasterAddressTracker for setting znode and backup master znode. bq. M src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java bq.Generated code. bq. M src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java bq.Fix up import because MasterAddressTracker moved. bq. A src/main/java/org/apache/hadoop/hbase/zookeeper/MasterAddressTracker.java bq.Add back MasterAddressTracker here. Add some utility functions for setting bq.and getting the master address. bq. M src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java bq.Move dataToServerName into ZKUtil. Its useful beyond this class only. bq. M src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bq.Go via accessor to get master znode path and use new utility method bq.to get current master address. dataToServerName moved here from bq.RootRegionTracker. bq. M src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java bq.Shutdown access to masterAddressZNode. Add an accessor method. bq. M src/main/protobuf/ZooKeeper.proto bq.Add new master message class. bq. M src/test/java/org/apache/hadoop/hbase/master/TestActiveMasterManager.java bq. M src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java bq.Go via new accessor and use utility over in MasterAddressTracker. bq. M src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java bq.Fix import to point at new location. bq. bq. bq. This addresses bug hbase-5709. bq. https://issues.apache.org/jira/browse/hbase-5709 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/MasterAddressTracker.java a246abe bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa30969 bq.src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java ef19cb9 bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java 8ff87fe bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 4f80999 bq. src/main/java/org/apache/hadoop/hbase/zookeeper/MasterAddressTracker.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java 6b2ea57 bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 037e354 bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 0f83655 bq.src/main/protobuf/ZooKeeper.proto 20f8eb0 bq. src/test/java/org/apache/hadoop/hbase/master/TestActiveMasterManager.java 05f6b1a bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java e91d83c bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java 8478260 bq. bq. Diff:
[jira] [Commented] (HBASE-5625) Avoid byte buffer allocations when reading a value from a Result object
[ https://issues.apache.org/jira/browse/HBASE-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246481#comment-13246481 ] jirapos...@reviews.apache.org commented on HBASE-5625: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4607/ --- (Updated 2012-04-04 17:08:03.581526) Review request for hbase. Changes --- Implemented coding style comments. Added separate unit tests for all modified or added methods. Summary --- When calling Result.getValue(), an extra dummy KeyValue and its associated underlying byte array are allocated, as well as a persistent buffer that will contain the returned value. These can be avoided by reusing a static array for the dummy object and by passing a ByteBuffer object as a value destination buffer to the read method. This addresses bug HBASE-5625. https://issues.apache.org/jira/browse/HBASE-5625 Diffs (updated) - src/main/java/org/apache/hadoop/hbase/KeyValue.java 243d76f src/main/java/org/apache/hadoop/hbase/client/Result.java df0b3ef src/test/java/org/apache/hadoop/hbase/TestKeyValue.java fae6902 src/test/java/org/apache/hadoop/hbase/client/TestResult.java f9e29c2 Diff: https://reviews.apache.org/r/4607/diff Testing --- Added value check to TestResult#testBasic and TestResult.testMultiVersion. Thanks, Tudor Avoid byte buffer allocations when reading a value from a Result object --- Key: HBASE-5625 URL: https://issues.apache.org/jira/browse/HBASE-5625 Project: HBase Issue Type: Improvement Components: client Affects Versions: 0.92.1 Reporter: Tudor Scurtu Assignee: Tudor Scurtu Labels: patch Fix For: 0.96.0 Attachments: 5625.txt, 5625v2.txt, 5625v3.txt, 5625v4.txt, 5625v5.txt, 5625v6.txt When calling Result.getValue(), an extra dummy KeyValue and its associated underlying byte array are allocated, as well as a persistent buffer that will contain the returned value. These can be avoided by reusing a static array for the dummy object and by passing a ByteBuffer object as a value destination buffer to the read method. The current functionality is maintained, and we have added a separate method call stack that employs the described changes. I will provide more details with the patch. Running tests with a profiler, the reduction of read time seems to be of up to 40%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5625) Avoid byte buffer allocations when reading a value from a Result object
[ https://issues.apache.org/jira/browse/HBASE-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246480#comment-13246480 ] jirapos...@reviews.apache.org commented on HBASE-5625: -- bq. On 2012-04-02 17:34:38, Michael Stack wrote: bq. Looks good. Some comments below. Thanks! Please read the changes below. bq. On 2012-04-02 17:34:38, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/KeyValue.java, line 531 bq. https://reviews.apache.org/r/4607/diff/1/?file=97954#file97954line531 bq. bq. How do we know this buffer is big enough? Maybe should add an override that takes an offset into the buffer? Added overload. Added exception comment for when there is insufficient space remaining in the buffer. bq. On 2012-04-02 17:34:38, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/KeyValue.java, line 1443 bq. https://reviews.apache.org/r/4607/diff/1/?file=97954#file97954line1443 bq. bq. Is this an override? Yes. Modified the original method to call the new one with default parameters. bq. On 2012-04-02 17:34:38, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/KeyValue.java, line 1448 bq. https://reviews.apache.org/r/4607/diff/1/?file=97954#file97954line1448 bq. bq. Why pull out all these values here? Maybe we'll fail the first test (q1 is not used by the first test and these extractions cost.. they create byte arrays...) Moved. bq. On 2012-04-02 17:34:38, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/KeyValue.java, line 2001 bq. https://reviews.apache.org/r/4607/diff/1/?file=97954#file97954line2001 bq. bq. How do I know the buffer is big enough? Should there be an override that takes an offset? Added buffer offset parameter. We know exactly how many bytes we have to write and how much free space we have in the buffer. An 'IllegalArgumentException' is thrown when there isn't enough free space. Is that what you were asking? bq. On 2012-04-02 17:34:38, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/KeyValue.java, line 2002 bq. https://reviews.apache.org/r/4607/diff/1/?file=97954#file97954line2002 bq. bq. Please follow the formatting that is present in the rest of the file. Notice placement of exceptions and params. Do not add your own style. Thanks. Hope I fixed this. bq. On 2012-04-02 17:34:38, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/client/Result.java, line 251 bq. https://reviews.apache.org/r/4607/diff/1/?file=97955#file97955line251 bq. bq. Please follow convention that the rest of the file has. Hope I fixed this. bq. On 2012-04-02 17:34:38, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/client/Result.java, line 255 bq. https://reviews.apache.org/r/4607/diff/1/?file=97955#file97955line255 bq. bq. ditto Hope I fixed this. bq. On 2012-04-02 17:34:38, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/client/Result.java, line 478 bq. https://reviews.apache.org/r/4607/diff/1/?file=97955#file97955line478 bq. bq. Rename hasColumn or isColumn. This overloads an original method. A refactoring here would be a major non-backwards compatible change. bq. On 2012-04-02 17:34:38, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/client/Result.java, line 398 bq. https://reviews.apache.org/r/4607/diff/1/?file=97955#file97955line398 bq. bq. Rename hasContent or isNotEmpty or isNotEmptyColumn This method was named to mirror 'containsColumn()'. See below. bq. On 2012-04-02 17:34:38, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/client/Result.java, line 431 bq. https://reviews.apache.org/r/4607/diff/1/?file=97955#file97955line431 bq. bq. Why we have both isNonEmptyColumn and isEmptyColumn? Why not just one and then check return with a !? They are not complementary. containsColumn = value exists containsEmptyColumn = value exists is empty byte array containsNonEmptyColumn = value exists is not empty byte array The value could be missing, in which case all methods would return false. bq. On 2012-04-02 17:34:38, Michael Stack wrote: bq. src/test/java/org/apache/hadoop/hbase/client/TestResult.java, line 117 bq. https://reviews.apache.org/r/4607/diff/1/?file=97956#file97956line117 bq. bq. Or leave it and change the name of the test so it doesn't have the test prefix. Add a main and have it call this. I think the method useful. Might as well keep it. Refactored as 'doReadBenchmark()'; called from 'main()'; bq. On 2012-04-02 17:34:38, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/KeyValue.java, line 2056 bq. https://reviews.apache.org/r/4607/diff/1/?file=97954#file97954line2056 bq. bq. This is a critical base class. I'm nervous when it gets
[jira] [Commented] (HBASE-5625) Avoid byte buffer allocations when reading a value from a Result object
[ https://issues.apache.org/jira/browse/HBASE-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246479#comment-13246479 ] jirapos...@reviews.apache.org commented on HBASE-5625: -- bq. On 2012-04-02 16:50:48, Ted Yu wrote: bq. I ran TestTableMapReduce and TestMultithreadedTableMapper with patch v5. bq. They passed. bq. bq. Some minor comments below. Thanks! Fixed all except Result#259 and TestResult#117. bq. On 2012-04-02 16:50:48, Ted Yu wrote: bq. src/test/java/org/apache/hadoop/hbase/client/TestResult.java, line 117 bq. https://reviews.apache.org/r/4607/diff/1/?file=97956#file97956line117 bq. bq. Since benchmarking is hard to do, this test case can be dropped. Refactored as 'doReadBenchmark()'; called from 'main()'. bq. On 2012-04-02 16:50:48, Ted Yu wrote: bq. src/main/java/org/apache/hadoop/hbase/client/Result.java, line 259 bq. https://reviews.apache.org/r/4607/diff/1/?file=97955#file97955line259 bq. bq. Is this comment needed ? This was copied from original. Should I remove both occurences? - Tudor --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4607/#review6622 --- On 2012-04-02 14:22:48, Tudor Scurtu wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4607/ bq. --- bq. bq. (Updated 2012-04-02 14:22:48) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. When calling Result.getValue(), an extra dummy KeyValue and its associated underlying byte array are allocated, as well as a persistent buffer that will contain the returned value. bq. bq. These can be avoided by reusing a static array for the dummy object and by passing a ByteBuffer object as a value destination buffer to the read method. bq. bq. bq. This addresses bug HBASE-5625. bq. https://issues.apache.org/jira/browse/HBASE-5625 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/KeyValue.java 243d76f bq.src/main/java/org/apache/hadoop/hbase/client/Result.java df0b3ef bq.src/test/java/org/apache/hadoop/hbase/client/TestResult.java f9e29c2 bq. bq. Diff: https://reviews.apache.org/r/4607/diff bq. bq. bq. Testing bq. --- bq. bq. Added value check to TestResult#testBasic and TestResult.testMultiVersion. bq. bq. bq. Thanks, bq. bq. Tudor bq. bq. Avoid byte buffer allocations when reading a value from a Result object --- Key: HBASE-5625 URL: https://issues.apache.org/jira/browse/HBASE-5625 Project: HBase Issue Type: Improvement Components: client Affects Versions: 0.92.1 Reporter: Tudor Scurtu Assignee: Tudor Scurtu Labels: patch Fix For: 0.96.0 Attachments: 5625.txt, 5625v2.txt, 5625v3.txt, 5625v4.txt, 5625v5.txt, 5625v6.txt When calling Result.getValue(), an extra dummy KeyValue and its associated underlying byte array are allocated, as well as a persistent buffer that will contain the returned value. These can be avoided by reusing a static array for the dummy object and by passing a ByteBuffer object as a value destination buffer to the read method. The current functionality is maintained, and we have added a separate method call stack that employs the described changes. I will provide more details with the patch. Running tests with a profiler, the reduction of read time seems to be of up to 40%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4348) Add metrics for regions in transition
[ https://issues.apache.org/jira/browse/HBASE-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246761#comment-13246761 ] jirapos...@reviews.apache.org commented on HBASE-4348: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4402/ --- (Updated 2012-04-04 21:47:04.382172) Review request for hbase. Changes --- Changed the name of the property for defining the threshold time limit to show the Region in transitions in UI; ans other nits from Ted. Summary --- This patch is for adding Region in transition metrics to the HMaster metrics system. It also adds these metrics in the master ui, in the Region in transition section. I have attached the proposed new format in the jira 4348. This addresses bug HBase-4348. https://issues.apache.org/jira/browse/HBase-4348 Diffs (updated) - src/main/jamon/org/apache/hadoop/hbase/tmpl/master/AssignmentManagerStatusTmpl.jamon 0dc0691 src/main/java/org/apache/hadoop/hbase/HConstants.java 21ac4ba src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 64def15 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 9bd4ace src/main/java/org/apache/hadoop/hbase/master/metrics/MasterMetrics.java 83abc52 src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 91dce36 Diff: https://reviews.apache.org/r/4402/diff Testing --- Ran on a 5 node cluster and kill region servers randomly to observe the changes in the RIT metrics as emitted out by the Master's mxbean; mvn test passes without any failure. Thanks, Himanshu Add metrics for regions in transition - Key: HBASE-4348 URL: https://issues.apache.org/jira/browse/HBASE-4348 Project: HBase Issue Type: Improvement Components: metrics Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Himanshu Vashishtha Priority: Minor Labels: noob Fix For: 0.96.0 Attachments: 4348-metrics-v3.patch, 4348-v1.patch, 4348-v2.patch, RITs.png, RegionInTransitions2.png, metrics-v2.patch The following metrics would be useful for monitoring the master: - the number of regions in transition - the number of regions in transition that have been in transition for more than a minute - how many seconds has the oldest region-in-transition been in transition -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5717) Scanner metrics are only reported if you get to the end of a scanner
[ https://issues.apache.org/jira/browse/HBASE-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246772#comment-13246772 ] jirapos...@reviews.apache.org commented on HBASE-5717: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4640/ --- Review request for hbase. Summary --- Fix for persistence of scan metrics when the scanner doesn't run to exhaustion. This addresses bug HBASE-5717. https://issues.apache.org/jira/browse/HBASE-5717 Diffs - /src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java 1309585 /src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1309585 Diff: https://reviews.apache.org/r/4640/diff Testing --- Altered the scan metrics unit test to show this problem (now fails without changes to ClientScanner.java). Thanks, Ian Scanner metrics are only reported if you get to the end of a scanner Key: HBASE-5717 URL: https://issues.apache.org/jira/browse/HBASE-5717 Project: HBase Issue Type: Bug Components: client, metrics Reporter: Ian Varley Priority: Minor Attachments: ClientScanner_HBASE_5717.patch Original Estimate: 4h Remaining Estimate: 4h When you turn on Scanner Metrics, the metrics are currently only made available if you run over all records available in the scanner. If you stop iterating before the end, the values are never flushed into the metrics object (in the Scan attribute). Will supply a patch with fix and test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5717) Scanner metrics are only reported if you get to the end of a scanner
[ https://issues.apache.org/jira/browse/HBASE-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246776#comment-13246776 ] jirapos...@reviews.apache.org commented on HBASE-5717: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4640/#review6697 --- /src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java https://reviews.apache.org/r/4640/#comment14500 No need for this here any more, since it's called in close() /src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java https://reviews.apache.org/r/4640/#comment14501 Linebreak change (100 characters now) /src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java https://reviews.apache.org/r/4640/#comment14502 Important change: writing the scan metrics on scanner close /src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java https://reviews.apache.org/r/4640/#comment14503 This would fail w/o the changes to ClientScanner, because neither next() nor close() would persist the scan metrics out if we don't run off the end of the available records. - Ian On 2012-04-04 22:00:42, Ian Varley wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4640/ bq. --- bq. bq. (Updated 2012-04-04 22:00:42) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Fix for persistence of scan metrics when the scanner doesn't run to exhaustion. bq. bq. bq. This addresses bug HBASE-5717. bq. https://issues.apache.org/jira/browse/HBASE-5717 bq. bq. bq. Diffs bq. - bq. bq./src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java 1309585 bq./src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1309585 bq. bq. Diff: https://reviews.apache.org/r/4640/diff bq. bq. bq. Testing bq. --- bq. bq. Altered the scan metrics unit test to show this problem (now fails without changes to ClientScanner.java). bq. bq. bq. Thanks, bq. bq. Ian bq. bq. Scanner metrics are only reported if you get to the end of a scanner Key: HBASE-5717 URL: https://issues.apache.org/jira/browse/HBASE-5717 Project: HBase Issue Type: Bug Components: client, metrics Reporter: Ian Varley Priority: Minor Attachments: ClientScanner_HBASE_5717.patch Original Estimate: 4h Remaining Estimate: 4h When you turn on Scanner Metrics, the metrics are currently only made available if you run over all records available in the scanner. If you stop iterating before the end, the values are never flushed into the metrics object (in the Scan attribute). Will supply a patch with fix and test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5717) Scanner metrics are only reported if you get to the end of a scanner
[ https://issues.apache.org/jira/browse/HBASE-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246781#comment-13246781 ] jirapos...@reviews.apache.org commented on HBASE-5717: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4640/#review6698 --- /src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java https://reviews.apache.org/r/4640/#comment14504 Should this call be placed in a finally block ? - Ted On 2012-04-04 22:00:42, Ian Varley wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4640/ bq. --- bq. bq. (Updated 2012-04-04 22:00:42) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Fix for persistence of scan metrics when the scanner doesn't run to exhaustion. bq. bq. bq. This addresses bug HBASE-5717. bq. https://issues.apache.org/jira/browse/HBASE-5717 bq. bq. bq. Diffs bq. - bq. bq./src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java 1309585 bq./src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1309585 bq. bq. Diff: https://reviews.apache.org/r/4640/diff bq. bq. bq. Testing bq. --- bq. bq. Altered the scan metrics unit test to show this problem (now fails without changes to ClientScanner.java). bq. bq. bq. Thanks, bq. bq. Ian bq. bq. Scanner metrics are only reported if you get to the end of a scanner Key: HBASE-5717 URL: https://issues.apache.org/jira/browse/HBASE-5717 Project: HBase Issue Type: Bug Components: client, metrics Reporter: Ian Varley Priority: Minor Attachments: ClientScanner_HBASE_5717.patch Original Estimate: 4h Remaining Estimate: 4h When you turn on Scanner Metrics, the metrics are currently only made available if you run over all records available in the scanner. If you stop iterating before the end, the values are never flushed into the metrics object (in the Scan attribute). Will supply a patch with fix and test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5717) Scanner metrics are only reported if you get to the end of a scanner
[ https://issues.apache.org/jira/browse/HBASE-5717?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246790#comment-13246790 ] jirapos...@reviews.apache.org commented on HBASE-5717: -- bq. On 2012-04-04 22:07:52, Ted Yu wrote: bq. /src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java, line 376 bq. https://reviews.apache.org/r/4640/diff/1/?file=100049#file100049line376 bq. bq. Should this call be placed in a finally block ? Yes, you're right, that'd be cleaner (I took a shortcut putting it here b/c it also throws IOException, and we'd have to swallow it the same way in another try block inside the finally block, but you're right, that's more correct). Will append the patch with that. - Ian --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4640/#review6698 --- On 2012-04-04 22:00:42, Ian Varley wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4640/ bq. --- bq. bq. (Updated 2012-04-04 22:00:42) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Fix for persistence of scan metrics when the scanner doesn't run to exhaustion. bq. bq. bq. This addresses bug HBASE-5717. bq. https://issues.apache.org/jira/browse/HBASE-5717 bq. bq. bq. Diffs bq. - bq. bq./src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java 1309585 bq./src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide.java 1309585 bq. bq. Diff: https://reviews.apache.org/r/4640/diff bq. bq. bq. Testing bq. --- bq. bq. Altered the scan metrics unit test to show this problem (now fails without changes to ClientScanner.java). bq. bq. bq. Thanks, bq. bq. Ian bq. bq. Scanner metrics are only reported if you get to the end of a scanner Key: HBASE-5717 URL: https://issues.apache.org/jira/browse/HBASE-5717 Project: HBase Issue Type: Bug Components: client, metrics Reporter: Ian Varley Priority: Minor Attachments: ClientScanner_HBASE_5717.patch Original Estimate: 4h Remaining Estimate: 4h When you turn on Scanner Metrics, the metrics are currently only made available if you run over all records available in the scanner. If you stop iterating before the end, the values are never flushed into the metrics object (in the Scan attribute). Will supply a patch with fix and test. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5719) Enhance hbck to sideline overlapped mega regions
[ https://issues.apache.org/jira/browse/HBASE-5719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246861#comment-13246861 ] jirapos...@reviews.apache.org commented on HBASE-5719: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4649/ --- Review request for hbase. Summary --- Make it configurable to sideline some regions in big overlapped groups which hbck doesn't handle currently. The regions chose to sideline are those which overlap with most other regions. This addresses bug HBASE-5719. https://issues.apache.org/jira/browse/HBASE-5719 Diffs - src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 54f9b21 src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java 17678dd Diff: https://reviews.apache.org/r/4649/diff Testing --- mvn -PlocalTests -Dtest=TestHBaseFsck* clean test Also tested in real system to fix inconsistencies. Thanks, Jimmy Enhance hbck to sideline overlapped mega regions Key: HBASE-5719 URL: https://issues.apache.org/jira/browse/HBASE-5719 Project: HBase Issue Type: New Feature Components: hbck Affects Versions: 0.94.0, 0.96.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: hbase-5719.patch If there are too many regions in one overlapped group (by default, more than 10), hbck currently doesn't merge them since it takes time. In this case, we can sideline some regions in the group and break the overlapping to fix the inconsistency. Later on, sidelined regions can be bulk loaded manually. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5719) Enhance hbck to sideline overlapped mega regions
[ https://issues.apache.org/jira/browse/HBASE-5719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246864#comment-13246864 ] jirapos...@reviews.apache.org commented on HBASE-5719: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4649/ --- (Updated 2012-04-04 23:52:50.930344) Review request for hbase and jmhsieh. Summary --- Make it configurable to sideline some regions in big overlapped groups which hbck doesn't handle currently. The regions chose to sideline are those which overlap with most other regions. This addresses bug HBASE-5719. https://issues.apache.org/jira/browse/HBASE-5719 Diffs - src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 54f9b21 src/main/java/org/apache/hadoop/hbase/util/RegionSplitCalculator.java 17678dd Diff: https://reviews.apache.org/r/4649/diff Testing --- mvn -PlocalTests -Dtest=TestHBaseFsck* clean test Also tested in real system to fix inconsistencies. Thanks, Jimmy Enhance hbck to sideline overlapped mega regions Key: HBASE-5719 URL: https://issues.apache.org/jira/browse/HBASE-5719 Project: HBase Issue Type: New Feature Components: hbck Affects Versions: 0.94.0, 0.96.0 Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 Attachments: hbase-5719.patch If there are too many regions in one overlapped group (by default, more than 10), hbck currently doesn't merge them since it takes time. In this case, we can sideline some regions in the group and break the overlapping to fix the inconsistency. Later on, sidelined regions can be bulk loaded manually. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4348) Add metrics for regions in transition
[ https://issues.apache.org/jira/browse/HBASE-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246887#comment-13246887 ] jirapos...@reviews.apache.org commented on HBASE-4348: -- bq. On 2012-04-04 01:55:42, Himanshu Vashishtha wrote: bq. src/main/java/org/apache/hadoop/hbase/HConstants.java, line 655 bq. https://reviews.apache.org/r/4402/diff/6/?file=97548#file97548line655 bq. bq. Jon: This is not a refresh metrics per se. Its for defining what a user considers to be a _longer_ period for a Region to be in a transition state, and show these regions count as a metric. bq. So, it can be hbase.metrics.rit.stuck.warning.threshold? bq. Long name, though. Please let me know amd I will change accordingly and revise the patch. got it. - jmhsieh --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4402/#review6669 --- On 2012-04-04 21:47:04, Himanshu Vashishtha wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4402/ bq. --- bq. bq. (Updated 2012-04-04 21:47:04) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This patch is for adding Region in transition metrics to the HMaster metrics system. It also adds these metrics in the master ui, in the Region in transition section. I have attached the proposed new format in the jira 4348. bq. bq. bq. This addresses bug HBase-4348. bq. https://issues.apache.org/jira/browse/HBase-4348 bq. bq. bq. Diffs bq. - bq. bq. src/main/jamon/org/apache/hadoop/hbase/tmpl/master/AssignmentManagerStatusTmpl.jamon 0dc0691 bq.src/main/java/org/apache/hadoop/hbase/HConstants.java 21ac4ba bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 64def15 bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 9bd4ace bq.src/main/java/org/apache/hadoop/hbase/master/metrics/MasterMetrics.java 83abc52 bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 91dce36 bq. bq. Diff: https://reviews.apache.org/r/4402/diff bq. bq. bq. Testing bq. --- bq. bq. Ran on a 5 node cluster and kill region servers randomly to observe the changes in the RIT metrics as emitted out by the Master's mxbean; bq. bq. mvn test passes without any failure. bq. bq. bq. Thanks, bq. bq. Himanshu bq. bq. Add metrics for regions in transition - Key: HBASE-4348 URL: https://issues.apache.org/jira/browse/HBASE-4348 Project: HBase Issue Type: Improvement Components: metrics Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Himanshu Vashishtha Priority: Minor Labels: noob Fix For: 0.96.0 Attachments: 4348-metrics-v3.patch, 4348-v1.patch, 4348-v2.patch, RITs.png, RegionInTransitions2.png, metrics-v2.patch The following metrics would be useful for monitoring the master: - the number of regions in transition - the number of regions in transition that have been in transition for more than a minute - how many seconds has the oldest region-in-transition been in transition -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4348) Add metrics for regions in transition
[ https://issues.apache.org/jira/browse/HBASE-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246892#comment-13246892 ] jirapos...@reviews.apache.org commented on HBASE-4348: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4402/#review6703 --- Ship it! I'm happy with this. Ted do you have comments? If no, I'll commit in about 24 hours. src/main/java/org/apache/hadoop/hbase/HConstants.java https://reviews.apache.org/r/4402/#comment14507 I like this name. - jmhsieh On 2012-04-04 21:47:04, Himanshu Vashishtha wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4402/ bq. --- bq. bq. (Updated 2012-04-04 21:47:04) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This patch is for adding Region in transition metrics to the HMaster metrics system. It also adds these metrics in the master ui, in the Region in transition section. I have attached the proposed new format in the jira 4348. bq. bq. bq. This addresses bug HBase-4348. bq. https://issues.apache.org/jira/browse/HBase-4348 bq. bq. bq. Diffs bq. - bq. bq. src/main/jamon/org/apache/hadoop/hbase/tmpl/master/AssignmentManagerStatusTmpl.jamon 0dc0691 bq.src/main/java/org/apache/hadoop/hbase/HConstants.java 21ac4ba bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 64def15 bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 9bd4ace bq.src/main/java/org/apache/hadoop/hbase/master/metrics/MasterMetrics.java 83abc52 bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 91dce36 bq. bq. Diff: https://reviews.apache.org/r/4402/diff bq. bq. bq. Testing bq. --- bq. bq. Ran on a 5 node cluster and kill region servers randomly to observe the changes in the RIT metrics as emitted out by the Master's mxbean; bq. bq. mvn test passes without any failure. bq. bq. bq. Thanks, bq. bq. Himanshu bq. bq. Add metrics for regions in transition - Key: HBASE-4348 URL: https://issues.apache.org/jira/browse/HBASE-4348 Project: HBase Issue Type: Improvement Components: metrics Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Himanshu Vashishtha Priority: Minor Labels: noob Fix For: 0.96.0 Attachments: 4348-metrics-v3.patch, 4348-v1.patch, 4348-v2.patch, RITs.png, RegionInTransitions2.png, metrics-v2.patch The following metrics would be useful for monitoring the master: - the number of regions in transition - the number of regions in transition that have been in transition for more than a minute - how many seconds has the oldest region-in-transition been in transition -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5715) Revert 'Instant schema alter' for now, HBASE-4213
[ https://issues.apache.org/jira/browse/HBASE-5715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13247022#comment-13247022 ] jirapos...@reviews.apache.org commented on HBASE-5715: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4656/ --- Review request for hbase. Summary --- Revert hbase-4213 instant schema alter. Reviewing, you probably want to do follow along w/ the original 4213 patch v7. This patch is a reverse of that. This addresses bug hbase-5715. https://issues.apache.org/jira/browse/hbase-5715 Diffs - src/main/java/org/apache/hadoop/hbase/LocalHBaseCluster.java 9c9c7cc src/main/java/org/apache/hadoop/hbase/executor/EventHandler.java 6229858 src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java ea7ae45 src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 26e9552 src/main/java/org/apache/hadoop/hbase/master/HMaster.java daf3b07 src/main/java/org/apache/hadoop/hbase/master/MasterFileSystem.java 8a7da2e src/main/java/org/apache/hadoop/hbase/master/MasterServices.java e1e6685 src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 3515d4a src/main/java/org/apache/hadoop/hbase/master/handler/DeleteTableHandler.java 02bec37 src/main/java/org/apache/hadoop/hbase/master/handler/ModifyTableHandler.java 4a735ec src/main/java/org/apache/hadoop/hbase/master/handler/TableAddFamilyHandler.java 4ec3d68 src/main/java/org/apache/hadoop/hbase/master/handler/TableDeleteFamilyHandler.java bfa624b src/main/java/org/apache/hadoop/hbase/master/handler/TableEventHandler.java 6d173bb src/main/java/org/apache/hadoop/hbase/master/handler/TableModifyFamilyHandler.java b4f8cd4 src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java 2913c2b src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 4f80999 src/main/java/org/apache/hadoop/hbase/regionserver/OnlineRegions.java da9e5cf src/main/java/org/apache/hadoop/hbase/zookeeper/MasterSchemaChangeTracker.java 6c723b4 src/main/java/org/apache/hadoop/hbase/zookeeper/SchemaChangeTracker.java 48d4ff7 src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 0f83655 src/main/resources/hbase-default.xml 341431a src/test/java/org/apache/hadoop/hbase/client/InstantSchemaChangeTestBase.java 378c2b4 src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChange.java 4ac2847 src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChangeFailover.java c1490eb src/test/java/org/apache/hadoop/hbase/client/TestInstantSchemaChangeSplit.java 8f3124b src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java d2b3060 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 227c5f2 src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java bb3ddd7 Diff: https://reviews.apache.org/r/4656/diff Testing --- Thanks, Michael Revert 'Instant schema alter' for now, HBASE-4213 - Key: HBASE-5715 URL: https://issues.apache.org/jira/browse/HBASE-5715 Project: HBase Issue Type: Task Reporter: stack Attachments: revert.txt, revert.v2.txt, revert.v3.txt, revert.v4.txt See this discussion: http://search-hadoop.com/m/NxCQh1KlSxR1/Pull+instant+schema+updating+out%253Fsubj=Pull+instant+schema+updating+out+ Pull out hbase-4213 for now. Can add it back later. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5444) Add PB-based calls to HMasterRegionInterface
[ https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13245013#comment-13245013 ] jirapos...@reviews.apache.org commented on HBASE-5444: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4463/#review6651 --- Looks great. On commit will everything be broke? pom.xml https://reviews.apache.org/r/4463/#comment14348 We don't need this anymore now we are checking in generated files. src/main/java/org/apache/hadoop/hbase/HConstants.java https://reviews.apache.org/r/4463/#comment14349 Should this top level class have references down into protobufs? Maybe the empty server load should be in ServerLoad or at least under o.a.h.h.protobuf src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java https://reviews.apache.org/r/4463/#comment14350 Radical! Jimmy didn't remove his interface. Maybe he should have? src/main/java/org/apache/hadoop/hbase/master/HMaster.java https://reviews.apache.org/r/4463/#comment14351 Should it return the Response builder or the Response (don't they have same underlying 'Interface'? Return that?) Oh, I think I understand... must be a Builder in case we add config later? src/main/java/org/apache/hadoop/hbase/master/HMaster.java https://reviews.apache.org/r/4463/#comment14352 ok, yeah, here is the actual instance, not the builder src/main/java/org/apache/hadoop/hbase/protobuf/MasterRegionInterface.java https://reviews.apache.org/r/4463/#comment14354 You think this package right place for this? How about in master package or up above at o.a.h.h? Is MasterRegionInterface a good name for this Interface even? The name was made long long time ago. Didn't make sense then. Still doesn't. Should it be called RegionServerInterface or RegionServer in the master package -- its the Interface RegionServers invoke on the master... whats a good name for that? RegionServerService or RegionServersInterface src/main/java/org/apache/hadoop/hbase/protobuf/PBHelper.java https://reviews.apache.org/r/4463/#comment14353 Something called ProtobufUtil.java was added since you posted your patch. Maybe this stuff belongs in there? src/main/java/org/apache/hadoop/hbase/protobuf/PBHelper.java https://reviews.apache.org/r/4463/#comment14355 Why not have this as static in ServerLoad? src/main/java/org/apache/hadoop/hbase/protobuf/PBHelper.java https://reviews.apache.org/r/4463/#comment14356 ditto src/main/proto/MasterRegionProtocol.proto https://reviews.apache.org/r/4463/#comment14357 MasterRegionProtocol doesn't seem like good name for this file? MasterRegionServer or RegionServerToMasterProtocol or Interface? Jimmy left of the 'Protocol' in his patch. src/main/proto/MasterRegionProtocol.proto https://reviews.apache.org/r/4463/#comment14358 Seems like a bad name. No Region on Master? src/main/proto/MasterRegionProtocol.proto https://reviews.apache.org/r/4463/#comment14359 What is this again? Should this be ServerName from hbase.proto? src/main/proto/MasterRegionProtocol.proto https://reviews.apache.org/r/4463/#comment14360 ditto src/main/proto/MasterRegionProtocol.proto https://reviews.apache.org/r/4463/#comment14361 RegionServerService? src/main/proto/hbase.proto https://reviews.apache.org/r/4463/#comment14362 A file of this name has gone in already. Need to rejigger your patch? src/main/proto/hbase.proto https://reviews.apache.org/r/4463/#comment14363 Use Jimmy's regionspec? src/main/proto/hbase.proto https://reviews.apache.org/r/4463/#comment14364 I think jimmy made one of these already. Coordinate. I like the name of yours... - Michael On 2012-03-23 19:55:28, Gregory Chanan wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4463/ bq. --- bq. bq. (Updated 2012-03-23 19:55:28) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Adds PB-based calls replacing HMasterRegionInterface. bq. bq. There are some temporary hacks, e.g. converting PB-based ServerLoad to existing HServerLoad so I didn't need to convert ClusterStatus (which brings in a lot of other changes). That will be cleaned up in HBASE-5445. bq. bq. bq. This addresses bug HBASE-5444. bq. https://issues.apache.org/jira/browse/HBASE-5444 bq. bq. bq. Diffs bq. - bq. bq.pom.xml 10b13ef bq. src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 69434f7 bq.
[jira] [Commented] (HBASE-5444) Add PB-based calls to HMasterRegionInterface
[ https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13245463#comment-13245463 ] jirapos...@reviews.apache.org commented on HBASE-5444: -- bq. On 2012-04-03 06:07:58, Michael Stack wrote: bq. src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java, line 78 bq. https://reviews.apache.org/r/4463/diff/2/?file=95093#file95093line78 bq. bq. Radical! bq. bq. Jimmy didn't remove his interface. Maybe he should have? I haven't removed HRegionInterface yet. I can remove it after the admin part is also completed. bq. On 2012-04-03 06:07:58, Michael Stack wrote: bq. src/main/proto/hbase.proto, line 116 bq. https://reviews.apache.org/r/4463/diff/2/?file=95105#file95105line116 bq. bq. I think jimmy made one of these already. Coordinate. I like the name of yours... Sure, I can rename mine. - Jimmy --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4463/#review6651 --- On 2012-03-23 19:55:28, Gregory Chanan wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4463/ bq. --- bq. bq. (Updated 2012-03-23 19:55:28) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Adds PB-based calls replacing HMasterRegionInterface. bq. bq. There are some temporary hacks, e.g. converting PB-based ServerLoad to existing HServerLoad so I didn't need to convert ClusterStatus (which brings in a lot of other changes). That will be cleaned up in HBASE-5445. bq. bq. bq. This addresses bug HBASE-5444. bq. https://issues.apache.org/jira/browse/HBASE-5444 bq. bq. bq. Diffs bq. - bq. bq.pom.xml 10b13ef bq. src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 69434f7 bq. src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon ae76204 bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 9df4c10 bq.src/main/java/org/apache/hadoop/hbase/HConstants.java 347 bq.src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java cbfa489 bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 0db2760 bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 2916d68 bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java fd97830 bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java f1f06b0 bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java d47ef10 bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java cd1755f bq.src/main/java/org/apache/hadoop/hbase/master/MXBean.java 7f44dc2 bq.src/main/java/org/apache/hadoop/hbase/master/MXBeanImpl.java 45b8fe7 bq.src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java be63838 bq.src/main/java/org/apache/hadoop/hbase/master/ServerManager.java cbd55f7 bq. src/main/java/org/apache/hadoop/hbase/protobuf/MasterRegionInterface.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/protobuf/PBHelper.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e0af8fb bq.src/main/proto/MasterRegionProtocol.proto PRE-CREATION bq.src/main/proto/hbase.proto PRE-CREATION bq.src/main/resources/hbase-webapps/master/table.jsp 811df46 bq.src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 6af9188 bq.src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java 368a0e5 bq.src/test/java/org/apache/hadoop/hbase/io/TestHbaseObjectWritable.java f2f8ee3 bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 841649a bq.src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java 379f70c bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java e99d251 bq. bq. Diff: https://reviews.apache.org/r/4463/diff bq. bq. bq. Testing bq. --- bq. bq. Ran jenkins job, all unit tests passed. bq. bq. bq. Thanks, bq. bq. Gregory bq. bq. Add PB-based calls to HMasterRegionInterface Key: HBASE-5444 URL: https://issues.apache.org/jira/browse/HBASE-5444 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: Gregory Chanan -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA
[jira] [Commented] (HBASE-5707) Move clusterid and clusterup (shutdown) znodes over to pb
[
https://issues.apache.org/jira/browse/HBASE-5707?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13245737#comment-13245737
]
jirapos...@reviews.apache.org commented on HBASE-5707:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4627/
---
Review request for hbase.
Summary
---
Move two more znodes to serialize their content as pb.
Here are the changes:
{code}
M src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
Get the clusterid via utility in the ClusterId class rather than
get it directly.
M src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java
M src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterStatusTracker.java
Serialize and deserialize using new ClusterId pb message.
Include the PBUF preamble on znode content.
M src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java
Rename getRootRegionZNodeContent to getZNodeData so it aligns in name
with the other cases where we compose the znode content up in ClusterId
and in ClusterStatusTracker.
M src/main/protobuf/ZooKeeper.proto
Add new messages for ClusterId and ClusterUp
{code}
This addresses bug hbase-5707.
https://issues.apache.org/jira/browse/hbase-5707
Diffs
-
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa30969
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java
8ff87fe
src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterId.java 3fa83e6
src/main/java/org/apache/hadoop/hbase/zookeeper/ClusterStatusTracker.java
61e7367
src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java
6b2ea57
src/main/protobuf/ZooKeeper.proto 20f8eb0
Diff: https://reviews.apache.org/r/4627/diff
Testing
---
Thanks,
Michael
Move clusterid and clusterup (shutdown) znodes over to pb
-
Key: HBASE-5707
URL: https://issues.apache.org/jira/browse/HBASE-5707
Project: HBase
Issue Type: Task
Reporter: stack
Attachments: 5707.txt
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5598) Analyse and fix the findbugs reporting by QA and add invalid bugs into findbugs-excludeFilter file
[ https://issues.apache.org/jira/browse/HBASE-5598?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13245859#comment-13245859 ] jirapos...@reviews.apache.org commented on HBASE-5598: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4612/ --- Review request for hbase. Summary --- Fixes for issues reported in findbugs for Correctness category. This addresses bug HBASE-5598. https://issues.apache.org/jira/browse/HBASE-5598 Diffs - dev-support/test-patch.properties 2209d27 src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon ae76204 src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 37bcaba src/main/java/org/apache/hadoop/hbase/mapreduce/CellCounter.java 32d66fb src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 64def15 src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java 1676832 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 9bd4ace src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 3515d4a src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8ae60a3 src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreLAB.java 2694897 src/main/java/org/apache/hadoop/hbase/regionserver/ShutdownHook.java e3b230e src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 0c7b396 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 2e98b39 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 698bb3d src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java 8950c9f src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java 62cf6ac src/main/java/org/apache/hadoop/hbase/util/FSUtils.java aebe5b0 src/main/java/org/apache/hadoop/hbase/util/JVMClusterUtil.java c21377c src/main/java/org/apache/hadoop/hbase/util/Merge.java 04f15d4 src/main/java/org/apache/hadoop/hbase/util/PoolMap.java 8e2a856 Diff: https://reviews.apache.org/r/4612/diff Testing --- Ran test-patch.sh and unit tests. Thanks, David Analyse and fix the findbugs reporting by QA and add invalid bugs into findbugs-excludeFilter file -- Key: HBASE-5598 URL: https://issues.apache.org/jira/browse/HBASE-5598 Project: HBase Issue Type: Bug Components: scripts Affects Versions: 0.92.1, 0.94.0, 0.96.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G Priority: Minor Attachments: HBASE-5598.patch, findbugs-gui-report.jpg There are many findbugs errors reporting by HbaseQA. HBASE-5597 is going to up the OK count. This may lead to other issues when we re-factor the code, if we induce new valid ones and remove invalid bugs also can not be reported by QA. So, I would propose to add the exclude filter file for findbugs(for the invalid bugs). If we find any valid ones, we can fix under this JIRA. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5645) [findbugs] Fix correctness warnings
[ https://issues.apache.org/jira/browse/HBASE-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13245862#comment-13245862 ] jirapos...@reviews.apache.org commented on HBASE-5645: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4612/ --- (Updated 2012-04-03 22:54:54.016499) Review request for hbase. Changes --- Corrected JIRA number. Summary (updated) --- Fixes for issues reported in findbugs for Correctness category. This addresses bug HBASE-5645. https://issues.apache.org/jira/browse/HBASE-5645 Diffs - dev-support/test-patch.properties 2209d27 src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon ae76204 src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 37bcaba src/main/java/org/apache/hadoop/hbase/mapreduce/CellCounter.java 32d66fb src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 64def15 src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java 1676832 src/main/java/org/apache/hadoop/hbase/master/HMaster.java 9bd4ace src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 3515d4a src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8ae60a3 src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreLAB.java 2694897 src/main/java/org/apache/hadoop/hbase/regionserver/ShutdownHook.java e3b230e src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 0c7b396 src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 2e98b39 src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 698bb3d src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java 8950c9f src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java 62cf6ac src/main/java/org/apache/hadoop/hbase/util/FSUtils.java aebe5b0 src/main/java/org/apache/hadoop/hbase/util/JVMClusterUtil.java c21377c src/main/java/org/apache/hadoop/hbase/util/Merge.java 04f15d4 src/main/java/org/apache/hadoop/hbase/util/PoolMap.java 8e2a856 Diff: https://reviews.apache.org/r/4612/diff Testing --- Ran test-patch.sh and unit tests. Thanks, David [findbugs] Fix correctness warnings --- Key: HBASE-5645 URL: https://issues.apache.org/jira/browse/HBASE-5645 Project: HBase Issue Type: Sub-task Components: scripts Reporter: Jonathan Hsieh Assignee: David S. Wang Attachments: HBASE-5645.patch See https://builds.apache.org/job/PreCommit-HBASE-Build/1313//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Fix the warnings in the correctness section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5620) Convert the client protocol of HRegionInterface to PB
[ https://issues.apache.org/jira/browse/HBASE-5620?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13245888#comment-13245888 ] jirapos...@reviews.apache.org commented on HBASE-5620: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4629/ --- Review request for hbase. Summary --- This is the client protocol part of region interface. The admin protocol part will be done in a different jira. The HRegionInterface is still there since the admin part is not done yet. The other reason is that in case some people still wants the old interface Filters, comparators and coprocessor parameters are still Writable. They will be addressed in different jiras. The existing client interface is not changed so that we don't break existing clients. This addresses bug HBASE-5620. https://issues.apache.org/jira/browse/HBASE-5620 Diffs - src/main/java/org/apache/hadoop/hbase/HConstants.java 21ac4ba src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java 0129ee9 src/main/java/org/apache/hadoop/hbase/client/ClientScanner.java 97293aa src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 src/main/java/org/apache/hadoop/hbase/client/HConnection.java 5d43086 src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa30969 src/main/java/org/apache/hadoop/hbase/client/HTable.java 3db0295 src/main/java/org/apache/hadoop/hbase/client/ScannerCallable.java 9903df3 src/main/java/org/apache/hadoop/hbase/client/ServerCallable.java ddcf9ad src/main/java/org/apache/hadoop/hbase/filter/ParseConstants.java 1acbdab src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java cbfa489 src/main/java/org/apache/hadoop/hbase/io/TimeRange.java d135393 src/main/java/org/apache/hadoop/hbase/ipc/ExecRPCInvoker.java 05ae717 src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java f1f06b0 src/main/java/org/apache/hadoop/hbase/ipc/RegionProtocols.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/ipc/WritableRpcEngine.java 1316457 src/main/java/org/apache/hadoop/hbase/mapreduce/LoadIncrementalHFiles.java 19ae18c src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java 2eb57de src/main/java/org/apache/hadoop/hbase/protobuf/RegionAdminProtocol.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/protobuf/RegionClientProtocol.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/protobuf/ResponseConverter.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/protobuf/generated/HBaseProtos.java 4026da0 src/main/java/org/apache/hadoop/hbase/protobuf/generated/RegionClientProtos.java b36a9c0 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 4f80999 src/main/java/org/apache/hadoop/hbase/regionserver/HRegionThriftServer.java 703e73d src/main/java/org/apache/hadoop/hbase/regionserver/Leases.java b520f3f src/main/java/org/apache/hadoop/hbase/regionserver/RegionServer.java PRE-CREATION src/main/protobuf/RegionClient.proto 358382b src/main/protobuf/hbase.proto da78788 src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java c7284dc src/test/java/org/apache/hadoop/hbase/catalog/TestMetaReaderEditorNoCluster.java 0042468 src/test/java/org/apache/hadoop/hbase/client/HConnectionTestingUtility.java e34d8bc src/test/java/org/apache/hadoop/hbase/io/TestHbaseObjectWritable.java f2f8ee3 src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java d2b3060 src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 91dce36 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 227c5f2 src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java ceca6f5 src/test/java/org/apache/hadoop/hbase/regionserver/OOMERegionServer.java cac2989 src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegionServerBulkLoad.java a1bf73b Diff: https://reviews.apache.org/r/4629/diff Testing --- mvn -PrunAllTests clean test is green, except some flaky tests which need to run again. Also tested it on a real cluster with ycsb and bigtop. Thanks, Jimmy Convert the client protocol of HRegionInterface to PB - Key: HBASE-5620 URL: https://issues.apache.org/jira/browse/HBASE-5620 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Jimmy Xiang Assignee: Jimmy Xiang Fix For: 0.96.0 -- This message is automatically generated by JIRA. If you think it
[jira] [Commented] (HBASE-5709) Move active master and backup master znodes to use pbs
[ https://issues.apache.org/jira/browse/HBASE-5709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13245893#comment-13245893 ] jirapos...@reviews.apache.org commented on HBASE-5709: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4630/ --- Review request for hbase. Summary --- Serialize active master using pb. Do same for backup masters. D src/main/java/org/apache/hadoop/hbase/MasterAddressTracker.java Moved under zookeeper. All other Trackers are under zookeeper but this one. M src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java Use new utility method for getting current master that is over in MasterAddressTracker. M src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java Move dup code out into a new handle function. watcher.masterAddressZNode is no longer public. Use accessor. Use new setMasterAddress,getMasterAddress function on MasterAddressTracker for setting znode and backup master znode. M src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java Generated code. M src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java Fix up import because MasterAddressTracker moved. A src/main/java/org/apache/hadoop/hbase/zookeeper/MasterAddressTracker.java Add back MasterAddressTracker here. Add some utility functions for setting and getting the master address. M src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java Move dataToServerName into ZKUtil. Its useful beyond this class only. M src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java Go via accessor to get master znode path and use new utility method to get current master address. dataToServerName moved here from RootRegionTracker. M src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java Shutdown access to masterAddressZNode. Add an accessor method. M src/main/protobuf/ZooKeeper.proto Add new master message class. M src/test/java/org/apache/hadoop/hbase/master/TestActiveMasterManager.java M src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java Go via new accessor and use utility over in MasterAddressTracker. M src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java Fix import to point at new location. This addresses bug hbase-5709. https://issues.apache.org/jira/browse/hbase-5709 Diffs - src/main/java/org/apache/hadoop/hbase/MasterAddressTracker.java a246abe src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa30969 src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java ef19cb9 src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java 8ff87fe src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 4f80999 src/main/java/org/apache/hadoop/hbase/zookeeper/MasterAddressTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java 6b2ea57 src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 037e354 src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 0f83655 src/main/protobuf/ZooKeeper.proto 20f8eb0 src/test/java/org/apache/hadoop/hbase/master/TestActiveMasterManager.java 05f6b1a src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java e91d83c src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java 8478260 Diff: https://reviews.apache.org/r/4630/diff Testing --- Thanks, Michael Move active master and backup master znodes to use pbs -- Key: HBASE-5709 URL: https://issues.apache.org/jira/browse/HBASE-5709 Project: HBase Issue Type: Task Reporter: stack Assignee: stack Attachments: 5709.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5645) [findbugs] Fix correctness warnings
[ https://issues.apache.org/jira/browse/HBASE-5645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13245898#comment-13245898 ] jirapos...@reviews.apache.org commented on HBASE-5645: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4612/#review --- Ship it! Looks good to me. dev-support/test-patch.properties https://reviews.apache.org/r/4612/#comment14421 Hurray! - Michael On 2012-04-03 22:54:54, David Wang wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4612/ bq. --- bq. bq. (Updated 2012-04-03 22:54:54) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Fixes for issues reported in findbugs for Correctness category. bq. bq. bq. This addresses bug HBASE-5645. bq. https://issues.apache.org/jira/browse/HBASE-5645 bq. bq. bq. Diffs bq. - bq. bq.dev-support/test-patch.properties 2209d27 bq. src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon ae76204 bq.src/main/java/org/apache/hadoop/hbase/io/hfile/HFile.java 37bcaba bq.src/main/java/org/apache/hadoop/hbase/mapreduce/CellCounter.java 32d66fb bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 64def15 bq.src/main/java/org/apache/hadoop/hbase/master/DefaultLoadBalancer.java 1676832 bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 9bd4ace bq.src/main/java/org/apache/hadoop/hbase/master/ServerManager.java 3515d4a bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 8ae60a3 bq.src/main/java/org/apache/hadoop/hbase/regionserver/MemStoreLAB.java 2694897 bq.src/main/java/org/apache/hadoop/hbase/regionserver/ShutdownHook.java e3b230e bq.src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 0c7b396 bq.src/main/java/org/apache/hadoop/hbase/regionserver/StoreFile.java 2e98b39 bq.src/main/java/org/apache/hadoop/hbase/regionserver/wal/HLog.java 698bb3d bq. src/main/java/org/apache/hadoop/hbase/replication/regionserver/ReplicationSource.java 8950c9f bq.src/main/java/org/apache/hadoop/hbase/util/FSTableDescriptors.java 62cf6ac bq.src/main/java/org/apache/hadoop/hbase/util/FSUtils.java aebe5b0 bq.src/main/java/org/apache/hadoop/hbase/util/JVMClusterUtil.java c21377c bq.src/main/java/org/apache/hadoop/hbase/util/Merge.java 04f15d4 bq.src/main/java/org/apache/hadoop/hbase/util/PoolMap.java 8e2a856 bq. bq. Diff: https://reviews.apache.org/r/4612/diff bq. bq. bq. Testing bq. --- bq. bq. Ran test-patch.sh and unit tests. bq. bq. bq. Thanks, bq. bq. David bq. bq. [findbugs] Fix correctness warnings --- Key: HBASE-5645 URL: https://issues.apache.org/jira/browse/HBASE-5645 Project: HBase Issue Type: Sub-task Components: scripts Reporter: Jonathan Hsieh Assignee: David S. Wang Attachments: HBASE-5645.patch See https://builds.apache.org/job/PreCommit-HBASE-Build/1313//artifact/trunk/patchprocess/newPatchFindbugsWarnings.html Fix the warnings in the correctness section. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5709) Move active master and backup master znodes to use pbs
[ https://issues.apache.org/jira/browse/HBASE-5709?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13245922#comment-13245922 ] jirapos...@reviews.apache.org commented on HBASE-5709: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4630/#review6667 --- For ServerName, port and startcode are unassigned in pb, how do you handle those -1 default values? For example, hbck uses -1 port. - Jimmy On 2012-04-03 23:38:15, Michael Stack wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4630/ bq. --- bq. bq. (Updated 2012-04-03 23:38:15) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Serialize active master using pb. Do same for backup masters. bq. bq. D src/main/java/org/apache/hadoop/hbase/MasterAddressTracker.java bq.Moved under zookeeper. All other Trackers are under zookeeper but this one. bq. M src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java bq.Use new utility method for getting current master that is over in MasterAddressTracker. bq. M src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java bq.Move dup code out into a new handle function. bq.watcher.masterAddressZNode is no longer public. Use accessor. bq.Use new setMasterAddress,getMasterAddress function on bq.MasterAddressTracker for setting znode and backup master znode. bq. M src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java bq.Generated code. bq. M src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java bq.Fix up import because MasterAddressTracker moved. bq. A src/main/java/org/apache/hadoop/hbase/zookeeper/MasterAddressTracker.java bq.Add back MasterAddressTracker here. Add some utility functions for setting bq.and getting the master address. bq. M src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java bq.Move dataToServerName into ZKUtil. Its useful beyond this class only. bq. M src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java bq.Go via accessor to get master znode path and use new utility method bq.to get current master address. dataToServerName moved here from bq.RootRegionTracker. bq. M src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java bq.Shutdown access to masterAddressZNode. Add an accessor method. bq. M src/main/protobuf/ZooKeeper.proto bq.Add new master message class. bq. M src/test/java/org/apache/hadoop/hbase/master/TestActiveMasterManager.java bq. M src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java bq.Go via new accessor and use utility over in MasterAddressTracker. bq. M src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java bq.Fix import to point at new location. bq. bq. bq. This addresses bug hbase-5709. bq. https://issues.apache.org/jira/browse/hbase-5709 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/MasterAddressTracker.java a246abe bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java aa30969 bq.src/main/java/org/apache/hadoop/hbase/master/ActiveMasterManager.java ef19cb9 bq. src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java 8ff87fe bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 4f80999 bq. src/main/java/org/apache/hadoop/hbase/zookeeper/MasterAddressTracker.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java 6b2ea57 bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZKUtil.java 037e354 bq.src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 0f83655 bq.src/main/protobuf/ZooKeeper.proto 20f8eb0 bq. src/test/java/org/apache/hadoop/hbase/master/TestActiveMasterManager.java 05f6b1a bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestMasterAddressManager.java e91d83c bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestRSStatusServlet.java 8478260 bq. bq. Diff: https://reviews.apache.org/r/4630/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Michael bq. bq. Move active master and backup master znodes to use pbs -- Key: HBASE-5709 URL: https://issues.apache.org/jira/browse/HBASE-5709 Project: HBase Issue Type: Task Reporter: stack
[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode
[ https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13245952#comment-13245952 ] jirapos...@reviews.apache.org commented on HBASE-5547: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4633/ --- Review request for hbase, Michael Stack and Lars Hofhansl. Summary --- Essentially, whenever an hfile would be deleted, it is instead moved to the archive directory. In this impl, the archive directory is on a per table basis, but defaults to '.archive'. Removing hfiles occurs in three places - compaction, merge and catalog janitor. The former and two latter are distinctly different code paths, but but did pull out some similarities. The latter two end up calling the same method, so there should be a reasonable amount of overlap. Implementation wise: Updated the HMasterInterface to pass the calls onto the zookeeper. Added a zk listener to handle updates from the master to the RS to backup. Added a utility for removing files and finding archive directories Added tests for the regionserver and catalogjanitor approaches. Added creation of manager in regionserver. This addresses bug HBASE-5547. https://issues.apache.org/jira/browse/HBASE-5547 Diffs - src/main/java/org/apache/hadoop/hbase/HConstants.java 21ac4ba src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java 16e4017 src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java ea7ae45 src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java 79d5fdd src/main/java/org/apache/hadoop/hbase/master/HMaster.java 9bd4ace src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java 1d11f8f src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 4f80999 src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java 6884d53 src/main/java/org/apache/hadoop/hbase/regionserver/Store.java 0c7b396 src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java PRE-CREATION src/main/java/org/apache/hadoop/hbase/zookeeper/ZooKeeperWatcher.java 0f83655 src/main/resources/hbase-default.xml 341431a src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java d2b3060 src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java 227c5f2 src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionHFileArchiving.java PRE-CREATION src/test/java/org/apache/hadoop/hbase/util/MockRegionServerServices.java bb3ddd7 Diff: https://reviews.apache.org/r/4633/diff Testing --- Added two tests for the separate cases - archiving via the regionserver and for the catalog tracker. Former runs in a mini cluster and also touches the changes to HMasterInterface and zookeeper. Thanks, Jesse Don't delete HFiles when in backup mode - Key: HBASE-5547 URL: https://issues.apache.org/jira/browse/HBASE-5547 Project: HBase Issue Type: New Feature Reporter: Lars Hofhansl Assignee: Jesse Yates This came up in a discussion I had with Stack. It would be nice if HBase could be notified that a backup is in progress (via a znode for example) and in that case either: 1. rename HFiles to be delete to file.bck 2. rename the HFiles into a special directory 3. rename them to a general trash directory (which would not need to be tied to backup mode). That way it should be able to get a consistent backup based on HFiles (HDFS snapshots or hard links would be better options here, but we do not have those). #1 makes cleanup a bit harder. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-4348) Add metrics for regions in transition
[ https://issues.apache.org/jira/browse/HBASE-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13245962#comment-13245962 ] jirapos...@reviews.apache.org commented on HBASE-4348: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4402/#review6669 --- src/main/java/org/apache/hadoop/hbase/HConstants.java https://reviews.apache.org/r/4402/#comment14427 Jon: This is not a refresh metrics per se. Its for defining what a user considers to be a _longer_ period for a Region to be in a transition state, and show these regions count as a metric. So, it can be hbase.metrics.rit.stuck.warning.threshold? Long name, though. Please let me know amd I will change accordingly and revise the patch. src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java https://reviews.apache.org/r/4402/#comment14428 They are single line if statement, but will do. src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java https://reviews.apache.org/r/4402/#comment14429 Not following, sorry. Please explain. src/main/java/org/apache/hadoop/hbase/master/HMaster.java https://reviews.apache.org/r/4402/#comment14430 ok, will do. - Himanshu On 2012-03-30 05:21:12, Himanshu Vashishtha wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4402/ bq. --- bq. bq. (Updated 2012-03-30 05:21:12) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. This patch is for adding Region in transition metrics to the HMaster metrics system. It also adds these metrics in the master ui, in the Region in transition section. I have attached the proposed new format in the jira 4348. bq. bq. bq. This addresses bug HBase-4348. bq. https://issues.apache.org/jira/browse/HBase-4348 bq. bq. bq. Diffs bq. - bq. bq. src/main/jamon/org/apache/hadoop/hbase/tmpl/master/AssignmentManagerStatusTmpl.jamon 0dc0691 bq.src/main/java/org/apache/hadoop/hbase/HConstants.java 21ac4ba bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 64def15 bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 9bd4ace bq.src/main/java/org/apache/hadoop/hbase/master/metrics/MasterMetrics.java 83abc52 bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 91dce36 bq. bq. Diff: https://reviews.apache.org/r/4402/diff bq. bq. bq. Testing bq. --- bq. bq. Ran on a 5 node cluster and kill region servers randomly to observe the changes in the RIT metrics as emitted out by the Master's mxbean; bq. bq. mvn test passes without any failure. bq. bq. bq. Thanks, bq. bq. Himanshu bq. bq. Add metrics for regions in transition - Key: HBASE-4348 URL: https://issues.apache.org/jira/browse/HBASE-4348 Project: HBase Issue Type: Improvement Components: metrics Affects Versions: 0.92.0 Reporter: Todd Lipcon Assignee: Himanshu Vashishtha Priority: Minor Labels: noob Fix For: 0.96.0 Attachments: 4348-metrics-v3.patch, 4348-v1.patch, 4348-v2.patch, RITs.png, RegionInTransitions2.png, metrics-v2.patch The following metrics would be useful for monitoring the master: - the number of regions in transition - the number of regions in transition that have been in transition for more than a minute - how many seconds has the oldest region-in-transition been in transition -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5547) Don't delete HFiles when in backup mode
[
https://issues.apache.org/jira/browse/HBASE-5547?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13245987#comment-13245987
]
jirapos...@reviews.apache.org commented on HBASE-5547:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4633/#review6672
---
src/main/java/org/apache/hadoop/hbase/HConstants.java
https://reviews.apache.org/r/4633/#comment14456
There is no mentioning of HFile in the name of znode.
What if another archive scheme comes up in the future ?
src/main/java/org/apache/hadoop/hbase/ipc/HMasterInterface.java
https://reviews.apache.org/r/4633/#comment14433
What is the counterpart for disableHFileBackup() where backup for all
tables is enabled ?
src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java
https://reviews.apache.org/r/4633/#comment14435
License.
src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java
https://reviews.apache.org/r/4633/#comment14436
'or not' is not needed.
src/main/java/org/apache/hadoop/hbase/regionserver/HFileArchiveManager.java
https://reviews.apache.org/r/4633/#comment14437
Remove empty line.
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
https://reviews.apache.org/r/4633/#comment14438
white space.
src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java
https://reviews.apache.org/r/4633/#comment14441
Should we log if rss == null ?
Since in that case we cannot perform desired archiving.
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
https://reviews.apache.org/r/4633/#comment14439
Should they be named hFileArchive{Manager,Tracker} ?
src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java
https://reviews.apache.org/r/4633/#comment14440
The leading 'get ' is not necessary.
src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
https://reviews.apache.org/r/4633/#comment14442
License.
src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
https://reviews.apache.org/r/4633/#comment14443
Please check the return value from mkdirs().
src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
https://reviews.apache.org/r/4633/#comment1
Please check the return value.
Should resolveAndCopyFile() be enclosed in a try block ?
src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
https://reviews.apache.org/r/4633/#comment14445
The code at line 92 doesn't 'keep' HFiles.
src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
https://reviews.apache.org/r/4633/#comment14446
This doesn't match the fs.rename() call.
src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
https://reviews.apache.org/r/4633/#comment14447
What about non-directory files ?
src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
https://reviews.apache.org/r/4633/#comment14448
Should include file in log message.
src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
https://reviews.apache.org/r/4633/#comment14449
Move this to line 197.
src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
https://reviews.apache.org/r/4633/#comment14450
Should this be contingent on success being true ?
src/main/java/org/apache/hadoop/hbase/util/HFileArchiveUtil.java
https://reviews.apache.org/r/4633/#comment14451
Move else if to line 219 and enclose LOG.debug in curly braces.
src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment14434
Add license, please.
src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment14452
Should this method return a boolean which indicates whether the tracker is
enabled ?
src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment14453
variable table should be included in the log.
src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment14454
Remove first 'the'.
src/main/java/org/apache/hadoop/hbase/zookeeper/HFileArchiveTracker.java
https://reviews.apache.org/r/4633/#comment14455
Why do we return when archiveHFileZNode exists ?
- Ted
On 2012-04-04 01:23:29, Jesse Yates wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4633/
bq. ---
bq.
bq. (Updated 2012-04-04 01:23:29)
bq.
bq.
bq. Review request for hbase, Michael Stack and
[jira] [Commented] (HBASE-5451) Switch RPC call envelope/headers to PBs
[ https://issues.apache.org/jira/browse/HBASE-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244006#comment-13244006 ] jirapos...@reviews.apache.org commented on HBASE-5451: -- bq. On 2012-04-02 00:21:20, Michael Stack wrote: bq. Some more questions. Just being careful DD. That's fine. Hope the answers below are okay. Please let me know your response soon so that I can submit another patch. bq. On 2012-04-02 00:21:20, Michael Stack wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/DataOutputOutputStream.java, line 25 bq. https://reviews.apache.org/r/4096/diff/3/?file=97739#file97739line25 bq. bq. We should just be using the hadoop DOOS... looks like no diff (when I diff them). I'll make an issue to remove. cool bq. On 2012-04-02 00:21:20, Michael Stack wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java, line 446 bq. https://reviews.apache.org/r/4096/diff/3/?file=97740#file97740line446 bq. bq. Is this written up anywhere? That its hrpc, then version, then a length, then a protobuf? bq. bq. I see it in the proto definition. That'll do. cool bq. On 2012-04-02 00:21:20, Michael Stack wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java, line 548 bq. https://reviews.apache.org/r/4096/diff/3/?file=97740#file97740line548 bq. bq. We have an issue for removing this Invocation stuff? No not yet. But I'll create one to do with this issue once this patch is committed. bq. On 2012-04-02 00:21:20, Michael Stack wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto, line 25 bq. https://reviews.apache.org/r/4096/diff/3/?file=97744#file97744line25 bq. bq. Should we just remove them in the next iteration on rpc since 0.96 is to be a singularity? Why even bother trying to keep compatibility w/ older clients? bq. bq. What is 'failure compatibility'? We are telling the client to go away, nicely (smile). bq. bq. What you think we should replace hrpc0x0005 with? bq. bq. this - these Yeah, valid points. We can remove this version string and all in a follow up patch. bq. On 2012-04-02 00:21:20, Michael Stack wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto, line 28 bq. https://reviews.apache.org/r/4096/diff/3/?file=97744#file97744line28 bq. bq. How does RpcRequestWithHeaderProto relate to ConnectionHeaderProto? This text should say? bq. bq. Would be nice to have illustration on how the back and forth work. The latter is used only while establishing connections and the former for exchanging RPC requests/responses over a channel that is connected. Okay, will add some text. bq. On 2012-04-02 00:21:20, Michael Stack wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto, line 55 bq. https://reviews.apache.org/r/4096/diff/3/?file=97744#file97744line55 bq. bq. We'll send this String each time? Actually, I could make this field 'optional' since this has a default value. Will do so. bq. On 2012-04-02 00:21:20, Michael Stack wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto, line 66 bq. https://reviews.apache.org/r/4096/diff/3/?file=97744#file97744line66 bq. bq. Which part in here is the 'header'? How does it relate to ConnectionHeaderProto? bq. bq. request can be an Invocation/Writable? Or a protobuf? Do we need a length in here? Today the only 'header' is the callId.. There is no relation to ConnectionHeaderProto. If the 'header' is confusing, I can take it off the object name. Let me know. 'request' in this patch is only a Invocation/Writable. In theory, it could be a protobuf object as well (since it is just bytes), but, for protobuf, we could make things more explicit by defining a protobuf object rather than a opaque set of bytes. But that's another jira (ProtoBufRpcEngine implementation similar to Hadoop). Length is not needed - the protobuf serialization/deserialization will take care of it.. bq. On 2012-04-02 00:21:20, Michael Stack wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto, line 93 bq. https://reviews.apache.org/r/4096/diff/3/?file=97744#file97744line93 bq. bq. Should this precede the response? So if false, a response follows else an exception? Do we need a length here? Where is the header that the message name refers too? Length will be taken care of by the protobuf serialization/deserialization. The header is the combination of callId, error. If the 'header' is confusing,
[jira] [Commented] (HBASE-5625) Avoid byte buffer allocations when reading a value from a Result object
[ https://issues.apache.org/jira/browse/HBASE-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244219#comment-13244219 ] jirapos...@reviews.apache.org commented on HBASE-5625: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4607/ --- Review request for hbase. Summary --- When calling Result.getValue(), an extra dummy KeyValue and its associated underlying byte array are allocated, as well as a persistent buffer that will contain the returned value. These can be avoided by reusing a static array for the dummy object and by passing a ByteBuffer object as a value destination buffer to the read method. This addresses bug HBASE-5625. https://issues.apache.org/jira/browse/HBASE-5625 Diffs - src/main/java/org/apache/hadoop/hbase/KeyValue.java 243d76f src/main/java/org/apache/hadoop/hbase/client/Result.java df0b3ef src/test/java/org/apache/hadoop/hbase/client/TestResult.java f9e29c2 Diff: https://reviews.apache.org/r/4607/diff Testing --- Added value check to TestResult#testBasic and TestResult.testMultiVersion. Thanks, Tudor Avoid byte buffer allocations when reading a value from a Result object --- Key: HBASE-5625 URL: https://issues.apache.org/jira/browse/HBASE-5625 Project: HBase Issue Type: Improvement Components: client Affects Versions: 0.92.1 Reporter: Tudor Scurtu Assignee: Tudor Scurtu Labels: patch Attachments: 5625.txt, 5625v2.txt, 5625v3.txt, 5625v4.txt When calling Result.getValue(), an extra dummy KeyValue and its associated underlying byte array are allocated, as well as a persistent buffer that will contain the returned value. These can be avoided by reusing a static array for the dummy object and by passing a ByteBuffer object as a value destination buffer to the read method. The current functionality is maintained, and we have added a separate method call stack that employs the described changes. I will provide more details with the patch. Running tests with a profiler, the reduction of read time seems to be of up to 40%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5451) Switch RPC call envelope/headers to PBs
[ https://issues.apache.org/jira/browse/HBASE-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244255#comment-13244255 ] jirapos...@reviews.apache.org commented on HBASE-5451: -- bq. On 2012-04-02 00:21:20, Michael Stack wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java, line 548 bq. https://reviews.apache.org/r/4096/diff/3/?file=97740#file97740line548 bq. bq. We have an issue for removing this Invocation stuff? bq. bq. Devaraj Das wrote: bq. No not yet. But I'll create one to do with this issue once this patch is committed. Thanks bq. On 2012-04-02 00:21:20, Michael Stack wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto, line 25 bq. https://reviews.apache.org/r/4096/diff/3/?file=97744#file97744line25 bq. bq. Should we just remove them in the next iteration on rpc since 0.96 is to be a singularity? Why even bother trying to keep compatibility w/ older clients? bq. bq. What is 'failure compatibility'? We are telling the client to go away, nicely (smile). bq. bq. What you think we should replace hrpc0x0005 with? bq. bq. this - these bq. bq. Devaraj Das wrote: bq. Yeah, valid points. We can remove this version string and all in a follow up patch. Lets discuss in another jira. A hrpc version followed by something that says its protobuf that follows, etc., bq. On 2012-04-02 00:21:20, Michael Stack wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto, line 28 bq. https://reviews.apache.org/r/4096/diff/3/?file=97744#file97744line28 bq. bq. How does RpcRequestWithHeaderProto relate to ConnectionHeaderProto? This text should say? bq. bq. Would be nice to have illustration on how the back and forth work. bq. bq. Devaraj Das wrote: bq. The latter is used only while establishing connections and the former for exchanging RPC requests/responses over a channel that is connected. Okay, will add some text. Thanks bq. On 2012-04-02 00:21:20, Michael Stack wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto, line 55 bq. https://reviews.apache.org/r/4096/diff/3/?file=97744#file97744line55 bq. bq. We'll send this String each time? bq. bq. Devaraj Das wrote: bq. Actually, I could make this field 'optional' since this has a default value. Will do so. That'd be a good idea I think. The other protocols are less used and can include the String bq. On 2012-04-02 00:21:20, Michael Stack wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto, line 66 bq. https://reviews.apache.org/r/4096/diff/3/?file=97744#file97744line66 bq. bq. Which part in here is the 'header'? How does it relate to ConnectionHeaderProto? bq. bq. request can be an Invocation/Writable? Or a protobuf? Do we need a length in here? bq. bq. Devaraj Das wrote: bq. Today the only 'header' is the callId.. There is no relation to ConnectionHeaderProto. If the 'header' is confusing, I can take it off the object name. Let me know. bq. bq. 'request' in this patch is only a Invocation/Writable. In theory, it could be a protobuf object as well (since it is just bytes), but, for protobuf, we could make things more explicit by defining a protobuf object rather than a opaque set of bytes. But that's another jira (ProtoBufRpcEngine implementation similar to Hadoop). Length is not needed - the protobuf serialization/deserialization will take care of it.. I think taking the 'Header' off Request/Response would be best (Did I ask you add it previous? If so, sorry... I misunderstood. Thanks for being accomodating). Yes, on a new issue to make it pb rather than opaque bytes. Do you have to do something here -- make bytes optional? -- to allow for the later pb replacement? On length, thats probably good to keep. For us, we'll give the stream to a pb deserializer but other clients might want to know how many bytes on the line so keep it I'd say. bq. On 2012-04-02 00:21:20, Michael Stack wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto, line 93 bq. https://reviews.apache.org/r/4096/diff/3/?file=97744#file97744line93 bq. bq. Should this precede the response? So if false, a response follows else an exception? Do we need a length here? Where is the header that the message name refers too? bq. bq. Devaraj Das wrote: bq. Length will be taken care of by the protobuf serialization/deserialization. The header is the combination of callId, error. If the 'header' is confusing, I can take it off the object name. Let me know. Yeah, take away
[jira] [Commented] (HBASE-5625) Avoid byte buffer allocations when reading a value from a Result object
[ https://issues.apache.org/jira/browse/HBASE-5625?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244324#comment-13244324 ] jirapos...@reviews.apache.org commented on HBASE-5625: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4607/#review6622 --- I ran TestTableMapReduce and TestMultithreadedTableMapper with patch v5. They passed. Some minor comments below. src/main/java/org/apache/hadoop/hbase/KeyValue.java https://reviews.apache.org/r/4607/#comment14303 Please include vlength in the exception message src/main/java/org/apache/hadoop/hbase/KeyValue.java https://reviews.apache.org/r/4607/#comment14304 Should read 'BufferOverflowException if there' src/main/java/org/apache/hadoop/hbase/KeyValue.java https://reviews.apache.org/r/4607/#comment14305 Add a space between comma and fl. src/main/java/org/apache/hadoop/hbase/client/Result.java https://reviews.apache.org/r/4607/#comment14306 Is this comment needed ? src/main/java/org/apache/hadoop/hbase/client/Result.java https://reviews.apache.org/r/4607/#comment14307 This line can be removed. src/test/java/org/apache/hadoop/hbase/client/TestResult.java https://reviews.apache.org/r/4607/#comment14308 white space. src/test/java/org/apache/hadoop/hbase/client/TestResult.java https://reviews.apache.org/r/4607/#comment14309 Since benchmarking is hard to do, this test case can be dropped. - Ted On 2012-04-02 14:22:48, Tudor Scurtu wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4607/ bq. --- bq. bq. (Updated 2012-04-02 14:22:48) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. When calling Result.getValue(), an extra dummy KeyValue and its associated underlying byte array are allocated, as well as a persistent buffer that will contain the returned value. bq. bq. These can be avoided by reusing a static array for the dummy object and by passing a ByteBuffer object as a value destination buffer to the read method. bq. bq. bq. This addresses bug HBASE-5625. bq. https://issues.apache.org/jira/browse/HBASE-5625 bq. bq. bq. Diffs bq. - bq. bq.src/main/java/org/apache/hadoop/hbase/KeyValue.java 243d76f bq.src/main/java/org/apache/hadoop/hbase/client/Result.java df0b3ef bq.src/test/java/org/apache/hadoop/hbase/client/TestResult.java f9e29c2 bq. bq. Diff: https://reviews.apache.org/r/4607/diff bq. bq. bq. Testing bq. --- bq. bq. Added value check to TestResult#testBasic and TestResult.testMultiVersion. bq. bq. bq. Thanks, bq. bq. Tudor bq. bq. Avoid byte buffer allocations when reading a value from a Result object --- Key: HBASE-5625 URL: https://issues.apache.org/jira/browse/HBASE-5625 Project: HBase Issue Type: Improvement Components: client Affects Versions: 0.92.1 Reporter: Tudor Scurtu Assignee: Tudor Scurtu Labels: patch Attachments: 5625.txt, 5625v2.txt, 5625v3.txt, 5625v4.txt, 5625v5.txt When calling Result.getValue(), an extra dummy KeyValue and its associated underlying byte array are allocated, as well as a persistent buffer that will contain the returned value. These can be avoided by reusing a static array for the dummy object and by passing a ByteBuffer object as a value destination buffer to the read method. The current functionality is maintained, and we have added a separate method call stack that employs the described changes. I will provide more details with the patch. Running tests with a profiler, the reduction of read time seems to be of up to 40%. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5451) Switch RPC call envelope/headers to PBs
[ https://issues.apache.org/jira/browse/HBASE-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244362#comment-13244362 ] jirapos...@reviews.apache.org commented on HBASE-5451: -- bq. On 2012-04-02 00:21:20, Michael Stack wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto, line 93 bq. https://reviews.apache.org/r/4096/diff/3/?file=97744#file97744line93 bq. bq. Should this precede the response? So if false, a response follows else an exception? Do we need a length here? Where is the header that the message name refers too? bq. bq. Devaraj Das wrote: bq. Length will be taken care of by the protobuf serialization/deserialization. The header is the combination of callId, error. If the 'header' is confusing, I can take it off the object name. Let me know. bq. bq. Michael Stack wrote: bq. Yeah, take away the header. Length I think is good. Makes it more robust (IIRC, we went out of our way to add length to the old RPC to help clients figure how much to pull). The argument above for 'length' applies here too... bq. On 2012-04-02 00:21:20, Michael Stack wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto, line 66 bq. https://reviews.apache.org/r/4096/diff/3/?file=97744#file97744line66 bq. bq. Which part in here is the 'header'? How does it relate to ConnectionHeaderProto? bq. bq. request can be an Invocation/Writable? Or a protobuf? Do we need a length in here? bq. bq. Devaraj Das wrote: bq. Today the only 'header' is the callId.. There is no relation to ConnectionHeaderProto. If the 'header' is confusing, I can take it off the object name. Let me know. bq. bq. 'request' in this patch is only a Invocation/Writable. In theory, it could be a protobuf object as well (since it is just bytes), but, for protobuf, we could make things more explicit by defining a protobuf object rather than a opaque set of bytes. But that's another jira (ProtoBufRpcEngine implementation similar to Hadoop). Length is not needed - the protobuf serialization/deserialization will take care of it.. bq. bq. Michael Stack wrote: bq. I think taking the 'Header' off Request/Response would be best (Did I ask you add it previous? If so, sorry... I misunderstood. Thanks for being accomodating). Yes, on a new issue to make it pb rather than opaque bytes. Do you have to do something here -- make bytes optional? -- to allow for the later pb replacement? bq. bq. On length, thats probably good to keep. For us, we'll give the stream to a pb deserializer but other clients might want to know how many bytes on the line so keep it I'd say. Yes, I'll take off the 'header' from the message name. I could make the 'bytes' field optional. Actually, on the length, I am not sure I understand why we need it in the PB model. Generally speaking, clients talking to servers have to be aware of the PB encoding in order for them to make any sense of the PB data.. The PB type 'bytes' has the length taken care of in the implementation of serialization/deserialization internally. In that sense, I don't think having an explicit length field is required. Does this reasoning make sense? (Also note that the top level RPC request envelope has the length preceding the request data) - Devaraj --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4096/#review6613 --- On 2012-03-30 23:29:32, Devaraj Das wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4096/ bq. --- bq. bq. (Updated 2012-03-30 23:29:32) bq. bq. bq. Review request for Michael Stack and Benoit Sigoure. bq. bq. bq. Summary bq. --- bq. bq. Switch RPC call envelope/headers to PBs bq. bq. bq. This addresses bug HBASE-5451. bq. https://issues.apache.org/jira/browse/HBASE-5451 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/DataOutputOutputStream.java 1307644 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1307644 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 1307644 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCMessageProtos.java PRE-CREATION bq.
[jira] [Commented] (HBASE-5451) Switch RPC call envelope/headers to PBs
[ https://issues.apache.org/jira/browse/HBASE-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244365#comment-13244365 ] jirapos...@reviews.apache.org commented on HBASE-5451: -- bq. On 2012-04-02 00:21:20, Michael Stack wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto, line 66 bq. https://reviews.apache.org/r/4096/diff/3/?file=97744#file97744line66 bq. bq. Which part in here is the 'header'? How does it relate to ConnectionHeaderProto? bq. bq. request can be an Invocation/Writable? Or a protobuf? Do we need a length in here? bq. bq. Devaraj Das wrote: bq. Today the only 'header' is the callId.. There is no relation to ConnectionHeaderProto. If the 'header' is confusing, I can take it off the object name. Let me know. bq. bq. 'request' in this patch is only a Invocation/Writable. In theory, it could be a protobuf object as well (since it is just bytes), but, for protobuf, we could make things more explicit by defining a protobuf object rather than a opaque set of bytes. But that's another jira (ProtoBufRpcEngine implementation similar to Hadoop). Length is not needed - the protobuf serialization/deserialization will take care of it.. bq. bq. Michael Stack wrote: bq. I think taking the 'Header' off Request/Response would be best (Did I ask you add it previous? If so, sorry... I misunderstood. Thanks for being accomodating). Yes, on a new issue to make it pb rather than opaque bytes. Do you have to do something here -- make bytes optional? -- to allow for the later pb replacement? bq. bq. On length, thats probably good to keep. For us, we'll give the stream to a pb deserializer but other clients might want to know how many bytes on the line so keep it I'd say. bq. bq. Devaraj Das wrote: bq. Yes, I'll take off the 'header' from the message name. I could make the 'bytes' field optional. bq. bq. Actually, on the length, I am not sure I understand why we need it in the PB model. Generally speaking, clients talking to servers have to be aware of the PB encoding in order for them to make any sense of the PB data.. The PB type 'bytes' has the length taken care of in the implementation of serialization/deserialization internally. In that sense, I don't think having an explicit length field is required. Does this reasoning make sense? bq. bq. (Also note that the top level RPC request envelope has the length preceding the request data) If the top level rpc request envelope has the length, then I agree w/ you, its not needed as prefix on pb messages. bq. On 2012-04-02 00:21:20, Michael Stack wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto, line 93 bq. https://reviews.apache.org/r/4096/diff/3/?file=97744#file97744line93 bq. bq. Should this precede the response? So if false, a response follows else an exception? Do we need a length here? Where is the header that the message name refers too? bq. bq. Devaraj Das wrote: bq. Length will be taken care of by the protobuf serialization/deserialization. The header is the combination of callId, error. If the 'header' is confusing, I can take it off the object name. Let me know. bq. bq. Michael Stack wrote: bq. Yeah, take away the header. Length I think is good. Makes it more robust (IIRC, we went out of our way to add length to the old RPC to help clients figure how much to pull). bq. bq. Devaraj Das wrote: bq. The argument above for 'length' applies here too... Agreed. So high level, the response and request have a length of the total message? If so, don't need it down inside preceeding pb messages. - Michael --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4096/#review6613 --- On 2012-03-30 23:29:32, Devaraj Das wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4096/ bq. --- bq. bq. (Updated 2012-03-30 23:29:32) bq. bq. bq. Review request for Michael Stack and Benoit Sigoure. bq. bq. bq. Summary bq. --- bq. bq. Switch RPC call envelope/headers to PBs bq. bq. bq. This addresses bug HBASE-5451. bq. https://issues.apache.org/jira/browse/HBASE-5451 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/DataOutputOutputStream.java 1307644 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java
[jira] [Commented] (HBASE-5451) Switch RPC call envelope/headers to PBs
[ https://issues.apache.org/jira/browse/HBASE-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244368#comment-13244368 ] jirapos...@reviews.apache.org commented on HBASE-5451: -- bq. On 2012-04-02 00:21:20, Michael Stack wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto, line 66 bq. https://reviews.apache.org/r/4096/diff/3/?file=97744#file97744line66 bq. bq. Which part in here is the 'header'? How does it relate to ConnectionHeaderProto? bq. bq. request can be an Invocation/Writable? Or a protobuf? Do we need a length in here? bq. bq. Devaraj Das wrote: bq. Today the only 'header' is the callId.. There is no relation to ConnectionHeaderProto. If the 'header' is confusing, I can take it off the object name. Let me know. bq. bq. 'request' in this patch is only a Invocation/Writable. In theory, it could be a protobuf object as well (since it is just bytes), but, for protobuf, we could make things more explicit by defining a protobuf object rather than a opaque set of bytes. But that's another jira (ProtoBufRpcEngine implementation similar to Hadoop). Length is not needed - the protobuf serialization/deserialization will take care of it.. bq. bq. Michael Stack wrote: bq. I think taking the 'Header' off Request/Response would be best (Did I ask you add it previous? If so, sorry... I misunderstood. Thanks for being accomodating). Yes, on a new issue to make it pb rather than opaque bytes. Do you have to do something here -- make bytes optional? -- to allow for the later pb replacement? bq. bq. On length, thats probably good to keep. For us, we'll give the stream to a pb deserializer but other clients might want to know how many bytes on the line so keep it I'd say. bq. bq. Devaraj Das wrote: bq. Yes, I'll take off the 'header' from the message name. I could make the 'bytes' field optional. bq. bq. Actually, on the length, I am not sure I understand why we need it in the PB model. Generally speaking, clients talking to servers have to be aware of the PB encoding in order for them to make any sense of the PB data.. The PB type 'bytes' has the length taken care of in the implementation of serialization/deserialization internally. In that sense, I don't think having an explicit length field is required. Does this reasoning make sense? bq. bq. (Also note that the top level RPC request envelope has the length preceding the request data) bq. bq. Michael Stack wrote: bq. If the top level rpc request envelope has the length, then I agree w/ you, its not needed as prefix on pb messages. cool bq. On 2012-04-02 00:21:20, Michael Stack wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto, line 93 bq. https://reviews.apache.org/r/4096/diff/3/?file=97744#file97744line93 bq. bq. Should this precede the response? So if false, a response follows else an exception? Do we need a length here? Where is the header that the message name refers too? bq. bq. Devaraj Das wrote: bq. Length will be taken care of by the protobuf serialization/deserialization. The header is the combination of callId, error. If the 'header' is confusing, I can take it off the object name. Let me know. bq. bq. Michael Stack wrote: bq. Yeah, take away the header. Length I think is good. Makes it more robust (IIRC, we went out of our way to add length to the old RPC to help clients figure how much to pull). bq. bq. Devaraj Das wrote: bq. The argument above for 'length' applies here too... bq. bq. Michael Stack wrote: bq. Agreed. So high level, the response and request have a length of the total message? If so, don't need it down inside preceeding pb messages. I meant the argument on the PB encoding.. The RPC response envelope, even today, doesn't include the length. For instance, the client side of the method HBaseClient.receiveResponse starts with reading the callId. - Devaraj --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4096/#review6613 --- On 2012-03-30 23:29:32, Devaraj Das wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4096/ bq. --- bq. bq. (Updated 2012-03-30 23:29:32) bq. bq. bq. Review request for Michael Stack and Benoit Sigoure. bq. bq. bq. Summary bq. --- bq. bq. Switch RPC call envelope/headers to PBs bq. bq. bq. This addresses bug HBASE-5451. bq.
[jira] [Commented] (HBASE-5451) Switch RPC call envelope/headers to PBs
[ https://issues.apache.org/jira/browse/HBASE-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244377#comment-13244377 ] jirapos...@reviews.apache.org commented on HBASE-5451: -- bq. On 2012-04-02 00:21:20, Michael Stack wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto, line 93 bq. https://reviews.apache.org/r/4096/diff/3/?file=97744#file97744line93 bq. bq. Should this precede the response? So if false, a response follows else an exception? Do we need a length here? Where is the header that the message name refers too? bq. bq. Devaraj Das wrote: bq. Length will be taken care of by the protobuf serialization/deserialization. The header is the combination of callId, error. If the 'header' is confusing, I can take it off the object name. Let me know. bq. bq. Michael Stack wrote: bq. Yeah, take away the header. Length I think is good. Makes it more robust (IIRC, we went out of our way to add length to the old RPC to help clients figure how much to pull). bq. bq. Devaraj Das wrote: bq. The argument above for 'length' applies here too... bq. bq. Michael Stack wrote: bq. Agreed. So high level, the response and request have a length of the total message? If so, don't need it down inside preceeding pb messages. bq. bq. Devaraj Das wrote: bq. I meant the argument on the PB encoding.. bq. bq. The RPC response envelope, even today, doesn't include the length. For instance, the client side of the method HBaseClient.receiveResponse starts with reading the callId. Ok. We are replicating what was there previous. Lets make new jira for doing things like a length prefix. - Michael --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4096/#review6613 --- On 2012-03-30 23:29:32, Devaraj Das wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4096/ bq. --- bq. bq. (Updated 2012-03-30 23:29:32) bq. bq. bq. Review request for Michael Stack and Benoit Sigoure. bq. bq. bq. Summary bq. --- bq. bq. Switch RPC call envelope/headers to PBs bq. bq. bq. This addresses bug HBASE-5451. bq. https://issues.apache.org/jira/browse/HBASE-5451 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/DataOutputOutputStream.java 1307644 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1307644 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 1307644 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCMessageProtos.java PRE-CREATION bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java 1307644 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/4096/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Devaraj bq. bq. Switch RPC call envelope/headers to PBs --- Key: HBASE-5451 URL: https://issues.apache.org/jira/browse/HBASE-5451 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Affects Versions: 0.94.0 Reporter: Todd Lipcon Assignee: Devaraj Das Fix For: 0.96.0 Attachments: rpc-proto.2.txt, rpc-proto.3.txt, rpc-proto.patch.1_2, rpc-proto.r5.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5451) Switch RPC call envelope/headers to PBs
[ https://issues.apache.org/jira/browse/HBASE-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244378#comment-13244378 ] jirapos...@reviews.apache.org commented on HBASE-5451: -- bq. On 2012-04-02 00:21:20, Michael Stack wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto, line 93 bq. https://reviews.apache.org/r/4096/diff/3/?file=97744#file97744line93 bq. bq. Should this precede the response? So if false, a response follows else an exception? Do we need a length here? Where is the header that the message name refers too? bq. bq. Devaraj Das wrote: bq. Length will be taken care of by the protobuf serialization/deserialization. The header is the combination of callId, error. If the 'header' is confusing, I can take it off the object name. Let me know. bq. bq. Michael Stack wrote: bq. Yeah, take away the header. Length I think is good. Makes it more robust (IIRC, we went out of our way to add length to the old RPC to help clients figure how much to pull). bq. bq. Devaraj Das wrote: bq. The argument above for 'length' applies here too... bq. bq. Michael Stack wrote: bq. Agreed. So high level, the response and request have a length of the total message? If so, don't need it down inside preceeding pb messages. bq. bq. Devaraj Das wrote: bq. I meant the argument on the PB encoding.. bq. bq. The RPC response envelope, even today, doesn't include the length. For instance, the client side of the method HBaseClient.receiveResponse starts with reading the callId. bq. bq. Michael Stack wrote: bq. Ok. We are replicating what was there previous. Lets make new jira for doing things like a length prefix. Okay let's discuss that in a separate jira.. Otherwise, do you think the patch is good to go? If so, I'll submit a new patch with some of the comments incorporated. - Devaraj --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4096/#review6613 --- On 2012-03-30 23:29:32, Devaraj Das wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4096/ bq. --- bq. bq. (Updated 2012-03-30 23:29:32) bq. bq. bq. Review request for Michael Stack and Benoit Sigoure. bq. bq. bq. Summary bq. --- bq. bq. Switch RPC call envelope/headers to PBs bq. bq. bq. This addresses bug HBASE-5451. bq. https://issues.apache.org/jira/browse/HBASE-5451 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/DataOutputOutputStream.java 1307644 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1307644 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 1307644 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCMessageProtos.java PRE-CREATION bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java 1307644 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/4096/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Devaraj bq. bq. Switch RPC call envelope/headers to PBs --- Key: HBASE-5451 URL: https://issues.apache.org/jira/browse/HBASE-5451 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Affects Versions: 0.94.0 Reporter: Todd Lipcon Assignee: Devaraj Das Fix For: 0.96.0 Attachments: rpc-proto.2.txt, rpc-proto.3.txt, rpc-proto.patch.1_2, rpc-proto.r5.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5451) Switch RPC call envelope/headers to PBs
[ https://issues.apache.org/jira/browse/HBASE-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244389#comment-13244389 ] jirapos...@reviews.apache.org commented on HBASE-5451: -- bq. On 2012-04-02 00:21:20, Michael Stack wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto, line 93 bq. https://reviews.apache.org/r/4096/diff/3/?file=97744#file97744line93 bq. bq. Should this precede the response? So if false, a response follows else an exception? Do we need a length here? Where is the header that the message name refers too? bq. bq. Devaraj Das wrote: bq. Length will be taken care of by the protobuf serialization/deserialization. The header is the combination of callId, error. If the 'header' is confusing, I can take it off the object name. Let me know. bq. bq. Michael Stack wrote: bq. Yeah, take away the header. Length I think is good. Makes it more robust (IIRC, we went out of our way to add length to the old RPC to help clients figure how much to pull). bq. bq. Devaraj Das wrote: bq. The argument above for 'length' applies here too... bq. bq. Michael Stack wrote: bq. Agreed. So high level, the response and request have a length of the total message? If so, don't need it down inside preceeding pb messages. bq. bq. Devaraj Das wrote: bq. I meant the argument on the PB encoding.. bq. bq. The RPC response envelope, even today, doesn't include the length. For instance, the client side of the method HBaseClient.receiveResponse starts with reading the callId. bq. bq. Michael Stack wrote: bq. Ok. We are replicating what was there previous. Lets make new jira for doing things like a length prefix. bq. bq. Devaraj Das wrote: bq. Okay let's discuss that in a separate jira.. bq. bq. Otherwise, do you think the patch is good to go? If so, I'll submit a new patch with some of the comments incorporated. There items above you said you'd address such as removing Header from the request and response and cleaning up doc in the .proto file, right? - Michael --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4096/#review6613 --- On 2012-03-30 23:29:32, Devaraj Das wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4096/ bq. --- bq. bq. (Updated 2012-03-30 23:29:32) bq. bq. bq. Review request for Michael Stack and Benoit Sigoure. bq. bq. bq. Summary bq. --- bq. bq. Switch RPC call envelope/headers to PBs bq. bq. bq. This addresses bug HBASE-5451. bq. https://issues.apache.org/jira/browse/HBASE-5451 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/DataOutputOutputStream.java 1307644 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1307644 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 1307644 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCMessageProtos.java PRE-CREATION bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java 1307644 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/4096/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Devaraj bq. bq. Switch RPC call envelope/headers to PBs --- Key: HBASE-5451 URL: https://issues.apache.org/jira/browse/HBASE-5451 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Affects Versions: 0.94.0 Reporter: Todd Lipcon Assignee: Devaraj Das Fix For: 0.96.0 Attachments: rpc-proto.2.txt, rpc-proto.3.txt, rpc-proto.patch.1_2, rpc-proto.r5.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5451) Switch RPC call envelope/headers to PBs
[ https://issues.apache.org/jira/browse/HBASE-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244400#comment-13244400 ] jirapos...@reviews.apache.org commented on HBASE-5451: -- bq. On 2012-04-02 00:21:20, Michael Stack wrote: bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto, line 93 bq. https://reviews.apache.org/r/4096/diff/3/?file=97744#file97744line93 bq. bq. Should this precede the response? So if false, a response follows else an exception? Do we need a length here? Where is the header that the message name refers too? bq. bq. Devaraj Das wrote: bq. Length will be taken care of by the protobuf serialization/deserialization. The header is the combination of callId, error. If the 'header' is confusing, I can take it off the object name. Let me know. bq. bq. Michael Stack wrote: bq. Yeah, take away the header. Length I think is good. Makes it more robust (IIRC, we went out of our way to add length to the old RPC to help clients figure how much to pull). bq. bq. Devaraj Das wrote: bq. The argument above for 'length' applies here too... bq. bq. Michael Stack wrote: bq. Agreed. So high level, the response and request have a length of the total message? If so, don't need it down inside preceeding pb messages. bq. bq. Devaraj Das wrote: bq. I meant the argument on the PB encoding.. bq. bq. The RPC response envelope, even today, doesn't include the length. For instance, the client side of the method HBaseClient.receiveResponse starts with reading the callId. bq. bq. Michael Stack wrote: bq. Ok. We are replicating what was there previous. Lets make new jira for doing things like a length prefix. bq. bq. Devaraj Das wrote: bq. Okay let's discuss that in a separate jira.. bq. bq. Otherwise, do you think the patch is good to go? If so, I'll submit a new patch with some of the comments incorporated. bq. bq. Michael Stack wrote: bq. There items above you said you'd address such as removing Header from the request and response and cleaning up doc in the .proto file, right? Correct .. that's what I meant to include in the new patch. - Devaraj --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4096/#review6613 --- On 2012-03-30 23:29:32, Devaraj Das wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4096/ bq. --- bq. bq. (Updated 2012-03-30 23:29:32) bq. bq. bq. Review request for Michael Stack and Benoit Sigoure. bq. bq. bq. Summary bq. --- bq. bq. Switch RPC call envelope/headers to PBs bq. bq. bq. This addresses bug HBASE-5451. bq. https://issues.apache.org/jira/browse/HBASE-5451 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/DataOutputOutputStream.java 1307644 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1307644 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 1307644 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCMessageProtos.java PRE-CREATION bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java 1307644 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/4096/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Devaraj bq. bq. Switch RPC call envelope/headers to PBs --- Key: HBASE-5451 URL: https://issues.apache.org/jira/browse/HBASE-5451 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Affects Versions: 0.94.0 Reporter: Todd Lipcon Assignee: Devaraj Das Fix For: 0.96.0 Attachments: rpc-proto.2.txt, rpc-proto.3.txt, rpc-proto.patch.1_2, rpc-proto.r5.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5688) Convert zk root-region-server znode content to pb
[
https://issues.apache.org/jira/browse/HBASE-5688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244612#comment-13244612
]
jirapos...@reviews.apache.org commented on HBASE-5688:
--
bq. On 2012-04-01 16:41:00, Ted Yu wrote:
bq. src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java, line 48
bq. https://reviews.apache.org/r/4600/diff/1/?file=97843#file97843line48
bq.
bq. I think prefixedWithPBMagic would be a better name for this method.
Disagree.
bq. On 2012-04-01 16:41:00, Ted Yu wrote:
bq.
src/test/java/org/apache/hadoop/hbase/zookeeper/TestRootRegionTracker.java,
line 28
bq. https://reviews.apache.org/r/4600/diff/1/?file=97852#file97852line28
bq.
bq. Javadoc would be desirable.
Classname says what it does.
bq. On 2012-04-01 16:41:00, Ted Yu wrote:
bq.
src/test/java/org/apache/hadoop/hbase/zookeeper/TestRootRegionTracker.java,
line 43
bq. https://reviews.apache.org/r/4600/diff/1/?file=97852#file97852line43
bq.
bq. White space.
Will fix on commit.
- Michael
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4600/#review6606
---
On 2012-04-01 00:18:54, Michael Stack wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4600/
bq. ---
bq.
bq. (Updated 2012-04-01 00:18:54)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Changes the content of the root location znode, root-region-server, to be
bq. four magic bytes ('PBUF') followed by a protobuf message that holds the
bq. ServerName of the server currently hosting root.
bq.
bq. D src/main/java/org/apache/hadoop/hbase/catalog/RootLocationEditor.java
bq.Removed. Had two methods, one to add root-region-server znode and another
bq.to removed it. Rather, put these methods in RootRegionTracker. It
bq.tracks root-region-server znode. Having all to do w/ root-region-server
bq.is more cohesive. Also makes it so can encapsulate in one class
bq.all to do w/ create, delete, and reading of root-region-server.
bq.We also want to purge the catalog package (See note at head of
bq.CatalogTracker).
bq. M src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
bq. M src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
bq. M src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
bq.Get root region location from RootRegionTracker rather than from
RootLocationEditor.
bq. A src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
bq.Utility to do w/ protobuf handling. Has methods to help prefixing
bq.and stripping from serialized protobuf messages some 'magic'.
bq. A
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java
bq.PB generated.
bq. M src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
bq.Use new RootRegionTracker method for getting content of znode rather
bq.than do it all here (going via RootRegionTracker, we can keep how
bq.the znode content is serialized private to the RootRegionTracker class.
bq. M src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java
bq.Has the methods that used to be in RootLocationEditor plus a new
bq.
bq.
bq. This addresses bug hbase-5688.
bq. https://issues.apache.org/jira/browse/hbase-5688
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/main/java/org/apache/hadoop/hbase/catalog/RootLocationEditor.java
c90864a
bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
b2a5463
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
64def15
bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
PRE-CREATION
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
9c215b4
bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 2f05005
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java
33e4e71
bq.src/main/protobuf/ZooKeeper.proto PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java
533b2bf
bq.
src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
fe37156
bq.src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java
2132036
bq.
src/test/java/org/apache/hadoop/hbase/zookeeper/TestRootRegionTracker.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/4600/diff
bq.
bq.
bq. Testing
bq. ---
bq.
[jira] [Commented] (HBASE-5451) Switch RPC call envelope/headers to PBs
[ https://issues.apache.org/jira/browse/HBASE-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244882#comment-13244882 ] jirapos...@reviews.apache.org commented on HBASE-5451: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4096/ --- (Updated 2012-04-03 00:54:26.369788) Review request for Michael Stack and Benoit Sigoure. Changes --- This addresses the last set of comments from Stack (to do with changing the message names and marking certain fields optional, documentation on RPC spec). Summary --- Switch RPC call envelope/headers to PBs This addresses bug HBASE-5451. https://issues.apache.org/jira/browse/HBASE-5451 Diffs (updated) - http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/DataOutputOutputStream.java 1307644 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1307644 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 1307644 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCMessageProtos.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java 1307644 http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto PRE-CREATION Diff: https://reviews.apache.org/r/4096/diff Testing --- Thanks, Devaraj Switch RPC call envelope/headers to PBs --- Key: HBASE-5451 URL: https://issues.apache.org/jira/browse/HBASE-5451 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Affects Versions: 0.94.0 Reporter: Todd Lipcon Assignee: Devaraj Das Fix For: 0.96.0 Attachments: rpc-proto.2.txt, rpc-proto.3.txt, rpc-proto.patch.1_2, rpc-proto.r5.txt -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5444) Add PB-based calls to HMasterRegionInterface
[ https://issues.apache.org/jira/browse/HBASE-5444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13244995#comment-13244995 ] jirapos...@reviews.apache.org commented on HBASE-5444: -- bq. On 2012-03-23 17:01:13, Jimmy Xiang wrote: bq. src/main/java/org/apache/hadoop/hbase/protobuf/MasterRegionInterface.java, line 34 bq. https://reviews.apache.org/r/4463/diff/1/?file=94838#file94838line34 bq. bq. Should it be Private? Should we call it MasterRegionProtocol? bq. bq. Gregory Chanan wrote: bq. I'll make it private. bq. bq. I'll leave it named Interface for now, just because it is replacing HMasterRegionInterface. I agree we should be consistent. Perhaps someone else can chime in with an opinion on Interface vs Protocol. IMO, Protocol would be better. Interface is way overloaded. So is protocol but less so and this seems like usage seems particularly apt. Just my twosense (This was an old comment I never published) - Michael --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4463/#review6286 --- On 2012-03-23 19:55:28, Gregory Chanan wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4463/ bq. --- bq. bq. (Updated 2012-03-23 19:55:28) bq. bq. bq. Review request for hbase. bq. bq. bq. Summary bq. --- bq. bq. Adds PB-based calls replacing HMasterRegionInterface. bq. bq. There are some temporary hacks, e.g. converting PB-based ServerLoad to existing HServerLoad so I didn't need to convert ClusterStatus (which brings in a lot of other changes). That will be cleaned up in HBASE-5445. bq. bq. bq. This addresses bug HBASE-5444. bq. https://issues.apache.org/jira/browse/HBASE-5444 bq. bq. bq. Diffs bq. - bq. bq.pom.xml 10b13ef bq. src/main/jamon/org/apache/hadoop/hbase/tmpl/master/MasterStatusTmpl.jamon 69434f7 bq. src/main/jamon/org/apache/hadoop/hbase/tmpl/regionserver/RSStatusTmpl.jamon ae76204 bq.src/main/java/org/apache/hadoop/hbase/ClusterStatus.java 9df4c10 bq.src/main/java/org/apache/hadoop/hbase/HConstants.java 347 bq.src/main/java/org/apache/hadoop/hbase/io/HbaseObjectWritable.java cbfa489 bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseRpcMetrics.java 0db2760 bq.src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 2916d68 bq.src/main/java/org/apache/hadoop/hbase/ipc/HMasterRegionInterface.java fd97830 bq.src/main/java/org/apache/hadoop/hbase/ipc/Invocation.java f1f06b0 bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java d47ef10 bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java cd1755f bq.src/main/java/org/apache/hadoop/hbase/master/MXBean.java 7f44dc2 bq.src/main/java/org/apache/hadoop/hbase/master/MXBeanImpl.java 45b8fe7 bq.src/main/java/org/apache/hadoop/hbase/master/MasterDumpServlet.java be63838 bq.src/main/java/org/apache/hadoop/hbase/master/ServerManager.java cbd55f7 bq. src/main/java/org/apache/hadoop/hbase/protobuf/MasterRegionInterface.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/protobuf/PBHelper.java PRE-CREATION bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java e0af8fb bq.src/main/proto/MasterRegionProtocol.proto PRE-CREATION bq.src/main/proto/hbase.proto PRE-CREATION bq.src/main/resources/hbase-webapps/master/table.jsp 811df46 bq.src/test/java/org/apache/hadoop/hbase/MiniHBaseCluster.java 6af9188 bq.src/test/java/org/apache/hadoop/hbase/coprocessor/TestClassLoading.java 368a0e5 bq.src/test/java/org/apache/hadoop/hbase/io/TestHbaseObjectWritable.java f2f8ee3 bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java 841649a bq.src/test/java/org/apache/hadoop/hbase/master/TestMXBean.java 379f70c bq. src/test/java/org/apache/hadoop/hbase/regionserver/TestServerCustomProtocol.java e99d251 bq. bq. Diff: https://reviews.apache.org/r/4463/diff bq. bq. bq. Testing bq. --- bq. bq. Ran jenkins job, all unit tests passed. bq. bq. bq. Thanks, bq. bq. Gregory bq. bq. Add PB-based calls to HMasterRegionInterface Key: HBASE-5444 URL: https://issues.apache.org/jira/browse/HBASE-5444 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Reporter: Todd Lipcon Assignee: Gregory Chanan -- This message is automatically generated
[jira] [Commented] (HBASE-4348) Add metrics for regions in transition
[
https://issues.apache.org/jira/browse/HBASE-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13243745#comment-13243745
]
jirapos...@reviews.apache.org commented on HBASE-4348:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4402/#review6604
---
src/main/java/org/apache/hadoop/hbase/HConstants.java
https://reviews.apache.org/r/4402/#comment14269
I think the trailing '.time' isn't needed. Take a look at existing config
parameter names involving threshold:
{code}
this.thresholdIdleConnections = conf.getInt(ipc.client.idlethreshold,
4000);
src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
putsortreducer.row.threshold, 2L * (130));
src/main/java/org/apache/hadoop/hbase/mapreduce/PutSortReducer.java
{code}
src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
https://reviews.apache.org/r/4402/#comment14270
Please add curly braces around the following line.
src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
https://reviews.apache.org/r/4402/#comment14271
Lift this line to line 2733.
src/main/java/org/apache/hadoop/hbase/master/HMaster.java
https://reviews.apache.org/r/4402/#comment14272
'out' isn't needed here. It would be nice to combine this sentence into the
comment for this method.
- Ted
On 2012-03-30 05:21:12, Himanshu Vashishtha wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4402/
bq. ---
bq.
bq. (Updated 2012-03-30 05:21:12)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. This patch is for adding Region in transition metrics to the HMaster
metrics system. It also adds these metrics in the master ui, in the Region in
transition section. I have attached the proposed new format in the jira 4348.
bq.
bq.
bq. This addresses bug HBase-4348.
bq. https://issues.apache.org/jira/browse/HBase-4348
bq.
bq.
bq. Diffs
bq. -
bq.
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/master/AssignmentManagerStatusTmpl.jamon
0dc0691
bq.src/main/java/org/apache/hadoop/hbase/HConstants.java 21ac4ba
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
64def15
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 9bd4ace
bq.src/main/java/org/apache/hadoop/hbase/master/metrics/MasterMetrics.java
83abc52
bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
91dce36
bq.
bq. Diff: https://reviews.apache.org/r/4402/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Ran on a 5 node cluster and kill region servers randomly to observe the
changes in the RIT metrics as emitted out by the Master's mxbean;
bq.
bq. mvn test passes without any failure.
bq.
bq.
bq. Thanks,
bq.
bq. Himanshu
bq.
bq.
Add metrics for regions in transition
-
Key: HBASE-4348
URL: https://issues.apache.org/jira/browse/HBASE-4348
Project: HBase
Issue Type: Improvement
Components: metrics
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: Himanshu Vashishtha
Priority: Minor
Labels: noob
Fix For: 0.96.0
Attachments: 4348-metrics-v3.patch, 4348-v1.patch, 4348-v2.patch,
RITs.png, RegionInTransitions2.png, metrics-v2.patch
The following metrics would be useful for monitoring the master:
- the number of regions in transition
- the number of regions in transition that have been in transition for more
than a minute
- how many seconds has the oldest region-in-transition been in transition
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5688) Convert zk root-region-server znode content to pb
[
https://issues.apache.org/jira/browse/HBASE-5688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13243769#comment-13243769
]
jirapos...@reviews.apache.org commented on HBASE-5688:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4600/#review6605
---
Looks good to me.
- Jimmy
On 2012-04-01 00:18:54, Michael Stack wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4600/
bq. ---
bq.
bq. (Updated 2012-04-01 00:18:54)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Changes the content of the root location znode, root-region-server, to be
bq. four magic bytes ('PBUF') followed by a protobuf message that holds the
bq. ServerName of the server currently hosting root.
bq.
bq. D src/main/java/org/apache/hadoop/hbase/catalog/RootLocationEditor.java
bq.Removed. Had two methods, one to add root-region-server znode and another
bq.to removed it. Rather, put these methods in RootRegionTracker. It
bq.tracks root-region-server znode. Having all to do w/ root-region-server
bq.is more cohesive. Also makes it so can encapsulate in one class
bq.all to do w/ create, delete, and reading of root-region-server.
bq.We also want to purge the catalog package (See note at head of
bq.CatalogTracker).
bq. M src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
bq. M src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
bq. M src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
bq.Get root region location from RootRegionTracker rather than from
RootLocationEditor.
bq. A src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
bq.Utility to do w/ protobuf handling. Has methods to help prefixing
bq.and stripping from serialized protobuf messages some 'magic'.
bq. A
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java
bq.PB generated.
bq. M src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
bq.Use new RootRegionTracker method for getting content of znode rather
bq.than do it all here (going via RootRegionTracker, we can keep how
bq.the znode content is serialized private to the RootRegionTracker class.
bq. M src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java
bq.Has the methods that used to be in RootLocationEditor plus a new
bq.
bq.
bq. This addresses bug hbase-5688.
bq. https://issues.apache.org/jira/browse/hbase-5688
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/main/java/org/apache/hadoop/hbase/catalog/RootLocationEditor.java
c90864a
bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
b2a5463
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
64def15
bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
PRE-CREATION
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
9c215b4
bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 2f05005
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java
33e4e71
bq.src/main/protobuf/ZooKeeper.proto PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java
533b2bf
bq.
src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
fe37156
bq.src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java
2132036
bq.
src/test/java/org/apache/hadoop/hbase/zookeeper/TestRootRegionTracker.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/4600/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq.
bq. Thanks,
bq.
bq. Michael
bq.
bq.
Convert zk root-region-server znode content to pb
-
Key: HBASE-5688
URL: https://issues.apache.org/jira/browse/HBASE-5688
Project: HBase
Issue Type: Task
Reporter: stack
Assignee: stack
Fix For: 0.96.0
Attachments: 5688.txt, 5688v4.txt
Move the root-region-server znode content from the versioned bytes that
ServerName.getVersionedBytes outputs to instead be pb.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on
[jira] [Commented] (HBASE-5688) Convert zk root-region-server znode content to pb
[
https://issues.apache.org/jira/browse/HBASE-5688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13243774#comment-13243774
]
jirapos...@reviews.apache.org commented on HBASE-5688:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4600/#review6606
---
src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
https://reviews.apache.org/r/4600/#comment14273
I think prefixedWithPBMagic would be a better name for this method.
src/test/java/org/apache/hadoop/hbase/zookeeper/TestRootRegionTracker.java
https://reviews.apache.org/r/4600/#comment14274
Javadoc would be desirable.
src/test/java/org/apache/hadoop/hbase/zookeeper/TestRootRegionTracker.java
https://reviews.apache.org/r/4600/#comment14275
White space.
- Ted
On 2012-04-01 00:18:54, Michael Stack wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4600/
bq. ---
bq.
bq. (Updated 2012-04-01 00:18:54)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. Changes the content of the root location znode, root-region-server, to be
bq. four magic bytes ('PBUF') followed by a protobuf message that holds the
bq. ServerName of the server currently hosting root.
bq.
bq. D src/main/java/org/apache/hadoop/hbase/catalog/RootLocationEditor.java
bq.Removed. Had two methods, one to add root-region-server znode and another
bq.to removed it. Rather, put these methods in RootRegionTracker. It
bq.tracks root-region-server znode. Having all to do w/ root-region-server
bq.is more cohesive. Also makes it so can encapsulate in one class
bq.all to do w/ create, delete, and reading of root-region-server.
bq.We also want to purge the catalog package (See note at head of
bq.CatalogTracker).
bq. M src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
bq. M src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
bq. M src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
bq.Get root region location from RootRegionTracker rather than from
RootLocationEditor.
bq. A src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
bq.Utility to do w/ protobuf handling. Has methods to help prefixing
bq.and stripping from serialized protobuf messages some 'magic'.
bq. A
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java
bq.PB generated.
bq. M src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
bq.Use new RootRegionTracker method for getting content of znode rather
bq.than do it all here (going via RootRegionTracker, we can keep how
bq.the znode content is serialized private to the RootRegionTracker class.
bq. M src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java
bq.Has the methods that used to be in RootLocationEditor plus a new
bq.
bq.
bq. This addresses bug hbase-5688.
bq. https://issues.apache.org/jira/browse/hbase-5688
bq.
bq.
bq. Diffs
bq. -
bq.
bq.src/main/java/org/apache/hadoop/hbase/catalog/RootLocationEditor.java
c90864a
bq.src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
b2a5463
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
64def15
bq.src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
PRE-CREATION
bq.
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java
PRE-CREATION
bq.src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
9c215b4
bq.src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 2f05005
bq.src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java
33e4e71
bq.src/main/protobuf/ZooKeeper.proto PRE-CREATION
bq.src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java
533b2bf
bq.
src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
fe37156
bq.src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java
2132036
bq.
src/test/java/org/apache/hadoop/hbase/zookeeper/TestRootRegionTracker.java
PRE-CREATION
bq.
bq. Diff: https://reviews.apache.org/r/4600/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq.
bq. Thanks,
bq.
bq. Michael
bq.
bq.
Convert zk root-region-server znode content to pb
-
Key: HBASE-5688
URL: https://issues.apache.org/jira/browse/HBASE-5688
Project: HBase
Issue Type: Task
Reporter:
[jira] [Commented] (HBASE-4348) Add metrics for regions in transition
[
https://issues.apache.org/jira/browse/HBASE-4348?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13243798#comment-13243798
]
jirapos...@reviews.apache.org commented on HBASE-4348:
--
bq. On 2012-04-01 15:13:59, Ted Yu wrote:
bq. src/main/java/org/apache/hadoop/hbase/HConstants.java, line 655
bq. https://reviews.apache.org/r/4402/diff/6/?file=97548#file97548line655
bq.
bq. I think the trailing '.time' isn't needed. Take a look at existing
config parameter names involving threshold:
bq. {code}
bq. this.thresholdIdleConnections =
conf.getInt(ipc.client.idlethreshold, 4000);
bq. src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java
bq. putsortreducer.row.threshold, 2L * (130));
bq. src/main/java/org/apache/hadoop/hbase/mapreduce/PutSortReducer.java
bq. {code}
bq.
Actually, looking at the metric name without context,
hbase.metrics.rit.threshold makes me think this is a count of the number of max
regions in transition. With the .time suffix, it makes me think it is the max
time for an RIT which also isn't quite right. If all things are in millis than
we probably don't need units but it doesn't hurt IMO. What do you think of
something like: hbase.metrics.rit.refresh.millis,
hbase.metrics.rit.refresh.threshold.millis, or
hbase.metrics.rit.refresh.threshold?
- jmhsieh
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4402/#review6604
---
On 2012-03-30 05:21:12, Himanshu Vashishtha wrote:
bq.
bq. ---
bq. This is an automatically generated e-mail. To reply, visit:
bq. https://reviews.apache.org/r/4402/
bq. ---
bq.
bq. (Updated 2012-03-30 05:21:12)
bq.
bq.
bq. Review request for hbase.
bq.
bq.
bq. Summary
bq. ---
bq.
bq. This patch is for adding Region in transition metrics to the HMaster
metrics system. It also adds these metrics in the master ui, in the Region in
transition section. I have attached the proposed new format in the jira 4348.
bq.
bq.
bq. This addresses bug HBase-4348.
bq. https://issues.apache.org/jira/browse/HBase-4348
bq.
bq.
bq. Diffs
bq. -
bq.
bq.
src/main/jamon/org/apache/hadoop/hbase/tmpl/master/AssignmentManagerStatusTmpl.jamon
0dc0691
bq.src/main/java/org/apache/hadoop/hbase/HConstants.java 21ac4ba
bq.src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
64def15
bq.src/main/java/org/apache/hadoop/hbase/master/HMaster.java 9bd4ace
bq.src/main/java/org/apache/hadoop/hbase/master/metrics/MasterMetrics.java
83abc52
bq.src/test/java/org/apache/hadoop/hbase/master/TestAssignmentManager.java
91dce36
bq.
bq. Diff: https://reviews.apache.org/r/4402/diff
bq.
bq.
bq. Testing
bq. ---
bq.
bq. Ran on a 5 node cluster and kill region servers randomly to observe the
changes in the RIT metrics as emitted out by the Master's mxbean;
bq.
bq. mvn test passes without any failure.
bq.
bq.
bq. Thanks,
bq.
bq. Himanshu
bq.
bq.
Add metrics for regions in transition
-
Key: HBASE-4348
URL: https://issues.apache.org/jira/browse/HBASE-4348
Project: HBase
Issue Type: Improvement
Components: metrics
Affects Versions: 0.92.0
Reporter: Todd Lipcon
Assignee: Himanshu Vashishtha
Priority: Minor
Labels: noob
Fix For: 0.96.0
Attachments: 4348-metrics-v3.patch, 4348-v1.patch, 4348-v2.patch,
RITs.png, RegionInTransitions2.png, metrics-v2.patch
The following metrics would be useful for monitoring the master:
- the number of regions in transition
- the number of regions in transition that have been in transition for more
than a minute
- how many seconds has the oldest region-in-transition been in transition
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5451) Switch RPC call envelope/headers to PBs
[ https://issues.apache.org/jira/browse/HBASE-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13243919#comment-13243919 ] jirapos...@reviews.apache.org commented on HBASE-5451: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4096/#review6613 --- Some more questions. Just being careful DD. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/DataOutputOutputStream.java https://reviews.apache.org/r/4096/#comment14285 We should just be using the hadoop DOOS... looks like no diff (when I diff them). I'll make an issue to remove. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java https://reviews.apache.org/r/4096/#comment14278 Is this written up anywhere? That its hrpc, then version, then a length, then a protobuf? I see it in the proto definition. That'll do. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java https://reviews.apache.org/r/4096/#comment14279 We have an issue for removing this Invocation stuff? http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto https://reviews.apache.org/r/4096/#comment14280 Should we just remove them in the next iteration on rpc since 0.96 is to be a singularity? Why even bother trying to keep compatibility w/ older clients? What is 'failure compatibility'? We are telling the client to go away, nicely (smile). What you think we should replace hrpc0x0005 with? this - these http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto https://reviews.apache.org/r/4096/#comment14281 How does RpcRequestWithHeaderProto relate to ConnectionHeaderProto? This text should say? Would be nice to have illustration on how the back and forth work. http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto https://reviews.apache.org/r/4096/#comment14282 We'll send this String each time? http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto https://reviews.apache.org/r/4096/#comment14283 Which part in here is the 'header'? How does it relate to ConnectionHeaderProto? request can be an Invocation/Writable? Or a protobuf? Do we need a length in here? http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto https://reviews.apache.org/r/4096/#comment14284 Should this precede the response? So if false, a response follows else an exception? Do we need a length here? Where is the header that the message name refers too? - Michael On 2012-03-30 23:29:32, Devaraj Das wrote: bq. bq. --- bq. This is an automatically generated e-mail. To reply, visit: bq. https://reviews.apache.org/r/4096/ bq. --- bq. bq. (Updated 2012-03-30 23:29:32) bq. bq. bq. Review request for Michael Stack and Benoit Sigoure. bq. bq. bq. Summary bq. --- bq. bq. Switch RPC call envelope/headers to PBs bq. bq. bq. This addresses bug HBASE-5451. bq. https://issues.apache.org/jira/browse/HBASE-5451 bq. bq. bq. Diffs bq. - bq. bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/DataOutputOutputStream.java 1307644 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1307644 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 1307644 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCMessageProtos.java PRE-CREATION bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java 1307644 bq. http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto PRE-CREATION bq. bq. Diff: https://reviews.apache.org/r/4096/diff bq. bq. bq. Testing bq. --- bq. bq. bq. Thanks, bq. bq. Devaraj bq. bq. Switch RPC call envelope/headers to PBs --- Key: HBASE-5451 URL: https://issues.apache.org/jira/browse/HBASE-5451 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Affects Versions: 0.94.0 Reporter: Todd Lipcon Assignee: Devaraj Das Fix For: 0.96.0 Attachments: rpc-proto.2.txt, rpc-proto.3.txt,
[jira] [Commented] (HBASE-5688) Convert zk root-region-server znode content to pb
[
https://issues.apache.org/jira/browse/HBASE-5688?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13243618#comment-13243618
]
jirapos...@reviews.apache.org commented on HBASE-5688:
--
---
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/4600/
---
Review request for hbase.
Summary
---
Changes the content of the root location znode, root-region-server, to be
four magic bytes ('PBUF') followed by a protobuf message that holds the
ServerName of the server currently hosting root.
D src/main/java/org/apache/hadoop/hbase/catalog/RootLocationEditor.java
Removed. Had two methods, one to add root-region-server znode and another
to removed it. Rather, put these methods in RootRegionTracker. It
tracks root-region-server znode. Having all to do w/ root-region-server
is more cohesive. Also makes it so can encapsulate in one class
all to do w/ create, delete, and reading of root-region-server.
We also want to purge the catalog package (See note at head of
CatalogTracker).
M src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java
M src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java
M src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java
Get root region location from RootRegionTracker rather than from
RootLocationEditor.
A src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java
Utility to do w/ protobuf handling. Has methods to help prefixing
and stripping from serialized protobuf messages some 'magic'.
A src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java
PB generated.
M src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java
Use new RootRegionTracker method for getting content of znode rather
than do it all here (going via RootRegionTracker, we can keep how
the znode content is serialized private to the RootRegionTracker class.
M src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java
Has the methods that used to be in RootLocationEditor plus a new
This addresses bug hbase-5688.
https://issues.apache.org/jira/browse/hbase-5688
Diffs
-
src/main/java/org/apache/hadoop/hbase/catalog/RootLocationEditor.java c90864a
src/main/java/org/apache/hadoop/hbase/client/HConnectionManager.java b2a5463
src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java 64def15
src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java PRE-CREATION
src/main/java/org/apache/hadoop/hbase/protobuf/generated/ZooKeeperProtos.java
PRE-CREATION
src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java 9c215b4
src/main/java/org/apache/hadoop/hbase/util/HBaseFsck.java 2f05005
src/main/java/org/apache/hadoop/hbase/zookeeper/RootRegionTracker.java
33e4e71
src/main/protobuf/ZooKeeper.proto PRE-CREATION
src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTracker.java 533b2bf
src/test/java/org/apache/hadoop/hbase/catalog/TestCatalogTrackerOnCluster.java
fe37156
src/test/java/org/apache/hadoop/hbase/master/TestMasterNoCluster.java 2132036
src/test/java/org/apache/hadoop/hbase/zookeeper/TestRootRegionTracker.java
PRE-CREATION
Diff: https://reviews.apache.org/r/4600/diff
Testing
---
Thanks,
Michael
Convert zk root-region-server znode content to pb
-
Key: HBASE-5688
URL: https://issues.apache.org/jira/browse/HBASE-5688
Project: HBase
Issue Type: Task
Reporter: stack
Assignee: stack
Fix For: 0.96.0
Attachments: 5688.txt, 5688v4.txt
Move the root-region-server znode content from the versioned bytes that
ServerName.getVersionedBytes outputs to instead be pb.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-5451) Switch RPC call envelope/headers to PBs
[ https://issues.apache.org/jira/browse/HBASE-5451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13242848#comment-13242848 ] jirapos...@reviews.apache.org commented on HBASE-5451: -- --- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/4096/ --- (Updated 2012-03-30 23:29:32.254174) Review request for Michael Stack and Benoit Sigoure. Changes --- This should take care of the comments. Summary --- Switch RPC call envelope/headers to PBs This addresses bug HBASE-5451. https://issues.apache.org/jira/browse/HBASE-5451 Diffs (updated) - http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/io/DataOutputOutputStream.java 1307644 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseClient.java 1307644 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/ipc/HBaseServer.java 1307644 http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/protobuf/generated/RPCMessageProtos.java PRE-CREATION http://svn.apache.org/repos/asf/hbase/trunk/src/main/java/org/apache/hadoop/hbase/security/User.java 1307644 http://svn.apache.org/repos/asf/hbase/trunk/src/main/protobuf/RPCMessageProto.proto PRE-CREATION Diff: https://reviews.apache.org/r/4096/diff Testing --- Thanks, Devaraj Switch RPC call envelope/headers to PBs --- Key: HBASE-5451 URL: https://issues.apache.org/jira/browse/HBASE-5451 Project: HBase Issue Type: Sub-task Components: ipc, master, migration, regionserver Affects Versions: 0.94.0 Reporter: Todd Lipcon Assignee: Devaraj Das Fix For: 0.96.0 Attachments: rpc-proto.2.txt, rpc-proto.3.txt, rpc-proto.patch.1_2 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
