[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13815867#comment-13815867 ] Jean-Marc Spaggiari commented on HBASE-7403: Just to provide more details ;) If you look at the Fix version/s field, you will see 0.95.0. 0.95.0 is the dev version of 0.96.0. Everything with has been fixed in 0.95.0 is there in 0.96.0. Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.95.0 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.98.0, 0.95.0 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403-v5.txt, 7403v5.diff, 7403v5.txt, hbase-7403-0.95.patch, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, hbase-7403-trunkv19.patch, hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch, hbase-7403-trunkv23.patch, hbase-7403-trunkv24.patch, hbase-7403-trunkv26.patch, hbase-7403-trunkv28.patch, hbase-7403-trunkv29.patch, hbase-7403-trunkv30.patch, hbase-7403-trunkv31.patch, hbase-7403-trunkv32.patch, hbase-7403-trunkv33.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf Support executing region merge transaction on Regionserver, similar with split transaction Process of merging two regions: a.client sends RPC (dispatch merging regions) to master b.master moves the regions together (on the same regionserver where the more heavily loaded region resided) c.master sends RPC (merge regions) to this regionserver d.Regionserver executes the region merge transaction in the thread pool e.the above b,c,d run asynchronously Process of region merge transaction: a.Construct a new region merge transaction. b.prepare for the merge transaction, the transaction will be canceled if it is unavailable, e.g. two regions don't belong to same table; two regions are not adjacent in a non-compulsory merge; region is closed or has reference c.execute the transaction as the following: /** * Set region as in transition, set it into MERGING state. */ SET_MERGING_IN_ZK, /** * We created the temporary merge data directory. */ CREATED_MERGE_DIR, /** * Closed the merging region A. */ CLOSED_REGION_A, /** * The merging region A has been taken out of the server's online regions list. */ OFFLINED_REGION_A, /** * Closed the merging region B. */ CLOSED_REGION_B, /** * The merging region B has been taken out of the server's online regions list. */ OFFLINED_REGION_B, /** * Started in on creation of the merged region. */ STARTED_MERGED_REGION_CREATION, /** * Point of no return. If we got here, then transaction is not recoverable * other than by crashing out the regionserver. */ PONR d.roll back if step c throws exception Usage: HBaseAdmin#mergeRegions See more details from the patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13815706#comment-13815706 ] Asaf Mesika commented on HBASE-7403: 0.96.0 has this too? Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.95.0 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.98.0, 0.95.0 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403-v5.txt, 7403v5.diff, 7403v5.txt, hbase-7403-0.95.patch, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, hbase-7403-trunkv19.patch, hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch, hbase-7403-trunkv23.patch, hbase-7403-trunkv24.patch, hbase-7403-trunkv26.patch, hbase-7403-trunkv28.patch, hbase-7403-trunkv29.patch, hbase-7403-trunkv30.patch, hbase-7403-trunkv31.patch, hbase-7403-trunkv32.patch, hbase-7403-trunkv33.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf Support executing region merge transaction on Regionserver, similar with split transaction Process of merging two regions: a.client sends RPC (dispatch merging regions) to master b.master moves the regions together (on the same regionserver where the more heavily loaded region resided) c.master sends RPC (merge regions) to this regionserver d.Regionserver executes the region merge transaction in the thread pool e.the above b,c,d run asynchronously Process of region merge transaction: a.Construct a new region merge transaction. b.prepare for the merge transaction, the transaction will be canceled if it is unavailable, e.g. two regions don't belong to same table; two regions are not adjacent in a non-compulsory merge; region is closed or has reference c.execute the transaction as the following: /** * Set region as in transition, set it into MERGING state. */ SET_MERGING_IN_ZK, /** * We created the temporary merge data directory. */ CREATED_MERGE_DIR, /** * Closed the merging region A. */ CLOSED_REGION_A, /** * The merging region A has been taken out of the server's online regions list. */ OFFLINED_REGION_A, /** * Closed the merging region B. */ CLOSED_REGION_B, /** * The merging region B has been taken out of the server's online regions list. */ OFFLINED_REGION_B, /** * Started in on creation of the merged region. */ STARTED_MERGED_REGION_CREATION, /** * Point of no return. If we got here, then transaction is not recoverable * other than by crashing out the regionserver. */ PONR d.roll back if step c throws exception Usage: HBaseAdmin#mergeRegions See more details from the patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13815714#comment-13815714 ] stack commented on HBASE-7403: -- [~asafm] yes Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.95.0 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.98.0, 0.95.0 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403-v5.txt, 7403v5.diff, 7403v5.txt, hbase-7403-0.95.patch, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, hbase-7403-trunkv19.patch, hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch, hbase-7403-trunkv23.patch, hbase-7403-trunkv24.patch, hbase-7403-trunkv26.patch, hbase-7403-trunkv28.patch, hbase-7403-trunkv29.patch, hbase-7403-trunkv30.patch, hbase-7403-trunkv31.patch, hbase-7403-trunkv32.patch, hbase-7403-trunkv33.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf Support executing region merge transaction on Regionserver, similar with split transaction Process of merging two regions: a.client sends RPC (dispatch merging regions) to master b.master moves the regions together (on the same regionserver where the more heavily loaded region resided) c.master sends RPC (merge regions) to this regionserver d.Regionserver executes the region merge transaction in the thread pool e.the above b,c,d run asynchronously Process of region merge transaction: a.Construct a new region merge transaction. b.prepare for the merge transaction, the transaction will be canceled if it is unavailable, e.g. two regions don't belong to same table; two regions are not adjacent in a non-compulsory merge; region is closed or has reference c.execute the transaction as the following: /** * Set region as in transition, set it into MERGING state. */ SET_MERGING_IN_ZK, /** * We created the temporary merge data directory. */ CREATED_MERGE_DIR, /** * Closed the merging region A. */ CLOSED_REGION_A, /** * The merging region A has been taken out of the server's online regions list. */ OFFLINED_REGION_A, /** * Closed the merging region B. */ CLOSED_REGION_B, /** * The merging region B has been taken out of the server's online regions list. */ OFFLINED_REGION_B, /** * Started in on creation of the merged region. */ STARTED_MERGED_REGION_CREATION, /** * Point of no return. If we got here, then transaction is not recoverable * other than by crashing out the regionserver. */ PONR d.roll back if step c throws exception Usage: HBaseAdmin#mergeRegions See more details from the patch -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13616895#comment-13616895 ] Enis Soztutar commented on HBASE-7403: -- I've created a backport patch for 0.94 to see whether there is any interest, HBASE-8217. [~zjushch], [~lhofhansl] wdyt? Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.95.0 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.95.0, 0.98.0 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-0.95.patch, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch, hbase-7403-trunkv23.patch, hbase-7403-trunkv24.patch, hbase-7403-trunkv26.patch, hbase-7403-trunkv28.patch, hbase-7403-trunkv29.patch, hbase-7403-trunkv30.patch, hbase-7403-trunkv31.patch, hbase-7403-trunkv32.patch, hbase-7403-trunkv33.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf Support executing region merge transaction on Regionserver, similar with split transaction Process of merging two regions: a.client sends RPC (dispatch merging regions) to master b.master moves the regions together (on the same regionserver where the more heavily loaded region resided) c.master sends RPC (merge regions) to this regionserver d.Regionserver executes the region merge transaction in the thread pool e.the above b,c,d run asynchronously Process of region merge transaction: a.Construct a new region merge transaction. b.prepare for the merge transaction, the transaction will be canceled if it is unavailable, e.g. two regions don't belong to same table; two regions are not adjacent in a non-compulsory merge; region is closed or has reference c.execute the transaction as the following: /** * Set region as in transition, set it into MERGING state. */ SET_MERGING_IN_ZK, /** * We created the temporary merge data directory. */ CREATED_MERGE_DIR, /** * Closed the merging region A. */ CLOSED_REGION_A, /** * The merging region A has been taken out of the server's online regions list. */ OFFLINED_REGION_A, /** * Closed the merging region B. */ CLOSED_REGION_B, /** * The merging region B has been taken out of the server's online regions list. */ OFFLINED_REGION_B, /** * Started in on creation of the merged region. */ STARTED_MERGED_REGION_CREATION, /** * Point of no return. If we got here, then transaction is not recoverable * other than by crashing out the regionserver. */ PONR d.roll back if step c throws exception Usage: HBaseAdmin#mergeRegions See more details from the patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13612064#comment-13612064 ] chunhui shen commented on HBASE-7403: - Committed to trunk and 0.95. Thanks all for the review, Ted, Stack, Ram, Enis, Jean-Marc, Jonathan, Matteo Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.95.0 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.95.0, 0.98.0 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-0.95.patch, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch, hbase-7403-trunkv23.patch, hbase-7403-trunkv24.patch, hbase-7403-trunkv26.patch, hbase-7403-trunkv28.patch, hbase-7403-trunkv29.patch, hbase-7403-trunkv30.patch, hbase-7403-trunkv31.patch, hbase-7403-trunkv32.patch, hbase-7403-trunkv33.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf Support executing region merge transaction on Regionserver, similar with split transaction Process of merging two regions: a.client sends RPC (dispatch merging regions) to master b.master moves the regions together (on the same regionserver where the more heavily loaded region resided) c.master sends RPC (merge regions) to this regionserver d.Regionserver executes the region merge transaction in the thread pool e.the above b,c,d run asynchronously Process of region merge transaction: a.Construct a new region merge transaction. b.prepare for the merge transaction, the transaction will be canceled if it is unavailable, e.g. two regions don't belong to same table; two regions are not adjacent in a non-compulsory merge; region is closed or has reference c.execute the transaction as the following: /** * Set region as in transition, set it into MERGING state. */ SET_MERGING_IN_ZK, /** * We created the temporary merge data directory. */ CREATED_MERGE_DIR, /** * Closed the merging region A. */ CLOSED_REGION_A, /** * The merging region A has been taken out of the server's online regions list. */ OFFLINED_REGION_A, /** * Closed the merging region B. */ CLOSED_REGION_B, /** * The merging region B has been taken out of the server's online regions list. */ OFFLINED_REGION_B, /** * Started in on creation of the merged region. */ STARTED_MERGED_REGION_CREATION, /** * Point of no return. If we got here, then transaction is not recoverable * other than by crashing out the regionserver. */ PONR d.roll back if step c throws exception Usage: HBaseAdmin#mergeRegions See more details from the patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13612076#comment-13612076 ] Hudson commented on HBASE-7403: --- Integrated in hbase-0.95-on-hadoop2 #40 (See [https://builds.apache.org/job/hbase-0.95-on-hadoop2/40/]) HBASE-7403 Online Merge(Chunhui shen) (Revision 1460310) Result = FAILURE zjushch : Files : * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/HRegionInfo.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/catalog/MetaReader.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/client/HBaseAdmin.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/executor/EventType.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/ProtobufUtil.java * /hbase/branches/0.95/hbase-client/src/main/java/org/apache/hadoop/hbase/protobuf/RequestConverter.java * /hbase/branches/0.95/hbase-common/src/main/java/org/apache/hadoop/hbase/HConstants.java * /hbase/branches/0.95/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/AdminProtos.java * /hbase/branches/0.95/hbase-protocol/src/main/java/org/apache/hadoop/hbase/protobuf/generated/MasterAdminProtos.java * /hbase/branches/0.95/hbase-protocol/src/main/protobuf/Admin.proto * /hbase/branches/0.95/hbase-protocol/src/main/protobuf/MasterAdmin.proto * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/catalog/MetaEditor.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/AssignmentManager.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/CatalogJanitor.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterServices.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/ServerManager.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/DispatchMergingRegionHandler.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/master/handler/MergedRegionHandler.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CompactSplitThread.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegion.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/HRegionServer.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionMergeRequest.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionMergeTransaction.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/RegionServerServices.java * /hbase/branches/0.95/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SplitTransaction.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/HBaseTestingUtility.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/MockRegionServer.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/master/TestCatalogJanitor.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransaction.java * /hbase/branches/0.95/hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransactionOnCluster.java Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.95.0 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.95.0, 0.98.0 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-0.95.patch, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch, hbase-7403-trunkv23.patch, hbase-7403-trunkv24.patch, hbase-7403-trunkv26.patch, hbase-7403-trunkv28.patch, hbase-7403-trunkv29.patch, hbase-7403-trunkv30.patch, hbase-7403-trunkv31.patch, hbase-7403-trunkv32.patch, hbase-7403-trunkv33.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf Support executing region merge transaction on Regionserver, similar with
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13611659#comment-13611659 ] chunhui shen commented on HBASE-7403: - Shell commend and documentation is done in HBASE-8189 HBASE-8190 Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.95.0 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.95.0, 0.98.0 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch, hbase-7403-trunkv23.patch, hbase-7403-trunkv24.patch, hbase-7403-trunkv26.patch, hbase-7403-trunkv28.patch, hbase-7403-trunkv29.patch, hbase-7403-trunkv30.patch, hbase-7403-trunkv31.patch, hbase-7403-trunkv32.patch, hbase-7403-trunkv33.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf Support executing region merge transaction on Regionserver, similar with split transaction Process of merging two regions: a.client sends RPC (dispatch merging regions) to master b.master moves the regions together (on the same regionserver where the more heavily loaded region resided) c.master sends RPC (merge regions) to this regionserver d.Regionserver executes the region merge transaction in the thread pool e.the above b,c,d run asynchronously Process of region merge transaction: a.Construct a new region merge transaction. b.prepare for the merge transaction, the transaction will be canceled if it is unavailable, e.g. two regions don't belong to same table; two regions are not adjacent in a non-compulsory merge; region is closed or has reference c.execute the transaction as the following: /** * Set region as in transition, set it into MERGING state. */ SET_MERGING_IN_ZK, /** * We created the temporary merge data directory. */ CREATED_MERGE_DIR, /** * Closed the merging region A. */ CLOSED_REGION_A, /** * The merging region A has been taken out of the server's online regions list. */ OFFLINED_REGION_A, /** * Closed the merging region B. */ CLOSED_REGION_B, /** * The merging region B has been taken out of the server's online regions list. */ OFFLINED_REGION_B, /** * Started in on creation of the merged region. */ STARTED_MERGED_REGION_CREATION, /** * Point of no return. If we got here, then transaction is not recoverable * other than by crashing out the regionserver. */ PONR d.roll back if step c throws exception Usage: HBaseAdmin#mergeRegions See more details from the patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13611967#comment-13611967 ] stack commented on HBASE-7403: -- +1 on 0.95 and trunk commit. Doesn't really touch core. This is a vital missing feature. Please mark it 'experimental' in the release notes. Excellent work. Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.95.0 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.95.0, 0.98.0 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch, hbase-7403-trunkv23.patch, hbase-7403-trunkv24.patch, hbase-7403-trunkv26.patch, hbase-7403-trunkv28.patch, hbase-7403-trunkv29.patch, hbase-7403-trunkv30.patch, hbase-7403-trunkv31.patch, hbase-7403-trunkv32.patch, hbase-7403-trunkv33.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf Support executing region merge transaction on Regionserver, similar with split transaction Process of merging two regions: a.client sends RPC (dispatch merging regions) to master b.master moves the regions together (on the same regionserver where the more heavily loaded region resided) c.master sends RPC (merge regions) to this regionserver d.Regionserver executes the region merge transaction in the thread pool e.the above b,c,d run asynchronously Process of region merge transaction: a.Construct a new region merge transaction. b.prepare for the merge transaction, the transaction will be canceled if it is unavailable, e.g. two regions don't belong to same table; two regions are not adjacent in a non-compulsory merge; region is closed or has reference c.execute the transaction as the following: /** * Set region as in transition, set it into MERGING state. */ SET_MERGING_IN_ZK, /** * We created the temporary merge data directory. */ CREATED_MERGE_DIR, /** * Closed the merging region A. */ CLOSED_REGION_A, /** * The merging region A has been taken out of the server's online regions list. */ OFFLINED_REGION_A, /** * Closed the merging region B. */ CLOSED_REGION_B, /** * The merging region B has been taken out of the server's online regions list. */ OFFLINED_REGION_B, /** * Started in on creation of the merged region. */ STARTED_MERGED_REGION_CREATION, /** * Point of no return. If we got here, then transaction is not recoverable * other than by crashing out the regionserver. */ PONR d.roll back if step c throws exception Usage: HBaseAdmin#mergeRegions See more details from the patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13610017#comment-13610017
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574965/hbase-7403-trunkv33.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4964//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4964//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4964//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4964//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4964//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4964//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4964//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4964//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4964//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4964//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.95.0
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.95.0, 0.98.0
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch,
hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch,
hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch,
hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch,
hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch,
hbase-7403-trunkv23.patch, hbase-7403-trunkv24.patch,
hbase-7403-trunkv26.patch, hbase-7403-trunkv28.patch,
hbase-7403-trunkv29.patch, hbase-7403-trunkv30.patch,
hbase-7403-trunkv31.patch, hbase-7403-trunkv32.patch,
hbase-7403-trunkv33.patch, hbase-7403-trunkv5.patch,
hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch,
hbase-7403-trunkv9.patch, merge region.pdf
Support executing region merge transaction on Regionserver, similar with
split transaction
Process of merging two regions:
a.client sends RPC (dispatch merging regions) to master
b.master moves the regions together (on the same regionserver where the more
heavily loaded region resided)
c.master sends RPC (merge regions) to this regionserver
d.Regionserver executes the region merge transaction in the thread pool
e.the above b,c,d run asynchronously
Process of region merge transaction:
a.Construct a new region merge transaction.
b.prepare for the merge transaction, the transaction will be canceled if it
is unavailable,
e.g. two regions don't belong to same table; two regions are not adjacent in
a non-compulsory merge; region is closed or has reference
c.execute the transaction as the following:
/**
* Set region as in transition, set it into MERGING state.
*/
SET_MERGING_IN_ZK,
/**
* We
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13610252#comment-13610252 ] Jean-Marc Spaggiari commented on HBASE-7403: Any chance to fix the lines longer than 100 to get a full +1 from Hadoop QA? I'm excited to see this in! I already have many plans for it! ;) Very good job [~zjushch]! Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.95.0 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.95.0, 0.98.0 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch, hbase-7403-trunkv23.patch, hbase-7403-trunkv24.patch, hbase-7403-trunkv26.patch, hbase-7403-trunkv28.patch, hbase-7403-trunkv29.patch, hbase-7403-trunkv30.patch, hbase-7403-trunkv31.patch, hbase-7403-trunkv32.patch, hbase-7403-trunkv33.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf Support executing region merge transaction on Regionserver, similar with split transaction Process of merging two regions: a.client sends RPC (dispatch merging regions) to master b.master moves the regions together (on the same regionserver where the more heavily loaded region resided) c.master sends RPC (merge regions) to this regionserver d.Regionserver executes the region merge transaction in the thread pool e.the above b,c,d run asynchronously Process of region merge transaction: a.Construct a new region merge transaction. b.prepare for the merge transaction, the transaction will be canceled if it is unavailable, e.g. two regions don't belong to same table; two regions are not adjacent in a non-compulsory merge; region is closed or has reference c.execute the transaction as the following: /** * Set region as in transition, set it into MERGING state. */ SET_MERGING_IN_ZK, /** * We created the temporary merge data directory. */ CREATED_MERGE_DIR, /** * Closed the merging region A. */ CLOSED_REGION_A, /** * The merging region A has been taken out of the server's online regions list. */ OFFLINED_REGION_A, /** * Closed the merging region B. */ CLOSED_REGION_B, /** * The merging region B has been taken out of the server's online regions list. */ OFFLINED_REGION_B, /** * Started in on creation of the merged region. */ STARTED_MERGED_REGION_CREATION, /** * Point of no return. If we got here, then transaction is not recoverable * other than by crashing out the regionserver. */ PONR d.roll back if step c throws exception Usage: HBaseAdmin#mergeRegions See more details from the patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13611336#comment-13611336 ] Ted Yu commented on HBASE-7403: --- +1 from me. Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.95.0 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.95.0, 0.98.0 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch, hbase-7403-trunkv23.patch, hbase-7403-trunkv24.patch, hbase-7403-trunkv26.patch, hbase-7403-trunkv28.patch, hbase-7403-trunkv29.patch, hbase-7403-trunkv30.patch, hbase-7403-trunkv31.patch, hbase-7403-trunkv32.patch, hbase-7403-trunkv33.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf Support executing region merge transaction on Regionserver, similar with split transaction Process of merging two regions: a.client sends RPC (dispatch merging regions) to master b.master moves the regions together (on the same regionserver where the more heavily loaded region resided) c.master sends RPC (merge regions) to this regionserver d.Regionserver executes the region merge transaction in the thread pool e.the above b,c,d run asynchronously Process of region merge transaction: a.Construct a new region merge transaction. b.prepare for the merge transaction, the transaction will be canceled if it is unavailable, e.g. two regions don't belong to same table; two regions are not adjacent in a non-compulsory merge; region is closed or has reference c.execute the transaction as the following: /** * Set region as in transition, set it into MERGING state. */ SET_MERGING_IN_ZK, /** * We created the temporary merge data directory. */ CREATED_MERGE_DIR, /** * Closed the merging region A. */ CLOSED_REGION_A, /** * The merging region A has been taken out of the server's online regions list. */ OFFLINED_REGION_A, /** * Closed the merging region B. */ CLOSED_REGION_B, /** * The merging region B has been taken out of the server's online regions list. */ OFFLINED_REGION_B, /** * Started in on creation of the merged region. */ STARTED_MERGED_REGION_CREATION, /** * Point of no return. If we got here, then transaction is not recoverable * other than by crashing out the regionserver. */ PONR d.roll back if step c throws exception Usage: HBaseAdmin#mergeRegions See more details from the patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13611410#comment-13611410 ] chunhui shen commented on HBASE-7403: - line longer than 100 is generated because of protobuf I will commit it tomorrow if no further more comments Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.95.0 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.95.0, 0.98.0 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch, hbase-7403-trunkv23.patch, hbase-7403-trunkv24.patch, hbase-7403-trunkv26.patch, hbase-7403-trunkv28.patch, hbase-7403-trunkv29.patch, hbase-7403-trunkv30.patch, hbase-7403-trunkv31.patch, hbase-7403-trunkv32.patch, hbase-7403-trunkv33.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf Support executing region merge transaction on Regionserver, similar with split transaction Process of merging two regions: a.client sends RPC (dispatch merging regions) to master b.master moves the regions together (on the same regionserver where the more heavily loaded region resided) c.master sends RPC (merge regions) to this regionserver d.Regionserver executes the region merge transaction in the thread pool e.the above b,c,d run asynchronously Process of region merge transaction: a.Construct a new region merge transaction. b.prepare for the merge transaction, the transaction will be canceled if it is unavailable, e.g. two regions don't belong to same table; two regions are not adjacent in a non-compulsory merge; region is closed or has reference c.execute the transaction as the following: /** * Set region as in transition, set it into MERGING state. */ SET_MERGING_IN_ZK, /** * We created the temporary merge data directory. */ CREATED_MERGE_DIR, /** * Closed the merging region A. */ CLOSED_REGION_A, /** * The merging region A has been taken out of the server's online regions list. */ OFFLINED_REGION_A, /** * Closed the merging region B. */ CLOSED_REGION_B, /** * The merging region B has been taken out of the server's online regions list. */ OFFLINED_REGION_B, /** * Started in on creation of the merged region. */ STARTED_MERGED_REGION_CREATION, /** * Point of no return. If we got here, then transaction is not recoverable * other than by crashing out the regionserver. */ PONR d.roll back if step c throws exception Usage: HBaseAdmin#mergeRegions See more details from the patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13611479#comment-13611479 ] Jonathan Hsieh commented on HBASE-7403: --- This is out of scope for the current patch but can we add some follow on issues to add documentation to the book, add shell stuff commands, tooling in the webui (MERGING in the gui, or possibly displaying regions in merging similar to RITs), and access control coproc hooks? Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.95.0 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.95.0, 0.98.0 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch, hbase-7403-trunkv23.patch, hbase-7403-trunkv24.patch, hbase-7403-trunkv26.patch, hbase-7403-trunkv28.patch, hbase-7403-trunkv29.patch, hbase-7403-trunkv30.patch, hbase-7403-trunkv31.patch, hbase-7403-trunkv32.patch, hbase-7403-trunkv33.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf Support executing region merge transaction on Regionserver, similar with split transaction Process of merging two regions: a.client sends RPC (dispatch merging regions) to master b.master moves the regions together (on the same regionserver where the more heavily loaded region resided) c.master sends RPC (merge regions) to this regionserver d.Regionserver executes the region merge transaction in the thread pool e.the above b,c,d run asynchronously Process of region merge transaction: a.Construct a new region merge transaction. b.prepare for the merge transaction, the transaction will be canceled if it is unavailable, e.g. two regions don't belong to same table; two regions are not adjacent in a non-compulsory merge; region is closed or has reference c.execute the transaction as the following: /** * Set region as in transition, set it into MERGING state. */ SET_MERGING_IN_ZK, /** * We created the temporary merge data directory. */ CREATED_MERGE_DIR, /** * Closed the merging region A. */ CLOSED_REGION_A, /** * The merging region A has been taken out of the server's online regions list. */ OFFLINED_REGION_A, /** * Closed the merging region B. */ CLOSED_REGION_B, /** * The merging region B has been taken out of the server's online regions list. */ OFFLINED_REGION_B, /** * Started in on creation of the merged region. */ STARTED_MERGED_REGION_CREATION, /** * Point of no return. If we got here, then transaction is not recoverable * other than by crashing out the regionserver. */ PONR d.roll back if step c throws exception Usage: HBaseAdmin#mergeRegions See more details from the patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13608760#comment-13608760
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574745/hbase-7403-trunkv30.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4940//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4940//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4940//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4940//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4940//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4940//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4940//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4940//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4940//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4940//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.95.0
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.95.0, 0.98.0
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch,
hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch,
hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch,
hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch,
hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch,
hbase-7403-trunkv23.patch, hbase-7403-trunkv24.patch,
hbase-7403-trunkv26.patch, hbase-7403-trunkv28.patch,
hbase-7403-trunkv29.patch, hbase-7403-trunkv30.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf
Support executing region merge transaction on Regionserver, similar with
split transaction
Process of merging two regions:
a.client sends RPC (dispatch merging regions) to master
b.master moves the regions together (on the same regionserver where the more
heavily loaded region resided)
c.master sends RPC (merge regions) to this regionserver
d.Regionserver executes the region merge transaction in the thread pool
e.the above b,c,d run asynchronously
Process of region merge transaction:
a.Construct a new region merge transaction.
b.prepare for the merge transaction, the transaction will be canceled if it
is unavailable,
e.g. two regions don't belong to same table; two regions are not adjacent in
a non-compulsory merge; region is closed or has reference
c.execute the transaction as the following:
/**
* Set region as in transition, set it into MERGING state.
*/
SET_MERGING_IN_ZK,
/**
* We created the temporary merge data directory.
*/
CREATED_MERGE_DIR,
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13608812#comment-13608812
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574758/hbase-7403-trunkv31.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestMasterFailover
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4941//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4941//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4941//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4941//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4941//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4941//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4941//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4941//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4941//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4941//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.95.0
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.95.0, 0.98.0
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch,
hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch,
hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch,
hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch,
hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch,
hbase-7403-trunkv23.patch, hbase-7403-trunkv24.patch,
hbase-7403-trunkv26.patch, hbase-7403-trunkv28.patch,
hbase-7403-trunkv29.patch, hbase-7403-trunkv30.patch,
hbase-7403-trunkv31.patch, hbase-7403-trunkv5.patch,
hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch,
hbase-7403-trunkv9.patch, merge region.pdf
Support executing region merge transaction on Regionserver, similar with
split transaction
Process of merging two regions:
a.client sends RPC (dispatch merging regions) to master
b.master moves the regions together (on the same regionserver where the more
heavily loaded region resided)
c.master sends RPC (merge regions) to this regionserver
d.Regionserver executes the region merge transaction in the thread pool
e.the above b,c,d run asynchronously
Process of region merge transaction:
a.Construct a new region merge transaction.
b.prepare for the merge transaction, the transaction will be canceled if it
is unavailable,
e.g. two regions don't belong to same table; two regions are not adjacent in
a non-compulsory merge; region is closed or has reference
c.execute the transaction as the following:
/**
* Set region as in transition, set it into MERGING state.
*/
SET_MERGING_IN_ZK,
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13609905#comment-13609905
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574944/hbase-7403-trunkv32.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestAssignmentManagerOnCluster
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4960//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4960//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4960//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4960//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4960//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4960//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4960//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4960//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4960//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4960//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.95.0
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.95.0, 0.98.0
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch,
hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch,
hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch,
hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch,
hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch,
hbase-7403-trunkv23.patch, hbase-7403-trunkv24.patch,
hbase-7403-trunkv26.patch, hbase-7403-trunkv28.patch,
hbase-7403-trunkv29.patch, hbase-7403-trunkv30.patch,
hbase-7403-trunkv31.patch, hbase-7403-trunkv32.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf
Support executing region merge transaction on Regionserver, similar with
split transaction
Process of merging two regions:
a.client sends RPC (dispatch merging regions) to master
b.master moves the regions together (on the same regionserver where the more
heavily loaded region resided)
c.master sends RPC (merge regions) to this regionserver
d.Regionserver executes the region merge transaction in the thread pool
e.the above b,c,d run asynchronously
Process of region merge transaction:
a.Construct a new region merge transaction.
b.prepare for the merge transaction, the transaction will be canceled if it
is unavailable,
e.g. two regions don't belong to same table; two regions are not adjacent in
a non-compulsory merge; region is closed or has reference
c.execute the transaction as the following:
/**
* Set region as in transition, set it into MERGING state.
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13607510#comment-13607510
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574516/hbase-7403-trunkv28.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 site{color}. The mvn site goal succeeds with this patch.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4907//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4907//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4907//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4907//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4907//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4907//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4907//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4907//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4907//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4907//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.95.0
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.95.0, 0.98.0
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch,
hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch,
hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch,
hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch,
hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch,
hbase-7403-trunkv23.patch, hbase-7403-trunkv24.patch,
hbase-7403-trunkv26.patch, hbase-7403-trunkv28.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf
Support executing region merge transaction on Regionserver, similar with
split transaction
Process of merging two regions:
a.client sends RPC (dispatch merging regions) to master
b.master moves the regions together (on the same regionserver where the more
heavily loaded region resided)
c.master sends RPC (merge regions) to this regionserver
d.Regionserver executes the region merge transaction in the thread pool
e.the above b,c,d run asynchronously
Process of region merge transaction:
a.Construct a new region merge transaction.
b.prepare for the merge transaction, the transaction will be canceled if it
is unavailable,
e.g. two regions don't belong to same table; two regions are not adjacent in
a non-compulsory merge; region is closed or has reference
c.execute the transaction as the following:
/**
* Set region as in transition, set it into MERGING state.
*/
SET_MERGING_IN_ZK,
/**
* We created the temporary merge data directory.
*/
CREATED_MERGE_DIR,
/**
* Closed the merging region A.
*/
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13608660#comment-13608660
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574731/hbase-7403-trunkv29.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4938//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.95.0
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.95.0, 0.98.0
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch,
hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch,
hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch,
hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch,
hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch,
hbase-7403-trunkv23.patch, hbase-7403-trunkv24.patch,
hbase-7403-trunkv26.patch, hbase-7403-trunkv28.patch,
hbase-7403-trunkv29.patch, hbase-7403-trunkv5.patch,
hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch,
hbase-7403-trunkv9.patch, merge region.pdf
Support executing region merge transaction on Regionserver, similar with
split transaction
Process of merging two regions:
a.client sends RPC (dispatch merging regions) to master
b.master moves the regions together (on the same regionserver where the more
heavily loaded region resided)
c.master sends RPC (merge regions) to this regionserver
d.Regionserver executes the region merge transaction in the thread pool
e.the above b,c,d run asynchronously
Process of region merge transaction:
a.Construct a new region merge transaction.
b.prepare for the merge transaction, the transaction will be canceled if it
is unavailable,
e.g. two regions don't belong to same table; two regions are not adjacent in
a non-compulsory merge; region is closed or has reference
c.execute the transaction as the following:
/**
* Set region as in transition, set it into MERGING state.
*/
SET_MERGING_IN_ZK,
/**
* We created the temporary merge data directory.
*/
CREATED_MERGE_DIR,
/**
* Closed the merging region A.
*/
CLOSED_REGION_A,
/**
* The merging region A has been taken out of the server's online regions
list.
*/
OFFLINED_REGION_A,
/**
* Closed the merging region B.
*/
CLOSED_REGION_B,
/**
* The merging region B has been taken out of the server's online regions
list.
*/
OFFLINED_REGION_B,
/**
* Started in on creation of the merged region.
*/
STARTED_MERGED_REGION_CREATION,
/**
* Point of no return. If we got here, then transaction is not recoverable
* other than by crashing out the regionserver.
*/
PONR
d.roll back if step c throws exception
Usage:
HBaseAdmin#mergeRegions
See more details from the patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13606382#comment-13606382
]
Ted Yu commented on HBASE-7403:
---
From https://builds.apache.org/job/PreCommit-HBASE-Build/4884/console :
{code}
pool-1-thread-1 prio=10 tid=0x773cec00 nid=0x5cfc in Object.wait()
[0x7747]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on 0x7f0afe98 (a
org.apache.hadoop.hbase.util.JVMClusterUtil$RegionServerThread)
at java.lang.Thread.join(Thread.java:1194)
- locked 0x7f0afe98 (a
org.apache.hadoop.hbase.util.JVMClusterUtil$RegionServerThread)
at
org.apache.hadoop.hbase.util.JVMClusterUtil.shutdown(JVMClusterUtil.java:252)
at
org.apache.hadoop.hbase.LocalHBaseCluster.shutdown(LocalHBaseCluster.java:430)
at
org.apache.hadoop.hbase.MiniHBaseCluster.shutdown(MiniHBaseCluster.java:501)
at
org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:919)
at
org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:889)
at
org.apache.hadoop.hbase.mapreduce.TestTableInputFormatScan.tearDownAfterClass(TestTableInputFormatScan.java:84)
{code}
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.95.0
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.95.0, 0.98.0
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch,
hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch,
hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch,
hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch,
hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch,
hbase-7403-trunkv23.patch, hbase-7403-trunkv24.patch,
hbase-7403-trunkv26.patch, hbase-7403-trunkv5.patch,
hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch,
hbase-7403-trunkv9.patch, merge region.pdf
Support executing region merge transaction on Regionserver, similar with
split transaction
Process of merging two regions:
a.client sends RPC (dispatch merging regions) to master
b.master moves the regions together (on the same regionserver where the more
heavily loaded region resided)
c.master sends RPC (merge regions) to this regionserver
d.Regionserver executes the region merge transaction in the thread pool
e.the above b,c,d run asynchronously
Process of region merge transaction:
a.Construct a new region merge transaction.
b.prepare for the merge transaction, the transaction will be canceled if it
is unavailable,
e.g. two regions don't belong to same table; two regions are not adjacent in
a non-compulsory merge; region is closed or has reference
c.execute the transaction as the following:
/**
* Set region as in transition, set it into MERGING state.
*/
SET_MERGING_IN_ZK,
/**
* We created the temporary merge data directory.
*/
CREATED_MERGE_DIR,
/**
* Closed the merging region A.
*/
CLOSED_REGION_A,
/**
* The merging region A has been taken out of the server's online regions
list.
*/
OFFLINED_REGION_A,
/**
* Closed the merging region B.
*/
CLOSED_REGION_B,
/**
* The merging region B has been taken out of the server's online regions
list.
*/
OFFLINED_REGION_B,
/**
* Started in on creation of the merged region.
*/
STARTED_MERGED_REGION_CREATION,
/**
* Point of no return. If we got here, then transaction is not recoverable
* other than by crashing out the regionserver.
*/
PONR
d.roll back if step c throws exception
Usage:
HBaseAdmin#mergeRegions
See more details from the patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13606072#comment-13606072
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574320/hbase-7403-trunkv26.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4884//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4884//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4884//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4884//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4884//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4884//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4884//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4884//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4884//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4884//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.95.0
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.95.0, 0.98.0
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch,
hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch,
hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch,
hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch,
hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch,
hbase-7403-trunkv23.patch, hbase-7403-trunkv24.patch,
hbase-7403-trunkv26.patch, hbase-7403-trunkv5.patch,
hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch,
hbase-7403-trunkv9.patch, merge region.pdf
Support executing region merge transaction on Regionserver, similar with
split transaction
Process of merging two regions:
a.client sends RPC (dispatch merging regions) to master
b.master moves the regions together (on the same regionserver where the more
heavily loaded region resided)
c.master sends RPC (merge regions) to this regionserver
d.Regionserver executes the region merge transaction in the thread pool
e.the above b,c,d run asynchronously
Process of region merge transaction:
a.Construct a new region merge transaction.
b.prepare for the merge transaction, the transaction will be canceled if it
is unavailable,
e.g. two regions don't belong to same table; two regions are not adjacent in
a non-compulsory merge; region is closed or has reference
c.execute the transaction as the following:
/**
* Set region as in transition, set it into MERGING state.
*/
SET_MERGING_IN_ZK,
/**
* We created the temporary merge data directory.
*/
CREATED_MERGE_DIR,
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13604526#comment-13604526
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12574034/hbase-7403-trunkv24.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4853//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4853//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4853//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4853//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4853//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4853//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4853//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4853//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4853//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4853//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.95.0
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.95.0, 0.98.0
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch,
hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch,
hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch,
hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch,
hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch,
hbase-7403-trunkv23.patch, hbase-7403-trunkv24.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf
Support executing region merge transaction on Regionserver, similar with
split transaction
Process of merging two regions:
a.client sends RPC (dispatch merging regions) to master
b.master moves the regions together (on the same regionserver where the more
heavily loaded region resided)
c.master sends RPC (merge regions) to this regionserver
d.Regionserver executes the region merge transaction in the thread pool
e.the above b,c,d run asynchronously
Process of region merge transaction:
a.Construct a new region merge transaction.
b.prepare for the merge transaction, the transaction will be canceled if it
is unavailable,
e.g. two regions don't belong to same table; two regions are not adjacent in
a non-compulsory merge; region is closed or has reference
c.execute the transaction as the following:
/**
* Set region as in transition, set it into MERGING state.
*/
SET_MERGING_IN_ZK,
/**
* We created the temporary merge data directory.
*/
CREATED_MERGE_DIR,
/**
* Closed the merging region A.
*/
CLOSED_REGION_A,
/**
* The merging region A
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13604147#comment-13604147 ] Ted Yu commented on HBASE-7403: --- Thanks for the quick response. I will try to go over the new tests. This is mostly ready to go. Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.95.0 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.95.0, 0.98.0 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch, hbase-7403-trunkv23.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf Support executing region merge transaction on Regionserver, similar with split transaction Process of merging two regions: a.client send RPC(dispacth merging regions) to master b.master move the regions together (on the same regionserver) c.master send RPC(merge regions) to regionserver d.Regionserver execute the regions merge transaction in the thread pool e.the above b,c,d run asynchronously Process of region merge transaction: a.Construct a new region merge transaction. b.prepare for the merge transaction, the transaction will be canceled if it is unavailable, e.g. two regions don't belong to same table; two regions are not adjacent in a non-compulsory merge; region is closed or has reference c.execute the transaction as the following: /** * Set region as in transition, set it into MERGING state. */ SET_MERGING_IN_ZK, /** * We created the temporary merge data directory. */ CREATED_MERGE_DIR, /** * Closed the merging region A. */ CLOSED_REGION_A, /** * The merging region A has been taken out of the server's online regions list. */ OFFLINED_REGION_A, /** * Closed the merging region B. */ CLOSED_REGION_B, /** * The merging region B has been taken out of the server's online regions list. */ OFFLINED_REGION_B, /** * Started in on creation of the merged region. */ STARTED_MERGED_REGION_CREATION, /** * Point of no return. If we got here, then transaction is not recoverable * other than by crashing out the regionserver. */ PONR d.roll back if step c throws exception Usage: HBaseAdmin#mergeRegions See more details from the patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13604126#comment-13604126
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12573993/hbase-7403-trunkv23.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4846//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4846//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4846//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4846//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4846//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4846//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4846//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4846//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4846//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4846//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.95.0
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.95.0, 0.98.0
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch,
hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch,
hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch,
hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch,
hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch,
hbase-7403-trunkv23.patch, hbase-7403-trunkv5.patch,
hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch,
hbase-7403-trunkv9.patch, merge region.pdf
Support executing region merge transaction on Regionserver, similar with
split transaction
Process of merging two regions:
a.client send RPC(dispacth merging regions) to master
b.master move the regions together (on the same regionserver)
c.master send RPC(merge regions) to regionserver
d.Regionserver execute the regions merge transaction in the thread pool
e.the above b,c,d run asynchronously
Process of region merge transaction:
a.Construct a new region merge transaction.
b.prepare for the merge transaction, the transaction will be canceled if it
is unavailable,
e.g. two regions don't belong to same table; two regions are not adjacent in
a non-compulsory merge; region is closed or has reference
c.execute the transaction as the following:
/**
* Set region as in transition, set it into MERGING state.
*/
SET_MERGING_IN_ZK,
/**
* We created the temporary merge data directory.
*/
CREATED_MERGE_DIR,
/**
* Closed the merging region A.
*/
CLOSED_REGION_A,
/**
* The merging region A has been taken out of the server's online regions
list.
*/
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13600958#comment-13600958
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12573486/hbase-7403-trunkv22.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4796//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4796//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4796//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4796//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4796//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4796//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4796//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4796//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4796//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4796//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.95.0
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.95.0, 0.98.0
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch,
hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch,
hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch,
hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch,
hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf
Support executing region merge transaction on Regionserver, similar with
split transaction
Process of merging two regions:
a.client send RPC(dispacth merging regions) to master
b.master move the regions together (on the same regionserver)
c.master send RPC(merge regions) to regionserver
d.Regionserver execute the regions merge transaction in the thread pool
e.the above b,c,d run asynchronously
Process of region merge transaction:
a.Construct a new region merge transaction.
b.prepare for the merge transaction, the transaction will be canceled if it
is unavailable,
e.g. two regions don't belong to same table; two regions are not adjacent in
a non-compulsory merge; region is closed or has reference
c.execute the transaction as the following:
/**
* Set region as in transition, set it into MERGING state.
*/
SET_MERGING_IN_ZK,
/**
* We created the temporary merge data directory.
*/
CREATED_MERGE_DIR,
/**
* Closed the merging region A.
*/
CLOSED_REGION_A,
/**
* The merging region A has been taken out of the server's online regions
list.
*/
OFFLINED_REGION_A,
/**
* Closed
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13600986#comment-13600986 ] chunhui shen commented on HBASE-7403: - This could be ready to go. Anybody to take a look? Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.95.0 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.95.0, 0.98.0 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf Support executing region merge transaction on Regionserver, similar with split transaction Process of merging two regions: a.client send RPC(dispacth merging regions) to master b.master move the regions together (on the same regionserver) c.master send RPC(merge regions) to regionserver d.Regionserver execute the regions merge transaction in the thread pool e.the above b,c,d run asynchronously Process of region merge transaction: a.Construct a new region merge transaction. b.prepare for the merge transaction, the transaction will be canceled if it is unavailable, e.g. two regions don't belong to same table; two regions are not adjacent in a non-compulsory merge; region is closed or has reference c.execute the transaction as the following: /** * Set region as in transition, set it into MERGING state. */ SET_MERGING_IN_ZK, /** * We created the temporary merge data directory. */ CREATED_MERGE_DIR, /** * Closed the merging region A. */ CLOSED_REGION_A, /** * The merging region A has been taken out of the server's online regions list. */ OFFLINED_REGION_A, /** * Closed the merging region B. */ CLOSED_REGION_B, /** * The merging region B has been taken out of the server's online regions list. */ OFFLINED_REGION_B, /** * Started in on creation of the merged region. */ STARTED_MERGED_REGION_CREATION, /** * Point of no return. If we got here, then transaction is not recoverable * other than by crashing out the regionserver. */ PONR d.roll back if step c throws exception Usage: HBaseAdmin#mergeRegions See more details from the patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13601944#comment-13601944 ] chunhui shen commented on HBASE-7403: - [~enis] Could you take a review ? Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.95.0 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.95.0, 0.98.0 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv20.patch, hbase-7403-trunkv22.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf Support executing region merge transaction on Regionserver, similar with split transaction Process of merging two regions: a.client send RPC(dispacth merging regions) to master b.master move the regions together (on the same regionserver) c.master send RPC(merge regions) to regionserver d.Regionserver execute the regions merge transaction in the thread pool e.the above b,c,d run asynchronously Process of region merge transaction: a.Construct a new region merge transaction. b.prepare for the merge transaction, the transaction will be canceled if it is unavailable, e.g. two regions don't belong to same table; two regions are not adjacent in a non-compulsory merge; region is closed or has reference c.execute the transaction as the following: /** * Set region as in transition, set it into MERGING state. */ SET_MERGING_IN_ZK, /** * We created the temporary merge data directory. */ CREATED_MERGE_DIR, /** * Closed the merging region A. */ CLOSED_REGION_A, /** * The merging region A has been taken out of the server's online regions list. */ OFFLINED_REGION_A, /** * Closed the merging region B. */ CLOSED_REGION_B, /** * The merging region B has been taken out of the server's online regions list. */ OFFLINED_REGION_B, /** * Started in on creation of the merged region. */ STARTED_MERGED_REGION_CREATION, /** * Point of no return. If we got here, then transaction is not recoverable * other than by crashing out the regionserver. */ PONR d.roll back if step c throws exception Usage: HBaseAdmin#mergeRegions See more details from the patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598191#comment-13598191
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12572950/hbase-7403-trunkv20.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.constraint.TestConstraint
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4745//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4745//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4745//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4745//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4745//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4745//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4745//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4745//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4745//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4745//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.95.0, 0.94.6
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.95.0
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch,
hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch,
hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch,
hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch,
hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv20.patch, hbase-7403-trunkv5.patch,
hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch,
hbase-7403-trunkv9.patch, merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
Usage:
1.Tool:
bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show]
table-name region-encodedname-1 region-encodedname-2
2.API: static void MergeManager#createMergeRequest
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598161#comment-13598161 ] Ted Yu commented on HBASE-7403: --- bq. master move the regions together(on the regionserver) What's the implication of the above action ? If one (or both) of the regions were receiving non-trivial load prior to this action, would client(s) be affected ? Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.95.0, 0.94.6 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.95.0 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions Usage: 1.Tool: bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show] table-name region-encodedname-1 region-encodedname-2 2.API: static void MergeManager#createMergeRequest We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598164#comment-13598164 ] chunhui shen commented on HBASE-7403: - bq.If one (or both) of the regions were receiving non-trivial load prior to this action, would client(s) be affected ? Yes, region would be off services in a short time, it is equal with moving region, e.g balance a region Compared with split, region merge operation would take a overhead of moving one region Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.95.0, 0.94.6 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.95.0 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions Usage: 1.Tool: bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show] table-name region-encodedname-1 region-encodedname-2 2.API: static void MergeManager#createMergeRequest We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598165#comment-13598165
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12572947/hbase-7403-trunkv19.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 3
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 site{color}. The patch appears to cause mvn site goal to
fail.
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.backup.TestHFileArchiving
org.apache.hadoop.hbase.client.TestHTableMultiplexer
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.client.TestMultiParallel.testBatchWithDelete(TestMultiParallel.java:343)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4744//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4744//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4744//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-client.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4744//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4744//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4744//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4744//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4744//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4744//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4744//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.95.0, 0.94.6
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.95.0
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch,
hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch,
hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch,
hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch,
hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
Usage:
1.Tool:
bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show]
table-name region-encodedname-1 region-encodedname-2
2.API: static void MergeManager#createMergeRequest
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598168#comment-13598168
]
Ted Yu commented on HBASE-7403:
---
{code}
+if (!onSameRS) {
+ // Move region_b to region a's location
+ RegionPlan regionPlan = new RegionPlan(region_b, region_b_location,
+ region_a_location);
{code}
Can we consider metrics so that region with less load is moved onto region
server where region with more load resides ?
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.95.0, 0.94.6
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.95.0
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch,
hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch,
hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch,
hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch,
hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
Usage:
1.Tool:
bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show]
table-name region-encodedname-1 region-encodedname-2
2.API: static void MergeManager#createMergeRequest
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it until successful if it throws exception or abort or server restart, but
couldn’t be rolled back.
It depends on
Use zookeeper to record the transaction journal state, make redo easier
Use zookeeper to send/receive merge request
Merge transaction is executed on the master
Support calling merge request through API or shell tool
About the merge process, please see the attachment and patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13598173#comment-13598173 ] chunhui shen commented on HBASE-7403: - bq.Can we consider metrics so that region with less load is moved onto region server where region with more load resides ? It seems better. -1 site. The patch appears to cause mvn site goal to fail. I couldn't find the reason of this warning... Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.95.0, 0.94.6 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.95.0 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, hbase-7403-trunkv19.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions Usage: 1.Tool: bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show] table-name region-encodedname-1 region-encodedname-2 2.API: static void MergeManager#createMergeRequest We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13594883#comment-13594883 ] Daniel Einspanjer commented on HBASE-7403: -- What is the likely-hood of being able to apply this patch to run against CDH 0.90.6-cdh3u4 ? The codebase has changed too much for the old online_merge.rb script to run without significant overhauling, and we need to do some region reduction sooner than we'd be able to upgrade to CDH4. I'm just wondering if it is a worthy task to investigate or if it would be a deep dark rabbit hole. :) Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.95.0, 0.94.6 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.95.0 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions Usage: 1.Tool: bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show] table-name region-encodedname-1 region-encodedname-2 2.API: static void MergeManager#createMergeRequest We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13594944#comment-13594944 ] Ted Yu commented on HBASE-7403: --- @Daniel: 0.90.x is too different from 0.94, let alone trunk. 0.94.y, a future release, would be more realistic target. Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.95.0, 0.94.6 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.95.0 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions Usage: 1.Tool: bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show] table-name region-encodedname-1 region-encodedname-2 2.API: static void MergeManager#createMergeRequest We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13595007#comment-13595007 ] Lars Hofhansl commented on HBASE-7403: -- Let's prove this out in trunk first. Then we can make separate decisions about other release. 0.90 and 0.92 are out of the question (imho), so might 0.94 (as this needs a lot of other plumbing - table locks, atomic META updates, etc) Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.95.0, 0.94.6 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.95.0 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions Usage: 1.Tool: bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show] table-name region-encodedname-1 region-encodedname-2 2.API: static void MergeManager#createMergeRequest We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586691#comment-13586691 ] Enis Soztutar commented on HBASE-7403: -- Lars, we can actually backport this without the dependency on HBASE-7721. Just like online schema alter, we can say that while running the merge, you should ensure that no splits, or alter tables are going on. Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.96.0 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions Usage: 1.Tool: bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show] table-name region-encodedname-1 region-encodedname-2 2.API: static void MergeManager#createMergeRequest We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13586727#comment-13586727 ] Lars Hofhansl commented on HBASE-7403: -- If you want it in, that's fine too. Like online-schema-alter I feel that without HBASE-7721 this is a recipe for disaster. Personally I would love to get HBASE-7721 into 0.94, that would allow for safe online-schema-alter and online-merge. Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.96.0 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions Usage: 1.Tool: bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show] table-name region-encodedname-1 region-encodedname-2 2.API: static void MergeManager#createMergeRequest We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13580581#comment-13580581 ] chunhui shen commented on HBASE-7403: - I'm waiting HBASE-7721 to commit. Current patch use BlockingMetaScannerVisitor to provide consistency view of META entries of table, it is similar with split Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.96.0, 0.94.6 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions Usage: 1.Tool: bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show] table-name region-encodedname-1 region-encodedname-2 2.API: static void MergeManager#createMergeRequest We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13580344#comment-13580344 ] Lars Hofhansl commented on HBASE-7403: -- Where are we with this? [~zjushch] Are you waiting for more comments? Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.96.0, 0.94.6 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch, hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions Usage: 1.Tool: bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show] table-name region-encodedname-1 region-encodedname-2 2.API: static void MergeManager#createMergeRequest We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13573180#comment-13573180
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12568364/hbase-7403-trunkv16.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4363//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4363//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4363//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4363//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4363//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-prefix-tree.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4363//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4363//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4363//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4363//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.96.0, 0.94.6
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch,
hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch,
hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch,
hbase-7403-trunkv15.patch, hbase-7403-trunkv16.patch,
hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch,
hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch,
merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
Usage:
1.Tool:
bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show]
table-name region-encodedname-1 region-encodedname-2
2.API: static void MergeManager#createMergeRequest
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13571173#comment-13571173
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12567976/hbase-7403-trunkv15.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4328//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4328//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4328//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4328//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4328//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4328//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4328//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4328//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.96.0, 0.94.6
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch,
hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch,
hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch,
hbase-7403-trunkv15.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
Usage:
1.Tool:
bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show]
table-name region-encodedname-1 region-encodedname-2
2.API: static void MergeManager#createMergeRequest
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13568600#comment-13568600
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12567543/hbase-7403-trunkv14.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestMasterFailover
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hbase.master.TestMasterFailover.testMasterFailoverWithMockedRIT(TestMasterFailover.java:391)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4290//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4290//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4290//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4290//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4290//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4290//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4290//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4290//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.96.0, 0.94.6
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch,
hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch,
hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch,
hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch,
hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch,
merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
Usage:
1.Tool:
bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show]
table-name region-encodedname-1 region-encodedname-2
2.API: static void MergeManager#createMergeRequest
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13568918#comment-13568918
]
Ted Yu commented on HBASE-7403:
---
I ran TestMasterFailover locally with patch and it passed.
From https://builds.apache.org/job/PreCommit-HBASE-Build/4290/consoleFull :
{code}
NumberFormatException: Thread 1802
(MASTER_OPEN_REGION-asf002.sp2.ygridcore.net,36491,1359710601974-2):
State: WAITING
Blocked count: 3
Waited count: 3
Waiting on
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject@d8e38a
Stack:
sun.misc.Unsafe.park(Native Method)
java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.await(AbstractQueuedSynchronizer.java:1987)
java.util.concurrent.LinkedBlockingQueue.take(LinkedBlockingQueue.java:399)
java.util.concurrent.ThreadPoolExecutor.getTask(ThreadPoolExecutor.java:947)
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:907)
java.lang.Thread.run(Thread.java:662)
{code}
I wonder what the above tells us.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.96.0, 0.94.6
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch,
hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch,
hbase-7403-trunkv13.patch, hbase-7403-trunkv14.patch,
hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch,
hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch,
merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
Usage:
1.Tool:
bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show]
table-name region-encodedname-1 region-encodedname-2
2.API: static void MergeManager#createMergeRequest
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it until successful if it throws exception or abort or server restart, but
couldn’t be rolled back.
It depends on
Use zookeeper to record the transaction journal state, make redo easier
Use zookeeper to send/receive merge request
Merge transaction is executed on the master
Support calling merge request through API or shell tool
About the merge process, please see the attachment and patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13569381#comment-13569381
]
chunhui shen commented on HBASE-7403:
-
The above stack seems normal.
The failed test is
{code}
testMasterFailoverWithMockedRIT(org.apache.hadoop.hbase.master.TestMasterFailover):
test timed out after 18 milliseconds
-1 core zombie tests{color}. There are 1 zombie test(s): at
org.apache.hadoop.hbase.master.TestMasterFailover.testMasterFailoverWithMockedRIT(TestMasterFailover.java:391){code}
{code}
Thread-613 prio=10 tid=0x766df800 nid=0xae3 in Object.wait() [0x770fe000]
java.lang.Thread.State: TIMED_WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on lt;0x83a5cad8 (a
org.apache.hadoop.hbase.util.JVMClusterUtil$RegionServerThread)
at java.lang.Thread.join(Thread.java:1194)
- locked lt;0x83a5cad8 (a
org.apache.hadoop.hbase.util.JVMClusterUtil$RegionServerThread)
at
org.apache.hadoop.hbase.util.Threads.threadDumpingIsAlive(Threads.java:119)
at
org.apache.hadoop.hbase.LocalHBaseCluster.join(LocalHBaseCluster.java:399)
at
org.apache.hadoop.hbase.MiniHBaseCluster.waitUntilShutDown(MiniHBaseCluster.java:676)
at
org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniHBaseCluster(HBaseTestingUtility.java:895)
at
org.apache.hadoop.hbase.HBaseTestingUtility.shutdownMiniCluster(HBaseTestingUtility.java:863)
at
org.apache.hadoop.hbase.master.TestMasterFailover.testMasterFailoverWithMockedRIT(TestMasterFailover.java:391)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
From{code}
Automatic Stack Trace every 60 seconds waiting on
RegionServer:0;asf002.sp2.ygridcore.net,58219,1359710594587
{code}
We could see regionserver is hanged when stopping, causing test failed.
Through the Stack Trace every 60 seconds, we could find that
RegionServer:0;asf002.sp2.ygridcore.net,58219,1359710594587 is hanged by the
following stack:
{code}
NumberFormatException: Thread 1251
(RegionServer:0;asf002.sp2.ygridcore.net,58219,1359710594587):
State: TIMED_WAITING
Blocked count: 41
Waited count: 1179
Stack:
java.lang.Thread.sleep(Native Method)
org.apache.hadoop.hbase.util.Threads.sleep(Threads.java:133)
org.apache.hadoop.hbase.regionserver.HRegionServer.waitOnAllRegionsToClose(HRegionServer.java:1127)
org.apache.hadoop.hbase.regionserver.HRegionServer.run(HRegionServer.java:963)
org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.runRegionServer(MiniHBaseCluster.java:151)
org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.access$000(MiniHBaseCluster.java:103)
org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer$1.run(MiniHBaseCluster.java:135)
java.security.AccessController.doPrivileged(Native Method)
javax.security.auth.Subject.doAs(Subject.java:337)
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1118)
sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
java.lang.reflect.Method.invoke(Method.java:597)
org.apache.hadoop.hbase.util.Methods.call(Methods.java:41)
org.apache.hadoop.hbase.security.User.call(User.java:404)
org.apache.hadoop.hbase.security.User.access$300(User.java:52)
org.apache.hadoop.hbase.security.User$SecureHadoopUser.runAs(User.java:244)
org.apache.hadoop.hbase.MiniHBaseCluster$MiniHBaseClusterRegionServer.run(MiniHBaseCluster.java:133)
java.lang.Thread.run(Thread.java:662)
NumberFormatException: Thread 1747
(RS_CLOSE_REGION-asf002.sp2.ygridcore.net,58219,1359710594587-0):
State: RUNNABLE
Blocked count: 5
Waited count: 7
Stack:
java.util.TreeMap.fixAfterDeletion(TreeMap.java:2193)
java.util.TreeMap.deleteEntry(TreeMap.java:2151)
java.util.TreeMap.remove(TreeMap.java:585)
java.util.TreeSet.remove(TreeSet.java:259)
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13567457#comment-13567457
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12567312/hbase-7403-trunkv13.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 9 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.wal.TestHLog
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4269//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4269//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4269//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4269//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4269//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4269//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4269//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4269//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Priority: Critical
Fix For: 0.96.0, 0.94.6
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch,
hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch,
hbase-7403-trunkv13.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
Usage:
1.Tool:
bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show]
table-name region-encodedname-1 region-encodedname-2
2.API: static void MergeManager#createMergeRequest
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13566304#comment-13566304 ] chunhui shen commented on HBASE-7403: - bq.The introduction of the two new qualifiers (mergeA,mergeB) for merged region in .META. I think the new qualifiers wouldn't affect current logic, or give an example? Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.96.0, 0.94.6 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions Usage: 1.Tool: bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show] table-name region-encodedname-1 region-encodedname-2 2.API: static void MergeManager#createMergeRequest We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13565502#comment-13565502
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12566995/hbase-7403-trunkv12.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.master.TestCatalogJanitor
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4234//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4234//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4234//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4234//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4234//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4234//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4234//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4234//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.6
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch,
hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch,
hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch,
hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch,
merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
Usage:
1.Tool:
bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show]
table-name region-encodedname-1 region-encodedname-2
2.API: static void MergeManager#createMergeRequest
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it until successful if it throws exception or
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13565619#comment-13565619 ] Ted Yu commented on HBASE-7403: --- I put some comment on review board. The introduction of the two new qualifiers (mergeA,mergeB) for merged region in .META. makes split/merge scenarios more complicated. Suppose we go along this route, I would expect hbck to be enhanced so that it can recognize / repair merge-related errors. Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.6 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions Usage: 1.Tool: bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show] table-name region-encodedname-1 region-encodedname-2 2.API: static void MergeManager#createMergeRequest We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13566079#comment-13566079 ] chunhui shen commented on HBASE-7403: - Thanks Ted I realized a potential bug when fixing problem of overlapping regions. I will re-modify it as review comments Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Priority: Critical Fix For: 0.96.0, 0.94.6 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv12.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions Usage: 1.Tool: bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show] table-name region-encodedname-1 region-encodedname-2 2.API: static void MergeManager#createMergeRequest We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13561868#comment-13561868 ] Enis Soztutar commented on HBASE-7403: -- bq. Looks like you have 2 +1's already. I had put up some comments in RB. It is better to address them, or explicitly defer to another jira without going with commit. Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.5 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions Usage: 1.Tool: bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show] table-name region-encodedname-1 region-encodedname-2 2.API: static void MergeManager#createMergeRequest We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13560793#comment-13560793 ] ramkrishna.s.vasudevan commented on HBASE-7403: --- I dont have much comments here. clarified my doubts with Chunhui. Overall the functionality seems fine and the scenarios have been taken care. So +1 from me. Thanks Chunhui. Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.5 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions Usage: 1.Tool: bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show] table-name region-encodedname-1 region-encodedname-2 2.API: static void MergeManager#createMergeRequest We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13561190#comment-13561190 ] Ted Yu commented on HBASE-7403: --- @Chunhui: Looks like you have 2 +1's already. Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.5 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions Usage: 1.Tool: bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show] table-name region-encodedname-1 region-encodedname-2 2.API: static void MergeManager#createMergeRequest We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13561288#comment-13561288 ] Matteo Bertozzi commented on HBASE-7403: Merging two regions by moving the hfiles means breaking snapshots and cloned tables. since the snapshot want expect the hfile in a location like /table/region/family and the region now is changed. Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.5 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions Usage: 1.Tool: bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show] table-name region-encodedname-1 region-encodedname-2 2.API: static void MergeManager#createMergeRequest We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13561405#comment-13561405 ] ramkrishna.s.vasudevan commented on HBASE-7403: --- @Matteo Oh I see. I am totally not aware of the snapshot feature. So previously when we do offline merge, was it impacting snapshot? I will read and understand the snapshot feature sooner. Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.5 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions Usage: 1.Tool: bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show] table-name region-encodedname-1 region-encodedname-2 2.API: static void MergeManager#createMergeRequest We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13561410#comment-13561410 ] chunhui shen commented on HBASE-7403: - [~mbertozzi] bq.Merging two regions by moving the hfiles means breaking snapshots and cloned tables. Thanks for reminding me about snapshots. I think there are two solutions: 1.Disable merging regions if it would break snapshots or cloned tables. 2.Creating reference files instead of moving the hfiles when merging. I will update patch later with solution 2 if you also think so. Is there any other better suggestion? Thanks Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.5 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions Usage: 1.Tool: bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show] table-name region-encodedname-1 region-encodedname-2 2.API: static void MergeManager#createMergeRequest We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13561421#comment-13561421 ] Matteo Bertozzi commented on HBASE-7403: Yeah sorry I forgot to post the easy solution. Since the snapshot relies on region/family/file an hfile can't change location so you need to use the reference file (as done in the split) to move the file in another region. (If the offline merge version perform a rename, it has the same problem) Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.5 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions Usage: 1.Tool: bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show] table-name region-encodedname-1 region-encodedname-2 2.API: static void MergeManager#createMergeRequest We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13553614#comment-13553614
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12564899/hbase-7403-trunkv11.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/4021//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4021//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4021//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4021//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4021//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4021//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4021//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/4021//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/4021//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.5
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch,
hbase-7403-trunkv11.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
Usage:
1.Tool:
bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show]
table-name region-encodedname-1 region-encodedname-2
2.API: static void MergeManager#createMergeRequest
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13554150#comment-13554150 ] ramkrishna.s.vasudevan commented on HBASE-7403: --- Checking out some testcases and how its working. Wil complete by tomorrow. By the time if reviews are complete you can go ahead and merge the same. Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.5 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv11.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions Usage: 1.Tool: bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show] table-name region-encodedname-1 region-encodedname-2 2.API: static void MergeManager#createMergeRequest We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13553012#comment-13553012 ] Enis Soztutar commented on HBASE-7403: -- bq. If we send the merge request through RPC, we should also initiate the merge on zk by master. We can keep the current infra for handling the state in zk, but not expose these operations to the client though a master API. As you said, upon getting the call in master, it can just initiate the transaction in zk. bq. Additionally, using ZK won't break the RPC protocol, and it's easy to backport I don't think it will make it harder to backport. We need to isolate the client from directly initiating zk requests. If we expose much internal zk state, it will be way harder to keep BC with older clients. Can you please also put the patch in review board. Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.5 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions Usage: 1.Tool: bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show] table-name region-encodedname-1 region-encodedname-2 2.API: static void MergeManager#createMergeRequest We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13553017#comment-13553017 ] Ted Yu commented on HBASE-7403: --- Here is review request: https://reviews.apache.org/r/8716/ Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.5 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions Usage: 1.Tool: bin/hbase org.apache.hadoop.hbase.util.OnlineMerge [-force] [-async] [-show] table-name region-encodedname-1 region-encodedname-2 2.API: static void MergeManager#createMergeRequest We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13551730#comment-13551730 ] Enis Soztutar commented on HBASE-7403: -- Sorry, I came into this late. I noticed that the merge is initiated through zk. Wouldn't it be better to add a merge() RPC to the master interface, and the simplify the invocation from the client to just call that? Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.5 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13551775#comment-13551775 ] chunhui shen commented on HBASE-7403: - [~enis] We would use ZK to record merge transaction state for redo if server abort. If we send the merge request through RPC, we should also initiate the merge on zk by master. Additionally, using ZK won't break the RPC protocol, and it's easy to backport Thanks Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.5 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549464#comment-13549464
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12564137/hbase-7403-trunkv9.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplicationWithCompression
org.apache.hadoop.hbase.client.TestMultiParallel
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3959//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3959//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3959//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3959//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3959//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3959//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3959//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3959//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3959//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.5
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it until successful if it throws exception or abort or server restart, but
couldn’t
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13550881#comment-13550881
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12564361/hbase-7403-trunkv10.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestLocalHBaseCluster
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
at
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup.testBalancerWithRackLocality(TestBalancerWithNodeGroup.java:220)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3976//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3976//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3976//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3976//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3976//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3976//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3976//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3976//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3976//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.5
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch,
hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch,
hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch,
merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13550885#comment-13550885 ] Ted Yu commented on HBASE-7403: --- +1 on v10 Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.5 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv10.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, hbase-7403-trunkv9.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548588#comment-13548588
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12563889/hbase-7403-trunkv8.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 1
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestLocalHBaseCluster
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
at
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup.testBalancerWithRackLocality(TestBalancerWithNodeGroup.java:220)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3949//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3949//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3949//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3949//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3949//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3949//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3949//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3949//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3949//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.5
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
hbase-7403-trunkv8.patch, merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13548705#comment-13548705
]
Ted Yu commented on HBASE-7403:
---
In restartServersAfterExpectedState():
{code}
+if (currentMergeTransaction.isSuccessful()) {
+ waitAndVerifyRegionNum(master, tablename, regionNumBeforeMerging - 1);
+}
{code}
Under what scenario we expect merge transaction to be unsuccessful ?
{code}
+ * Restart master and regionserver when do the merging
{code}
'do the merging' - 'doing the merge'
{code}
+ListJVMClusterUtil.RegionServerThread aliveRSs = cluster
+.getLiveRegionServerThreads();
{code}
aliveRSs - liveRSs
{code}
+HRegionServer rs = aliveRSs.get(0).getRegionServer();
+rs.abort(Aborting for tests, new Exception(Trace info));
{code}
Can we use SecureRandom to randomize the region server which gets aborted ?
{code}
+ListJVMClusterUtil.MasterThread aliveMasters = cluster
{code}
aliveMasters - liveMasters
{code}
+while (!master.isInitialized() System.currentTimeMillis() timeout) {
+ Thread.sleep(250);
{code}
Can we use shorter sleep interval ?
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.5
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
hbase-7403-trunkv8.patch, merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it until successful if it throws exception or abort or server restart, but
couldn’t be rolled back.
It depends on
Use zookeeper to record the transaction journal state, make redo easier
Use zookeeper to send/receive merge request
Merge transaction is executed on the master
Support calling merge request through API or shell tool
About the merge process, please see the attachment and patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549214#comment-13549214 ] Ted Yu commented on HBASE-7403: --- In TestMergeTransaction, can you add test where region splitting stops merging and merge can resume to completion ? Thanks Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.5 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549331#comment-13549331 ] chunhui shen commented on HBASE-7403: - bq.Under what scenario we expect merge transaction to be unsuccessful ? If merge transaction is finished with state CANCEL_MERGE, it means this merge is unsuccessful bq.can you add test where region splitting stops merging and merge can resume to completion I'm not clear about region splitting stops mergin, is exception injection an kind of stopping merging? I will address other comments in new patch. Thanks. Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.5 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549334#comment-13549334
]
chunhui shen commented on HBASE-7403:
-
bq.Under what scenario we expect merge transaction to be unsuccessful ?
In the testing case, merge transaction would enter the state CANCEL_MERGING if
region has split log files
{code}
MergeTransaction#offlineRegion
else if ((conditionA = containsEditsLogFile(this.regionA))
|| containsEditsLogFile(this.regionB)) {
LOG.info(Canceling merge because region
+ (conditionA ? regionA.getEncodedName() : regionB.getEncodedName())
+ has edits log file);
this.currentState = JournalState.CANCEL_MERGING;
}
{code}
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.5
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
hbase-7403-trunkv8.patch, merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it until successful if it throws exception or abort or server restart, but
couldn’t be rolled back.
It depends on
Use zookeeper to record the transaction journal state, make redo easier
Use zookeeper to send/receive merge request
Merge transaction is executed on the master
Support calling merge request through API or shell tool
About the merge process, please see the attachment and patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549337#comment-13549337 ] Ted Yu commented on HBASE-7403: --- Since the patch doesn't use lock manager that makes region splitting and merging exclusive, it is possible that regionA or regionB undergoes splitting when merging is in progress. Looking at MergeTransaction#offlineRegion(), I see that this case is handled. It would be nice if TestMergeTransaction covers this scenario. Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.5 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13549346#comment-13549346 ] chunhui shen commented on HBASE-7403: - bq.Since the patch doesn't use lock manager that makes region splitting and merging exclusive I'm clear that you mean concurrent region splitting and region merging. Ok, I will add some tests to cover this scenario Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.5 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, hbase-7403-trunkv8.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13545706#comment-13545706
]
chunhui shen commented on HBASE-7403:
-
bq.When can we get this below situation?
{code}
if ((regionInfoA == null || regionInfoB == null)) {
{code}
After deleting the table or redo merging from the state COMPLETE_MERGING, this
situation is OK.
bq.Should we not check this even before the merge has started
We receive the merge request in a zookeeper event handler, and I don't want to
check the above situation in that handler because that handler is synchronized
bq.Y should not we resubmit here?
It is a new merge request, but we can't get the regioninfo of merging regions,
it seems user send a wrong request.
bq.Do we need to wait for the main region to be assigned
It's no necessary
bq.In EXECUTE_MERGING do we need to handle any failure saying it must move to
CANCEL_MERGING.
Once we entered the state EXECUTE_MERGING, we couldn't CANCEL_MERGING, we only
ensure that we could redo it until successful after abort or failture
bq.will it be reassigned by SSH?
It won't happen, the merging regions won't be assigned to anywhere, see
SSH#processDeadRegion and RegionMergeManager#mergingRegions
bq.I think the Merger thread should read the znode and try to restart the merge
right?
Yes, bingo! We record the merge state on ZK, so we can be easy to redo it
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.5
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it until successful if it throws exception or abort or server restart, but
couldn’t be rolled back.
It depends on
Use zookeeper to record the transaction journal state, make redo easier
Use zookeeper to send/receive merge request
Merge transaction is executed on the master
Support calling merge request through API or shell tool
About the merge process, please see the attachment and patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13545731#comment-13545731
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12563531/hbase-7403-trunkv7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.regionserver.TestSplitTransaction
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3889//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3889//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3889//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3889//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3889//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3889//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3889//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3889//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3889//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.5
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it until successful if it throws exception or abort or server restart, but
couldn’t be rolled back.
It depends on
Use zookeeper to
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13545749#comment-13545749
]
Hadoop QA commented on HBASE-7403:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12563531/hbase-7403-trunkv7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3888//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3888//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3888//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3888//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3888//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3888//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3888//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3888//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3888//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.5
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it until successful if it throws exception or abort or server restart, but
couldn’t be rolled back.
It depends on
Use zookeeper to record the transaction journal state, make redo easier
Use zookeeper to send/receive merge request
Merge transaction is executed on the master
Support
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13545784#comment-13545784
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12563531/hbase-7403-trunkv7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor
{color:red}-1 core zombie tests{color}. There are 6 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3890//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3890//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3890//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3890//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3890//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3890//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3890//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3890//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3890//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.5
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it until successful if it throws exception or abort or server restart, but
couldn’t be rolled back.
It depends on
Use
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13545786#comment-13545786
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12563531/hbase-7403-trunkv7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 core tests{color}. The patch passed unit tests in .
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3891//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3891//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3891//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3891//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3891//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3891//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3891//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3891//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3891//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.5
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it until successful if it throws exception or abort or server restart, but
couldn’t be rolled back.
It depends on
Use zookeeper to record the transaction journal state, make redo easier
Use zookeeper to
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13545794#comment-13545794
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12563531/hbase-7403-trunkv7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestFromClientSide
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3892//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3892//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3892//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3892//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3892//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3892//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3892//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3892//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3892//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.5
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it until successful if it throws exception or abort or server restart, but
couldn’t be rolled back.
It depends on
Use zookeeper to record the transaction journal state, make redo easier
Use zookeeper to send/receive
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13545833#comment-13545833
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12563531/hbase-7403-trunkv7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestFromClientSideWithCoprocessor
{color:red}-1 core zombie tests{color}. There are 6 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3893//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3893//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3893//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3893//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3893//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3893//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3893//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3893//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3893//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.5
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it until successful if it throws exception or abort or server restart, but
couldn’t be rolled back.
It depends on
Use
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13545835#comment-13545835
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12563531/hbase-7403-trunkv7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestFromClientSide
org.apache.hadoop.hbase.replication.TestReplication
{color:red}-1 core zombie tests{color}. There are 1 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3894//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3894//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3894//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3894//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3894//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3894//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3894//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3894//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3894//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.5
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it until successful if it throws exception or abort or server restart, but
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13545838#comment-13545838
]
Hadoop QA commented on HBASE-7403:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12563531/hbase-7403-trunkv7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3895//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3895//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3895//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3895//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3895//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3895//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3895//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3895//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3895//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.5
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it until successful if it throws exception or abort or server restart, but
couldn’t be rolled back.
It depends on
Use zookeeper to record the transaction journal state, make redo easier
Use zookeeper to send/receive merge request
Merge transaction is executed on the master
Support
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13545854#comment-13545854
]
Hadoop QA commented on HBASE-7403:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12563531/hbase-7403-trunkv7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3896//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3896//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3896//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3896//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3896//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3896//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3896//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3896//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3896//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.5
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it until successful if it throws exception or abort or server restart, but
couldn’t be rolled back.
It depends on
Use zookeeper to record the transaction journal state, make redo easier
Use zookeeper to send/receive merge request
Merge transaction is executed on the master
Support
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13545895#comment-13545895
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12563531/hbase-7403-trunkv7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplication
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3897//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3897//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3897//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.5
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it until successful if it throws exception or abort or server restart, but
couldn’t be rolled back.
It depends on
Use zookeeper to record the transaction journal state, make redo easier
Use zookeeper to
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13545929#comment-13545929
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12563531/hbase-7403-trunkv7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestFromClientSide
{color:red}-1 core zombie tests{color}. There are 7 zombie test(s):
at
org.apache.hadoop.hbase.catalog.TestCatalogTracker.testServerNotRunningIOException(TestCatalogTracker.java:250)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3898//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3898//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3898//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3898//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3898//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3898//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3898//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3898//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3898//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.5
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it until successful if
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13545933#comment-13545933
]
Hadoop QA commented on HBASE-7403:
--
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12563531/hbase-7403-trunkv7.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:green}+1 core tests{color}. The patch passed unit tests in .
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3899//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3899//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3899//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3899//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3899//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3899//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3899//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3899//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3899//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.5
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it until successful if it throws exception or abort or server restart, but
couldn’t be rolled back.
It depends on
Use zookeeper to record the transaction journal state, make redo easier
Use zookeeper to send/receive merge request
Merge transaction is executed on the master
Support
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546068#comment-13546068 ] ramkrishna.s.vasudevan commented on HBASE-7403: --- bq.We receive the merge request in a zookeeper event handler, and I don't want to check the above situation in that handler because that handler is synchronized Ok makes sense. bq.It is a new merge request, but we can't get the regioninfo of merging regions, it seems user send a wrong request. But i felt like, while doing complete_merging if due to some reason we dont get regioninfo and we retry why not the same thing happen here in CREATE_MERGE too. bq.see SSH#processDeadRegion and RegionMergeManager#mergingRegions Thanks. I missed it in the code. bq.I think the Merger thread should read the znode and try to restart the merge right? Don't think am asking this again. My point was does the failed merge transaction, restart again on master restart? Because the resubmit queue is going to be empty right? Should we forcefully read the merge znode and completed the failed merge as do for SPLIT. Thanks for your answers. Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.5 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546611#comment-13546611 ] chunhui shen commented on HBASE-7403: - bq.while doing complete_merging if due to some reason we dont get regioninfo In the state of complete_merging, we will delete the merging regions from .META., so it is avaiable bq.My point was does the failed merge transaction, restart again on master restart? Yes, it will. The failed merge transaction not means it will be always failed. So we redo it Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.5 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13546640#comment-13546640
]
Ted Yu commented on HBASE-7403:
---
TestMergeTransaction is marked as MediumTests
I originally expected a large test, due to the various scenarios that should be
covered.
{code}
+ public void testRedoMergeWhenOfflineRegion() throws Exception {
{code}
Should the above method be called
testRedoMergeWhenOfflineRegionEncountersException() ?
{code}
+// Throw exception when offline region
+throwExceptionStep = 1;
{code}
Can you use enum or add javadoc explaining what each step does ?
{code}
+ public void testRedoMergeWhenExecuteMerge() throws Exception {
{code}
There should be a better name for the above method - we redo merge when there
is exception from merge execution.
For testRedoMergeWhenCancelMerge(),
{code}
+// Throw exception when complete merge
+throwExceptionStep = 4;
{code}
Does the comment match method name ?
Overall, testing is done through fault injection. Can we add more test(s) for
master failover scenario, etc ?
Thanks
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.5
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, hbase-7403-trunkv7.patch,
merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it until successful if it throws exception or abort or server restart, but
couldn’t be rolled back.
It depends on
Use zookeeper to record the transaction journal state, make redo easier
Use zookeeper to send/receive merge request
Merge transaction is executed on the master
Support calling merge request through API or shell tool
About the merge process, please see the attachment and patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13545434#comment-13545434
]
ramkrishna.s.vasudevan commented on HBASE-7403:
---
{code}
if (!elements[3].equals(true) !elements[3].equals(false)) {
return false;
}
{code}
May be we can use Boolean.valueOf
{code}
LOG.warn(Merge request + transactionData + try to merge region
+ (mergingRegions.contains(regionA) ? regionA : regionB)
+ where other transaction contains);
{code}
Could you make this log msg more clear.
{code}
LOG.debug(Merge reuest arrived or completed );
{code}
It should be 'request'.
When can we get this below situation?
{code}
if ((regionInfoA == null || regionInfoB == null)) {
{code}
Should we not check this even before the merge has started. If we had already
checked
what scenario we could get to this situation?
{code}
else if (this.initialState == JournalState.CREATE_MERGE) {
String msg = Skip merging + transactionData
+ , because couldn't get the regioninfo, initialState=
+ initialState;
{code}
Y should not we resubmit here?
{code}
mergeManager.setJournalStateOnZK(transactionData.getTransactionName(),
this.currentState);
{code}
Nice.. we need not pass the state everytime...But should we have a clear log
msg. Should be of great help.
{code}
masterServices.getAssignmentManager().assign(this.mergedRegion, true);
{code}
Do we need to wait for the main region to be assigned
In EXECUTE_MERGING do we need to handle any failure saying it must move to
CANCEL_MERGING.
This will help us to do remedial steps as done for other Merge Transaction
states.
Failure while deleting the region in META means before the delete from one of
the META happens, will it be reassigned by SSH? Hope you will be knowing this
scenario and should have handled, just wanted to know.
Similarly on master restart what are we doing if the Merge znode is in some
state related to online merge? do we try to start from there?
I think the Merger thread should read the znode and try to restart the merge
right?
Will try to review once more tomorrow. Nice feature. Thanks Chunhui.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.5
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to tabke care the events like
Server Dead, Balance, Split, Disabing/Enabing table, no need to take care
whether you send a wrong merge request, it has alread done for you
5.Only little offline time for two merging regions
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it until successful if it throws exception or abort or server restart, but
couldn’t be rolled back.
It depends on
Use zookeeper to record the transaction journal state, make redo easier
Use zookeeper to send/receive merge request
Merge transaction is executed on the master
Support calling merge request through API or shell tool
About the merge process, please see the attachment and patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13544795#comment-13544795 ] ramkrishna.s.vasudevan commented on HBASE-7403: --- @Chunhui I have started with my review. I will update on this tomorrow. Could not spend much time today. Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.5 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13545276#comment-13545276 ] chunhui shen commented on HBASE-7403: - OK Thanks ^_^ Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.5 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to tabke care the events like Server Dead, Balance, Split, Disabing/Enabing table, no need to take care whether you send a wrong merge request, it has alread done for you 5.Only little offline time for two merging regions We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13544592#comment-13544592 ] chunhui shen commented on HBASE-7403: - [~stack] [~ram_krish] Could you give any comments? Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.5 Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to care event like Server Dead, Balance, Split, Disabing/Enabing table, no need to care whether you send a wrong merge request, it alread have done for you 5.Only little offline time for two merging regions We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13541062#comment-13541062
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562710/7403-trunkv6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestMultiParallel
org.apache.hadoop.hbase.replication.TestReplication
org.apache.hadoop.hbase.client.TestFromClientSide
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3763//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3763//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3763//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3763//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3763//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3763//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3763//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3763//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3763//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.5
Attachments: 7403-trunkv5.patch, 7403-trunkv6.patch, 7403v5.diff,
7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to care event like Server Dead,
Balance, Split, Disabing/Enabing table, no need to care whether you send a
wrong merge request, it alread have done for you
5.Only little offline time for two merging regions
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it until successful if it throws exception or abort or
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540908#comment-13540908
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562667/7403v5.diff
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated 8
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestMultiParallel
org.apache.hadoop.hbase.replication.TestReplicationWithCompression
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3755//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3755//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3755//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3755//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3755//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3755//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3755//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3755//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3755//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.5
Attachments: 7403-trunkv5.patch, 7403v5.diff, 7403-v5.txt,
7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv5.patch, merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to care event like Server Dead,
Balance, Split, Disabing/Enabing table, no need to care whether you send a
wrong merge request, it alread have done for you
5.Only little offline time for two merging regions
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it until successful if it throws exception or abort or server restart, but
couldn’t be rolled back.
It depends on
Use zookeeper to record the transaction journal
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540912#comment-13540912 ] Ted Yu commented on HBASE-7403: --- There are lines longer than 100 characters in patch v5. Please shorten them. Thanks Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.5 Attachments: 7403-trunkv5.patch, 7403v5.diff, 7403-v5.txt, 7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to care event like Server Dead, Balance, Split, Disabing/Enabing table, no need to care whether you send a wrong merge request, it alread have done for you 5.Only little offline time for two merging regions We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13541040#comment-13541040
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562704/hbase-7403-trunkv6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 lineLengths{color}. The patch does not introduce lines
longer than 100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.TestDrainingServer
{color:red}-1 core zombie tests{color}. There are 3 zombie test(s):
at
org.apache.hadoop.hbase.io.encoding.TestUpgradeFromHFileV1ToEncoding.testUpgrade(TestUpgradeFromHFileV1ToEncoding.java:83)
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3762//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3762//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3762//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.5
Attachments: 7403-trunkv5.patch, 7403v5.diff, 7403-v5.txt,
7403v5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch,
hbase-7403-trunkv5.patch, hbase-7403-trunkv6.patch, merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to care event like Server Dead,
Balance, Split, Disabing/Enabing table, no need to care whether you send a
wrong merge request, it alread have done for you
5.Only little offline time for two merging regions
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it until successful if it throws exception or abort or server restart, but
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540515#comment-13540515
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562584/hbase-7403-trunkv5.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
10 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestMultiParallel
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3734//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3734//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3734//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.5
Attachments: 7403-trunkv5.patch, hbase-7403-94v1.patch,
hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv5.txt,
merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to care event like Server Dead,
Balance, Split, Disabing/Enabing table, no need to care whether you send a
wrong merge request, it alread have done for you
5.Only little offline time for two merging regions
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it until successful if it throws exception or abort or server restart, but
couldn’t be rolled back.
It depends on
Use zookeeper to record the transaction journal state, make redo easier
Use zookeeper to send/receive merge request
Merge transaction
[jira] [Commented] (HBASE-7403) Online Merge
[ https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540750#comment-13540750 ] Ted Yu commented on HBASE-7403: --- @Chunhui: TestMultiParallel is a flaky test. See HBASE-7299 Online Merge Key: HBASE-7403 URL: https://issues.apache.org/jira/browse/HBASE-7403 Project: HBase Issue Type: New Feature Affects Versions: 0.94.3 Reporter: chunhui shen Assignee: chunhui shen Fix For: 0.96.0, 0.94.5 Attachments: 7403-trunkv5.patch, 7403-trunkv5.txt, hbase-7403-94v1.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, hbase-7403-trunkv5.txt, merge region.pdf The feature of this online merge: 1.Online,no necessary to disable table 2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90 3.Easy to call merege request, no need to input a long region name, only encoded name enough 4.No limit when operation, you don't need to care event like Server Dead, Balance, Split, Disabing/Enabing table, no need to care whether you send a wrong merge request, it alread have done for you 5.Only little offline time for two merging regions We need merge in the following cases： 1.Region hole or region overlap, can’t be fix by hbck 2.Region become empty because of TTL and not reasonable Rowkey design 3.Region is always empty or very small because of presplit when create table 4.Too many empty or small regions would reduce the system performance(e.g. mslab) Current merge tools only support offline and are not able to redo if exception is thrown in the process of merging, causing a dirty data For online system, we need a online merge. This implement logic of this patch for Online Merge is : For example, merge regionA and regionB into regionC 1.Offline the two regions A and B 2.Merge the two regions in the HDFS(Create regionC’s directory, move regionA’s and regionB’s file to regionC’s directory, delete regionA’s and regionB’s directory) 3.Add the merged regionC to .META. 4.Assign the merged regionC As design of this patch , once we do the merge work in the HDFS,we could redo it until successful if it throws exception or abort or server restart, but couldn’t be rolled back. It depends on Use zookeeper to record the transaction journal state, make redo easier Use zookeeper to send/receive merge request Merge transaction is executed on the master Support calling merge request through API or shell tool About the merge process, please see the attachment and patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540780#comment-13540780
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562644/7403-trunkv5.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
10 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.replication.TestReplicationWithCompression
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3745//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3745//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3745//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3745//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3745//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3745//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3745//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3745//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3745//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.5
Attachments: 7403-trunkv5.patch, 7403-trunkv5.txt,
hbase-7403-94v1.patch, hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch,
hbase-7403-trunkv5.txt, merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to care event like Server Dead,
Balance, Split, Disabing/Enabing table, no need to care whether you send a
wrong merge request, it alread have done for you
5.Only little offline time for two merging regions
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it until successful if it throws exception or abort or server restart, but
couldn’t be rolled back.
It depends on
Use zookeeper to record the transaction journal state, make redo easier
Use zookeeper to send/receive
[jira] [Commented] (HBASE-7403) Online Merge
[
https://issues.apache.org/jira/browse/HBASE-7403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13540801#comment-13540801
]
Hadoop QA commented on HBASE-7403:
--
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12562647/7403-v5.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified tests.
{color:green}+1 hadoop2.0{color}. The patch compiles against the hadoop
2.0 profile.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
10 warning messages.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 lineLengths{color}. The patch introduces lines longer than
100
{color:red}-1 core tests{color}. The patch failed these unit tests:
org.apache.hadoop.hbase.client.TestMultiParallel
org.apache.hadoop.hbase.client.TestFromClientSide
{color:red}-1 core zombie tests{color}. There are 2 zombie test(s):
Test results:
https://builds.apache.org/job/PreCommit-HBASE-Build/3747//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3747//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-protocol.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3747//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop2-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3747//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-examples.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3747//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop1-compat.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3747//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-common.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3747//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-server.html
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HBASE-Build/3747//artifact/trunk/patchprocess/newPatchFindbugsWarningshbase-hadoop-compat.html
Console output:
https://builds.apache.org/job/PreCommit-HBASE-Build/3747//console
This message is automatically generated.
Online Merge
Key: HBASE-7403
URL: https://issues.apache.org/jira/browse/HBASE-7403
Project: HBase
Issue Type: New Feature
Affects Versions: 0.94.3
Reporter: chunhui shen
Assignee: chunhui shen
Fix For: 0.96.0, 0.94.5
Attachments: 7403-trunkv5.patch, 7403-v5.txt, hbase-7403-94v1.patch,
hbase-7403-trunkv1.patch, hbase-7403-trunkv5.patch, merge region.pdf
The feature of this online merge:
1.Online,no necessary to disable table
2.Less change for current code, could applied in trunk,0.94 or 0.92,0.90
3.Easy to call merege request, no need to input a long region name, only
encoded name enough
4.No limit when operation, you don't need to care event like Server Dead,
Balance, Split, Disabing/Enabing table, no need to care whether you send a
wrong merge request, it alread have done for you
5.Only little offline time for two merging regions
We need merge in the following cases：
1.Region hole or region overlap, can’t be fix by hbck
2.Region become empty because of TTL and not reasonable Rowkey design
3.Region is always empty or very small because of presplit when create table
4.Too many empty or small regions would reduce the system performance(e.g.
mslab)
Current merge tools only support offline and are not able to redo if
exception is thrown in the process of merging, causing a dirty data
For online system, we need a online merge.
This implement logic of this patch for Online Merge is :
For example, merge regionA and regionB into regionC
1.Offline the two regions A and B
2.Merge the two regions in the HDFS(Create regionC’s directory, move
regionA’s and regionB’s file to regionC’s directory, delete regionA’s and
regionB’s directory)
3.Add the merged regionC to .META.
4.Assign the merged regionC
As design of this patch , once we do the merge work in the HDFS,we could redo
it until successful if it throws exception or abort or server restart, but
couldn’t be rolled back.
It depends on
Use zookeeper to record the transaction journal state, make redo easier
Use zookeeper to
