[ https://issues.apache.org/jira/browse/HBASE-28533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17842562#comment-17842562 ]
Hudson commented on HBASE-28533: -------------------------------- Results for branch branch-2 [build #1046 on builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1046/]: (x) *{color:red}-1 overall{color}* ---- details (if available): (/) {color:green}+1 general checks{color} -- For more information [see general report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1046/General_20Nightly_20Build_20Report/] (/) {color:green}+1 jdk8 hadoop2 checks{color} -- For more information [see jdk8 (hadoop2) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1046/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/] (/) {color:green}+1 jdk8 hadoop3 checks{color} -- For more information [see jdk8 (hadoop3) report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1046/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/] (x) {color:red}-1 jdk11 hadoop3 checks{color} -- For more information [see jdk11 report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/1046/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/] (/) {color:green}+1 source release artifact{color} -- See build output for details. (/) {color:green}+1 client integration test{color} > Split procedure rollback can leave parent region state in SPLITTING after > completion > ------------------------------------------------------------------------------------ > > Key: HBASE-28533 > URL: https://issues.apache.org/jira/browse/HBASE-28533 > Project: HBase > Issue Type: Bug > Components: Region Assignment > Affects Versions: 2.5.8, 3.0.0-beta-2 > Reporter: Daniel Roudnitsky > Assignee: Daniel Roudnitsky > Priority: Major > Labels: pull-request-available > Fix For: 2.4.18, 2.7.0, 3.0.0-beta-2, 2.6.1, 2.5.9 > > > Depending on where the split procedure fails, SplitTableRegionProcedure > rollback can leave the parent region's in memory RegionStateNode in SPLITTING > after rollback is complete, when the parent region is still online on the > assigned region server and according to meta. This leaves active HMaster > believing that the parent region is offline according to its RegionStates, > and causes subsequent procedures that require the region to be online like > merge/split/move to fail to start. One workaround is to restart active > HMaster to reset the in memory record of region states. > Two example scenarios where this can happen: > * If we get to SPLIT_TABLE_REGION_CLOSE_PARENT_REGION and the parent region > has a replica which is in transition, the unassign procedure in that step is > never created/rolled back and we are left with the parent region state in > splitting. > * If region quotas are enabled and a split is run for a region whose > namespace is at its maximum region quota limit we will fail in > SPLIT_TABLE_REGION_PRE_OPERATION with QuotaExceededException and we are left > with the parent region state in splitting > The region replica case is demonstrated in > TestSplitTableRegionProcedure.testRollbackForSplitTableRegionWithReplica. > To reproduce the region quota case in HBase shell: > {code:java} > > create_namespace 'test_ns', {'hbase.namespace.quota.maxregions'=> 2} > > create 'test_ns:test_table', 'f1', {NUMREGIONS => 2, SPLITALGO => > > 'UniformSplit'} > > region_a = <first region from list_regions 'test_ns:test_table'> > > region_b = <second region from list_regions 'test_ns:test_table'> > > split region_a, 'x' > # HMaster will report: > pid=405, state=ROLLEDBACK, > exception=org.apache.hadoop.hbase.quotas.QuotaExceededException via > master-split-regions:org.apache.hadoop.hbase.quotas.QuotaExceededException: > Region split not possible for :<region_a> as quota limits are exceeded ; > SplitTableRegionProcedure table=test_ns:test_table, parent=... > > merge_region region_a, region_b > ERROR: org.apache.hadoop.hbase.exceptions.MergeRegionException: > org.apache.hadoop.hbase.client.DoNotRetryRegionException: <region_a> is not > OPEN; state=SPLITTING > > stop_master # trigger hmaster failover > > merge_region region_a, region_b # merge now succeeds {code} -- This message was sent by Atlassian Jira (v8.20.10#820010)