[
https://issues.apache.org/jira/browse/HBASE-30344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18108893#comment-18108893
]
mazhengxuan commented on HBASE-30344:
-------------------------------------
Submitted PR: https://github.com/apache/hbase/pull/8578.
The patch waits for the RegionServer serving the test table to observe
DOWNGRADE_ACTIVE before verify() starts. The production-side STANDBY protection
is unchanged.
TestSyncReplicationStandbyKillMaster passed five consecutive runs with
surefire.rerunFailingTestsCount=0.
> [Flaky Test] TestSyncReplicationStandbyKillMaster may verify before
> RegionServer sync state converges
> -----------------------------------------------------------------------------------------------------
>
> Key: HBASE-30344
> URL: https://issues.apache.org/jira/browse/HBASE-30344
> Project: HBase
> Issue Type: Bug
> Components: Replication, test
> Environment: Apache HBase master
> GitHub Actions:
> JDK17 Hadoop3 Unit Check, large-wave-3
> Failed workflow job:
> https://github.com/apache/hbase/actions/runs/33056801754/job/98468827602
> Reporter: mazhengxuan
> Assignee: mazhengxuan
> Priority: Major
> Labels: pull-request-available
>
> TestSyncReplicationStandbyKillMaster is still flaky on master after
> HBASE-30249.
> It failed in the JDK17 Hadoop3 Unit Check for apache/hbase#8483:
> [https://github.com/apache/hbase/actions/runs/33056801754/job/98468827602]
> The first run failed in
> TestSyncReplicationStandbyKillMaster.testStandbyKillMaster. The test waits
> for the peer state reported by Admin to become DOWNGRADE_ACTIVE and then
> immediately verifies the replicated data:
> await().atMost(Duration.ofMinutes(3))
> .untilAsserted(() -> assertEquals(
> SyncReplicationState.DOWNGRADE_ACTIVE,
> UTIL2.getAdmin().getReplicationPeerSyncReplicationState(PEER_ID)));
> verify(UTIL2, 0, COUNT);
> The verification request was rejected by the RegionServer because its local
> sync replication state was still STANDBY:
> org.apache.hadoop.hbase.DoNotRetryIOException:
> SyncRep,... is in STANDBY state.
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.rejectIfInStandByState(...)
> at org.apache.hadoop.hbase.regionserver.RSRpcServices.get(...)
> The Admin result reflects the peer state on the Master side. The RegionServer
> learns about this state asynchronously, so observing DOWNGRADE_ACTIVE through
> Admin does not mean that every RegionServer has already applied the
> transition.
> The Surefire retries then failed during setup with:
> 1. java.io.IOException: Shutting down
> 2. java.lang.IllegalStateException: A mini-cluster is already running
> These look like follow-up failures caused by the cluster cleanup after the
> first run failed.
> HBASE-30249 made the Master failure injection deterministic, but the test
> still does not wait for the RegionServer-side state transition before calling
> verify().
> The test should wait, with a timeout, until the RegionServer serving the
> table no longer rejects requests because of the STANDBY state. This should be
> handled as test synchronization unless further investigation shows that the
> RegionServer state never converges.
> The failure can be reproduced with Surefire retries disabled:
> mvn -pl hbase-server -am \
> -Dtest=TestSyncReplicationStandbyKillMaster \
> -Dsurefire.failIfNoSpecifiedTests=false \
> -Dsurefire.rerunFailingTestsCount=0 \
> -DskipITs \
> test
> Related:
> * HBASE-30249
> * [https://github.com/apache/hbase/pull/8393]
> * [https://github.com/apache/hbase/pull/8483]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
