[
https://issues.apache.org/jira/browse/PHOENIX-8027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lokesh Khurana resolved PHOENIX-8027.
-------------------------------------
Resolution: Fixed
> Allow consistent failover to be initiated only from ACTIVE_IN_SYNC
> ------------------------------------------------------------------
>
> Key: PHOENIX-8027
> URL: https://issues.apache.org/jira/browse/PHOENIX-8027
> Project: Phoenix
> Issue Type: Sub-task
> Reporter: Lokesh Khurana
> Assignee: Lokesh Khurana
> Priority: Major
>
> Today initiate-failover is accepted from ACTIVE_NOT_IN_SYNC
> (store-and-forward) and moves the
> group to ACTIVE_NOT_IN_SYNC_TO_STANDBY while the forwarder drains the local
> backlog to the
> standby. In that window a single peer HDFS write failure is unrecoverable:
> 1. SyncAndForwardModeImpl.onFailure calls transitionToStoreAndForward, which
> asks the HAGroupStore to move the group to ACTIVE_NOT_IN_SYNC.
> 2. ACTIVE_NOT_IN_SYNC is not an allowed transition from
> ACTIVE_NOT_IN_SYNC_TO_STANDBY, so the update throws
> InvalidClusterRoleTransitionException.
> 3. The mode never changes, the local store-and-forward write is never
> attempted, and the writer goes fatal ("Both SYNC and SAF replication writes
> failed").
> 4. ReplicationLogGroup.abort() aborts the RegionServer. The RSs that open its
> regions replay the recovered edits in IndexRegionObserver.preWALRestore and
> abort the same way.
> abort-failover does not help: it only applies to a standby in
> STANDBY_TO_ACTIVE, and the
> standby never reaches that state.
> Testing found (FAILOVER policy, active cluster with 6 RegionServers, ~3
> rows/s client load):
> - Repro: set the active to ACTIVE_NOT_IN_SYNC, then to
> ACTIVE_NOT_IN_SYNC_TO_STANDBY, then put a 1-byte space quota on the peer's
> /phoenixHA/<group>/in.
> - The first RS aborted 0.13 s after the failed peer write. The other 5
> aborted 6 s later on
> RS_OPEN_REGION threads while opening its regions.
> - 18 RS restarts in about 2 min. The group's writes stopped for 92 s. Every
> region of every table on those RSs was reassigned, not only the HA group's.
> No data was lost.
> - With a healthy peer, the same steps drained in about 100 s and completed
> the failover.
> Proposal: allow initiate-failover only from ACTIVE_IN_SYNC, and reject
> ACTIVE_NOT_IN_SYNC with an error telling the operator to wait for
> ACTIVE_IN_SYNC, or to fix the peer first if the group is stuck. No capability
> is lost: the ANIS path already had to drain to ACTIVE_IN_SYNC_TO_STANDBY
> before cutover, so the operator now just waits for the drain before starting
> the failover. Keep the ACTIVE_NOT_IN_SYNC_TO_STANDBY state and its
> transitions so groups already in that state, and mixed-version clusters, keep
> working.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
