ritegarg opened a new pull request, #2547:
URL: https://github.com/apache/phoenix/pull/2547
### What changes were proposed in this pull request?
Make a `STANDBY` RegionServer's replication replay **fail closed** while its
peer cluster is not visible, and move all peer-connection handling out of
`HAGroupStoreClient` into a dedicated `PeerClusterWatcher`.
- **`PeerClusterWatcher` (new)** — owns the peer `PathChildrenCache` +
`PhoenixHAAdmin` for one HA group and reports peer state via a `Listener`
(`onPeerStateChanged` / `onPeerVisible` / `onPeerBlind`):
- *Retry when peer ZK is down* (e.g. unreachable at startup): if the cache
can't be built the watcher goes `BLIND`, and a per-client daemon retries the
build on `phoenix.ha.group.store.peer.cache.retry.interval.seconds` until the
peer returns, then goes `VISIBLE` — no restart needed.
- *Forced redelivery after reconnect*: peer records are de-duplicated by
znode version, with exactly one forced redelivery on reconnect so no transition
is missed across the disconnect.
- *Concurrency*: lock order `transitionLock → connectionStateLock`; the
blocking cache build and listener callbacks run outside `connectionStateLock`.
- **`HAGroupStoreCacheUtil` (new)** — shared helpers to parse a znode into
`(record, stat)` and build/start a `PathChildrenCache` (init latch released in
`finally`).
- **`HAGroupStoreClient`** — delegates peer handling to `PeerClusterWatcher`
via a `PeerListener`; adds the in-memory effective-state overlay
`getEffectiveHAGroupStoreRecord()` (reports a local `STANDBY` as
`DEGRADED_STANDBY` while peer-blind, never persisted); suppresses a real
`STANDBY` re-entry while peer-blind.
- **`HAGroupStoreManager` / `HAGroupStoreRecord`** — add
`getEffectiveHAGroupStoreRecord(haGroupName)` and `withHAGroupState(...)`
(immutable copy for the overlay), and document `DEGRADED_STANDBY`'s dual
nature. The `DEGRADED_STANDBY` state and the subscription framework are
pre-existing.
- **Replication** — `ReplicationLogGroup.init` and
`ReplicationLogDiscoveryReplay.getHAGroupRecord` read the **effective** record;
mode mapping is otherwise unchanged.
- **Config** — new
`phoenix.ha.group.store.peer.cache.retry.interval.seconds` (default 60s; `0`
disables retry), with jittered retry and rate-limited `WARN` (1st + every 10th
attempt).
### Why are the changes needed?
When the peer ZK is unreachable — including down at startup — a `STANDBY`
can't reliably determine peer state, so replay risked proceeding as if in sync
(fail-open). It must instead **fail closed** (`STORE_AND_FORWARD`) until the
peer is reachable, then recover automatically. Extracting the peer lifecycle
into `PeerClusterWatcher` decouples replay from peer connectivity and keeps
`HAGroupStoreClient` focused on the effective HA view.
### Does this PR introduce _any_ user-facing change?
Yes, within the unreleased consistent-failover feature branch (no change vs
released Phoenix):
- New config `phoenix.ha.group.store.peer.cache.retry.interval.seconds`
(default 60s).
- A `STANDBY` whose peer ZK is unreachable presents an effective
`DEGRADED_STANDBY` (in-memory only, never written to ZK), so replay fails
closed until the peer is visible again. Persisted wire format is unchanged.
### How was this patch tested?
New and existing unit/integration tests: `HAGroupStoreClientIT`,
`HAGroupStoreManagerIT`, `HAGroupStateSubscriptionIT`,
`ReplicationLogDiscoveryReplayTestIT`, `HAGroupStoreRecordTest`,
`PeerClusterWatcherTest`.
Key cases: peer-loss degrade/recover; peer ZK down at startup then retry
rebuilds the cache; in-memory-only overlay (persisted stays `STANDBY`);
`STANDBY` re-entry suppressed while peer-blind; forced reconnect redelivery;
visible/blind serialization; `close()` idempotency; replayer `degrade → abort →
recover`; and a listener throwing on the cache `INITIALIZED` event still
initializes healthy.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
