lokiore opened a new pull request, #2502: URL: https://github.com/apache/phoenix/pull/2502
### What changes were proposed in this pull request? <!-- Please clarify what changes you are proposing. --> Adds client-side and server-side observability metrics for the Consistent Failover (CCF) high-availability path. Internal mirror of bigdata-packaging/phoenix#1750. JIRA: https://issues.apache.org/jira/browse/PHOENIX-7872 **Tier-1 client-side counters** (in `GlobalClientMetrics` + `MetricType`): - `HA_POLLER_TICK_COUNT` — total poller ticks across all HA groups (incremented in `GetClusterRoleRecordUtil`'s polling task) - `HA_POLLER_TICK_FAILURES` — per-tick CRR fetch failures - `HA_FAILOVER_COUNT` — failover transitions executed by the client. Emitted from `HighAvailabilityGroup.applyClusterRoleRecord` only on actual ACTIVE → STANDBY or STANDBY → ACTIVE role transitions - `HA_MUTATION_BLOCKED_COUNT` — `MutationBlockedIOException` occurrences detected via the wrap-and-propagate path in `FailoverPhoenixConnection.wrapActionWhileFailover` **Tier-2 client-side metrics**: - `HA_FAILOVER_DURATION_MS` — failover end-to-end latency histogram (try/finally wrapper in `FailoverPhoenixConnection.failover`) - `HA_STALE_CRR_DETECTED_COUNT` — `StaleClusterRoleRecordException` occurrences detected in the wrap path - `HA_CRR_CACHE_AGE_MS` — gauge of staleness of the in-memory CRR cache, set on every successful CRR refresh in `HighAvailabilityGroup` **Tier-2 server-side counter** (new 3-file Hadoop-metrics2 source under `phoenix-core-server/.../hbase/index/metrics`): - `BYPASSED_MUTATION_BLOCK_COUNT` — emitted from `IndexRegionObserver` when a mutation bypasses the mutation-block check because no log group is present for the data table. Implemented as `MetricsHaBypassSource` (interface) + `MetricsHaBypassSourceFactory` (DefaultMetricsSystem-anchored, double-checked lock) + `MetricsHaBypassSourceImpl`. ### Why are the changes needed? <!-- Please clarify why the changes are needed. --> The CCF HA path previously had no observability for client-side polling cadence, failover frequency, failover latency, mutation-block fail-fast counts, stale-CRR detection, or CRR cache age. Operators investigating slow failovers or unexpected mutation rejections had to reconstruct event timelines from scattered DEBUG logs. These metrics close the gap on the dimensions the platform team needs for HA SLO tracking and incident triage: - Poller liveness & health (tick count + failures) - Failover frequency, duration, and trigger (CRR transition vs MBIOE-driven) - Stale-CRR detection rate as a leading indicator of failover-in-progress windows - Server-side rate of mutation-block bypass (regions without an HA log group attached) ### Does this PR introduce _any_ user-facing change? <!-- Note that it means *any* user-facing change including all aspects such as new features, bug fixes, or other behavior changes. --> No The new metrics are emitted via the existing `GlobalClientMetrics` (client-side) and Hadoop metrics2 (server-side) pipelines. No public-API change, no SQL surface change, no behavior change on the failover/poller paths beyond `getMetric().increment()` / `.update()` / `.set()` calls. ### How was this patch tested? <!-- If tests were added, say they were added here. --> **New unit tests**: - `HighAvailabilityUtilTest` — covers `RetriesExhaustedWithDetailsException` cause-chain MBIOE detection (the wrap-and-propagate path that fires `HA_MUTATION_BLOCKED_COUNT`) - `MetricsHaBypassSourceFactoryTest` — covers factory thread-safety (single-instance under concurrent `getInstance()`) **New ITs**: - `HAGroupMetricsIT` — covers all 8 client-side metrics across the CCF failover lifecycle (poller ticks, failover transitions, CRR cache age gauge, stale-CRR detection, MBIOE detection on the wrap path) - `BypassedMutationBlockMetricsIT` — covers server-side `BYPASSED_MUTATION_BLOCK_COUNT` emission when a mutation hits a region without an HA log group Local 13/13 PASS reproduction: ``` [INFO] Running org.apache.phoenix.jdbc.HighAvailabilityUtilTest [INFO] Tests run: 4, Failures: 0, Errors: 0, Skipped: 0 [INFO] Running org.apache.phoenix.hbase.index.metrics.MetricsHaBypassSourceFactoryTest [INFO] Tests run: 3, Failures: 0, Errors: 0, Skipped: 0 [INFO] Running org.apache.phoenix.jdbc.HAGroupMetricsIT [INFO] Tests run: 5, Failures: 0, Errors: 0, Skipped: 0 [INFO] Running org.apache.phoenix.end2end.index.BypassedMutationBlockMetricsIT [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0 [INFO] BUILD SUCCESS ``` ### Was this patch authored or co-authored using generative AI tooling? <!-- If generative AI tooling has been used in the process of authoring this patch, please include the phrase: 'Generated-by: ' followed by the name of the tool and its version. Please refer to the [ASF Generative Tooling Guidance](https://www.apache.org/legal/generative-tooling.html) for details. --> Generated-by: Claude Code (Opus 4.7) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
