unbridled-41 opened a new pull request, #5449:
URL: https://github.com/apache/rocketmq-dashboard/pull/5449
## Problem / Evidence
When a broker's runtime stats cannot be read at all, the disk/JVM/send-queue
metrics produce **no sample**, and alert reconciliation reads that as "the
value cleared" — resolving an active incident on the very pass that failed to
read it.
-
`cluster/metrics/collectors/ApacheRocketMqClusterMetricsCollector.java:105-128`
— all three failure branches (no master address, `fetchBrokerRuntimeStats`
returned null/empty, or it threw) add **only**
`unavailable(BROKER_AVAILABILITY, …)` and return. The success path deliberately
uses `metricOrUnavailable` for `broker.disk.usage_ratio`,
`broker.jvm.heap.usage_ratio` and `broker.send_queue.usage_ratio`.
- `CollectorScheduler.persist` declares the full metric scope rather than
deriving it from the samples (`:314-321`, `:335-340`), so the three keys count
as successfully collected.
- `NativeAlertProcessor.reconcileMissingActiveStates` (`:142-175`) skips
reconciliation only when an unavailable sample carries **the rule's own metric
key** and matching labels (`:154-158`); the per-broker `broker.availability`
sample has a different key, so the state advances with `clear` →
`AlertStateMachine.advanceClear` (`:91-98`) → RESOLVED, and a recovery flaps it
back.
- Contract: `docs/studio-native-alerting-design.md:242` ("Collection failure
records an unavailable sample and health diagnostic") and
`AlertStateMachine.java:46-47` ("A regular metric must not clear an alert
merely because collection failed").
- Trigger: an enabled CLUSTER rule on `broker.disk.usage_ratio` is FIRING
for broker X; X becomes unreachable while still listed in the NameServer
topology (so `examineBrokerClusterInfo` succeeds and `fetchBrokerRuntimeStats`
throws). The next 30s pass emits RESOLVED.
The same defect in the parse-miss branches was already fixed by `eda9884d`
(#4521); the three early-return/exception branches were missed.
## Root cause / Fix
Emit an unavailable sample for each runtime metric on those paths too, with
the same broker labels:
```java
samples.add(unavailable(BROKER_AVAILABILITY, instance, clusterId, labels,
collectedAt));
addUnavailableRuntimeMetrics(instance, clusterId, labels, collectedAt,
samples);
```
The reconciler's coverage guard then matches the rule's metric key and keeps
the active state until a successful collection can actually clear it.
## Priority & scoring
PRIORITY 58 = 影响 18 (a false RESOLVED plus notification for an active infra
alert, and a flap on recovery) + 波及 8 (three metrics for every broker of every
Apache instance) + 可复现 16 (deterministic: mock the admin to throw) + 维护价值 16
(mirrors the accepted rationale of #4521).
FIX_CONFIDENCE 82: the maintainers already accepted the same rationale for
the sibling branches; the change reuses the existing `unavailable` helper and
labels.
## Tests
```
cd server && mvn -o test -Dtest=ApacheRocketMqClusterMetricsCollectorTest
```
- Red before the fix:
`emitsUnavailableSamplesForEveryRuntimeMetricWhenTheStatsCallFailsTest` fails
(no `broker.disk.usage_ratio` sample).
- Green after: `Tests run: 5, Failures: 0, Errors: 0, Skipped: 0`.
- Wider sweep (`cluster.metrics.**` + `ops.alert.**`): 473 tests, 0
failures; the errors are the known `@SpringBootTest` classes that need the
MySQL datasource CI provisions.
- `mvn -o checkstyle:check` passes.
## Risk
Low. Only failure paths gain samples (extra unavailable rows), which is what
the design note prescribes; the success path and the parse-miss branches are
unchanged.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
