dodjdnh opened a new issue, #3764:
URL: https://github.com/apache/dubbo-go/issues/3764
## Motivation / Problem
The built-in Prometheus exporter in dubbo-go continues to export the QPS,
aggregate window count, and latency quantiles computed when the last request
was recorded, even after application traffic stops and all observations in the
sliding window expire. Even when the Prometheus server scrapes normally, the
exported metrics do not reflect the actual state of the current window.
As a result, idle applications still appear to have request load on
monitoring dashboards, and latency panels retain historical P50 / P90 / P95 /
P99 observations indefinitely. This is a data-refresh defect on the SDK
producer side and cannot be resolved solely by adjusting Dubbo Admin or Grafana
queries.
## Current State
In the verified baseline, `DefaultQpsMetricVec`,
`DefaultAggregateCounterVec`, and `DefaultQuantileMetricVec` calculate window
data in place only when `Record()` or `Inc()` is triggered and write the
snapshot to a static Gauge. Once traffic stops, no new records trigger
recalculation, so each Prometheus scrape can only read the last values written.
Although the underlying sliding window already excludes expired buckets when
actively read, the exported Gauge does not trigger a fresh window read during
scraping.
| **Metric** | **Actual behavior after all observations expire** |
**Expected behavior** |
| --- | --- | --- |
| **Provider / Consumer QPS** | Retains the nonzero value from the last
recording indefinitely | Automatically refreshes to `0` |
| **Window request count (`*_aggregate`)** | Retains the window count from
the last recording indefinitely | Automatically refreshes to `0` |
| **Window latency (P50 / P90 / P95 / P99)** | Retains the latency value
from the last recording indefinitely | Returns `NaN` (explicitly indicating no
samples in the window) |
| **Cumulative request Counter** | Retains the historical cumulative value |
**Unchanged** (must never reset to zero because traffic stops) |
_Note: This issue concerns only the three sliding-window metric categories
above. It does not imply that all latency metrics use this export mechanism.
Aggregate RT minimum, maximum, and average use a separate Collector._
## Desired Behavior
Window metrics exported by the built-in Prometheus implementation should
reflect the current valid window rather than the historical snapshot from the
last request. Even without new RPC calls, the next collection after all samples
expire should yield QPS `0`, window count `0`, and latency quantiles `NaN`.
After traffic resumes, only samples still within the valid window should be
counted. Repeated collection should neither create new observations nor
increase cumulative counts, and different label series should remain isolated.
HTTP scraping, local `Scrape()`, and the next Pushgateway push should have
consistent window semantics.
Preserve the existing metric contract, window rules, and source
compatibility for third-party registries. The fix should support concurrent
recording and collection and explicitly describe the capability boundary of
custom registries, without claiming that all implementations automatically gain
idle-window refresh.
## Scope
### Included
- Refreshing Provider / Consumer QPS, window request counts, and latency
quantiles in the built-in Prometheus implementation.
- Correctness for empty windows, resumption after idle periods, repeated
collection, label isolation, and concurrency safety.
- Verification of exported results through HTTP `/metrics`, local
`Scrape()`, and Pushgateway pushes.
- Compatibility regression coverage for cumulative Counters, ordinary
Gauges, and third-party registries.
### Not Included
- Fixing dropped metric events or drift in the `processing` Gauge.
- Fixing sentinel-value logic for aggregate RT extrema (Min / Max) in empty
windows.
- Redesigning the QPS denominator algorithm, bucket boundaries, or
window-duration configuration.
- Dynamic label eviction, cardinality cleanup, persistent background refresh
timers, or performance benchmarks.
- Changes to Dubbo Admin, the Java SDK, Grafana dashboards, Kubernetes, or
Prometheus server scrape rules.
- Browser-based end-to-end acceptance from real production RPCs to external
Prometheus and Admin / Grafana.
## Acceptance Criteria
1. After all samples expire for registered Provider / Consumer series, the
next collection returns QPS and window counts of `0` without new records.
2. P50 / P90 / P95 / P99 are all `NaN` in empty windows; with valid samples,
each reflects its corresponding quantile.
3. After recording resumes, window metrics reflect only valid samples;
repeated collection neither creates extra observations nor increments
cumulative counts.
4. Label series for different applications and methods remain isolated, and
focused race tests for concurrent recording and collection pass.
5. Actual exported results are verified through HTTP `/metrics`, `Scrape()`,
and Pushgateway pushes and follow the expiration and resumption semantics above.
6. Cumulative Counters retain their values while idle and increment normally
after traffic resumes; ordinary Gauge behavior and existing metric names,
labels, types, units, and window rules remain unchanged.
7. Custom implementations of the original `MetricRegistry` still compile and
run, with an explicit description of which implementations support idle-window
refresh.
8. Relevant metric-package unit tests and static checks pass; whether
repository-wide CI and external end-to-end verification were performed must be
stated separately, rather than substituting package-level tests for them.
## Context / Design Notes
- This is a data-quality issue in the dubbo-go SDK metric export layer. It
relates to Admin query compatibility, but the fix does not depend on pending
Admin PRs.
- The default window is 120 seconds with 10 buckets. Sample expiration
follows existing bucket rules rather than simply the time of the last request.
This change does not redefine the window or QPS denominator algorithm.
- Pushgateway stores the most recently pushed snapshot. Correct SDK exports
after traffic stops do not mean that an external gateway refreshes itself
without a subsequent push.
- See the corresponding PR for the collection mechanism, optional interface,
and concurrency protection; this Issue does not prescribe an implementation.
### Minimal Reproduction
This example verifies SDK export behavior directly using the built-in metric
vectors, without a registry center or RPC service:
```go
package main
import (
"fmt"
"time"
prom "github.com/prometheus/client_golang/prometheus"
"dubbo.apache.org/dubbo-go/v3/common"
"dubbo.apache.org/dubbo-go/v3/metrics"
dubboProm "dubbo.apache.org/dubbo-go/v3/metrics/prometheus"
)
func main() {
registry := dubboProm.NewPromMetricRegistry(prom.NewRegistry(),
common.NewURLWithOptions())
qps := metrics.NewQpsMetricVec(registry,
metrics.NewMetricKey("dubbo_provider_qps_total", "Window QPS"))
count := metrics.NewAggregateCounterVec(registry,
metrics.NewMetricKey("dubbo_provider_requests_total_aggregate", "Window request
count"))
p99 := metrics.NewQuantileMetricVec(registry,
[]*metrics.MetricKey{metrics.NewMetricKey("dubbo_provider_rt_milliseconds_p99",
"Window P99")},
[]float64{0.99})
labels := map[string]string{
"application_name": "idle-window-repro",
"interface": "DemoService",
"method": "Call",
}
// Simulate recording six consecutive requests (100ms each), then
scrape once.
for range 6 {
qps.Record(labels)
count.Inc(labels)
p99.Record(labels, 100)
}
printScrape(registry)
// Stop recording and wait 125 seconds to cover expiration of the
default window.
time.Sleep(125 * time.Second)
printScrape(registry)
}
func printScrape(registry interface{ Scrape() (string, error) }) {
text, err := registry.Scrape()
if err != nil {
panic(err)
}
fmt.Print(text)
}
```
- **Execution comparison**:
- **Pre-fix baseline**: The second scrape still retains the historical
QPS, window count, and P99 values from the initial recording.
- **Expected correct result**: On the second scrape, QPS is `0`, window
count is `0`, and P99 is `NaN`.
### Environment and Version Boundaries
- **Code baseline**: `apache/dubbo-go`, commit
`5d892b0a7fd4fb5e6679112a18eccbf8a6eabab2` (other older versions may exhibit
the same behavior).
- **Component configuration**: Built-in Prometheus Registry, default
120-second window / 10 buckets.
- **Local verification environment**: Go 1.26.5 on Windows; focused race
analysis used WSL Ubuntu, Go 1.25.0, and native x86-64 GCC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]