jerryshao opened a new issue, #12944:
URL: https://github.com/apache/gravitino/issues/12944
### Version
main branch
### Describe what's wrong
Request-duration timers — both the Jersey `@Timed` HTTP-duration timers
(`gravitino-server`, `iceberg-rest-server`) and the AspectJ `@Monitored` method
timers (`gravitino-relational-store`) — report `mean`/`p95`/`p99`/`max` as
`0.0` for any endpoint that hasn't been called within the reservoir's sliding
window (`gravitino.metrics.timeSlidingWindowSecs`, default 60s) at the moment
`/metrics` is scraped, even though `count` keeps incrementing correctly. In
practice, this affects nearly every real API endpoint, since most operations
aren't called at least once every 60 seconds, making `/metrics` unusable for
latency alerting or slow-operation investigation.
### Error message and/or stacktrace
N/A — this is a silent data-correctness issue, not a crash. `/metrics`
returns valid Prometheus output; the duration values within it are just wrong
(zero) for low-frequency series.
### How to reproduce
1. Start a Gravitino server with default metrics config.
2. Call a REST endpoint that isn't hit frequently (e.g. `GET
/api/metalakes/{metalake}/catalogs/{catalog}` to load a catalog) a few times.
3. Wait longer than `gravitino.metrics.timeSlidingWindowSecs` (default 60s)
without calling that endpoint again.
4. Scrape `GET /metrics`.
5. Observe: the corresponding timer (e.g.
`gravitino-server.load-catalog.http-request-duration-seconds`) shows a nonzero
`count` but `mean`/`p95`/`p99`/`max` all read `0.0`.
### Additional context
Root cause: `MetricsSource.getTimer()`/`getHistogram()`
(`core/src/main/java/org/apache/gravitino/metrics/source/MetricsSource.java`)
and `HttpServerMetricsSource`
(`server-common/src/main/java/org/apache/gravitino/server/web/HttpServerMetricsSource.java`)
back every timer with `com.codahale.metrics.SlidingTimeWindowArrayReservoir`.
Its `getSnapshot()` calls `trim()` first, discarding any sample older than the
window; if nothing lands inside the window, the snapshot is built from an empty
array, and dropwizard's `UniformSnapshot` hard-codes `0.0`/`0` for
`getMean()`/`getMax()`/`getValue(quantile)` when there are no values. `count`
comes from `Histogram`'s own separate, never-expiring `LongAdder`, so it's
unaffected and stays accurate while every duration statistic reads zero —
durations are captured correctly at write time and simply discarded before they
can be read back.
Options:
- Short-term mitigation: increase `gravitino.metrics.timeSlidingWindowSecs`
— reduces how often this triggers, but any endpoint called less often than the
configured window will still eventually read zero, and a larger window
increases reservoir memory for high-traffic endpoints.
- Real fix: switch the reservoir type to one that doesn't hard-expire to
empty on low-traffic endpoints, e.g. `ExponentiallyDecayingReservoir`
(dropwizard's own default) or `SlidingWindowReservoir(N)` (fixed sample count,
no time-based eviction). This changes the statistical semantics of exposed
percentiles/mean/max (no longer a strict "last N seconds" window), so it should
be called out as an intentional behavior change rather than a silent tweak.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
