AlinsRan opened a new pull request, #13992:
URL: https://github.com/apache/apisix/pull/13992
### Description
The Stream metrics are keyed by `listen_addr` (`apisix_stream_status`,
`apisix_stream_active_connections`, `apisix_stream_bandwidth`) or by `route`
(`apisix_stream_connection_total`). None of them says which Service a port's
traffic belongs to, although a Service can own several ports and a port can
carry Routes of several Services.
This adds `service` and `service_id` to all four Stream metrics:
| Metric | Labels |
|---|---|
| `apisix_stream_connection_total` | `route`, **`service`**,
**`service_id`** |
| `apisix_stream_active_connections` | `listen_addr`, **`service`**,
**`service_id`** |
| `apisix_stream_status` | `code`, `listen_addr`, **`service`**,
**`service_id`**, `node` |
| `apisix_stream_bandwidth` | `listen_addr`, **`service`**,
**`service_id`**, `type`, `side` |
**Label rule.** It is the same as for the HTTP metrics:
- `service_id` is the ID of the Service the session's Stream Route belongs
to;
- `service` is that ID too, or the Service's name when the Route's
`prometheus` Plugin sets `prefer_name: true`.
Both are empty for a session that never reached a Stream Route with a
Service, for example a failed TLS handshake or no matching Route. Such sessions
are only counted in the `listen_addr` total, so summing over `service` gives
the same values as before.
**How the zone metrics get the label.** `apisix_stream_active_connections`
and `apisix_stream_bandwidth` come from an NGINX shared memory zone that sums
every session of a listening address, so they cannot be split after the fact.
api7/apisix-nginx-module#127 (released in 1.19.11) lets a session be labelled
with an ordered array of values. From then on, its active count and the bytes
it moves are accounted on a `(listen_addr, labels)` slot. The split is
therefore live: a long-lived connection is visible under its Service while it
runs.
- **The Stream `prometheus` Plugin gains a `preread` phase.** It resolves
`{service, service_id}` from the session context and its own conf, and passes
them to `set_labels`. Core code is untouched.
- **The zone read takes the values straight from each dumped entry's
`labels`.** There is no side channel between the subsystems.
- **`apisix_stream_status` and `apisix_stream_connection_total`** resolve
the same labels in the log phase.
- **A Route with both `upstream_id` and `service_id`.** The Service is not
merged into such a Route. The Route's own `service_id` still labels it, as in
the HTTP metrics.
- **Bandwidth baselines follow the existing rule.** A labelled slot is
baselined the first time it is read, like every zone slot. A lost baseline only
rebaselines; it never replays a lifetime total.
**Runtime.** `.requirements` moves to APISIX-Runtime 1.3.19, the first
runtime carrying apisix-nginx-module 1.19.11. `ci/linux-install-openresty.sh`
gets the matching `apisix-runtime-debug` checksums.
**Trade-offs**
- **Sessions rejected before this Plugin runs.** A session rejected in
`preread` by a higher-priority Stream Plugin (`ip-restriction`, `limit-conn`)
is never labelled, so its active count and bytes stay in the unlabelled total.
`apisix_stream_status` still reports it under its Service.
- **Relabelled Services.** A renamed Service, or a `prefer_name` change,
gets a new zone slot; the old series stays (the gauge at 0). Slots are freed
only on restart.
- **Zone capacity.** Labels may take at most three quarters of the zone's
slots, so listening addresses added by a reload are still counted. `1m` holds
about 760 slots. When they run out, sessions of a new Service stay in their
`listen_addr` total, and a warning is logged once per worker.
#### Which issue(s) this PR fixes:
N/A
### Checklist
- [x] I have explained the need for this PR and the problem it solves
- [x] I have explained the changes or the new features added to this PR
- [x] I have added tests corresponding to this change
- [x] I have updated the documentation to reflect this change
- [x] I have verified that this change is backward compatible (If not,
please discuss on the [APISIX mailing
list](https://github.com/apache/apisix/tree/master#community) first)
**Backward compatibility.** Each existing series gains `service=""` and
`service_id=""`, and per-Service series appear next to it. Aggregations over
`listen_addr` (or `route`) return the same values. An alert on a single
unaggregated series needs its selector adjusted.
### Tests
- **`t/stream-plugin/prometheus-metrics.t`.** Existing assertions gain the
two labels. New cases cover:
- the status and connection count under a Service;
- `prefer_name`;
- a Route with both `upstream_id` and `service_id`.
- **`t/stream-plugin/prometheus-metrics-service.t`** (new). It covers:
- a live session split out under its Service, with the unlabelled series
at 0;
- a new slot baselined, then counted from the next session;
- the gauge back to 0 after close;
- a Route without a Service staying unlabelled;
- `prefer_name` on the live series and on the status;
- a slot that lost its baseline rebaselining instead of replaying.
- **`t/stream-plugin/prometheus.t`, `t/cli/test_prometheus_stream.sh`.**
Their `apisix_stream_connection_total` assertions gain the empty labels.
Run locally on the released APISIX-Runtime 1.3.19 deb: every assertion in
the three `t/stream-plugin` files passes. The only failures left are the
lua-resty-events `event worker failed` lines during HUP reloads, which CI
downgrades to `warn` after installing the runtime.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
