hangc0276 commented on code in PR #22778: URL: https://github.com/apache/pulsar/pull/22778#discussion_r1623714315
########## pip/pip-355.md: ########## @@ -0,0 +1,39 @@ +# PIP-355: Enhancing Broker-Level Metrics for Pulsar + +# Background Knowledge +Pulsar provides broker-level, namespace-level, and topic-level metrics to monitor and analyze the behavior of the Pulsar service. These metrics are accessible through the Prometheus metrics endpoint. Detailed explanations of all metrics can be found on the Pulsar website: [Pulsar Metrics Reference](https://pulsar.apache.org/docs/3.2.x/reference-metrics/) + +# Motivation +Within Pulsar's current metrics framework, the `pulsar_out_bytes_total` metric is utilized to expose the total bytes dispatched by the broker to consumers. However, there are notable limitations and challenges associated with this metric: +- Inclusion of system subscriptions in the total bytes out, alongside user subscriptions, complicates accurate calculation of user-specific data. +- The granularity of the metric (namespace-level vs. topic-subscription level) impacts the scalability and resource consumption when calculating cluster-level total out bytes. + +# Goals +This proposal aims to address the following objectives: +- Simplify the process of calculating cluster-level total out bytes. +- Enable the calculation of total out bytes dispatched to system subscriptions. + +# High-Level Design +To achieve the outlined goals, the proposal introduces three new broker-level metrics: +- `pulsar_broker_out_bytes_total`: Represents the total out bytes dispatched by the broker to consumers, encompassing both user and system subscriptions. Review Comment: From the usage perspective, we only care about the `system_subscription` in `pulsar_broker_out_bytes_total`, and do not need system_topic, which will make the metric generation complex. The following two metric is enough: ``` pulsar_broker_in_bytes_total{cluster=, topic=, system_topic=true} pulsar_broker_out_bytes_total{cluster=, topic=, subscription=, system_subscription=true} ``` But it will fall back to this pair of metrics having different labels. I'm not sure if it is acceptable. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: commits-unsubscr...@pulsar.apache.org For queries about this service, please contact Infrastructure at: us...@infra.apache.org
