lhotari commented on code in PR #24675: URL: https://github.com/apache/pulsar/pull/24675#discussion_r2313827984
########## pip/pip-438.md: ########## @@ -0,0 +1,71 @@ +# PIP-437: Add namespace topics limit metric + +# Background knowledge + +Pulsar namespaces can be configured with a maximum number of topics. When this limit is reached, producers fail to create new topics. Currently, operators lack visibility into namespace topic limits through metrics. + +# Motivation + +Operators cannot proactively monitor namespace capacity because there's no metric exposing the configured topic limit. This leads to reactive troubleshooting when producers start failing. + +# Goals + +## In Scope + +- Add `pulsar_namespace_max_topics_limit` metric exposing the configured maximum topics limit for each namespace +- Include cluster and namespace labels for correlation with existing topic count metrics + +## Out of Scope + +# High Level Design + +Add a new gauge metric that exposes the configured maximum topics limit for each namespace. It will be calculated on +each export of metrics. + +# Detailed Design + +## Design & Implementation Details + +- Create method `getNamespaceTopicLimit` method in `NamespaceStatsAggregator`, fetching from namespace policies their + max topics limit, falling back to the default value set in the broker configuration +- Create method `printNamespaceTopicLimitStats` in `NamespaceStatsAggregator` to print the configured limit for each + namespace Review Comment: This might require some more changes since topic stats contains only the topics that are currently loaded in the broker. Since this is a cluster wide limit, it would be unnecessary work for each broker in the cluster to calculate the current amount of topics that are actually created in the namespace. One possible solution would be to handle this on the "leader broker" so that it's the only broker that creates these statistics. The current solution for finding out the number of topics in the namespace might not be optimal. One possible solution would be to extend the MetadataStore interface to contain a method "getNumberOfChildren" which would use an optimal method for the particular MetadataStore implementation to find the number of child nodes. For example, on Zookeeper, there's a field on the parent ZNode that tells how many child znodes there are. This would be an efficient way at least on Zookeeper to find out how many topics there are in a namespace. update: Since there's MetadataCache layer to list the topics in the namespace, this might not be a significant bottleneck. However, it would lead to the situation where all namespaces would get listed on the leader broker and kept in memory. That could be a problem in some cases and therefore having a way to get the number of children directly would be better. For MetadataStore implementation that don't support getting the number of child nodes, the cache could be used. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
