gortiz commented on PR #9416: URL: https://github.com/apache/pinot/pull/9416#issuecomment-1573590631
Update on the issue: Metrics created using Dropwizard and Yammer are very close, but not exactly the same. By running using the new CompoundPinotMetricsFactory as factory, we can produce both metrics. After that, I used [jxmterm](https://cwiki.apache.org/confluence/display/KAFKA/jmxterm+quickstart) to read the JMX metrics exported by Yammer and Dropwizard. The first difference is the id of the beans created. JMX metrics have several coordinates. Some of them are standard (like domain, name) and some other are not. Specifically, our usage of Yammer and Dropwizard uses these three coordinates: - domain: - Yammer: we set the value to `"org.apache.pinot.common.metrics"` (note> we include the double quotes). - Dropwizard: the domain can be changed with `pinot.metrics.dropwizard.domain`. By default it is `metrics`. There is a Javadoc saying that we use that because it was the default in Pinot < 11. I've changed the default to `org.apache.pinot.common.metrics` (without double quotes) to make it closer to Yammer and more difficult to collide with other metrics running in the same system. - name: - Yammer: we wrap the names between double quotes. For example, metric `pinot.server.meetupRsvp_REALTIME.numSegmentsPrunedInvalid` is reported as `"pinot.server.meetupRsvp_REALTIME.numSegmentsPrunedInvalid"` by Yammer - Dropwizard: we reported the names without double quotes. For example, metric `pinot.server.meetupRsvp_REALTIME.numSegmentsPrunedInvalid` is reported as `pinot.server.meetupRsvp_REALTIME.numSegmentsPrunedInvalid` by Dropwizard. - type: - Yammer: We set as type either `ControllerMetrics`, `BrokerMetrics`, `MinionMetrics`, `ServerMetrics` or `ValidationMetrics`. - Dropwizard: The type is set by Dropwizard, using values like `gauge`, `meters`. It may be possible to change that, but doesn't seem trivial. As the last part, we configure the prometheus exporter to map JMX metrics to prometheus. In files like `docker/images/pinot/etc/jmx_prometheus_javaagent/configs/broker.yml` (and server.yml, controller.yml, etc) we configure rules that do that mapping. For example: ```yaml - pattern: "\"org.apache.pinot.common.metrics\"<type=\"BrokerMetrics\", name=\"pinot.broker.([^\\.]*?)\\.totalServerResponseSize\"><>(\\w+)" name: "pinot_broker_totalServerResponseSize_$2" cache: true labels: table: "$1" ``` which follows the path: `${domain}<type=${type}, name=${name}` In order to be able to read Prometheus metrics, we should change the pattern to something like: ```yaml - pattern: "\"?org.apache.pinot.common.metrics\"?<type=\\w, name=\"pinot.broker.([^\\.]*?)\\.totalServerResponseSize\"><>(\\w+)" ``` Probably we could even skip the type and generate something like: ```yaml - pattern: "\"?org.apache.pinot.common.metrics\"?<name=\"pinot.broker.([^\\.]*?)\\.totalServerResponseSize\"><>(\\w+)" ``` This changes should not be done when using `CompoundPinotMetricsFactory`, given that it would catch metrics twice (once the one generated by Yammer and once the one generated by Dropwizard) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
