GitHub user justin-lathrop added a comment to the discussion: Pulsar upgrade to 3.0.5 causes prometheus metrics timeouts on brokers
Hey @lhotari sorry its been a bit. But I was able to circle back to this and realized I am using [kafka-on-pulsar 3.0.0.4](https://github.com/streamnative/kop/tree/branch-3.0.0.4) on this install as well. I only realized when I added some logs to some of the pulsar-broker metrics classes and found the following exception pop up: ``` 2024-07-05T16:35:12,221Z [jdk.internal.loader.ClassLoaders$AppClassLoader@5bc2b487] error Uncaught exception in thread pool-15-thread-76: 'org.apache.pulsar.common.util.SimpleTextOutputStream org.apache.pulsar.common.util.SimpleTextOutputStream.write(java.lang.String)' java.lang.NoSuchMethodError: 'org.apache.pulsar.common.util.SimpleTextOutputStream org.apache.pulsar.common.util.SimpleTextOutputStream.write(java.lang.String)' at io.streamnative.pulsar.handlers.kop.stats.PrometheusTextFormatUtil.writeGauge(PrometheusTextFormatUtil.java:32) at io.streamnative.pulsar.handlers.kop.stats.PrometheusMetricsProvider.lambda$generate$1(PrometheusMetricsProvider.java:87) at java.base/java.util.concurrent.ConcurrentHashMap.forEach(ConcurrentHashMap.java:1603) at io.streamnative.pulsar.handlers.kop.stats.PrometheusMetricsProvider.generate(PrometheusMetricsProvider.java:87) at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsGenerator.generateMetrics(PrometheusMetricsGenerator.java:345) at org.apache.pulsar.broker.stats.prometheus.PrometheusMetricsGenerator.lambda$renderToBuffer$0(PrometheusMetricsGenerator.java:528) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) ``` I went ahead and removed the kop `.nar` from the installed image and the pulsar-broker metrics began to work again. Then I pulled down the kop repo and rebuilt it against pulsar 3.0.5.5 and used that new `.nar` and was able to get pulsar-broker metrics still. I know the kop repo is archived but it looks like that was my issue. GitHub link: https://github.com/apache/pulsar/discussions/22897#discussioncomment-9990054 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
