lelabo-marc commented on issue #3112: broker error for prometheus metrics URL: https://github.com/apache/pulsar/issues/3112#issuecomment-467810965 We have something similar: `text format parsing error in line 189: second TYPE line for metric name "pulsar_subscriptions_count", or TYPE reported after samples` When doing a port forward on my pod: `kubectl port-forward $(kubectl get pods -l component=broker -o jsonpath='{.items[*].metadata.name}') 8080` I can see on [http://localhost:8080/metrics/](http://localhost:8080/metrics/). Some of the metrics like `pulsar_subscriptions_count` are duplicated (one by default and one for each namespace). ``` # TYPE log4j2_appender_total counter log4j2_appender_total{cluster="pulsar",level="debug"} 0.0 log4j2_appender_total{cluster="pulsar",level="warn"} 4.0 log4j2_appender_total{cluster="pulsar",level="trace"} 0.0 log4j2_appender_total{cluster="pulsar",level="error"} 0.0 log4j2_appender_total{cluster="pulsar",level="fatal"} 0.0 log4j2_appender_total{cluster="pulsar",level="info"} 477.0 # TYPE jvm_buffer_pool_used_bytes gauge jvm_buffer_pool_used_bytes{cluster="pulsar",pool="direct"} 1374353.0 jvm_buffer_pool_used_bytes{cluster="pulsar",pool="mapped"} 0.0 # TYPE jvm_buffer_pool_capacity_bytes gauge jvm_buffer_pool_capacity_bytes{cluster="pulsar",pool="direct"} 1374352.0 jvm_buffer_pool_capacity_bytes{cluster="pulsar",pool="mapped"} 0.0 # TYPE jvm_buffer_pool_used_buffers gauge jvm_buffer_pool_used_buffers{cluster="pulsar",pool="direct"} 50.0 jvm_buffer_pool_used_buffers{cluster="pulsar",pool="mapped"} 0.0 # TYPE jvm_gc_collection_seconds summary jvm_gc_collection_seconds_count{cluster="pulsar",gc="G1 Young Generation"} 2.0 jvm_gc_collection_seconds_sum{cluster="pulsar",gc="G1 Young Generation"} 1.667 jvm_gc_collection_seconds_count{cluster="pulsar",gc="G1 Old Generation"} 0.0 jvm_gc_collection_seconds_sum{cluster="pulsar",gc="G1 Old Generation"} 0.0 # TYPE process_cpu_seconds_total counter process_cpu_seconds_total{cluster="pulsar"} 60.95 # TYPE process_start_time_seconds gauge process_start_time_seconds{cluster="pulsar"} 1.551261824492E9 # TYPE process_open_fds gauge process_open_fds{cluster="pulsar"} 367.0 # TYPE process_max_fds gauge process_max_fds{cluster="pulsar"} 8192.0 # TYPE process_virtual_memory_bytes gauge process_virtual_memory_bytes{cluster="pulsar"} 1.743130624E10 # TYPE process_resident_memory_bytes gauge process_resident_memory_bytes{cluster="pulsar"} 1.637597184E9 # TYPE zk_write_latency summary zk_write_latency{cluster="pulsar",quantile="0.5"} NaN zk_write_latency{cluster="pulsar",quantile="0.75"} NaN zk_write_latency{cluster="pulsar",quantile="0.95"} NaN zk_write_latency{cluster="pulsar",quantile="0.99"} NaN zk_write_latency{cluster="pulsar",quantile="0.999"} NaN zk_write_latency{cluster="pulsar",quantile="0.9999"} NaN zk_write_latency_count{cluster="pulsar"} 0.0 zk_write_latency_sum{cluster="pulsar"} 0.0 # TYPE zk_read_latency summary zk_read_latency{cluster="pulsar",quantile="0.5"} NaN zk_read_latency{cluster="pulsar",quantile="0.75"} NaN zk_read_latency{cluster="pulsar",quantile="0.95"} NaN zk_read_latency{cluster="pulsar",quantile="0.99"} NaN zk_read_latency{cluster="pulsar",quantile="0.999"} NaN zk_read_latency{cluster="pulsar",quantile="0.9999"} NaN zk_read_latency_count{cluster="pulsar"} 0.0 zk_read_latency_sum{cluster="pulsar"} 0.0 # TYPE jvm_classes_loaded gauge jvm_classes_loaded{cluster="pulsar"} 9550.0 # TYPE jvm_classes_loaded_total counter jvm_classes_loaded_total{cluster="pulsar"} 9550.0 # TYPE jvm_classes_unloaded_total counter jvm_classes_unloaded_total{cluster="pulsar"} 0.0 # TYPE jvm_threads_current gauge jvm_threads_current{cluster="pulsar"} 113.0 # TYPE jvm_threads_daemon gauge jvm_threads_daemon{cluster="pulsar"} 9.0 # TYPE jvm_threads_peak gauge jvm_threads_peak{cluster="pulsar"} 116.0 # TYPE jvm_threads_started_total counter jvm_threads_started_total{cluster="pulsar"} 144.0 # TYPE jvm_threads_deadlocked gauge jvm_threads_deadlocked{cluster="pulsar"} 0.0 # TYPE jvm_threads_deadlocked_monitor gauge jvm_threads_deadlocked_monitor{cluster="pulsar"} 0.0 # TYPE jvm_memory_direct_bytes_used gauge jvm_memory_direct_bytes_used{cluster="pulsar"} 8.388608E7 # TYPE jvm_memory_bytes_used gauge jvm_memory_bytes_used{cluster="pulsar",area="heap"} 1.003231264E9 jvm_memory_bytes_used{cluster="pulsar",area="nonheap"} 8.194224E7 # TYPE jvm_memory_bytes_committed gauge jvm_memory_bytes_committed{cluster="pulsar",area="heap"} 1.2884901888E10 jvm_memory_bytes_committed{cluster="pulsar",area="nonheap"} 8.5991424E7 # TYPE jvm_memory_bytes_max gauge jvm_memory_bytes_max{cluster="pulsar",area="heap"} 1.2884901888E10 jvm_memory_bytes_max{cluster="pulsar",area="nonheap"} -1.0 # TYPE jvm_memory_bytes_init gauge jvm_memory_bytes_init{cluster="pulsar",area="heap"} 1.2884901888E10 jvm_memory_bytes_init{cluster="pulsar",area="nonheap"} 2555904.0 # TYPE jvm_memory_pool_bytes_used gauge jvm_memory_pool_bytes_used{cluster="pulsar",pool="Code Cache"} 2.0103808E7 jvm_memory_pool_bytes_used{cluster="pulsar",pool="Metaspace"} 5.5292816E7 jvm_memory_pool_bytes_used{cluster="pulsar",pool="Compressed Class Space"} 6545616.0 jvm_memory_pool_bytes_used{cluster="pulsar",pool="G1 Eden Space"} 9.68884224E8 jvm_memory_pool_bytes_used{cluster="pulsar",pool="G1 Survivor Space"} 2.5165824E7 jvm_memory_pool_bytes_used{cluster="pulsar",pool="G1 Old Gen"} 9181216.0 # TYPE jvm_memory_pool_bytes_committed gauge jvm_memory_pool_bytes_committed{cluster="pulsar",pool="Code Cache"} 2.1037056E7 jvm_memory_pool_bytes_committed{cluster="pulsar",pool="Metaspace"} 5.7745408E7 jvm_memory_pool_bytes_committed{cluster="pulsar",pool="Compressed Class Space"} 7208960.0 jvm_memory_pool_bytes_committed{cluster="pulsar",pool="G1 Eden Space"} 6.740246528E9 jvm_memory_pool_bytes_committed{cluster="pulsar",pool="G1 Survivor Space"} 2.5165824E7 jvm_memory_pool_bytes_committed{cluster="pulsar",pool="G1 Old Gen"} 6.119489536E9 # TYPE jvm_memory_pool_bytes_max gauge jvm_memory_pool_bytes_max{cluster="pulsar",pool="Code Cache"} 2.5165824E8 jvm_memory_pool_bytes_max{cluster="pulsar",pool="Metaspace"} -1.0 jvm_memory_pool_bytes_max{cluster="pulsar",pool="Compressed Class Space"} 1.073741824E9 jvm_memory_pool_bytes_max{cluster="pulsar",pool="G1 Eden Space"} -1.0 jvm_memory_pool_bytes_max{cluster="pulsar",pool="G1 Survivor Space"} -1.0 jvm_memory_pool_bytes_max{cluster="pulsar",pool="G1 Old Gen"} 1.2884901888E10 # TYPE jvm_memory_pool_bytes_init gauge jvm_memory_pool_bytes_init{cluster="pulsar",pool="Code Cache"} 2555904.0 jvm_memory_pool_bytes_init{cluster="pulsar",pool="Metaspace"} 0.0 jvm_memory_pool_bytes_init{cluster="pulsar",pool="Compressed Class Space"} 0.0 jvm_memory_pool_bytes_init{cluster="pulsar",pool="G1 Eden Space"} 6.765412352E9 jvm_memory_pool_bytes_init{cluster="pulsar",pool="G1 Survivor Space"} 0.0 jvm_memory_pool_bytes_init{cluster="pulsar",pool="G1 Old Gen"} 6.119489536E9 # TYPE jvm_info gauge jvm_info{cluster="pulsar",version="1.8.0_181-8u181-b13-2~deb9u1-b13",vendor="Oracle Corporation",runtime="OpenJDK Runtime Environment"} 1.0 # TYPE topic_load_times summary topic_load_times{cluster="pulsar",quantile="0.5"} NaN topic_load_times{cluster="pulsar",quantile="0.75"} NaN topic_load_times{cluster="pulsar",quantile="0.95"} NaN topic_load_times{cluster="pulsar",quantile="0.99"} NaN topic_load_times{cluster="pulsar",quantile="0.999"} NaN topic_load_times{cluster="pulsar",quantile="0.9999"} NaN topic_load_times_count{cluster="pulsar"} 0.0 topic_load_times_sum{cluster="pulsar"} 0.0 # TYPE jvm_memory_direct_bytes_max gauge jvm_memory_direct_bytes_max{cluster="pulsar"} 1.5032385536E10 # TYPE jetty_requests_total counter jetty_requests_total{cluster="pulsar"} 239.0 # TYPE jetty_requests_active gauge jetty_requests_active{cluster="pulsar"} 1.0 # TYPE jetty_requests_active_max gauge jetty_requests_active_max{cluster="pulsar"} 1.0 # TYPE jetty_request_time_max_seconds gauge jetty_request_time_max_seconds{cluster="pulsar"} 0.101 # TYPE jetty_request_time_seconds_total counter jetty_request_time_seconds_total{cluster="pulsar"} 0.914 # TYPE jetty_dispatched_total counter jetty_dispatched_total{cluster="pulsar"} 239.0 # TYPE jetty_dispatched_active gauge jetty_dispatched_active{cluster="pulsar"} 0.0 # TYPE jetty_dispatched_active_max gauge jetty_dispatched_active_max{cluster="pulsar"} 1.0 # TYPE jetty_dispatched_time_max gauge jetty_dispatched_time_max{cluster="pulsar"} 101.0 # TYPE jetty_dispatched_time_seconds_total counter jetty_dispatched_time_seconds_total{cluster="pulsar"} 0.588 # TYPE jetty_async_requests_total counter jetty_async_requests_total{cluster="pulsar"} 80.0 # TYPE jetty_async_requests_waiting gauge jetty_async_requests_waiting{cluster="pulsar"} 1.0 # TYPE jetty_async_requests_waiting_max gauge jetty_async_requests_waiting_max{cluster="pulsar"} 1.0 # TYPE jetty_async_dispatches_total counter jetty_async_dispatches_total{cluster="pulsar"} 0.0 # TYPE jetty_expires_total counter jetty_expires_total{cluster="pulsar"} 0.0 # TYPE jetty_responses_total counter jetty_responses_total{cluster="pulsar",code="1xx"} 0.0 jetty_responses_total{cluster="pulsar",code="2xx"} 158.0 jetty_responses_total{cluster="pulsar",code="3xx"} 78.0 jetty_responses_total{cluster="pulsar",code="4xx"} 2.0 jetty_responses_total{cluster="pulsar",code="5xx"} 0.0 # TYPE jetty_stats_seconds gauge jetty_stats_seconds{cluster="pulsar"} 1524.577 # TYPE jetty_responses_bytes_total counter jetty_responses_bytes_total{cluster="pulsar"} 910107.0 # TYPE pulsar_topics_count gauge pulsar_topics_count{cluster="pulsar"} 0 1551263358513 # TYPE pulsar_subscriptions_count gauge pulsar_subscriptions_count{cluster="pulsar"} 0 1551263358513 # TYPE pulsar_producers_count gauge pulsar_producers_count{cluster="pulsar"} 0 1551263358513 # TYPE pulsar_consumers_count gauge pulsar_consumers_count{cluster="pulsar"} 0 1551263358513 # TYPE pulsar_rate_in gauge pulsar_rate_in{cluster="pulsar"} 0 1551263358513 # TYPE pulsar_rate_out gauge pulsar_rate_out{cluster="pulsar"} 0 1551263358513 # TYPE pulsar_throughput_in gauge pulsar_throughput_in{cluster="pulsar"} 0 1551263358513 # TYPE pulsar_throughput_out gauge pulsar_throughput_out{cluster="pulsar"} 0 1551263358513 # TYPE pulsar_storage_size gauge pulsar_storage_size{cluster="pulsar"} 0 1551263358513 # TYPE pulsar_storage_write_rate gauge pulsar_storage_write_rate{cluster="pulsar"} 0 1551263358513 # TYPE pulsar_storage_read_rate gauge pulsar_storage_read_rate{cluster="pulsar"} 0 1551263358513 # TYPE pulsar_msg_backlog gauge pulsar_msg_backlog{cluster="pulsar"} 0 1551263358513 # TYPE pulsar_subscriptions_count gauge pulsar_subscriptions_count{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 1.0 1551263358513 # TYPE pulsar_producers_count gauge pulsar_producers_count{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_consumers_count gauge pulsar_consumers_count{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_rate_in gauge pulsar_rate_in{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_rate_out gauge pulsar_rate_out{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_throughput_in gauge pulsar_throughput_in{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_throughput_out gauge pulsar_throughput_out{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_storage_size gauge pulsar_storage_size{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 3254281406.0 1551263358513 # TYPE pulsar_msg_backlog gauge pulsar_msg_backlog{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 589081.0 1551263358513 # TYPE pulsar_storage_write_latency_le_0_5 gauge pulsar_storage_write_latency_le_0_5{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_storage_write_latency_le_1 gauge pulsar_storage_write_latency_le_1{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_storage_write_latency_le_5 gauge pulsar_storage_write_latency_le_5{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_storage_write_latency_le_10 gauge pulsar_storage_write_latency_le_10{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_storage_write_latency_le_20 gauge pulsar_storage_write_latency_le_20{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_storage_write_latency_le_50 gauge pulsar_storage_write_latency_le_50{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_storage_write_latency_le_100 gauge pulsar_storage_write_latency_le_100{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_storage_write_latency_le_200 gauge pulsar_storage_write_latency_le_200{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_storage_write_latency_le_1000 gauge pulsar_storage_write_latency_le_1000{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_storage_write_latency_overflow gauge pulsar_storage_write_latency_overflow{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_storage_write_latency_count gauge pulsar_storage_write_latency_count{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_storage_write_latency_sum gauge pulsar_storage_write_latency_sum{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_entry_size_le_128 gauge pulsar_entry_size_le_128{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_entry_size_le_512 gauge pulsar_entry_size_le_512{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_entry_size_le_1_kb gauge pulsar_entry_size_le_1_kb{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_entry_size_le_2_kb gauge pulsar_entry_size_le_2_kb{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_entry_size_le_4_kb gauge pulsar_entry_size_le_4_kb{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_entry_size_le_16_kb gauge pulsar_entry_size_le_16_kb{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_entry_size_le_100_kb gauge pulsar_entry_size_le_100_kb{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_entry_size_le_1_mb gauge pulsar_entry_size_le_1_mb{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_entry_size_le_overflow gauge pulsar_entry_size_le_overflow{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_entry_size_count gauge pulsar_entry_size_count{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_entry_size_sum gauge pulsar_entry_size_sum{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic"} 0.0 1551263358513 # TYPE pulsar_subscription_back_log gauge pulsar_subscription_back_log{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic",subscription="mysubscriber"} 589081 1551263358513 # TYPE pulsar_subscription_msg_rate_redeliver gauge pulsar_subscription_msg_rate_redeliver{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic",subscription="mysubscriber"} 0.0 1551263358513 # TYPE pulsar_subscription_unacked_massages gauge pulsar_subscription_unacked_massages{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic",subscription="mysubscriber"} 0 1551263358513 # TYPE pulsar_subscription_blocked_on_unacked_messages gauge pulsar_subscription_blocked_on_unacked_messages{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic",subscription="mysubscriber"} 0 1551263358513 # TYPE pulsar_subscription_msg_rate_out gauge pulsar_subscription_msg_rate_out{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic",subscription="mysubscriber"} 0.0 1551263358513 # TYPE pulsar_subscription_msg_throughput_out gauge pulsar_subscription_msg_throughput_out{cluster="pulsar",namespace="ten/ns",topic="persistent://ten/ns/my-topic",subscription="mysubscriber"} 0.0 1551263358513 ```
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services
