2020-02-09 21:17:28 UTC - Greg Methvin: So we’ve been using delayed messages with Pulsar and have been occasionally noticing strange issues with the metrics (at least we think it’s only with the metrics). Ideally we’d like some metric to tell us the number of non-delayed messages in the backlog, so we can use that to set up alerts to know when topics are getting consumed. For now we’ve been using `msg_backlog - msg_delayed` as the metric, but it seems like occasionally the delayed messages metric for some topics gets out of whack, and we often see more delayed messages than the total backlog. It seems like it’s often made worse by restarting a lot (10+) of consumers at once. Has anyone else encountered this issue before? Or does anyone have an idea of what might cause it? ---- 2020-02-10 07:13:57 UTC - Jasper Li: Hi all, I was trying to deploy a testing environment with pulsar 2.5.0, but when I deployed it on GKE by helm install, I have got the following issue: ```[main] ERROR org.apache.bookkeeper.server.Main - Failed to build bookie server org.apache.bookkeeper.bookie.BookieException$InvalidCookieException: Cookie [4 bookieHost: "10.8.0.19:3181" journalDir: "data/bookkeeper/journal" ledgerDirs: "1\tdata/bookkeeper/ledgers" nstanceId: "f92894c6-05ea-42d2-8e3f-02a58f0caed8" ] is not matching with [4 bookieHost: "10.8.1.7:3181" journalDir: "data/bookkeeper/journal" ledgerDirs: "1\tdata/bookkeeper/ledgers" instanceId: "f92894c6-05ea-42d2-8e3f-02a58f0caed8" ] at org.apache.bookkeeper.bookie.Cookie.verifyInternal(Cookie.java:136) ~[org.apache.bookkeeper-bookkeeper-server-4.10.0.jar:4.10.0] ...``` Can anyone help me to solve the issue? ---- 2020-02-10 07:23:21 UTC - Shahar Fermon: had the same issue a few days ago (still not sure why) and i tried to format the cookie but the container kept crashing (found this command `bin/bookkeeper shell bookieformat` ) ended up deleting the storage and forcing the bookkeeper creating a new cookie ---- 2020-02-10 07:23:45 UTC - Shahar Fermon: not sure it’s the best solution but it worked for me ---- 2020-02-10 07:51:46 UTC - Sijie Guo: it is because the pod ip address was changed. bookie prevent itself to start when the bookie identity was changed.
Please make sure “useHostNameAsBookieID” is set to true. If you are using helm chart, there is a problem in current chart. because the setting names are changed to “PULSAR_PREFIX_useHostNameAsBookieID”. It causes the settings are not applied correctly. Please remove “PULSAR_PREFIX_” in the current values.yml file. ---- 2020-02-10 07:54:40 UTC - Sijie Guo: I think all the counters are “approximate” because they are counted since a topic is owned. I don’t think `msg_backlog - msg_delayed` can be reliably used for measuring non-delayed messages. we can add a metric to count the non-delayed messages. /cc @Penghui Li ---- 2020-02-10 08:38:11 UTC - Penghui Li: Ok, I added an issue <https://github.com/apache/pulsar/issues/6281> to track this task. ---- 2020-02-10 08:39:57 UTC - Vincent LE MAITRE: @Vincent LE MAITRE has joined the channel ----
