2020-06-25 09:31:54 UTC - prem m: @prem m has joined the channel ---- 2020-06-25 09:36:09 UTC - prem m: Hi All, may I know from pulsar functions how to publish message(s) to different pulsar topic partition based on hashing? . pulsar producer has various hashing schemes, like that i am looking for. ---- 2020-06-25 12:44:49 UTC - Rahul Vashishth: @sjmittal @Vladimir Shchur @dionjansen same issue for me too. did you find a fix or workaround? ---- 2020-06-25 12:45:15 UTC - Vladimir Shchur: Workaround is to rollback to 2.5.2 +1 : Rahul Vashishth ---- 2020-06-25 13:12:32 UTC - Rahul Vashishth: Even after <https://github.com/apache/pulsar-helm-chart/blob/master/charts/pulsar/values.yaml#L113|dashboard: false> in charts, but still i can see dashboard pods being created when i installed latest chart 2.6.0 ---- 2020-06-25 13:49:29 UTC - Matteo Merli: No, the termination is permanently sealing the topic ---- 2020-06-25 13:49:45 UTC - Jeff Schneller: We have a cluster of 3 brokers and trying to setup pulsar-manager to manage it. However pulsar-manager will not connect to the cluster. The 3 brokers have authentication and authorization turned on. For Pulsar Manager we created a keystore with our ca and the cert for the admin role. However it will not connect. Any ideas? What url do we use for the cluster environment? How is the cluster defined? We are able to run the admin-cli from one of the brokers without any issues. ---- 2020-06-25 14:07:58 UTC - Frank Kelly: Thanks for flagging this - we are planning a custom auth methodology that uses a new JWT format - hopefully we'll just be able to piggy back on the current Auth Token provider (wrapping a JWT token with different semantics). So this is good to know ---- 2020-06-25 14:22:58 UTC - Gavin Liu: Have you tried just connecting to one broker instead of 3, I know that for version 2.5.1 and 2.5.2, if the pulsar connection string contains more than 1 broker, you have trouble accessing some functionalities. ---- 2020-06-25 14:24:03 UTC - Jeff Schneller: using only 1. the manager connects to http/https like the admin cli not the pulsar ports. +1 : Gavin Liu ---- 2020-06-25 14:25:15 UTC - Gavin Liu: @Poul Henriksen Hey man, I searched the history and found that you asked about this as well, did you figure all out? ---- 2020-06-25 14:43:10 UTC - Poul Henriksen: no... ---- 2020-06-25 15:00:29 UTC - Charvak Patel: in the url, are you giving tcp port ( mostly 8080 )? you can see this as an example setup for pulsar manager <https://pulsar.apache.org/docs/en/kubernetes-helm/#step-4-use-pulsar-manager-to-manage-the-cluster> The url provided is <http://pulsar-mini-broker:8080> which is this NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE pulsar-mini-bookie ClusterIP None <none> 3181/TCP,8000/TCP 11m pulsar-mini-broker ClusterIP None <none> 8080/TCP,6650/TCP 11m ---- 2020-06-25 15:04:08 UTC - Gavin Liu: Bummer, I am almost affirmative that all these are on a partition level. Because if not, when over the limit, how the events are are deleted from the partitions? @Sijie Guo agree? ---- 2020-06-25 15:11:11 UTC - Jeff Schneller: not using kubernetes and 8080 probably won't work since we are using tls certs ---- 2020-06-25 15:15:36 UTC - Anup Ghatage: Hey @rani I wrote a doc about how to cleanly decomm bookies.
Check it out here: <https://pulsar.apache.org/docs/en/administration-zk-bk/#decommissioning-bookies-cleanly> ---- 2020-06-25 15:40:09 UTC - Rahul Vashishth: it seems similar issue was fixed in <https://github.com/apache/pulsar/issues/6355> and reintroduced again ---- 2020-06-25 17:41:35 UTC - Raphael Enns: I've been doing some local stability testing with our messaging tool, which is backed by a pulsar broker. I'm using java client 2.5.2 and running pulsar 2.5.2 in standalone mode. While running a test, suddenly I stopped receiving any messages from pulsar. I noticed that the thread "pulsar-external-listener-3-1" in pulsar-client stopped having any log entries at that point. I did a thread dump of the client and it looks like that thread is stuck waiting in "hasMessagesAvailable" (see below). There were no warnings or errors in the client logs or the pulsar broker logs. The only way to resolve it is to restart the pulsar client, which is definitely not ideal. ---- 2020-06-25 17:41:52 UTC - Raphael Enns: ---- 2020-06-25 17:42:52 UTC - Matteo Merli: With tokens, there will be no problem, you can use the regular auth provider on client side and a custom one on the broker. sunglasses : Frank Kelly ---- 2020-06-25 18:27:00 UTC - Rahul Vashishth: Is the helm chart repo for this <https://github.com/apache/pulsar-helm-chart/> is publically hosted somewhere, like github pages or helm hub? ---- 2020-06-25 18:29:29 UTC - Rahul Vashishth: @Sijie Guo @Adriaan de Haan i see the same error on current chart in pulsar-bookkeeper's init container (pulsar-bookkeeper-verify-clusterid) `[conf/bookkeeper.conf] Applying config zkServers = pulsar-zookeeper:2181` `[conf/bookkeeper.conf] Adding config journalDirectories = /pulsar/data/bookkeeper/journal` `JMX enabled by default` `Error: Could not find or load main class "-Xms1g` ---- 2020-06-25 18:46:33 UTC - Shilpa Shanbhag: @Shilpa Shanbhag has joined the channel ---- 2020-06-25 20:14:09 UTC - rwaweber: To speak to the same general issue — we created another topic, without a consumer, as a means to understand backlog retention policy behavior better, and it appears that the `pulsar_storage_size` metric is twice the value of what is reported through `./bin/pulsar-admin topics stats topic <topicName>` for the `storageSize` key. What calculations go into the `pulsar_storage_size` metric? ---- 2020-06-25 22:14:54 UTC - Kirill Merkushev: so even no way to recreate the topic? ---- 2020-06-25 22:15:48 UTC - Kirill Merkushev: Is there a way to unsubscribe exclusive consumer from the broker side? ---- 2020-06-25 22:31:36 UTC - Raphael Enns: Another thing I ran into was that I got an OutOfMemoryError in the Java pulsar client (2.5.2), which seemed due to being disconnected from the broker. Here are a few of the ERROR/WARN log entries from the java client: ---- 2020-06-25 22:31:52 UTC - Raphael Enns: ---- 2020-06-25 22:32:05 UTC - Raphael Enns: I removed duplicate log entries. ---- 2020-06-25 22:32:30 UTC - Matteo Merli: `pulsar-admin topics unsubscribe $TOPIC -s $SUB` +1 : Kirill Merkushev ---- 2020-06-25 22:33:07 UTC - Matteo Merli: Yes, you can delete the topic and re-create ---- 2020-06-25 22:33:14 UTC - Raphael Enns: From what I could gather, the OutOfMemoryError occurred because the java client kept accumulating either messages to be sent or readers or the data until it ran out of memory. ---- 2020-06-25 22:33:25 UTC - Raphael Enns: Do you have any suggestions on how to recover from cases like this? ---- 2020-06-25 22:34:03 UTC - Matteo Merli: You can control how many messages to keep in pending state, with the `producerQueueSize` setting when creating the producer. ---- 2020-06-25 22:35:16 UTC - Kirill Merkushev: This is odd, since I’ve got an error that topic was terminated after I recreated it…ok, anyway thanks for the quick response ---- 2020-06-25 23:05:39 UTC - Alan Kittel: @Alan Kittel has joined the channel ---- 2020-06-25 23:20:13 UTC - Matteo Merli: Ok, if that's the case it would be a bug :slightly_smiling_face: ---- 2020-06-25 23:30:46 UTC - Sijie Guo: it is per-topic(partition) level. ---- 2020-06-25 23:31:09 UTC - Sijie Guo: The namespace policy is applied to the topic (partitions) within it. ---- 2020-06-25 23:32:56 UTC - Sijie Guo: Are you using tls authentication? Or just using tls for encryption? ---- 2020-06-25 23:33:29 UTC - Jeff Schneller: Both ---- 2020-06-25 23:33:59 UTC - Sijie Guo: Not yet. It will be published to <https://pulsar.apache.org/charts>. ---- 2020-06-25 23:34:10 UTC - Sijie Guo: There is an outstanding PR to be merged to make it happen. ---- 2020-06-25 23:37:58 UTC - Kirill Merkushev: its saying that subscription has active consumers that way. Is there a force mode? ---- 2020-06-25 23:40:37 UTC - Sijie Guo: how did you get the `pulsar_storage_size` ? • did you curl the metrics endpoint? If so, can you attach the whole metric with all the labels? • did you get the storage_size from grafana? If so, it is worth to see if double-counts the metrics from prometheus. ---- 2020-06-25 23:44:40 UTC - Sijie Guo: if it is tls authentication, I don’t think Pulsar Manager supports tls authentication yet. ---- 2020-06-25 23:44:52 UTC - Sijie Guo: It only supports tls encryption + jwt authentication. ---- 2020-06-25 23:46:18 UTC - Jeff Schneller: Hmm. Ok. Will need to rethink things then. ---- 2020-06-25 23:47:37 UTC - Sijie Guo: Can you create an issue at <https://github.com/apache/pulsar-manager>? We can see how to add that support. ---- 2020-06-26 00:30:34 UTC - Gavin Liu: @Sijie Guo Thanks for the confirmation. :thumbsup: :thumbsup: ---- 2020-06-26 01:24:32 UTC - Jeff Schneller: Issue created ok_hand : Sijie Guo ---- 2020-06-26 04:09:18 UTC - Matteo Merli: Yes, just add `--force` ---- 2020-06-26 08:38:02 UTC - Sajith Sebastian: @Sijie Guo I ran into an issue with pulsar binary protocol when I configure AMQP connector. The moment we configure messagingProtocols=amqp default pulsar binary protocol is not listening in the port 6650 . Can you please share some light ----
