2020-09-22 12:48:30 UTC - Brent Evans: The namespace will have a maximum of ~32 topics of two types, these are split using key_shared between two consumers currently. The first type can have ~1 message a second, with a size of around 6KB and the second is ~6 messages a second with a size of around 8KB.
In terms of broker logs for a given time this problem has occurred: ```Sep 22 10:13:46 ip-X.eu-south-1.compute.internal pulsar[4613]: 10:13:46.978 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO org.eclipse.jetty.server.RequestLog - 194.19.1.168 - - [22/Sep/2020:10:13:46 +0000] "GET /admin/v2/namespaces/public/namespace/topics HTTP/1.1" 200 721 "-" "Pulsar-Java-v2.6.1" 2 Sep 22 10:14:17 ip-X.eu-south-1.compute.internal pulsar[4613]: 10:14:17.382 [pulsar-load-manager-3-1] INFO org.apache.pulsar.broker.loadbalance.impl.ModularLoadManagerImpl - Writing local data to ZooKeeper because time since last update exceeded threshold of 15 minutes Sep 22 10:14:17 ip-X.eu-south-1.compute.internal pulsar[4613]: 10:14:17.386 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO org.apache.pulsar.zookeeper.ZooKeeperDataCache - [State:CONNECTED Timeout:30000 sessionid:0x766660006 local:/172.16.2.202:34336 remoteserver:ip-X.eu-south-1.compute.internal/194.19.1.194:2181 lastZxid:4294995477 xid:669052 sent:669052 recv:677875 queuedpkts:0 pendingresp:0 queuedevents:0] Received ZooKeeper watch event: WatchedEvent state:SyncConnected type:NodeDataChanged path:/loadbalance/brokers/194.19.1.202:8080 Sep 22 10:15:46 ip-X.eu-south-1.compute.internal pulsar[4613]: 10:15:46.996 [pulsar-ordered-OrderedExecutor-4-0-EventThread] INFO org.eclipse.jetty.server.RequestLog - 194.19.1.168 - - [22/Sep/2020:10:15:46 +0000] "GET /admin/v2/namespaces/public/namespace/topics HTTP/1.1" 200 721 "-" "Pulsar-Java-v2.6.1" 2``` ---- 2020-09-22 15:22:48 UTC - Seun: ```Hi people, I got this error when I tried to connect to broker via proxy from Pulsar Go_client. Proxy is TLS and endpoint is <https://pulsar.example.io:6651> but broker is not under tls. So proxy should be able to access broker un-encrypted. level=warning msg="Failed to connect to broker." error=EOF remote_addr="<pulsar+ssl://pulsar.example.io:6651>"``` ---- 2020-09-22 15:24:16 UTC - Seun: I don't understand where error=EOF is coming from. WE are trying to connect via TLS certificate ---- 2020-09-22 15:25:44 UTC - Ebere Abanonu: end-of-file (*EOF*) ---- 2020-09-22 15:25:54 UTC - Ebere Abanonu: is <http://pulsar.example.io|pulsar.example.io> accessible? ---- 2020-09-22 15:26:25 UTC - Seun: No, I replaced the real domain with example ---- 2020-09-22 15:26:35 UTC - Seun: Yes its accessible ---- 2020-09-22 15:28:11 UTC - Ebere Abanonu: Have you tried connecting to the proxy without TLS? ---- 2020-09-22 15:28:54 UTC - Seun: seun@seun:~/pulsar/pulsar$ curl -Iv <https://pulsar.roava.io:6651|https://pulsar.example.io:6651> * Trying 35.193.118.x:6651... * TCP_NODELAY set * Connected to <http://pulsar.roava.io|pulsar.example.io> (35.193.118.x) port 6651 (#0) * ALPN, offering h2 * ALPN, offering http/1.1 * successfully set certificate verify locations: * CAfile: /etc/ssl/certs/ca-certificates.crt CApath: /etc/ssl/certs * TLSv1.3 (OUT), TLS handshake, Client hello (1): * OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to <http://pulsar.example.io:6651|pulsar.example.io:6651> * Closing connection 0 curl: (35) OpenSSL SSL_connect: SSL_ERROR_SYSCALL in connection to <http://pulsar.example.io:6651|pulsar.example.io:6651> ---- 2020-09-22 15:29:55 UTC - Seun: I really do not how to connect to proxy using TLS cert. THis error is coming from my developer who try to connect ---- 2020-09-22 15:30:17 UTC - Seun: I set up pulsar cluster myself ---- 2020-09-22 15:30:57 UTC - Ebere Abanonu: Ok....have you had any success connecting without TLS cert? ---- 2020-09-22 15:31:24 UTC - Ebere Abanonu: If you are using Helm Charts then disable tls ---- 2020-09-22 15:33:00 UTC - Adriaan de Haan: @Sijie Guo I have another update on this issue. After we spoke about this earlier, the issue went away again, so I never ended up enabling ackTimeout. Today I noticed that the issue is back again and I noticed that it occurred very suddenly and at the same time most of the subscriptions got "recreated". So I checked my Brokers and I noticed that what triggered the problem is a Broker restart (it's running in Kubernetes). ---- 2020-09-22 15:33:33 UTC - Seun: I have tried without tls before on a different cluster and it all worked fine ---- 2020-09-22 15:34:03 UTC - Seun: Then I set up another in a different namespace with tls and this problem occured ---- 2020-09-22 15:34:11 UTC - Adriaan de Haan: ---- 2020-09-22 15:34:39 UTC - Ebere Abanonu: Are you deploying with charts? ---- 2020-09-22 15:35:50 UTC - Seun: Yes ... with helm charts ---- 2020-09-22 15:36:18 UTC - Adriaan de Haan: ---- 2020-09-22 15:36:23 UTC - Seun: I want to be sure if the problem is from dev code or from my end ---- 2020-09-22 15:36:38 UTC - Seun: I just don't know how to figure it out ---- 2020-09-22 15:37:09 UTC - Ebere Abanonu: Ok, lets walk through it together....first you need to install cert-manager, did you do that? ---- 2020-09-22 15:37:13 UTC - Adriaan de Haan: Is this something that can be expected after a Broker dies and is replaced by a new one? ---- 2020-09-22 15:37:34 UTC - Adriaan de Haan: Or should I open a bug report on github? ---- 2020-09-22 15:37:57 UTC - Seun: Yea I did but I didnt enable it ---- 2020-09-22 15:38:06 UTC - Seun: could this be the problem? ---- 2020-09-22 15:39:41 UTC - Ebere Abanonu: Cert-manager is a different package: <https://cert-manager.io/docs/installation/kubernetes/> ---- 2020-09-22 15:40:23 UTC - Ebere Abanonu: Please confirm if you installed it as given in the site above ---- 2020-09-22 15:44:08 UTC - Seun: I installed cert-manager it cert-manager namespace earlier. I used to setup tls for spinnaker. I just didn't enable it in pulsar helm chart ---- 2020-09-22 15:44:29 UTC - Seun: Yes, I followed that guide for installation. ---- 2020-09-22 15:44:34 UTC - Ebere Abanonu: ok ---- 2020-09-22 15:45:02 UTC - Seun: Maybe I should enable it in pulsar helm chart and redeploy. right? ---- 2020-09-22 15:46:37 UTC - Ebere Abanonu: pulsar needs tls secret for each component you are going to enable tls for....in the tls section set internal issuer to true ---- 2020-09-22 15:47:37 UTC - Seun: Yes I only enable tls for proxy to start with. I will now enable internal issuer ---- 2020-09-22 15:47:40 UTC - Ebere Abanonu: ```internal_issuer: enabled: true component: internal-cert-issuer type: selfsigning``` ---- 2020-09-22 15:47:46 UTC - Seun: I will get back shortly ---- 2020-09-22 15:47:48 UTC - Seun: thanks ---- 2020-09-22 15:47:56 UTC - Seun: YES! ---- 2020-09-22 15:48:03 UTC - Ebere Abanonu: not done ---- 2020-09-22 15:48:14 UTC - Seun: Basically, that is all I need to do right? ---- 2020-09-22 15:48:54 UTC - Ebere Abanonu: Your proxy has domain name or just ip?\ ---- 2020-09-22 15:49:56 UTC - Ebere Abanonu: If yes, enable ```external_dns``` ---- 2020-09-22 15:50:40 UTC - Ebere Abanonu: and then configure: ```public_issuer: enabled: true component: public-cert-issuer type: acme``` ---- 2020-09-22 15:51:26 UTC - Ebere Abanonu: set your acme solvers ---- 2020-09-22 15:52:01 UTC - Ebere Abanonu: enable ```domain: enabled: true``` ---- 2020-09-22 15:52:11 UTC - Sijie Guo: @Adriaan de Haan Did you check `topics stats` and `topics stats-internal` to see which subscription has backlog? ---- 2020-09-22 15:53:51 UTC - Ebere Abanonu: disable tls for broker if you dont need ---- 2020-09-22 15:54:16 UTC - Adriaan de Haan: I have, and I have narrowed the problem down to being only related to the service making use of the "proxy" ---- 2020-09-22 15:54:31 UTC - Seun: Yes I set proxy IP address to <http://pulsar.example.io|pulsar.example.io> in google dns ---- 2020-09-22 15:54:55 UTC - Adriaan de Haan: I am running Pulsar in Kubernetes and most of my other services also run in Kubernetes and connect directly to the broker "service" ---- 2020-09-22 15:55:05 UTC - Seun: Right now tls is not enabled for broker ---- 2020-09-22 15:55:32 UTC - Ebere Abanonu: The domain name must already exist ---- 2020-09-22 15:55:44 UTC - Adriaan de Haan: I have one service running outside of Kubernetes which connects to the Proxy through a node-port ---- 2020-09-22 15:56:08 UTC - Adriaan de Haan: And whenever the Broker restart happens, it is only this service that's affected. ---- 2020-09-22 15:56:23 UTC - Seun: Yes it exist I ca ping it ---- 2020-09-22 15:56:38 UTC - Seun: can* ---- 2020-09-22 15:56:58 UTC - Ebere Abanonu: ping works with IP address ---- 2020-09-22 15:57:05 UTC - Adriaan de Haan: I haven't been able to wrap my head around what exactly the Proxy does, would it be a problem if I expose the Broker directly over a node-port and just connect to it instead? To see if it resolves this problem? ---- 2020-09-22 15:57:46 UTC - Seun: I mean I can ping the domain name itself ---- 2020-09-22 15:58:20 UTC - Ebere Abanonu: <http://pulsar.example.io|pulsar.example.io>? ---- 2020-09-22 16:00:06 UTC - Ebere Abanonu: You can setup a test domain name for testing from <https://www.freenom.com/en/index.html?lang=en> ---- 2020-09-22 16:00:33 UTC - Ebere Abanonu: Then host the domain name in google dns zone ---- 2020-09-22 16:01:12 UTC - Seun: I use <http://pulsar.example.io|pulsar.example.io> and its working. I don't need another domain ---- 2020-09-22 16:01:18 UTC - Seun: Its hosted on gcp ---- 2020-09-22 16:02:04 UTC - Seun: There is nothing like domain.enable in pulsar helm chart ---- 2020-09-22 16:02:51 UTC - Ebere Abanonu: I use this <https://github.com/streamnative/charts> ---- 2020-09-22 16:04:41 UTC - Ebere Abanonu: if you are going to use letsencrypt then check this fork out <https://github.com/eaba/charts> ---- 2020-09-22 16:06:26 UTC - Ebere Abanonu: <http://pulsar.example.io|pulsar.example.io> is not reachable from my side ---- 2020-09-22 16:20:45 UTC - Seun: the real domain is <http://pulsar.roava.io|pulsar.roava.io> ---- 2020-09-22 16:21:03 UTC - Seun: i just use the other as example ---- 2020-09-22 16:23:42 UTC - Ebere Abanonu: Ok...if you used the streamNative charts....you might get it to work.....if this error is coming from the client then it may be that tls has not be configured properly: ```level=warning msg="Failed to connect to broker." error=EOF remote_addr="<pulsar+ssl://pulsar.example.io:6651>"``` ---- 2020-09-22 16:27:52 UTC - Seun: I used official helm chat from apache github page ---- 2020-09-22 16:47:54 UTC - Seun: I just enabled cert-manager and redeployed the helm charts. Not sure if anything has changed ---- 2020-09-22 16:49:11 UTC - Seun: Is it ok if I changed issuers to clusterissuer instead of selfsigning ---- 2020-09-22 16:53:04 UTC - Seun: `certs:` `internal_issuer:` `enabled: false` `component: internal-cert-issuer` `type: acme` `issuers:` `clusterissuer:` Is this setting ok ---- 2020-09-22 16:53:25 UTC - Seun: Do I need to provide additional details for clusterissuer? ---- 2020-09-22 17:00:37 UTC - Ebere Abanonu: the internal_issuer helps you to generate tls secret for each component you enabled tls for ---- 2020-09-22 17:00:52 UTC - Ebere Abanonu: you need it enabled ---- 2020-09-22 17:01:33 UTC - Seun: Oh I already enabled it ---- 2020-09-22 17:01:51 UTC - Seun: `certs:` `internal_issuer:` `enabled: true` `component: internal-cert-issuer` `type: acme` `issuers:` `selfsigning:` ---- 2020-09-22 17:03:07 UTC - Ebere Abanonu: then you will need to setup public_issuer for your proxy ---- 2020-09-22 17:03:36 UTC - Ebere Abanonu: ```public_issuer: enabled: true component: public-cert-issuer type: acme issuers: selfsigning: acme: # You must replace this email address with your own. # Let's Encrypt will use this to contact you about expiring # certificates, and issues related to your account. email: <mailto:[email protected]|[email protected]> # change this to production endpoint once you successfully test it server: <https://acme-v02.api.letsencrypt.org/directory> # server: <https://acme-staging-v02.api.letsencrypt.org/directory> solver: azuredns solvers: clouddns: # TODO: add a link about how to configure this section project: "[YOUR GCP PROJECT ID]" serviceAccountSecretRef: name: "[NAME OF SECRET]" key: "[KEY OF SECRET]" # route53: # region: "[ROUTE53 REGION]" # secretAccessKeySecretRef: # name: "[NAME OF SECRET]" # key: "[KEY OF SECRET]" # role: "[ASSUME A ROLE]"``` ---- 2020-09-22 17:05:26 UTC - Ebere Abanonu: I work with microsoft azure so my knowledge is limited on how to setup acme resolvers for clouddns and route53 ---- 2020-09-22 17:06:46 UTC - Ebere Abanonu: use ```lets_encrypt: ``` for your proxy certs ---- 2020-09-22 17:08:35 UTC - Seun: I implemented letsencrypt when I deployed spinnaker via helm chart. Worked well. I just didn't do it in the helm charts values.yaml itself ---- 2020-09-22 17:08:46 UTC - Seun: I am going to have to google this again ---- 2020-09-22 17:33:20 UTC - Lari Hotari: @Devin G. Bost I have filed <https://github.com/apache/pulsar/issues/6869> some time ago. That might be the same issue that you have. It should be fairly easy to fix? @Penghui Li @Sijie Guo ---- 2020-09-22 18:06:11 UTC - Walter: How to get number of topics under the specific namespaces using pulsar-cli ---- 2020-09-22 18:08:25 UTC - Addison Higham: hrm... that seems strange, so it looks like the broker is returning a proper HTTP 200 but you still are seeing a 504 from the proxy? Essentially what happens with this API call is that you are connecting to zookeeper and listing the children nodes, which any broker should be able to do. Some other API calls need to be redirect to the correct broker, for example, to get the stats for a topic, since that isn't persisted anywhere, you need to go ask the broker who currently owns that topic for the in-memory stats it has and it does so with HTTP redirects. With the proxy involved, that means the proxy needs to follow re-directs, and if you have some name resolution issues, that can cause problems. However, in this case, it should be very straight forward... we haven't seen other reports of this sort of issue, but perhaps it just has something to do with proxy... I think what might be helpful @Brent Evans would be to find a time where I can jump on a call with you and perhaps we can poke at it together? ---- 2020-09-22 18:14:16 UTC - Alexander Brown: From the documentation... *Get the list of topics for a namespace* pulsar-admin namespaces topics tenant/namespace +1 : Rahul Vashishth ---- 2020-09-22 18:19:16 UTC - Walter: It's priniting all topics but i want to get count of topics. ---- 2020-09-22 18:25:14 UTC - Alexander Brown: can you pipe the command output to count the # of lines returned? pulsar-admin namespaces topics tenant/namespace | wc -l 100 : Walter ---- 2020-09-22 18:44:17 UTC - Walter: Thank you ---- 2020-09-22 19:00:43 UTC - Raman Gupta: Definitely part of the plan @Devin G. Bost, thanks! ---- 2020-09-22 20:18:50 UTC - Jim M.: in the helm chart, is there a way to init topics? ---- 2020-09-22 20:21:07 UTC - Addison Higham: no, not currently ---- 2020-09-22 20:21:35 UTC - Jim M.: whats the best way to setup topics then? ---- 2020-09-22 20:21:39 UTC - Jim M.: partitions and etc ---- 2020-09-22 20:22:09 UTC - Addison Higham: one thing to keep in mind is that pulsar supports automatic topic creation when you produce or consume from a topic that doesn't yet exist. There is also support for having automatically created topics to default to partitioned topics with a configurable number of partitions ---- 2020-09-22 20:22:31 UTC - Jim M.: in the java api producer/consumer? ---- 2020-09-22 20:22:41 UTC - Addison Higham: In all APIs, that is handled on the broker ---- 2020-09-22 20:22:43 UTC - Jim M.: I didn't see that document, can you point me to that ---- 2020-09-22 20:24:53 UTC - Addison Higham: <https://pulsar.apache.org/docs/en/concepts-messaging/#no-need-to-explicitly-create-new-topics> <- this section mentions it It can be configured with `allowAutoTopicCreation` and related configuration params as documented here: <https://pulsar.apache.org/docs/en/reference-configuration/#broker> ---- 2020-09-22 20:25:31 UTC - Jim M.: ty ---- 2020-09-22 20:25:34 UTC - Addison Higham: Also, if you do want to managed tenants and namespace and topics in a declarative fashion, there is <https://github.com/streamnative/terraform-provider-pulsar> raised_hands : Alexander Brown +1 : Rahul Vashishth ---- 2020-09-23 01:14:35 UTC - Penghui Li: > It should be fairly easy to fix? The solution I currently think of is add a field in the message metadata(a bitset). the bitset indicate which batch index is deleted. When the consumer get this message, we can filter out the deleted batch index. ---- 2020-09-23 01:18:13 UTC - Seun: Does anyone knows how to prevent apache proxy from getting public IP address when pulsar is deployed via helm charts? I tried to comment out : `service:` `annotations: {}` `#type: LoadBalancer` to see if this will work. Proxy still took public IP. I am doing this because I need to configure proxy as a backend service to Global load balancer. Does anyone knows how to disable this? ---- 2020-09-23 01:20:33 UTC - chris: which environment (aws, gcp, etc...) are you running in? ---- 2020-09-23 02:23:07 UTC - Luke Stephenson: You might want to do something similar to <https://github.com/apache/pulsar-helm-chart/pull/12> . But that is a big might without knowing more details about your issue. ---- 2020-09-23 04:32:51 UTC - Ali Ahmed: <https://www.pluralsight.com/courses/deploying-apache-pulsar-google-kubernetes-engine> ---- 2020-09-23 05:48:45 UTC - Walter: what is the meaning of this error *ERROR org.apache.pulsar.broker.service.BacklogQuotaManager - Failed to read policies data, will apply the default backlog quota:* java.util.concurrent.TimeoutException: null ---- 2020-09-23 06:09:50 UTC - Seun: @chris I am running this on gcp ---- 2020-09-23 06:12:48 UTC - Seun: @Luke Stephenson I am only interested in proxy not getting a public IP. This way, I can pass it to nginx ingress object as one of the backend services. That way I have a full TLS provision for proxy with letsencrypt. ---- 2020-09-23 08:07:20 UTC - Rahul Vashishth: How we can auto-generate new token for PulsarClient and PulsarAdminBuilder. For example, spring provides OAuth2RestTemplate which can generate/cache the token for us. Does pulsar client library provides such implementation? ---- 2020-09-23 08:14:58 UTC - Seun: I am really really interested in this portion that talks about proxy certificate. I am holding back my developers because they could not connect to pulsar over TLS. I can't get proxy to work with letsencript certificate. I am really frustrated. ---- 2020-09-23 08:15:43 UTC - Seun: And oh the course was released today! Great! ----
