2020-05-22 09:23:47 UTC - Matej Šaravanja: Hi, does anyone have experience with deploying pulsar to aws or similar service? I'm interested in recommended setup (which nodes, how many of them etc.) to try and calculate the cost vs. Kafka. Thanks! ---- 2020-05-22 10:13:32 UTC - Vladimir Shchur: Hi! Am I correct that deduplication doesn't work with partitioned topics? ---- 2020-05-22 10:15:07 UTC - shichao: @shichao has joined the channel ---- 2020-05-22 10:17:25 UTC - shichao: ```error while running "VolumeBinding" filter plugin for pod "pulsar-clus-bookie-0": pod has unbound immediate PersistentVolumeClaims``` Hi. I' m trying with k8s deployment and get erros like this with bookie pods. Has anyone seen this? ---- 2020-05-22 12:03:10 UTC - Deepa: @Sijie Guo one more question, When the keepAliveIntervalSeconds in broker is set to 30 seconds and client is set to 90 seconds, the connection is not getting terminated. Does that mean keep alive check happens from both broker and client separately as per its configuration (broker check at 30 seconds and client respond back and client check at 90 seconds and broker responds back) ? ---- 2020-05-22 12:33:43 UTC - Rakshit Menpara: Looks like your PersistentVolumeClaims are not being fulfilled. Can you describe your kubernetes environment? EKS/GKE/Other? What is your default StorageClass? ---- 2020-05-22 12:38:43 UTC - shichao: Thanks. It' s a local cluster with a storageclass installed with `helm install stable/nfs-client-provisioner --set nfs.server=x.x.x.x --set nfs.path=/exported/path` (<https://github.com/kubernetes-incubator/external-storage/tree/master/nfs-client>) . Previously I specified an nfs folder without write permissions, thus the pvcs get stuck. Now all items in pvc are green. ---- 2020-05-22 12:40:21 UTC - Rakshit Menpara: Okay, and Pulsar pods are still stuck? ---- 2020-05-22 12:40:56 UTC - Rakshit Menpara: What about zookeeper? All 3 nodes should be running fine before broker and bookies can start. ---- 2020-05-22 12:49:16 UTC - shichao: Yes. zookeeper in Services and Stateful Sets are green ---- 2020-05-22 12:52:18 UTC - shichao: Seems changed to Waiting: PodInitializing rather than the red warning ---- 2020-05-22 12:53:03 UTC - shichao: ---- 2020-05-22 12:53:33 UTC - shichao: ---- 2020-05-22 12:54:33 UTC - shichao: ---- 2020-05-22 13:22:01 UTC - Chris DiGiovanni: Hello,
We've recently been having issues with our Pulsar Proxies where after 10-12hrs of uptime, they start deteriorating and start producing 502 errors on requests to the admin api. When it first starts, it seems a 502 is occasional, though quickly the whole process becomes sick and is unable to handle requests. We tried increasing the log verbosity on these proxies to debug, but it hasn't produced any interesting logs to share. Curious if anyone has any ideas on how to troubleshoot the problem. We are collecting metrics via prometheus on the Proxies, if you know of any metrics I should be looking at to pinpoint the issue as well. eyes : Jake, Jared Mackey, Tanner Nilsson ---- 2020-05-22 15:08:25 UTC - Rakshit Menpara: The initcontainers of these statefulsets need zookeeper to be functional. Doesn't look like you have a quorum. All 3 instances should be alive. ---- 2020-05-22 15:08:54 UTC - Rakshit Menpara: You can verify the zookeeper logs to make sure the cluster is up. ---- 2020-05-22 15:22:20 UTC - rwaweber: Gotcha, thanks for the help! ---- 2020-05-22 15:46:29 UTC - Addison Higham: @Hiroyuki Yamada another thing to consider: for long term storage, you probably want to be considering storage offloading, which moves the data to s3/etc. Assuming S3 with really high durability as well as versioning/etc to prevent against accidental deletion, that really shrinks the window for data. In our case, we try to move to s3 after 8 hours +1 : Matteo Merli, Franck Schmidlin, Hiroyuki Yamada ---- 2020-05-22 17:01:53 UTC - David Kjerrumgaard: I believe that the `bookie-rack` only applies to the `bookkeeperClientRackawarePolicyEnabled` not the region-aware one you are using. ---- 2020-05-22 17:03:49 UTC - David Kjerrumgaard: We have deployed Pulsar on AWS's EKS platform and were able to get by with 3 EC2 nodes. The size of the nodes depends on the volume of traffic you are looking to serve and the volume of data you want to retain on the bookies. ---- 2020-05-22 21:59:52 UTC - Jared Mackey: Which version of pulsar are the proxies running? ---- 2020-05-22 22:12:39 UTC - Jared Mackey: I have written something like this. You can generate a unique topic on startup of the servers. Possibly using non durable subscriptions in a namespace that has low or no retention. ---- 2020-05-22 22:25:58 UTC - Jarrod Johnson: @Jarrod Johnson has joined the channel ---- 2020-05-22 22:51:39 UTC - Tanner Nilsson: When the problem happens, does the metrics endpoint still respond? We may be having a similar issue. ---- 2020-05-23 00:33:30 UTC - Hiroyuki Yamada: @Addison Higham Yes, I’m considering to use tiered storage. Good to know how you use it. Thank you ! ---- 2020-05-23 05:08:39 UTC - Hiroyuki Yamada: @Addison Higham Do you know if there is any way to put back offloaded segment to where it was ? I think that would be great in case we need to replay all the segments from the beginning for some reason. ---- 2020-05-23 06:49:08 UTC - Vladimir Shchur: @Sijie Guo are you aware of it? ----
