2018-11-26 09:28:27 UTC - jia zhai: @Olivier Chicha MultiTopicsConsumerImpl, should be able to achieve your objective. ---- 2018-11-26 09:57:27 UTC - Yuvaraj Loganathan: Thanks! @Sijie Guo ---- 2018-11-26 10:41:28 UTC - yichen: @yichen has joined the channel ---- 2018-11-26 14:37:37 UTC - Christophe Bornet: OK. I'll open an issue ---- 2018-11-26 15:18:10 UTC - Christophe Bornet: BTW the code seems to be in line with the behavior I see : <https://github.com/apache/pulsar/blob/0ab2325fa33231f1c69782e081483012467dfcab/pulsar-client/src/main/java/org/apache/pulsar/client/impl/UnAckedMessageTracker.java#L87> ---- 2018-11-26 17:08:44 UTC - Ganga Lakshmanasamy: How to clear old messages that are in pulsar topic? There were some error records pushed to topic as part of testing and we are not sure how to get that cleared. Also, I get this exception org.apache.pulsar.client.api.SchemaSerializationException: java.io.EOFException when I try to get message.getValue(). Is there any other way the message data can be pulled other than message.getdata(). I am using schema to pass objects. @Ravi ---- 2018-11-26 17:10:58 UTC - Christophe Bornet: <https://github.com/apache/pulsar/issues/3059> ---- 2018-11-26 18:10:23 UTC - bossbaby: thanks you bro, i understand it, can you send me blog? ---- 2018-11-26 18:10:38 UTC - Matteo Merli: @Ganga Lakshmanasamy you can skip these messages in few different ways: * `bin/pulsar-admin topics skip $MY_TOPIC --subscription $SUBSCRIPTION --count 1` * `bin/pulsar-admin topics clear-backlog $MY_TOPIC --subscription $SUBSCRIPTION` * .. ---- 2018-11-26 18:18:09 UTC - Jon Bock: Here’s a blog by Sijie that discusses failover: <https://streaml.io/blog/pulsar-segment-based-architecture> ---- 2018-11-26 18:21:16 UTC - Sijie Guo: thank you @Jon Bock
@bossbaby sorry that I forgot to attach the link ---- 2018-11-26 22:32:18 UTC - durga: Hi Guys - Currently we have a topic that spans multiple regions. Is there any API in Pulsar to find when a message is picked up by all regions? More details below… If I understand correctly, pulsar keeps track of acks from the subscribers for a published message and deletes the message from the the stream storage once after is acknowledged by all subscribers. That would be a good point for us to know that. Any api for it? Another alternative if I understand correctly is that pulsar maintains a cursor_ledger for each subscriber to track the messages that subscriber has acknowledged. Is there any API available to access the cursor ledger? I am assuming our workflow in this case would be to find all subscribers for a given topic and then poll these subscribers cursor ledger to find if a message is acknowledged by all subscribers. Not a great solution though. Any other suggestions are welcome. ---- 2018-11-26 22:35:19 UTC - Matteo Merli: @durga The “replication backlog” for each region is available in the topic stats, in the “replication” section. There, you can see the stats from the point of view of the current cluster (eg: outgoing replication backlog) ---- 2018-11-26 22:36:00 UTC - Matteo Merli: if that backlog is 0, and the local backlog in the other clusters is 0, that would signify all the messages were replicated and processed in all clusters ---- 2018-11-26 22:43:09 UTC - durga: @Matteo Merli - Thanks. The requirement is bit more tricky in that we have lot of messages streaming between regions on the topic in async manner but we need to know the global replication status for a subset of messages. This is both for latency calculations as well as to do some additional processing on the producer side after a message is confirmed to be successfully replicated to all regions. We do not need sync replication here. We need an async replication but with confirmation that some of the messages were replicated to all regions. I guess one alternative is to open another multi-region topic where each subscriber to publish a message after they have acknowledged with idea being the original producer to be the subscriber. But that is heavy weight and if I can avoid by tapping into what is done already by pulsar then it is great. ---- 2018-11-26 23:11:53 UTC - Matteo Merli: Got it. While it’s difficult to provide an ultra-general solution, it’s indeed simpler if the scope is reduced. It’s kind of made difficult by the fact that the different clusters are isolated, on purpose, to prevent outages to spread across clusters. I think the solution of publishing on a different topic is not necessarily heavy weight: you can just publish a watermark every x amount of time. ---- 2018-11-27 00:31:08 UTC - durga: @Matteo Merli - Ok. Thanks. ---- 2018-11-27 01:02:43 UTC - Emma Pollum: What features will be release in 2.3? ---- 2018-11-27 01:17:28 UTC - bossbaby: thanks all ---- 2018-11-27 04:34:17 UTC - anhldbk: Hi guys, would you please let me know if Pulsar clients (Java) are thread-safe? ---- 2018-11-27 04:48:41 UTC - Sijie Guo: @anhldbk yes it is thread safe ---- 2018-11-27 04:49:43 UTC - anhldbk: Thank you @Sijie Guo. It’s nice to have such feature ---- 2018-11-27 04:50:30 UTC - Sijie Guo: @Emma Pollum : there are a couple of things: debezium, hdfs connector; bunch of flink integration; python state; and functions change like metrics, secrets; and schema improvements and potentially schema support on non-java clients ---- 2018-11-27 05:49:16 UTC - Yifan: @Yifan has joined the channel ---- 2018-11-27 06:43:12 UTC - Yifan: Hi, is anyone using Pulsar on Azure AKS? How do you manage storage for bookie? ---- 2018-11-27 06:51:19 UTC - Sijie Guo: @Yifan: I am not aware of any production deployment on Azure AKS yet. but since it is kubernetes, it should be same as how it is deployed to aws eks or google gks. Pulsar has a helm chart for doing so (you can check it out here: <https://github.com/apache/pulsar/tree/master/deployment/kubernetes/helm> ). for managing storage for bookies, there are a couple of options: - use daemon set, since bookkeeper is a replicated storage, it handles data replication. - use stateful set with persistent volume - use stateful set with local volume the most common deployment on cloud, is using statefulset with persistent volume ---- 2018-11-27 06:54:03 UTC - Yifan: Thanks @Sijie Guo I guess I will do it with persistent volume then. +1 : Sijie Guo, jia zhai ---- 2018-11-27 06:59:08 UTC - Yifan: Another question about Pulsar, When I use Kafka, I have different consumer group will read from the same topic, and they can do things independently, and I will join the messages later. Is it true that I cannot have multiple different consumer to read from same topic in Pulsar? ---- 2018-11-27 07:00:53 UTC - Yifan: The real life scenario would be I may have a testing worker trying to read the same input as production for example without interfering the production worker. ---- 2018-11-27 07:00:56 UTC - Matteo Merli: @Yifan you can use different subscriptions (identified by different subscription names) to achieve that ---- 2018-11-27 07:03:25 UTC - Yifan: I see, thanks. that’s similar to the consumer group concept I guess ---- 2018-11-27 07:09:21 UTC - Sijie Guo: @Yifan - you can think about pulsar subscription ~= kafka consumer group ---- 2018-11-27 07:12:40 UTC - Matteo Merli: I’d say more that subscription is a superset of Kafka consumer group :wink: +1 : jia zhai, Sijie Guo ---- 2018-11-27 07:22:55 UTC - Yifan: That’s even better, @Matteo Merli :slightly_smiling_face: it clears my doubt of migrating ---- 2018-11-27 07:36:36 UTC - Sijie Guo: @Matteo Merli lol ---- 2018-11-27 08:00:22 UTC - bossbaby: Hi all, i want run pulsar-admin in my pc, i read document and do it but pulsar-admin not start. Can you help me solve it? ---- 2018-11-27 08:02:25 UTC - Sijie Guo: what is the error you encountered? ---- 2018-11-27 08:04:49 UTC - bossbaby: i want testing in 1 cluster with 3 broker so i run pulsar-admin clusters list, but not cluster although i run pulsar standalone ---- 2018-11-27 08:08:53 UTC - Sijie Guo: @bossbaby - did you change conf/client.conf to point to the right cluster? if you don’t want to change conf/client.conf, you can specify --admin-url to point to the right pulsar service url ---- 2018-11-27 08:12:23 UTC - bossbaby: i wasn't change it, all default ---- 2018-11-27 08:16:25 UTC - Sijie Guo: then you need to change it to point to one of the broker, otherwise it might just talk to your standalone ---- 2018-11-27 08:19:47 UTC - bossbaby: thanks you, i try again ---- 2018-11-27 08:41:18 UTC - DucQuoc: @DucQuoc has joined the channel ---- 2018-11-27 08:56:07 UTC - bossbaby: hi @Sijie Guo i was run 1 broker in my pc, i wanna run 2 broker so i must start 2 zookeeper and map in to pulsar broker, it's right? ----
