2020-05-10 10:00:45 UTC - Liam Clarke: Hi all, I'm trying to understand how Pulsar subscriptions work, and while I wait for IDEA to import the Pulsar project, I wanted to ask about Exclusive and Key_Shared subscriptions, and ordering, when dealing with partitioned topics.
Firstly - if I have an exclusive consumer on a partitioned topic, then it will consume all partitions, but how will it consume? Does it round robin across the partitions? Or does it drain each partition before moving onto the next? I'm guessing that ordering isn't guaranteed in exclusive mode on a partitioned topic, because ordering is only guaranteed per partition, and how would the consumer know which partition this batch came from? Secondly - key shared. So it round robins, like Shared, but it sends the same key to the same consumer. Assuming that the producer was hashing keys to choose a partition, thus writing keyed messages to a given partition in order, am I right that key shared consumers would then implicitly get those keyed messages in order? ---- 2020-05-10 10:02:32 UTC - rani: @Sijie Guo, any ideas? ^ ---- 2020-05-10 10:03:20 UTC - Liam Clarke: Kia ora Marcio, Have you seen the examples in Pulsar's repo? For example: ---- 2020-05-10 10:05:56 UTC - Liam Clarke: <https://github.com/apache/pulsar/blob/master/examples/flink/src/main/java/org/apache/flink/streaming/connectors/pulsar/example/PulsarConsumerSourceWordCount.java> You'd need to have Lombok, Flink and Pulsar's Java client on the classpath for that one, but could easily import the pom.xml at the top to build the examples and go from there: <https://github.com/apache/pulsar/tree/master/examples/flink> ---- 2020-05-10 10:07:33 UTC - Liam Clarke: I'd copy that, remove the Lombok annotations and then all you need is the Flink and Pulsar dependencies. ---- 2020-05-10 10:54:11 UTC - JG: hey, nobody knows hot to directly subscribe on a topic in pulsar ? it seems it takes sometime when creating a new procuder. I would like to have the subscription directly/instantly after creating producer. ---- 2020-05-10 10:56:57 UTC - JG: this problem only occurs on function topics ---- 2020-05-10 13:37:45 UTC - Liam Clarke: Another question, if I'm wanting to add another bookie to BookKeeper, do I have to do anything to redistribute ledgers to the new node? Or will it happen automatically? I can't find anything definitive in the docs for BookKeeper on this. ---- 2020-05-10 13:40:35 UTC - Liam Clarke: Hi JG, I'm unsure what you mean. You're talking about subscriptions, a consumer concept, while talking about producers? If it helps, I can tell you the default behaviour is that messages produced onto a topic without any existing subscriptions are immediately discarded. But this can be overridden by setting a message retention at the namespace level. ---- 2020-05-10 13:41:17 UTC - Kashan Ali: @Kashan Ali has joined the channel ---- 2020-05-10 18:09:30 UTC - David Kjerrumgaard: After adding a bookie, ledgers will automatically start getting written to it after you have added it to the cluster. ---- 2020-05-10 20:25:09 UTC - Kirill Merkushev: the issue above is exactly about this - pulsar just don’t use function classloader properly somehow and ignores the class for the schema creation (when it finds it properly for the de/serialization itself) ---- 2020-05-10 21:10:58 UTC - Zuqing Li: @Zuqing Li has joined the channel ---- 2020-05-10 21:13:23 UTC - David Kjerrumgaard: and you specified the SerDe name when deploying the function, e.g. `--output-serde-classname` ? ---- 2020-05-11 00:48:19 UTC - Sijie Guo: Did you check the broker or proxy log to see if there are errors? One guess is - it seems to be related to authentication? Did you enable authentication? ---- 2020-05-11 02:00:03 UTC - Liam Clarke: Thanks @David Kjerrumgaard :) ---- 2020-05-11 07:08:36 UTC - Patrik Kleindl: @David Kjerrumgaard If I may add a follow-up question, if I understand correctly in Pulsar/Bookkeeper the distribution is on segment level, correct? This sounds good for new segments, but what if the (reading) traffic on existing segments is too much? Is there a way to redistribute those? ---- 2020-05-11 08:27:14 UTC - Kirill Merkushev: yep ---- 2020-05-11 08:29:58 UTC - Kirill Merkushev: you can take a look into code examples - they are runnable in one click. Will write a blog post on how to achieve that. Maybe you could add mentioning of testcontainers lib or admin api in your repo, as it seems nobody provide params with examples on how to really deploy the function ----
