Hi Israel, Pushkar, everyone! Sorry I didn't notice this message. Feel free to collaborate on the github discussions page about the project too.
And yes Israel is correct - offsets are handled correctly regardless of what modes you choose. Offsets are tracked individually, and committed to kafka inside the commit metadata payload - there is no message loss. Here's a conversation on the topic in the discussion forum: https://github.com/confluentinc/parallel-consumer/discussions/135#discussioncomment-1050437 Let me know if I can help with anything else! Cheers, Antony. On Thu, Jul 15, 2021 at 11:50 PM Israel Ekpo <israele...@gmail.com> wrote: > Hi Pushkar, > > Based on what I understand about the library, I don't think you need to > worry about data loss because there are mechanisms in place to track which > offsets have been processed in the event something goes wrong during > processing. > > If the processing client goes offline or is unresponsive and has to be > resumed or restarted, it should restart from where it left off since the > offsets are being tracked. > > > https://www.confluent.io/events/kafka-summit-europe-2021/introducing-confluent-labs-parallel-consumer-client/ > > This should be true regardless of the ordering strategy you select. > > Take a look at the talk from the most recent Kafka Summit for additional > details on this > > The github documentation for the project also covers this briefly > > > https://github.com/confluentinc/parallel-consumer/blob/master/README.adoc#commit-mode > > I hope this was helpful. If you still have additional questions, please > continue to bring it up. > > Thanks. > > On Thu, Jul 15, 2021 at 11:21 AM Pushkar Deole <pdeole2...@gmail.com> > wrote: > > > It is consumer client node that has received events and is processing > > those... > > > > On Thu, Jul 15, 2021 at 8:49 PM Israel Ekpo <israele...@gmail.com> > wrote: > > > > > Hi Pushkar, > > > > > > When you use the term “node/instance” are you referring to the Kafka > > > Brokers or the consuming clients that are retrieving events from the > > > broker? > > > > > > Please could you elaborate/clarify? > > > > > > On Thu, Jul 15, 2021 at 10:00 AM Pushkar Deole <pdeole2...@gmail.com> > > > wrote: > > > > > > > Well... with key-level ordering, i am mainly concerned about event > > loss, > > > if > > > > any, in below mentioned scenario: > > > > > > > > 1. since event1 with key1 and event2 with key2 are both part of the > > same > > > > partition1 > > > > 2. key1 event has offset 30 while key2 has offset 40 > > > > 3. key2 is processed by background thread and offset committed which > is > > > 40 > > > > 4. before key1 gets processed by background thread, the instance/node > > > goes > > > > down > > > > 5. partition1 gets rebalanced to node2 and start processing ahead of > > > offset > > > > 40, thus losing key1 > > > > > > > > > > > > > > > > On Thu, Jul 15, 2021 at 7:18 PM Israel Ekpo <israele...@gmail.com> > > > wrote: > > > > > > > > > Hi Pushkar, > > > > > > > > > > If you are selecting key-based ordering, you should not be > concerned > > > > about > > > > > the other keys from the same partitions being committed first > > > > > > > > > > If that is a concern for your use cases then you should go with > > > partition > > > > > based ordering to ensure that the events are processed in the > > sequence > > > > they > > > > > are picked up from the topic partition. > > > > > > > > > > For commit mode, you have the asynchronous, synchronous and > > > transactional > > > > > modes. I think if you are concerned with the order of commits you > > > should > > > > > look into the last two modes. > > > > > > > > > > My recommendation would be to go with the partition based ordering > > with > > > > > synchronous commits to start with. > > > > > > > > > > > > > > > > > > > > On Thu, Jul 15, 2021 at 7:36 AM Pushkar Deole < > pdeole2...@gmail.com> > > > > > wrote: > > > > > > > > > > > Hi All, and Antony (author of below article) > > > > > > > > > > > > i came across this article which seemed interesting: Introducing > > > > > > Confluent’s Parallel Consumer Message Processing Client > > > > > > < > > > > > > > > > > > > > > > > > > > > > https://www.confluent.io/blog/introducing-confluent-parallel-message-processing-client/ > > > > > > > > > > > > > > > > > > > I would like to use the key-level ordering strategy mentioned in > > the > > > > > > article to scale my consumers, however I would like to check how > > the > > > > > offset > > > > > > commits are handled in this strategy > > > > > > e.g. on partition 1, key1 has offsets 20 and 30 respectively and > on > > > the > > > > > > same partition key2 has offset 40. With key-level ordering model, > > > key2 > > > > > will > > > > > > be processed by a different thread in background and might gets > > > > processed > > > > > > before events related to key1, in this case offset for key2 will > be > > > > > > committed before key1 gets processed ? How is this handled? > > > > > > > > > > > > > > > > > > > > > -- <https://www.confluent.io> Antony Stubbs Follow us: Blog <https://confluent.io/blog?utm_source=footer&utm_medium=email&utm_campaign=ch.email-signature_type.community_content.blog> • Slack <https://slackpass.io/confluentcommunity> • Twitter <https://twitter.com/ConfluentInc> • YouTube <https://youtube.com/confluent>
