Hi Pushkar, Based on what I understand about the library, I don't think you need to worry about data loss because there are mechanisms in place to track which offsets have been processed in the event something goes wrong during processing.
If the processing client goes offline or is unresponsive and has to be resumed or restarted, it should restart from where it left off since the offsets are being tracked. https://www.confluent.io/events/kafka-summit-europe-2021/introducing-confluent-labs-parallel-consumer-client/ This should be true regardless of the ordering strategy you select. Take a look at the talk from the most recent Kafka Summit for additional details on this The github documentation for the project also covers this briefly https://github.com/confluentinc/parallel-consumer/blob/master/README.adoc#commit-mode I hope this was helpful. If you still have additional questions, please continue to bring it up. Thanks. On Thu, Jul 15, 2021 at 11:21 AM Pushkar Deole <pdeole2...@gmail.com> wrote: > It is consumer client node that has received events and is processing > those... > > On Thu, Jul 15, 2021 at 8:49 PM Israel Ekpo <israele...@gmail.com> wrote: > > > Hi Pushkar, > > > > When you use the term “node/instance” are you referring to the Kafka > > Brokers or the consuming clients that are retrieving events from the > > broker? > > > > Please could you elaborate/clarify? > > > > On Thu, Jul 15, 2021 at 10:00 AM Pushkar Deole <pdeole2...@gmail.com> > > wrote: > > > > > Well... with key-level ordering, i am mainly concerned about event > loss, > > if > > > any, in below mentioned scenario: > > > > > > 1. since event1 with key1 and event2 with key2 are both part of the > same > > > partition1 > > > 2. key1 event has offset 30 while key2 has offset 40 > > > 3. key2 is processed by background thread and offset committed which is > > 40 > > > 4. before key1 gets processed by background thread, the instance/node > > goes > > > down > > > 5. partition1 gets rebalanced to node2 and start processing ahead of > > offset > > > 40, thus losing key1 > > > > > > > > > > > > On Thu, Jul 15, 2021 at 7:18 PM Israel Ekpo <israele...@gmail.com> > > wrote: > > > > > > > Hi Pushkar, > > > > > > > > If you are selecting key-based ordering, you should not be concerned > > > about > > > > the other keys from the same partitions being committed first > > > > > > > > If that is a concern for your use cases then you should go with > > partition > > > > based ordering to ensure that the events are processed in the > sequence > > > they > > > > are picked up from the topic partition. > > > > > > > > For commit mode, you have the asynchronous, synchronous and > > transactional > > > > modes. I think if you are concerned with the order of commits you > > should > > > > look into the last two modes. > > > > > > > > My recommendation would be to go with the partition based ordering > with > > > > synchronous commits to start with. > > > > > > > > > > > > > > > > On Thu, Jul 15, 2021 at 7:36 AM Pushkar Deole <pdeole2...@gmail.com> > > > > wrote: > > > > > > > > > Hi All, and Antony (author of below article) > > > > > > > > > > i came across this article which seemed interesting: Introducing > > > > > Confluent’s Parallel Consumer Message Processing Client > > > > > < > > > > > > > > > > > > > > > https://www.confluent.io/blog/introducing-confluent-parallel-message-processing-client/ > > > > > > > > > > > > > > > > I would like to use the key-level ordering strategy mentioned in > the > > > > > article to scale my consumers, however I would like to check how > the > > > > offset > > > > > commits are handled in this strategy > > > > > e.g. on partition 1, key1 has offsets 20 and 30 respectively and on > > the > > > > > same partition key2 has offset 40. With key-level ordering model, > > key2 > > > > will > > > > > be processed by a different thread in background and might gets > > > processed > > > > > before events related to key1, in this case offset for key2 will be > > > > > committed before key1 gets processed ? How is this handled? > > > > > > > > > > > > > > >
