Well... with key-level ordering, i am mainly concerned about event loss, if any, in below mentioned scenario:
1. since event1 with key1 and event2 with key2 are both part of the same partition1 2. key1 event has offset 30 while key2 has offset 40 3. key2 is processed by background thread and offset committed which is 40 4. before key1 gets processed by background thread, the instance/node goes down 5. partition1 gets rebalanced to node2 and start processing ahead of offset 40, thus losing key1 On Thu, Jul 15, 2021 at 7:18 PM Israel Ekpo <[email protected]> wrote: > Hi Pushkar, > > If you are selecting key-based ordering, you should not be concerned about > the other keys from the same partitions being committed first > > If that is a concern for your use cases then you should go with partition > based ordering to ensure that the events are processed in the sequence they > are picked up from the topic partition. > > For commit mode, you have the asynchronous, synchronous and transactional > modes. I think if you are concerned with the order of commits you should > look into the last two modes. > > My recommendation would be to go with the partition based ordering with > synchronous commits to start with. > > > > On Thu, Jul 15, 2021 at 7:36 AM Pushkar Deole <[email protected]> > wrote: > > > Hi All, and Antony (author of below article) > > > > i came across this article which seemed interesting: Introducing > > Confluent’s Parallel Consumer Message Processing Client > > < > > > https://www.confluent.io/blog/introducing-confluent-parallel-message-processing-client/ > > > > > > > I would like to use the key-level ordering strategy mentioned in the > > article to scale my consumers, however I would like to check how the > offset > > commits are handled in this strategy > > e.g. on partition 1, key1 has offsets 20 and 30 respectively and on the > > same partition key2 has offset 40. With key-level ordering model, key2 > will > > be processed by a different thread in background and might gets processed > > before events related to key1, in this case offset for key2 will be > > committed before key1 gets processed ? How is this handled? > > >
