Hi Pushkar, When you use the term “node/instance” are you referring to the Kafka Brokers or the consuming clients that are retrieving events from the broker?
Please could you elaborate/clarify? On Thu, Jul 15, 2021 at 10:00 AM Pushkar Deole <[email protected]> wrote: > Well... with key-level ordering, i am mainly concerned about event loss, if > any, in below mentioned scenario: > > 1. since event1 with key1 and event2 with key2 are both part of the same > partition1 > 2. key1 event has offset 30 while key2 has offset 40 > 3. key2 is processed by background thread and offset committed which is 40 > 4. before key1 gets processed by background thread, the instance/node goes > down > 5. partition1 gets rebalanced to node2 and start processing ahead of offset > 40, thus losing key1 > > > > On Thu, Jul 15, 2021 at 7:18 PM Israel Ekpo <[email protected]> wrote: > > > Hi Pushkar, > > > > If you are selecting key-based ordering, you should not be concerned > about > > the other keys from the same partitions being committed first > > > > If that is a concern for your use cases then you should go with partition > > based ordering to ensure that the events are processed in the sequence > they > > are picked up from the topic partition. > > > > For commit mode, you have the asynchronous, synchronous and transactional > > modes. I think if you are concerned with the order of commits you should > > look into the last two modes. > > > > My recommendation would be to go with the partition based ordering with > > synchronous commits to start with. > > > > > > > > On Thu, Jul 15, 2021 at 7:36 AM Pushkar Deole <[email protected]> > > wrote: > > > > > Hi All, and Antony (author of below article) > > > > > > i came across this article which seemed interesting: Introducing > > > Confluent’s Parallel Consumer Message Processing Client > > > < > > > > > > https://www.confluent.io/blog/introducing-confluent-parallel-message-processing-client/ > > > > > > > > > > I would like to use the key-level ordering strategy mentioned in the > > > article to scale my consumers, however I would like to check how the > > offset > > > commits are handled in this strategy > > > e.g. on partition 1, key1 has offsets 20 and 30 respectively and on the > > > same partition key2 has offset 40. With key-level ordering model, key2 > > will > > > be processed by a different thread in background and might gets > processed > > > before events related to key1, in this case offset for key2 will be > > > committed before key1 gets processed ? How is this handled? > > > > > >
