It is consumer client node that has received events and is processing those...
On Thu, Jul 15, 2021 at 8:49 PM Israel Ekpo <[email protected]> wrote: > Hi Pushkar, > > When you use the term “node/instance” are you referring to the Kafka > Brokers or the consuming clients that are retrieving events from the > broker? > > Please could you elaborate/clarify? > > On Thu, Jul 15, 2021 at 10:00 AM Pushkar Deole <[email protected]> > wrote: > > > Well... with key-level ordering, i am mainly concerned about event loss, > if > > any, in below mentioned scenario: > > > > 1. since event1 with key1 and event2 with key2 are both part of the same > > partition1 > > 2. key1 event has offset 30 while key2 has offset 40 > > 3. key2 is processed by background thread and offset committed which is > 40 > > 4. before key1 gets processed by background thread, the instance/node > goes > > down > > 5. partition1 gets rebalanced to node2 and start processing ahead of > offset > > 40, thus losing key1 > > > > > > > > On Thu, Jul 15, 2021 at 7:18 PM Israel Ekpo <[email protected]> > wrote: > > > > > Hi Pushkar, > > > > > > If you are selecting key-based ordering, you should not be concerned > > about > > > the other keys from the same partitions being committed first > > > > > > If that is a concern for your use cases then you should go with > partition > > > based ordering to ensure that the events are processed in the sequence > > they > > > are picked up from the topic partition. > > > > > > For commit mode, you have the asynchronous, synchronous and > transactional > > > modes. I think if you are concerned with the order of commits you > should > > > look into the last two modes. > > > > > > My recommendation would be to go with the partition based ordering with > > > synchronous commits to start with. > > > > > > > > > > > > On Thu, Jul 15, 2021 at 7:36 AM Pushkar Deole <[email protected]> > > > wrote: > > > > > > > Hi All, and Antony (author of below article) > > > > > > > > i came across this article which seemed interesting: Introducing > > > > Confluent’s Parallel Consumer Message Processing Client > > > > < > > > > > > > > > > https://www.confluent.io/blog/introducing-confluent-parallel-message-processing-client/ > > > > > > > > > > > > > I would like to use the key-level ordering strategy mentioned in the > > > > article to scale my consumers, however I would like to check how the > > > offset > > > > commits are handled in this strategy > > > > e.g. on partition 1, key1 has offsets 20 and 30 respectively and on > the > > > > same partition key2 has offset 40. With key-level ordering model, > key2 > > > will > > > > be processed by a different thread in background and might gets > > processed > > > > before events related to key1, in this case offset for key2 will be > > > > committed before key1 gets processed ? How is this handled? > > > > > > > > > >
