How would you apply total ordering if multiple messages are being consumed in parallel? If message-1 and message-2 are being consumed in parallel, do you really mean you want to guarantee that message-1 is consumed before the consumption of message-2 begins?
On Tue, May 26, 2015 at 1:34 PM, Kumar Jayanti <kumar.jaya...@gmail.com> wrote: > Thanks for your response. > > The requirement is to have total ordering across partitions. I do have a > Key and all my consumers would be in the same consumer group. I am new to > kafka and so would like to reconfirm my understanding of your last > statement : > > "So assuming you have a key you can assign to organize them, you should be > able to use many topic partitions and consumers." > > Let me explore a bit more on your suggestion and come back if i have > further doubts. > > thanks. > > > > > > > > On Tue, May 26, 2015 at 10:42 PM, Ewen Cheslack-Postava <e...@confluent.io > > > wrote: > > > Messages are consumed in order from each topic partition. You can make > sure > > messages end up in the same topic partition by making them have the same > > key, or by explicitly mapping them to partitions yourself when producing > > them. > > > > To parallelize consumption, you use a consumer group. Each time you > create > > a consumer, you can specify the group it should be in and work is > > automatically balanced across all the members of the group. This is done > on > > a per topic-partition basis, so when the consumer group membership is > > stable, a single consumer will see all the messages from a given topic > > partition in order. > > > > So assuming you have a key you can assign to organize them, you should be > > able to use many topic partitions and consumers. > > > > On Tue, May 26, 2015 at 9:51 AM, Kumar Jayanti <kumar.jaya...@gmail.com> > > wrote: > > > > > Our system has a specific message type that requires total ordering. > > > Message-1 should be consumed (acted upon) before Message-2 is consumed. > > > However since we have a cluster of consumer nodes we would like to > > > round-robin the processing of messages if possible to different nodes > > while > > > maintaining the ordering. Is this possible ? > > > > > > From my reading it appears we have to pin it down to a Topic with > Single > > > Partition with Single Consumer Process but i may be wrong ?. > > > > > > > > > > > -- > > Thanks, > > Ewen > > >
