Thanks for your response. The requirement is to have total ordering across partitions. I do have a Key and all my consumers would be in the same consumer group. I am new to kafka and so would like to reconfirm my understanding of your last statement :
"So assuming you have a key you can assign to organize them, you should be able to use many topic partitions and consumers." Let me explore a bit more on your suggestion and come back if i have further doubts. thanks. On Tue, May 26, 2015 at 10:42 PM, Ewen Cheslack-Postava <[email protected]> wrote: > Messages are consumed in order from each topic partition. You can make sure > messages end up in the same topic partition by making them have the same > key, or by explicitly mapping them to partitions yourself when producing > them. > > To parallelize consumption, you use a consumer group. Each time you create > a consumer, you can specify the group it should be in and work is > automatically balanced across all the members of the group. This is done on > a per topic-partition basis, so when the consumer group membership is > stable, a single consumer will see all the messages from a given topic > partition in order. > > So assuming you have a key you can assign to organize them, you should be > able to use many topic partitions and consumers. > > On Tue, May 26, 2015 at 9:51 AM, Kumar Jayanti <[email protected]> > wrote: > > > Our system has a specific message type that requires total ordering. > > Message-1 should be consumed (acted upon) before Message-2 is consumed. > > However since we have a cluster of consumer nodes we would like to > > round-robin the processing of messages if possible to different nodes > while > > maintaining the ordering. Is this possible ? > > > > From my reading it appears we have to pin it down to a Topic with Single > > Partition with Single Consumer Process but i may be wrong ?. > > > > > > -- > Thanks, > Ewen >
