Re: Ordered Message Queue with Pool of Consumers
How would you apply total ordering if multiple messages are being consumed in parallel? If message-1 and message-2 are being consumed in parallel, do you really mean you want to guarantee that message-1 is consumed before the consumption of message-2 begins? On Tue, May 26, 2015 at 1:34 PM, Kumar Jayanti wrote: > Thanks for your response. > > The requirement is to have total ordering across partitions. I do have a > Key and all my consumers would be in the same consumer group. I am new to > kafka and so would like to reconfirm my understanding of your last > statement : > > "So assuming you have a key you can assign to organize them, you should be > able to use many topic partitions and consumers." > > Let me explore a bit more on your suggestion and come back if i have > further doubts. > > thanks. > > > > > > > > On Tue, May 26, 2015 at 10:42 PM, Ewen Cheslack-Postava > > wrote: > > > Messages are consumed in order from each topic partition. You can make > sure > > messages end up in the same topic partition by making them have the same > > key, or by explicitly mapping them to partitions yourself when producing > > them. > > > > To parallelize consumption, you use a consumer group. Each time you > create > > a consumer, you can specify the group it should be in and work is > > automatically balanced across all the members of the group. This is done > on > > a per topic-partition basis, so when the consumer group membership is > > stable, a single consumer will see all the messages from a given topic > > partition in order. > > > > So assuming you have a key you can assign to organize them, you should be > > able to use many topic partitions and consumers. > > > > On Tue, May 26, 2015 at 9:51 AM, Kumar Jayanti > > wrote: > > > > > Our system has a specific message type that requires total ordering. > > > Message-1 should be consumed (acted upon) before Message-2 is consumed. > > > However since we have a cluster of consumer nodes we would like to > > > round-robin the processing of messages if possible to different nodes > > while > > > maintaining the ordering. Is this possible ? > > > > > > From my reading it appears we have to pin it down to a Topic with > Single > > > Partition with Single Consumer Process but i may be wrong ?. > > > > > > > > > > > -- > > Thanks, > > Ewen > > >
Re: Ordered Message Queue with Pool of Consumers
Thanks for your response. The requirement is to have total ordering across partitions. I do have a Key and all my consumers would be in the same consumer group. I am new to kafka and so would like to reconfirm my understanding of your last statement : "So assuming you have a key you can assign to organize them, you should be able to use many topic partitions and consumers." Let me explore a bit more on your suggestion and come back if i have further doubts. thanks. On Tue, May 26, 2015 at 10:42 PM, Ewen Cheslack-Postava wrote: > Messages are consumed in order from each topic partition. You can make sure > messages end up in the same topic partition by making them have the same > key, or by explicitly mapping them to partitions yourself when producing > them. > > To parallelize consumption, you use a consumer group. Each time you create > a consumer, you can specify the group it should be in and work is > automatically balanced across all the members of the group. This is done on > a per topic-partition basis, so when the consumer group membership is > stable, a single consumer will see all the messages from a given topic > partition in order. > > So assuming you have a key you can assign to organize them, you should be > able to use many topic partitions and consumers. > > On Tue, May 26, 2015 at 9:51 AM, Kumar Jayanti > wrote: > > > Our system has a specific message type that requires total ordering. > > Message-1 should be consumed (acted upon) before Message-2 is consumed. > > However since we have a cluster of consumer nodes we would like to > > round-robin the processing of messages if possible to different nodes > while > > maintaining the ordering. Is this possible ? > > > > From my reading it appears we have to pin it down to a Topic with Single > > Partition with Single Consumer Process but i may be wrong ?. > > > > > > -- > Thanks, > Ewen >
Re: Ordered Message Queue with Pool of Consumers
Messages are consumed in order from each topic partition. You can make sure messages end up in the same topic partition by making them have the same key, or by explicitly mapping them to partitions yourself when producing them. To parallelize consumption, you use a consumer group. Each time you create a consumer, you can specify the group it should be in and work is automatically balanced across all the members of the group. This is done on a per topic-partition basis, so when the consumer group membership is stable, a single consumer will see all the messages from a given topic partition in order. So assuming you have a key you can assign to organize them, you should be able to use many topic partitions and consumers. On Tue, May 26, 2015 at 9:51 AM, Kumar Jayanti wrote: > Our system has a specific message type that requires total ordering. > Message-1 should be consumed (acted upon) before Message-2 is consumed. > However since we have a cluster of consumer nodes we would like to > round-robin the processing of messages if possible to different nodes while > maintaining the ordering. Is this possible ? > > From my reading it appears we have to pin it down to a Topic with Single > Partition with Single Consumer Process but i may be wrong ?. > -- Thanks, Ewen
Ordered Message Queue with Pool of Consumers
Our system has a specific message type that requires total ordering. Message-1 should be consumed (acted upon) before Message-2 is consumed. However since we have a cluster of consumer nodes we would like to round-robin the processing of messages if possible to different nodes while maintaining the ordering. Is this possible ? >From my reading it appears we have to pin it down to a Topic with Single Partition with Single Consumer Process but i may be wrong ?.
