Hello! For me it seems that rebalance is the cause that some of the messages are consumed by either one consumer or another. If you are using random client id, then it could happen that during the rebalance a client id will get a "lower" position than the consumer that was previously consumed the messages, and thus the new consumer will be the first that consumes, and the other one will be in standby.
Kafka rebalance algorithm orders alphabetically the client id per each groupid and allocates the partitions based on this order. I hope that it helps. Regards, Florin On Wed, Jul 13, 2016 at 8:59 AM, Gerard Klijs <gerard.kl...@dizzit.com> wrote: > Are you sure the topic itself has indeed 1 partition? > If so the only partition should be matched to either one till some > error/rebalance occurs, does this indeed happen (a lot)? > > On Wed, Jul 13, 2016 at 7:19 AM BYEONG-GI KIM <bg...@bluedigm.com> wrote: > > > Hello. > > > > I'm not sure whether it's a bug or not, but here is a something wrong; > > > > I set 2 consumer apps that have same consumer group id, and partition has > > been set to 1 on my Kafka Broker. > > > > In theory, the messages on the Kafka Broker must be consumed either one, > > which means it must not be consumed at both of them. But the messages > were > > sometimes consumed to both of them. > > > > I found the definition of the partition at the Kafka Website: > > *"Kafka only provides a total order over messages within a partition, not > > between different partitions in a topic. Per-partition ordering combined > > with the ability to partition data by key is sufficient for most > > applications. However, if you require a total order over messages this > can > > be achieved with a topic that has only one partition, though this will > mean > > only one consumer process per consumer group."* > > > > What's wrong with my setting? > > > > Regards > > > > KIM > > >
