Re: Message order different each time stream is replayed?
Ah, That explains it. I have 3 partitions. On 1 Dec 2016 5:42 a.m., "Apurva Mehta" wrote: How many partitions do you have in that topic. Kafka only guarantees a total ordering of messages within a partition, not across partitions of a topic. If you want total ordering over the entire topic, you need to create a topic with a single partition. On Wed, Nov 30, 2016 at 4:10 AM, Ali Akhtar wrote: > While I was connected to console-consumer.sh, I posted a few messages to a > Kafka topic, one message at a time, across a few hours. > > I'd post a message, see it arrive in console-consumer, a few mins later I'd > post the next message, and so on. > > They all arrived in order. > > However, when I now try to view console-consumer.sh --from-beginning, each > time I do so, the messages seem to be shown in a shuffled order. > > Also the last 2 messages appear to be missing, although its possible I > didn't see them in the shuffled output (the topic has over 10k messages > total which are all dumped to console when I view). > > Is this just due to network lag when viewing console-consumer, or should I > expect my actual consumers to also receive messages out of order each time > they replay a topic? >
Re: Message order different each time stream is replayed?
How many partitions do you have in that topic. Kafka only guarantees a total ordering of messages within a partition, not across partitions of a topic. If you want total ordering over the entire topic, you need to create a topic with a single partition. On Wed, Nov 30, 2016 at 4:10 AM, Ali Akhtar wrote: > While I was connected to console-consumer.sh, I posted a few messages to a > Kafka topic, one message at a time, across a few hours. > > I'd post a message, see it arrive in console-consumer, a few mins later I'd > post the next message, and so on. > > They all arrived in order. > > However, when I now try to view console-consumer.sh --from-beginning, each > time I do so, the messages seem to be shown in a shuffled order. > > Also the last 2 messages appear to be missing, although its possible I > didn't see them in the shuffled output (the topic has over 10k messages > total which are all dumped to console when I view). > > Is this just due to network lag when viewing console-consumer, or should I > expect my actual consumers to also receive messages out of order each time > they replay a topic? >
Message order different each time stream is replayed?
While I was connected to console-consumer.sh, I posted a few messages to a Kafka topic, one message at a time, across a few hours. I'd post a message, see it arrive in console-consumer, a few mins later I'd post the next message, and so on. They all arrived in order. However, when I now try to view console-consumer.sh --from-beginning, each time I do so, the messages seem to be shown in a shuffled order. Also the last 2 messages appear to be missing, although its possible I didn't see them in the shuffled output (the topic has over 10k messages total which are all dumped to console when I view). Is this just due to network lag when viewing console-consumer, or should I expect my actual consumers to also receive messages out of order each time they replay a topic?