Hi Israel, Thanks for your detailed explanation. I understand now that Kafka can't give me any guarantees with regards to ordering if my single consumer is consuming from multiple partitions.
Hi Sunil, Thanks for the thread suggestion. However I don't think increasing or decreasing the number of threads will have any impact on the ordering of the messages, which is my problem. Thanks! -R On Thu, Jan 6, 2022 at 12:42 PM sunil chaudhari <sunilmchaudhar...@gmail.com> wrote: > hi Roger, > What consumer u r using? > Is there a chance to mention consumer threads? > > Example: logstash kafka consumer has configurable number of threads under > each consumer instance. That may help up to some extent. > > > Regards, > Sunil. > On Thu, 6 Jan 2022 at 7:27 PM, Roger Kasinsky <roger.kasin...@gmail.com> > wrote: > > > Hi Luke, > > > > > The solution I can think of is to create only one partition for the > > topic. > > > > That would work, but then I lose the benefits of the partitions. > > > > > Or you can create 4 consumers in one group, to consume from 4 > partitions. > > That works, too. > > > > That does not work, because I need only one consumer receiving all the > > messages in the same order on every run. > > > > Hi Suni, > > > > > Why dont you provide new name to consumer group each time you restart > > your consumer? This new consumer group will not conflict with the earlier > > one and it will be treated as new consumer thread next time to get all > > messages again. > > > > That does not solve my problem, which is to have the same consumer > getting > > all the topic messages in the same order. I'm not worried about > conflicts. > > In fact, I want the exact same consumer to run twice in a row. Renaming > the > > consumer group does not help with anything related to message order. > > > > Thanks! > > > > -R > > > > > > > > On Thu, Jan 6, 2022 at 12:21 AM sunil chaudhari < > > sunilmchaudhar...@gmail.com> > > wrote: > > > > > hi, > > > Why dont you provide new name to consumer group each time you restart > > your > > > consumer? > > > This new consumer group will not conflict with the earlier one and it > > will > > > be treated as new consumer thread next time to get all messages again. > > > > > > > > > Regards, > > > Sunil. > > > > > > On Wed, 5 Jan 2022 at 10:45 PM, Roger Kasinsky < > roger.kasin...@gmail.com > > > > > > wrote: > > > > > > > Hi, > > > > > > > > I have a topic divided into 4 partitions. I have a consumer that > needs > > to > > > > consume all messages from the topic (all messages from all 4 > > partitions). > > > > So to do that I have this consumer sitting by itself in its own > > consumer > > > > group. I'm not committing any offsets, because I want to read all > > > messages > > > > again on every restart of the consumer. > > > > > > > > *Question:* *Can anything be said about the order of the messages > > > consumed > > > > by my consumer? Is there a way to enforce the same order of messages > > for > > > > every restart of my consumer?* > > > > > > > > Thanks! > > > > > > > > -R > > > > > > > > > >
