I turned off compression and still get duplicates, but only 1 from each topic. Should the initial fetch offset for a partition be committed offset +1 ?
Thanks, Damian On 15 September 2015 at 14:07, Damian Guy <[email protected]> wrote: > Hi, > > I've been trying out the new consumer and have noticed that i get > duplicate messages when i stop the consumer and then restart (different > processes, same consumer group). > > I consume all of the messages on the topic and commit the offsets for each > partition and stop the consumer. On the next run i expect to get 0 > messages, however i get a batch of records from each partition - in this > case works out 1020 messages. Run it again and i get the same batch of > records. > > My logging shows that i've received messages with offsets lower than were > previously committed. > > committed:{damian_test_one-2=137669} > committed:{damian_test_one-0=139161} > committed:{damian_test_one-1=137663} > > min offsets received: {damian_test_one-0=138824, damian_test_one-1=137321, > damian_test_one-2=137331} > > I've debugged the initial fetch requests for offsets and the offsets match > up with what has been committed. Is this expected behaviour? Something to > do with batching of compression of message sets? > > TIA, > Damian >
