You could set the auto.commit.interval.ms to a lower value, in your example it is 10 seconds, which can be a lot of messages. I don't really see how it could be prevented any further, since offset's can only committed by consumer to the partitions they are assigned to. I do believe there is some work in progress in which the assigned of partitions to consumers is somewhat sticky. In that case when a consumer has been assigned the same partitions after the rebalance as it has had before, and then it should not be necessary to consume the same data again in those partitions.
On Mon, Jul 11, 2016 at 3:18 PM Michael Luban <mluban....@gmail.com> wrote: > Using the 0.8.2.1 client. > > Is it possible to statistically minimize the possibility of duplication in > this scenario or has this behavior been corrected in a later client > version? Or is the test flawed? > > https://gist.github.com/mluban/03a5c0d9221182e6ddbc37189c4d3eb0 >
