I’ll look into assign - I think the issue I am having is often I will want to do these offset resets after the my consumer group has crashed. I guess what may be happening is that kafka still thinks there are active consumers even though they have crashed.
________________________________ From: Eric Azama <eazama...@gmail.com> Sent: Thursday, January 3, 2019 4:31 PM To: users@kafka.apache.org Subject: Re: Programmatic method of setting consumer groups offsets Adding on to Ryanne's point, if subscribe() isn't giving your consumer all of the partitions for a topic, that implies there are still active consumers running for that group. The consumer groups CLI command does not allow you to modify offsets for consumer groups that have active consumers. I believe this is because running consumers will not reset to the offsets that the command commits until they are restarted for some reason. Committing is presumably part of the consumer's processing loop, so they'll just overwrite whatever you committed with their current position. On Thu, Jan 3, 2019 at 4:25 PM Ryanne Dolan <ryannedo...@gmail.com> wrote: > Nimi, try using the low-level consumer API via assign() instead of > subscribe(), and then commit() each partition. I suspect you are trying to > commit partitions that the consumer is not subscribed to. > > Ryanne > > On Thu, Jan 3, 2019, 6:13 PM Nimi Wariboko < > nimi.warib...@livereachmedia.com > wrote: > > > Hi, > > > > Is there a guide or API that is implemented that can modify stored > offsets > > in Kafka? For example, I'm looking to set a consumer groups stored > offsets > > in Kafka to a specific value. I know there is the `kafka-consumer-groups` > > CLI command, but I'm looking for a way to do so from an application > > (without calling out to the shell). > > > > I know I could potentially create a consumer, while joining a consumer > and > > committing offsets this way, however I've been trying this and it seems > > error prone - very often I end up with issues where this method is only > > able to commit a subset of offsets. > > >
