Hi Sudev, My version is pretty much the same as https://github.com/goibibo/woof/tree/master/scripts/offset_commit_manual. So, little point posting code which does the same thing, also it python, although with a different method of parsing the params to it.
Sorry about the late reply, didn't remember to check this mailing list for a while. Regards -- Glen Ogilvie Open Systems Specialists Level 1, 162 Grafton Road http://www.oss.co.nz/ Ph: +64 9 984 3000 Mobile: +64 21 684 146 GPG Key: ACED9C17 From: "Sudev A C" <sudev...@goibibo.com> To: "users" <users@kafka.apache.org> Sent: Thursday, 9 March, 2017 7:25:21 PM Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X Hi Glen, Maybe have a look at this, slightly different version. Not doing anything more than what you have seen in gist. You should pass broker url as first parameter and and a json containing desired offsets as second parameter. https://github.com/goibibo/woof/tree/master/scripts/offset_commit_manual As suggested in previous mail it will be good to shut all consumers before reseting the offset. Thanks Sudev On Thu, 9 Mar 2017 at 11:41 AM, Manikumar <manikumar.re...@gmail.com> wrote: > A tool for reseting consumer group offsets is proposed here: > KIP-122: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-122%3A+Add+Reset+Consumer+Group+Offsets+tooling > > > KIP is currently in the voting stage. > > On Thu, Mar 9, 2017 at 7:35 AM, Jeff Widman <j...@netskope.com> wrote: > > > Did you have to do anything different beyond what was already in the > Gist? > > > > I'd be curious to see the code as I've considered putting together a > small > > repo of various python scripts I've found useful when working with Kafka > > > > On Wed, Mar 8, 2017 at 1:11 PM, Glen Ogilvie <glen.ogil...@oss.co.nz> > > wrote: > > > > > Thank you Jeff and Robert. > > > > > > I've had success in getting the offset position to seek to a position > in > > > the queue, when all other consumers in the group are off. > > > > > > Would anyone like me to tidy the tool up enough that it could be > included > > > in Kafka or the docs? > > > > > > Regards > > > -- > > > Glen Ogilvie > > > Open Systems Specialists > > > Level 1, 162 Grafton Road > > > http://www.oss.co.nz/ > > > > > > Ph: +64 9 984 3000 > > > Mobile: +64 21 684 146 > > > GPG Key: ACED9C17 > > > > > > > > > From: "Robert Quinlivan" <rquinli...@signal.co> > > > To: "users" <users@kafka.apache.org> > > > Sent: Thursday, 9 March, 2017 6:42:15 AM > > > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X > > > > > > The best approach would be: > > > - Have all consumers in your group shut down > > > - Have an offset reset tool join with the same group name as above > > > - Offset tool subscribes to all topic-partitions, seeks to the desired > > > offset, and commits. > > > - Offset tool shuts down > > > - Consumers then restart and re-join the consumer group, resuming at > the > > > offsets that were last committed for each topic-partition > > > > > > On Wed, Mar 8, 2017 at 10:51 AM, Jeff Widman <j...@netskope.com> > wrote: > > > > > > > Yeah, that gist looks like it *should* work. I haven't tested it so > > can't > > > > guarantee. > > > > > > > > On Tue, Mar 7, 2017 at 7:04 PM, Glen Ogilvie <glen.ogil...@oss.co.nz > > > > > > wrote: > > > > > > > > > Hi Jeff, > > > > > > > > > > Yes, the work I am doing is ops work. Logstash is consuming from > the > > > > topic > > > > > + consumer group, and I don't want it to start at the beginning, > but > > > > rather > > > > > at a specific offset, > > > > > so setting the offset for the consumer group externally, then > > starting > > > up > > > > > logstash is my goal. > > > > > > > > > > I'm still a little unclear as to how to do this. > > > > > > > > > > Is this python script: http://pastebin.com/tvxj1wTX > > > > > > > > > > The right way to go about getting the offset set to a specific > value > > > > > (12345678 in this example) for a specific consumer group? > > > > > > > > > > Regards > > > > > -- > > > > > Glen Ogilvie > > > > > Open Systems Specialists > > > > > Level 1, 162 Grafton Road > > > > > http://www.oss.co.nz/ > > > > > > > > > > Ph: +64 9 984 3000 > > > > > Mobile: +64 21 684 146 > > > > > GPG Key: ACED9C17 > > > > > > > > > > > > > > > From: "Jeff Widman" <j...@netskope.com> > > > > > To: "users" <users@kafka.apache.org> > > > > > Sent: Wednesday, 8 March, 2017 1:41:17 PM > > > > > Subject: Re: How to set offset for a consumer in Kafka 0.10.0.X > > > > > > > > > > Offsets for modern kafka consumers are stored in an internal Kafka > > > topic, > > > > > so they aren't as easy to change as zookeeper. > > > > > > > > > > To set a consumer offset, you need a consumer within a consumer > group > > > to > > > > > call commit() with your explicit offset. If needed, you can create > a > > > > dummy > > > > > consumer and tell it to join an existing consumer group. > > > > > > > > > > Take a look at this migration script, especially the part where it > > > > commits > > > > > the offset to see how it can work in Scala: > > > > > https://github.com/apache/kafka/pull/2615/files > > > > > > > > > > It's fairly straightforward to do this within most other clients as > > > well. > > > > > If you're doing some quick ops work where you don't want to spin up > > the > > > > > JVM, then it's fairly easy to do this using kafka-python. > > > > > > > > > > > > > > > > > > > > On Tue, Mar 7, 2017 at 4:08 PM, Glen Ogilvie < > glen.ogil...@oss.co.nz > > > > > > > > wrote: > > > > > > > > > > > Hi, > > > > > > > > > > > > We are running Kafka 0.10.0.X, with zookeeper. I'm trying to > figure > > > out > > > > > if > > > > > > I can manually > > > > > > set a consumer offset, for a specific consumer when that consumer > > is > > > > > > stopped. > > > > > > > > > > > > It looks like it used to be done using: > kafka.tools.ExportZkOffsets > > > and > > > > > > kafka.tools.ImportZkOffsets > > > > > > ( https://cwiki.apache.org/confluence/display/KAFKA/ > > > > > > System+Tools#SystemTools-ConsumerOffsetChecker ) > > > > > > > > > > > > However, if my version they don't work, because they try and read > > > from > > > > > > zookeeper /consumers which is empty.. I think they are old tools. > > > > > > > > > > > > Does anyone know where in zookeeper, where the current kafka > keeps > > > > > > consumer offsets? > > > > > > > > > > > > Regards > > > > > > -- > > > > > > Glen Ogilvie > > > > > > Open Systems Specialists > > > > > > Level 1, 162 Grafton Road > > > > > > http://www.oss.co.nz/ > > > > > > > > > > > > Ph: +64 9 984 3000 > > > > > > Mobile: +64 21 684 146 > > > > > > GPG Key: ACED9C17 > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > Robert Quinlivan > > > Software Engineer, Signal > > > > > >
