Yeah, that's a good point.
Some of the operations might make sense on multiple partitions at once. Moving
to a timestamp might apply to all partitions, moving backwards and forwards by
N offsets might apply to all partitions.
However, moving to a specific offset ("set to offset 43") would most likely
only make sense to one partition at time. It might make sense to *require*
topic and partition, when moving to a specific offset.
-James
> On Feb 8, 2017, at 3:36 PM, Gwen Shapira <g...@confluent.io> wrote:
>
> Just to clarify, we'll need to allow specifying topic and partition. I
> don't think we want this on ALL partitions at once.
>
> On Wed, Feb 8, 2017 at 3:35 PM, Gwen Shapira <g...@confluent.io> wrote:
>> That's what I'd like to see. For example, suppose a Connect task fails
>> because it can't deserialize an event from a partition. Stop
>> connector, move offset forward, start connector. Boom!
>>
>>
>> On Wed, Feb 8, 2017 at 3:22 PM, Matthias J. Sax <matth...@confluent.io>
>> wrote:
>>> I am not sure about --reset-plus and --reset-minus
>>>
>>> Would this skip n messages forward/backward for each partitions?
>>>
>>>
>>> -Matthias
>>>
>>> On 2/8/17 2:23 PM, Jorge Esteban Quilcate Otoya wrote:
>>>> Great. I think I got the idea. What about this options:
>>>>
>>>> Scenarios:
>>>>
>>>> 1. Current status
>>>>
>>>> ´kafka-consumer-groups.sh --reset-offset --group cg1´
>>>>
>>>> 2. To Datetime
>>>>
>>>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-datetime
>>>> 2017-01-01T00:00:00.000´
>>>>
>>>> 3. To Period
>>>>
>>>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-period P2D´
>>>>
>>>> 4. To Earliest
>>>>
>>>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-earliest´
>>>>
>>>> 5. To Latest
>>>>
>>>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to-latest´
>>>>
>>>> 6. Minus 'n' offsets
>>>>
>>>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-minus n´
>>>>
>>>> 7. Plus 'n' offsets
>>>>
>>>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-plus n´
>>>>
>>>> 8. To specific offset
>>>>
>>>> ´kafka-consumer-groups.sh --reset-offset --group cg1 --reset-to x´
>>>>
>>>> Scopes:
>>>>
>>>> a. All topics used by Consumer Group
>>>>
>>>> Don't specify --topics
>>>>
>>>> b. Specific List of Topics
>>>>
>>>> Add list of values in --topics t1,t2,tn
>>>>
>>>> c. One Topic, all Partitions
>>>>
>>>> Add one topic and no partitions values: --topic t1
>>>>
>>>> d. One Topic, List of Partitions
>>>>
>>>> Add one topic and partitions values: --topic t1 --partitions 0,1,2
>>>>
>>>> About Reset Plan (JSON file):
>>>>
>>>> I think is still valid to have the option to persist reset configuration as
>>>> a file, but I agree to give the option to run the tool without going down
>>>> to the JSON file.
>>>>
>>>> Execution options:
>>>>
>>>> 1. Without execution argument (No args):
>>>>
>>>> Print out results (reset plan)
>>>>
>>>> 2. With --execute argument:
>>>>
>>>> Run reset process
>>>>
>>>> 3. With --output argument:
>>>>
>>>> Save result in a JSON format.
>>>>
>>>> 4. Only with --execute option and --reset-file (path to JSON)
>>>>
>>>> Reset based on file
>>>>
>>>> 4. Only with --verify option and --reset-file (path to JSON)
>>>>
>>>> Verify file values with current offsets
>>>>
>>>> I think we can remove --generate-and-execute because is a bit clumsy.
>>>>
>>>> With this options we will be able to execute with manual JSON
>>>> configuration.
>>>>
>>>>
>>>> El mié., 8 feb. 2017 a las 22:43, Ben Stopford (<b...@confluent.io>)
>>>> escribió:
>>>>
>>>>> Yes - using a tool like this to skip a set of consumer groups over a
>>>>> corrupt/bad message is definitely appealing.
>>>>>
>>>>> B
>>>>>
>>>>> On Wed, Feb 8, 2017 at 9:37 PM Gwen Shapira <g...@confluent.io> wrote:
>>>>>
>>>>>> I like the --reset-to-earliest and --reset-to-latest. In general,
>>>>>> since the JSON route is the most challenging for users, we want to
>>>>>> provide a lot of ways to do useful things without going there.
>>>>>>
>>>>>> Two things that can help:
>>>>>>
>>>>>> 1. A lot of times, users want to skip few messages that cause issues
>>>>>> and continue. maybe just specifying the topic, partition and delta
>>>>>> will be better than having to find the offset and write a JSON and
>>>>>> validate the JSON etc.
>>>>>>
>>>>>> 2. Thinking if there are other common use-cases that we can make easy
>>>>>> rather than just one generic but not very usable method.
>>>>>>
>>>>>> Gwen
>>>>>>
>>>>>> On Wed, Feb 8, 2017 at 3:25 AM, Jorge Esteban Quilcate Otoya
>>>>>> <quilcate.jo...@gmail.com> wrote:
>>>>>>> Thanks for the feedback!
>>>>>>>
>>>>>>> @Onur, @Gwen:
>>>>>>>
>>>>>>> Agree. Actually at the first draft I considered to have it inside
>>>>>>> ´kafka-consumer-groups.sh´, but I decide to propose it as a standalone
>>>>>> tool
>>>>>>> to describe it clearly and focus it on reset functionality.
>>>>>>>
>>>>>>> But now that you mentioned, it does make sense to have it in
>>>>>>> ´kafka-consumer-groups.sh´. How would be a consistent way to introduce
>>>>>> it?
>>>>>>>
>>>>>>> Maybe something like this:
>>>>>>>
>>>>>>> ´kafka-consumer-groups.sh --reset-offset --generate --group cg1
>>>>> --topics
>>>>>> t1
>>>>>>> --reset-from 2017-01-01T00:00:00.000 --output plan.json´
>>>>>>>
>>>>>>> ´kafka-consumer-groups.sh --reset-offset --verify --reset-json-file
>>>>>>> plan.json´
>>>>>>>
>>>>>>> ´kafka-consumer-groups.sh --reset-offset --execute --reset-json-file
>>>>>>> plan.json´
>>>>>>>
>>>>>>> ´kafka-consumer-groups.sh --reset-offset --generate-and-execute --group
>>>>>> cg1
>>>>>>> --topics t1 --reset-from 2017-01-01T00:00:00.000´
>>>>>>>
>>>>>>> @Gwen:
>>>>>>>
>>>>>>>> It looks exactly like the replica assignment tool
>>>>>>>
>>>>>>> It was influenced by ;-) I use the generate-verify-execute process here
>>>>>> to
>>>>>>> make sure user will be aware of the result of this operation. At the
>>>>>>> beginning we considered only add a couple of options to Consumer Group
>>>>>>> Command:
>>>>>>>
>>>>>>> --rewind-to-timestamp and --rewind-to-period
>>>>>>>
>>>>>>> @Onur:
>>>>>>>
>>>>>>>> You can actually get away with overriding while members of the group
>>>>>> are live
>>>>>>> with method 2 by using group information from DescribeGroupsRequest.
>>>>>>>
>>>>>>> This means that we need to have Consumer Group stopped before executing
>>>>>> and
>>>>>>> start a new consumer internally to do this? Therefore, we won't be able
>>>>>> to
>>>>>>> consider executing reset when ConsumerGroup is active? (trying to
>>>>> relate
>>>>>> it
>>>>>>> with @Dong 5th question)
>>>>>>>
>>>>>>> @Dong:
>>>>>>>
>>>>>>>> Should we allow user to use wildcard to reset offset of all groups
>>>>> for a
>>>>>>> given topic as well?
>>>>>>>
>>>>>>> I haven't thought about this scenario. Could be interesting. Following
>>>>>> the
>>>>>>> recommendation to add it into Consumer Group Command, in this case
>>>>> Group
>>>>>>> argument will be optional if there are only 1 topic. I think for
>>>>> multiple
>>>>>>> topic won't be that useful.
>>>>>>>
>>>>>>>> Should we allow user to specify timestamp per topic partition in the
>>>>>> json
>>>>>>> file as well?
>>>>>>>
>>>>>>> Don't think this could be a valid from the tool, but if Reset Plan is
>>>>>>> generated, and user want to set the offset for a specific partition to
>>>>>>> other offset (eventually based on another timestamp), and execute it,
>>>>> it
>>>>>>> will be up to her/him.
>>>>>>>
>>>>>>>> Should the script take some credential file to make sure that this
>>>>>>> operation is authenticated given the potential impact of this
>>>>> operation?
>>>>>>>
>>>>>>> Haven't tried to secure brokers yet, but the tool should support
>>>>>>> authorization if it's enabled in the broker.
>>>>>>>
>>>>>>>> Should we provide constant to reset committed offset to
>>>>> earliest/latest
>>>>>>> offset of a partition, e.g. -1 indicates earliest offset and -2
>>>>> indicates
>>>>>>> latest offset.
>>>>>>>
>>>>>>> I will go for something like ´--reset-to-earliest´ and
>>>>>> ´--reset-to-latest´
>>>>>>>
>>>>>>>> Should we allow dynamic change of the comitted offset when consumer
>>>>> are
>>>>>>> running, such that consumer will seek to the newly committed offset and
>>>>>>> start consuming from there?
>>>>>>>
>>>>>>> Not sure about this. I will recommend to keep it simple and ask user to
>>>>>>> stop consumers first. But I would considered it if the trade-offs are
>>>>>>> clear.
>>>>>>>
>>>>>>> @Matthias
>>>>>>>
>>>>>>> Added :). And thanks a lot for your help to define this KIP!
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> El mié., 8 feb. 2017 a las 7:47, Gwen Shapira (<g...@confluent.io>)
>>>>>>> escribió:
>>>>>>>
>>>>>>>> As long as the CLI is a bit consistent? Like, not just adding 3
>>>>>>>> arguments and a JSON parser to the existing tool, right?
>>>>>>>>
>>>>>>>> On Tue, Feb 7, 2017 at 10:29 PM, Onur Karaman
>>>>>>>> <onurkaraman.apa...@gmail.com> wrote:
>>>>>>>>> I think it makes sense to just add the feature to
>>>>>>>> kafka-consumer-groups.sh
>>>>>>>>>
>>>>>>>>> On Tue, Feb 7, 2017 at 10:24 PM, Gwen Shapira <g...@confluent.io>
>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Thanks for the KIP. I'm super happy about adding the capability.
>>>>>>>>>>
>>>>>>>>>> I hate the interface, though. It looks exactly like the replica
>>>>>>>>>> assignment tool. A tool everyone loves so much that there are
>>>>>> multiple
>>>>>>>>>> projects, open and closed, that try to fix it.
>>>>>>>>>>
>>>>>>>>>> Can we swap it with something that looks a bit more like the
>>>>> consumer
>>>>>>>>>> group tool? or the kafka streams reset tool? Consistency is helpful
>>>>>> in
>>>>>>>>>> such cases. I spent some time learning existing tools and learning
>>>>>> yet
>>>>>>>>>> another one is a deterrent.
>>>>>>>>>>
>>>>>>>>>> Gwen
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Tue, Feb 7, 2017 at 6:43 PM, Jorge Esteban Quilcate Otoya
>>>>>>>>>> <quilcate.jo...@gmail.com> wrote:
>>>>>>>>>>> Hi all,
>>>>>>>>>>>
>>>>>>>>>>> I would like to propose a KIP to Add a tool to Reset Consumer
>>>>> Group
>>>>>>>>>> Offsets.
>>>>>>>>>>>
>>>>>>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-
>>>>>>>>>> 122%3A+Add+a+tool+to+Reset+Consumer+Group+Offsets
>>>>>>>>>>>
>>>>>>>>>>> Please, take a look at the proposal and share your feedback.
>>>>>>>>>>>
>>>>>>>>>>> Thanks,
>>>>>>>>>>> Jorge.
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> Gwen Shapira
>>>>>>>>>> Product Manager | Confluent
>>>>>>>>>> 650.450.2760 <(650)%20450-2760> <(650)%20450-2760>
>>>>> <(650)%20450-2760> | @gwenshap
>>>>>>>>>> Follow us: Twitter | blog
>>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> Gwen Shapira
>>>>>>>> Product Manager | Confluent
>>>>>>>> 650.450.2760 <(650)%20450-2760> <(650)%20450-2760>
>>>>> <(650)%20450-2760> | @gwenshap
>>>>>>>> Follow us: Twitter | blog
>>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Gwen Shapira
>>>>>> Product Manager | Confluent
>>>>>> 650.450.2760 <(650)%20450-2760> <(650)%20450-2760> | @gwenshap
>>>>>> Follow us: Twitter | blog
>>>>>>
>>>>>
>>>>
>>>
>>
>>
>>
>> --
>> Gwen Shapira
>> Product Manager | Confluent
>> 650.450.2760 | @gwenshap
>> Follow us: Twitter | blog
>
>
>
> --
> Gwen Shapira
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter | blog
