I have updated the KIP: - Change execution parameters, using `--dry-run` - Reference KAFKA-4327 - And advise about changes on `StreamResetter`
Also includes that it will cover a change on `ConsumerGroupCommand` to align execution options. El dom., 16 jul. 2017 a las 5:37, Matthias J. Sax (<matth...@confluent.io>) escribió: > Thanks a lot for the update! > > I like the KIP! > > One more question about `--dry-run` vs `--execute`: While I agree that > we should use the same flag for both tools, I am not sure which one is > the better one... My personal take is, that I like `--dry-run` better. > Not sure what others think. > > One more comment: with the removal of ZK, we can also tackle this JIRA: > https://issues.apache.org/jira/browse/KAFKA-4327 If we do so, I think we > should mention it in the KIP. > > I am also not sure about backward compatibility issue for this case. > Actually, I don't expect people to call `StreamsResetter` from Java > code, but you can never know. So if we break this, we need to make sure > to cover it in the KIP and later on in the release notes. > > > -Matthias > > On 7/14/17 7:15 AM, Jorge Esteban Quilcate Otoya wrote: > > Hi, > > > > KIP is updated. > > Changes: > > 1. Approach discussed to keep both tools (streams application resetter > and > > consumer group reset offset). > > 2. Options has been aligned between both tools. > > 3. Zookeeper option from streams-application-resetted will be removed, > and > > replaced internally for Kafka AdminClient. > > > > Looking forward to your feedback. > > > > El jue., 29 jun. 2017 a las 15:04, Matthias J. Sax (< > matth...@confluent.io>) > > escribió: > > > >> Damian, > >> > >>> resets everything and clears up > >>>> the state stores. > >> > >> That's not correct. The reset tool does not touch local store. For this, > >> we have `KafkaStreams#clenup` -- otherwise, you would need to run the > >> tool in every machine you run an application instance. > >> > >> With regard to state, the tool only deletes the underlying changelog > >> topics. > >> > >> Just to clarify. The idea of allowing to rest with different offset is > >> to clear all state but just use a different start offset (instead of > >> zero). This is for use case where your topic has a larger retention time > >> than the amount of data you want to reprocess. But we always need to > >> start with an empty state. (Resetting with consistent state is something > >> we might do at some point, but it's much hard and not part of this KIP) > >> > >>> @matthias, could we remove the ZK dependency from the streams reset > tool > >>> now? > >> > >> I think so. The new AdminClient provide the feature we need AFAIK. I > >> guess we can picky back this into the KIP (we would need a KIP anyway > >> because we deprecate `--zookeeper` what is an public API change). > >> > >> > >> I just want to point out, that even without ZK dependency, I prefer to > >> wrap the consumer offset tool and keep two tools. > >> > >> > >> -Matthias > >> > >> > >> On 6/29/17 9:14 AM, Damian Guy wrote: > >>> Hi, > >>> > >>> Thanks for the KIP. What is not clear is how is this going to handle > >> state > >>> stores? Right now the streams reset tool, resets everything and clears > up > >>> the state stores. What are we going to do if we reset to a particular > >>> offset? If we clear the state then we've lost any previously aggregated > >>> values (which may or may not be what is expected). If we don't clear > >> them, > >>> then we will end up with incorrect aggregates. > >>> > >>> @matthias, could we remove the ZK dependency from the streams reset > tool > >>> now? > >>> > >>> Thanks, > >>> Damian > >>> > >>> On Thu, 29 Jun 2017 at 15:22 Jorge Esteban Quilcate Otoya < > >>> quilcate.jo...@gmail.com> wrote: > >>> > >>>> You're right, I was not considering Zookeeper dependency. > >>>> > >>>> I'm starting to like the idea to wrap `reset-offset` from > >>>> `streams-application-reset`. > >>>> > >>>> I will wait a bit for more feedback and work on a draft with this > >> changes. > >>>> > >>>> > >>>> El mié., 28 jun. 2017 a las 0:20, Matthias J. Sax (< > >> matth...@confluent.io > >>>>> ) > >>>> escribió: > >>>> > >>>>> I agree, that we should not duplicate functionality. > >>>>> > >>>>> However, I am worried, that a non-streams users using the offset > reset > >>>>> tool might delete topics unintentionally (even if the changes are > >> pretty > >>>>> small). Also, currently the stream reset tool required Zookeeper > while > >>>>> the offset reset tool does not -- I don't think we should add this > >>>>> dependency to the offset reset tool. > >>>>> > >>>>> Thus, it think it might be better to keep both tools, but internally > >>>>> rewrite the streams reset entry class, to reuse as much as possible > >> from > >>>>> the offset reset tool. Ie. the streams tool would be a wrapper around > >>>>> the offset tool and add some functionality it needs that is Streams > >>>>> specific. > >>>>> > >>>>> I also think, that keeping separate tools for consumers and streams > is > >> a > >>>>> good thing. We might want to add new features that don't apply to > plain > >>>>> consumers -- note, a Streams applications is more than just a client. > >>>>> > >>>>> WDYT? > >>>>> > >>>>> Would be good to get some feedback from others, too. > >>>>> > >>>>> > >>>>> -Matthias > >>>>> > >>>>> > >>>>> On 6/27/17 9:05 AM, Jorge Esteban Quilcate Otoya wrote: > >>>>>> Thanks for the feedback Matthias! > >>>>>> > >>>>>> The main idea is to use only 1 tool to reset offsets and don't > >>>> replicate > >>>>>> functionality between tools. > >>>>>> Reset Offset (KIP-122) tool not only reset but support execute the > >>>> reset > >>>>>> but also export, import from files, etc. > >>>>>> If we extend the current tool (kafka-streams-application-reset.sh) > we > >>>>> will > >>>>>> have to duplicate all this functionality also. > >>>>>> Maybe another option is to move the current implementation into > >>>>>> `kafka-consumer-group` and add a new command > `--reset-offset-streams` > >>>>> with > >>>>>> the current implementation functionality and add `--reset-offset` > >>>> options > >>>>>> for input topics. Does this make sense? > >>>>>> > >>>>>> > >>>>>> El lun., 26 jun. 2017 a las 23:32, Matthias J. Sax (< > >>>>> matth...@confluent.io>) > >>>>>> escribió: > >>>>>> > >>>>>>> Jorge, > >>>>>>> > >>>>>>> thanks a lot for this KIP. Allowing the reset streams applications > >>>> with > >>>>>>> arbitrary start offset is something we got multiple requests > already. > >>>>>>> > >>>>>>> Couple of clarification question: > >>>>>>> > >>>>>>> - why do you want to deprecate the current tool instead of > extending > >>>>>>> the current tool with the stuff the offset reset tool can do (ie, > use > >>>>>>> the offset reset tool internally) > >>>>>>> > >>>>>>> - you suggest to extend the offset reset tool to replace the > stream > >>>>>>> reset tool: how would the reset tool know if it is resetting a > >> streams > >>>>>>> applications or a regular consumer group? > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -Matthias > >>>>>>> > >>>>>>> > >>>>>>> On 6/26/17 1:28 PM, Jorge Esteban Quilcate Otoya wrote: > >>>>>>>> Hi all, > >>>>>>>> > >>>>>>>> I'd like to start the discussion to add reset offset tooling for > >>>> Stream > >>>>>>>> applications. > >>>>>>>> The KIP can be found here: > >>>>>>>> > >>>>>>> > >>>>> > >>>> > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-171+-+Extend+Consumer+Group+Reset+Offset+for+Stream+Application > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Jorge. > >>>>>>>> > >>>>>>> > >>>>>>> > >>>>>> > >>>>> > >>>>> > >>>> > >>> > >> > >> > > > >
