Thanks for the update, Jukka! I'd be in favor of the current proposal. Not sure how the others feel. If people generally feel positive, it might be time to start a vote.
Thanks, -John On Sat, Sep 7, 2019 at 12:40 AM Jukka Karvanen <jukka.karva...@jukinimi.com> wrote: > > Hi, > > Sorry; I need to rollback right away the one method removal what I was > proposing. > > One constructor with Long restored to TestRecord, which is needed by > TestInputTopic. > > Regards, > Jukka > > la 7. syysk. 2019 klo 8.06 Jukka Karvanen (jukka.karva...@jukinimi.com) > kirjoitti: > > > Hi, > > > > Let's get back to this after summer break. > > Thanks Antony to offering help, it might be needed. > > > > I modified the KIP based on the feedback to be a mixture of variations 4 > > and 5. > > > > In TestInputTopic I removed deprecation from one variation with long > > timestamp and removed totally one version without key. > > The existing test code with it can be easily migrated to use remaining > > method adding null key. > > > > In TestRecord I removed constructors with Long timestamp from the public > > interface. You can migrate existing code > > with Long timestamp constructors to use constructors with ProducerRecord > > or ConsumerRecord. > > There is still Long timestamp(); method like in ProducerRecord / > > ConsumerRecord. > > > > Is this acceptable alternative? > > What else is needed to conclude the discussion phase and get to voting? > > > > Regards, > > Jukka > > > > to 5. syysk. 2019 klo 17.17 Antony Stubbs (ant...@confluent.io) kirjoitti: > > > >> Hi Jukka! I just came across your work - it looks great! I was taking a > >> stab at improving the existing API, but yours already looks great and just > >> about complete! Are you planning on continuing your work and submitting a > >> PR? If you want some help, I'd be happy to jump in. > >> > >> Regards, > >> Antony. > >> > >> On Thu, Aug 1, 2019 at 3:42 PM Bill Bejeck <bbej...@gmail.com> wrote: > >> > >> > Hi Jukka, > >> > > >> > I also think 3, 4, and 5 are all good options. > >> > > >> > My personal preference is 4, but I also wouldn't mind going with 5 if > >> that > >> > is what others want to do. > >> > > >> > Thanks, > >> > Bill > >> > > >> > On Tue, Jul 30, 2019 at 9:31 AM John Roesler <j...@confluent.io> wrote: > >> > > >> > > Hey Jukka, > >> > > > >> > > Sorry for the delay. > >> > > > >> > > For what it's worth, I think 3, 4, and 5 are all good options. I > >> guess my > >> > > own preference is 5. > >> > > > >> > > It seems like the migration pain is a one-time concern vs. having more > >> > > maintainable code for years thereafter. > >> > > > >> > > Thanks, > >> > > -John > >> > > > >> > > > >> > > > >> > > On Tue, Jul 2, 2019 at 4:03 AM Jukka Karvanen < > >> > jukka.karva...@jukinimi.com > >> > > > > >> > > wrote: > >> > > > >> > > > Hi Matthias, > >> > > > > >> > > > Generally I think using Instant and Duration make the test more > >> > readable > >> > > > and that's why I modified KIP based on your suggestion. > >> > > > Now a lot of code use time with long or Long and that make the > >> change > >> > > more > >> > > > complicated. > >> > > > > >> > > > What I tried to say about the migration is the lines without > >> timestamp > >> > or > >> > > > if long timestamp is supported can be migrated mainly with search & > >> > > > recplace: > >> > > > > >> > > > > >> > > > > >> > > > >> > > >> testDriver.pipeInput(recordFactory.create(WordCountLambdaExample.inputTopic, > >> > > > nullKey, "Hello", 1L)); > >> > > > > >> > > > -> > >> > > > > >> > > > *inputTopic*.pipeInput(nullKey, "Hello", 1L); > >> > > > > >> > > > If long is not supported as timestamp, the same is not so easy: > >> > > > > >> > > > > >> > > > > >> > > > >> > > >> testDriver.pipeInput(recordFactory.create(WordCountLambdaExample.inputTopic, > >> > > > nullKey, "Hello", 1L)); > >> > > > > >> > > > -> > >> > > > > >> > > > *inputTopic1*.pipeInput(nullKey, "Hello", Instant.ofEpochMilli(1L)); > >> > > > > >> > > > Also if you need to convert arbitrary long timestamps to proper time > >> > > > Instants, it require you need to understand the logic of the test. > >> So > >> > > > mechanical search & replace is not possible. > >> > > > > >> > > > > >> > > > I see there are several alternatives for long vs Instant / Duration: > >> > > > > >> > > > 1. All times as long/Long like in this version: > >> > > > > >> > > > > >> > > > >> > > >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=119550011 > >> > > > > >> > > > (startTimestampMs, autoAdvanceMs as parameter of createInputTopic > >> > > > instead of configureTiming) > >> > > > > >> > > > 2. Auto timestamping configured with Instant and Duration, pipeInput > >> > > > and TestRecord with long: > >> > > > > >> > > > > >> > > > >> > > >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=120722523 > >> > > > > >> > > > > >> > > > 3. (CURRENT) Auto timestamping configured with Instant and Duration, > >> > > > pipeInput and TestRecord with Instant, version with long deprecated: > >> > > > > >> > > > > >> > > > > >> > > > >> > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-470%3A+TopologyTestDriver+test+input+and+output+usability+improvements > >> > > > > >> > > > > >> > > > 4. Auto timestamping configured with Instant and Duration, pipeInput > >> > > > and TestRecord with Instant and long parallel (with long not > >> > > > deprecated): > >> > > > > >> > > > 5. Auto timestamping configured with Instant and Duration, pipeInput > >> > > > and TestRecord with Instant only > >> > > > > >> > > > I hope to get feedback. > >> > > > > >> > > > My own preference currently is alternative 3. or 4. > >> > > > > >> > > > > >> > > > If somebody want to test, there is a implementation of this version > >> > > > available in Github: > >> > > > > >> > > > https://github.com/jukkakarvanen/kafka-streams-test-topics > >> > > > > >> > > > which can be used directly from public Maven repository: > >> > > > > >> > > > <dependency> > >> > > > <groupId>com.github.jukkakarvanen</groupId> > >> > > > <artifactId>kafka-streams-test-topics</artifactId> > >> > > > <version>0.0.1-beta3</version> > >> > > > <scope>test</scope> > >> > > > </dependency> > >> > > > > >> > > > Also is this approach in KIP-470 preferred over KIP-456, so can we > >> > close > >> > > > it: > >> > > > > >> > > > > >> > > > > >> > > > >> > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-456%3A+Helper+classes+to+make+it+simpler+to+write+test+logic+with+TopologyTestDriver > >> > > > > >> > > > Jukka > >> > > > > >> > > > . > >> > > > > >> > > > > >> > > > pe 28. kesäk. 2019 klo 1.10 Matthias J. Sax (matth...@confluent.io) > >> > > > kirjoitti: > >> > > > > >> > > > > Thanks Jukka! > >> > > > > > >> > > > > The idea to use `Instant/Duration` was a proposal. If we think > >> it's > >> > not > >> > > > > a good one, we could still stay with `long`. Because > >> `ProducerRecord` > >> > > > > and `ConsumerRecord` are both based on `long,` it might make > >> sense to > >> > > > > keep `long`? > >> > > > > > >> > > > > > The result of converting millis to Instant directly generates > >> > > > > >> rather non readable test code and changing from long to Instant > >> > > > > correctly > >> > > > > >> require understand what is the case it is testing. > >> > > > > > >> > > > > This might be a good indicator the using `Instant/Duration` might > >> not > >> > > be > >> > > > > a good idea. > >> > > > > > >> > > > > Would be nice to get feedback from others. > >> > > > > > >> > > > > About adding new methods that we deprecate immediately: I don't > >> think > >> > > we > >> > > > > should do this. IMHO, there are two kind of users, one that > >> > immediately > >> > > > > rewrite their code to move off deprecated methods. Those won't use > >> > the > >> > > > > new+deprecated ones anyway. Other uses migrate their code slowly > >> and > >> > > > > would just not rewrite their code at all, and thus also not use > >> the > >> > > > > new+deprecated methods. > >> > > > > > >> > > > > > Checking my own tests I was able to migrate the most of my code > >> > with > >> > > > > > search&replace without further thinking about the logic to this > >> new > >> > > > > > approach. The result of converting millis to Instant directly > >> > > generates > >> > > > > > rather non readable test code and changing from long to Instant > >> > > > correctly > >> > > > > > require understand what is the case it is testing. > >> > > > > > >> > > > > Not sure if I can follow here. You first say, you could easily > >> > migrate > >> > > > > your code, but than you say it was not easily possible? Can you > >> > clarify > >> > > > > your experience upgrading your test code? > >> > > > > > >> > > > > > >> > > > > -Matthias > >> > > > > > >> > > > > > >> > > > > On 6/27/19 12:21 AM, Jukka Karvanen wrote: > >> > > > > > Hi, > >> > > > > > > >> > > > > >>> (4) Should we switch from `long` for timestamps to `Instant` > >> and > >> > > > > > `Duration` ? > >> > > > > >> This version startTimestamp is Instant and autoAdvance > >> Duration in > >> > > > > > Initialization and with manual configured collection pipe > >> methods. > >> > > > > >> Now timestamp of TestRecord is still Long and similarly single > >> > > record > >> > > > > > pipeInput still has long as parameter. > >> > > > > >> Should these also converted to to Instant type? > >> > > > > >> Should there be both long and Instant parallel? > >> > > > > >> I expect there are existing test codebase which would be > >> easier to > >> > > > > migrate > >> > > > > > if long could be still used. > >> > > > > > Now added Instant version to TestRecord and pipeInput method. > >> > > > > > > >> > > > > > Checking my own tests I was able to migrate the most of my code > >> > with > >> > > > > > search&replace without further thinking about the logic to this > >> new > >> > > > > > approach. The result of converting millis to Instant directly > >> > > generates > >> > > > > > rather non readable test code and changing from long to Instant > >> > > > correctly > >> > > > > > require understand what is the case it is testing. > >> > > > > > > >> > > > > > That is why version with long left still as deprecated for > >> easier > >> > > > > migration > >> > > > > > for existing test. > >> > > > > > Also TopologyTestDriver constructor and advanceWallClockTime > >> > method > >> > > > > > modified with same approach. > >> > > > > > > >> > > > > > Jukka > >> > > > > > > >> > > > > > > >> > > > > > ma 24. kesäk. 2019 klo 16.47 Bill Bejeck (bbej...@gmail.com) > >> > > > kirjoitti: > >> > > > > > > >> > > > > >> Hi Jukka > >> > > > > >> > >> > > > > >>> These topic objects are only interfacing TopologyTestDriver, > >> not > >> > > > > >> affecting > >> > > > > >>> the internal functionality of it. In my plan the internal data > >> > > > > structures > >> > > > > >>> are using those Producer/ConsumerRecords as earlier. That way > >> I > >> > > don't > >> > > > > see > >> > > > > >>> how those could be affected. > >> > > > > >> > >> > > > > >> I mistakenly thought the KIP was proposing to completely remove > >> > > > > >> Producer/ConsumerRecords in favor of TestRecord. But taking > >> > another > >> > > > > quick > >> > > > > >> look I can see the plan for using the TestRecord objects. > >> Thanks > >> > > for > >> > > > > the > >> > > > > >> clarification. > >> > > > > >> > >> > > > > >> -Bill > >> > > > > >> > >> > > > > >> On Sat, Jun 22, 2019 at 2:26 AM Jukka Karvanen < > >> > > > > >> jukka.karva...@jukinimi.com> > >> > > > > >> wrote: > >> > > > > >> > >> > > > > >>> Hi All, > >> > > > > >>> Hi Matthias, > >> > > > > >>> > >> > > > > >>>> (1) It's a little confusing that you list all method > >> (existing, > >> > > > > proposed > >> > > > > >>>> to deprecate, and new one) of `TopologyTestDriver` in the > >> KIP. > >> > > Maybe > >> > > > > >>>> only list the ones you propose to deprecate and the new ones > >> you > >> > > > want > >> > > > > to > >> > > > > >>>> add? > >> > > > > >>> > >> > > > > >>> Ok. Unmodified methods removed. > >> > > > > >>> > >> > > > > >>>> (2) `TopologyTestDriver#createInputTopic`: might it be worth > >> to > >> > > add > >> > > > > >>>> overload to initialize the timetamp and auto-advance feature > >> > > > directly? > >> > > > > >>>> Otherwise, uses always need to call `configureTiming` as an > >> > extra > >> > > > > call? > >> > > > > >>> > >> > > > > >>> Added with Instant and Duration parameters. > >> > > > > >>> > >> > > > > >>>> (3) `TestInputTopic#configureTiming()`: maybe rename to > >> > > > > >>> `reconfigureTiming()` ? > >> > > > > >>> > >> > > > > >>> I removed this method when we have now initialization in > >> > > constructor. > >> > > > > >>> You can recreate TestInputTopic if needing to reconfigure > >> timing. > >> > > > > >>> > >> > > > > >>> > >> > > > > >>>> (4) Should we switch from `long` for timestamps to `Instant` > >> and > >> > > > > >>> `Duration` ? > >> > > > > >>> This version startTimestamp is Instant and autoAdvance > >> Duration > >> > in > >> > > > > >>> Initialization and with manual configured collection pipe > >> > methods. > >> > > > > >>> Now timestamp of TestRecord is still Long and similarly single > >> > > record > >> > > > > >>> pipeInput still has long as parameter. > >> > > > > >>> Should these also converted to to Instant type? > >> > > > > >>> Should there be both long and Instant parallel? > >> > > > > >>> I expect there are existing test codebase which would be > >> easier > >> > to > >> > > > > >> migrate > >> > > > > >>> if long could be still used. > >> > > > > >>> > >> > > > > >>> Also should advanceWallClockTime in TopologyTestDriver > >> > changed(or > >> > > > > added > >> > > > > >>> alternative) for Duration parameter. > >> > > > > >>> > >> > > > > >>> > >> > > > > >>>> (5) Why do we have redundant getters? Or set with `getX()` > >> and > >> > one > >> > > > > >>> set without `get`-prefix? > >> > > > > >>> > >> > > > > >>> The methods without get-prefix are for compatibility with > >> > > > > >> ProducerRecord / > >> > > > > >>> ConsumerRecord and I expect would make migration to TestRecord > >> > > > easier. > >> > > > > >>> Standard getters in TestRecord enable writing test ignoring > >> > > selected > >> > > > > >> fields > >> > > > > >>> with hamcrest like this: > >> > > > > >>> > >> > > > > >>> assertThat(outputTopic.readRecord(), allOf( > >> > > > > >>> hasProperty("key", equalTo(1L)), > >> > > > > >>> hasProperty("value", equalTo("Hello")), > >> > > > > >>> hasProperty("headers", equalTo(headers)))); > >> > > > > >>> > >> > > > > >>> > >> > > > > >>> That's why I have currently both methods. > >> > > > > >>> > >> > > > > >>> Jukka > >> > > > > >>> > >> > > > > >>> > >> > > > > >>> pe 21. kesäk. 2019 klo 22.20 Matthias J. Sax ( > >> > > matth...@confluent.io) > >> > > > > >>> kirjoitti: > >> > > > > >>> > >> > > > > >>>> Thanks for the KIP. The idea to add InputTopic and > >> OutputTopic > >> > > > > >>>> abstractions is really neat! > >> > > > > >>>> > >> > > > > >>>> > >> > > > > >>>> Couple of minor comment: > >> > > > > >>>> > >> > > > > >>>> (1) It's a little confusing that you list all method > >> (existing, > >> > > > > >> proposed > >> > > > > >>>> to deprecate, and new one) of `TopologyTestDriver` in the > >> KIP. > >> > > Maybe > >> > > > > >>>> only list the ones you propose to deprecate and the new ones > >> you > >> > > > want > >> > > > > >> to > >> > > > > >>>> add? > >> > > > > >>>> > >> > > > > >>>> (Or mark all existing methods clearly -- atm, I need to got > >> back > >> > > to > >> > > > > the > >> > > > > >>>> code to read the KIP and to extract what changes are > >> proposed). > >> > > > > >>>> > >> > > > > >>>> > >> > > > > >>>> (2) `TopologyTestDriver#createInputTopic`: might it be worth > >> to > >> > > add > >> > > > > >>>> overload to initialize the timetamp and auto-advance feature > >> > > > directly? > >> > > > > >>>> Otherwise, uses always need to call `configureTiming` as an > >> > extra > >> > > > > call? > >> > > > > >>>> > >> > > > > >>>> > >> > > > > >>>> (3) `TestInputTopic#configureTiming()`: maybe rename to > >> > > > > >>>> `reconfigureTiming()` ? > >> > > > > >>>> > >> > > > > >>>> > >> > > > > >>>> (4) Should we switch from `long` for timestamps to `Instant` > >> and > >> > > > > >>>> `Duration` ? > >> > > > > >>>> > >> > > > > >>>> > >> > > > > >>>> (5) Why do we have redundant getters? Or set with `getX()` > >> and > >> > one > >> > > > set > >> > > > > >>>> without `get`-prefix? > >> > > > > >>>> > >> > > > > >>>> > >> > > > > >>>> > >> > > > > >>>> -Matthias > >> > > > > >>>> > >> > > > > >>>> > >> > > > > >>>> > >> > > > > >>>> > >> > > > > >>>> On 6/21/19 10:57 AM, Bill Bejeck wrote: > >> > > > > >>>>> Jukka, > >> > > > > >>>>> > >> > > > > >>>>> Thanks for the KIP. I like the changes overall. > >> > > > > >>>>> One thing I wanted to confirm, and this may be me being > >> > paranoid, > >> > > > but > >> > > > > >>>> will > >> > > > > >>>>> the changes for input/output topic affect how the > >> > > > TopologyTestDriver > >> > > > > >>>> works > >> > > > > >>>>> with internal topics when there are sub-topologies created? > >> > > > > >>>>> > >> > > > > >>>>> On Fri, Jun 21, 2019 at 12:05 PM Guozhang Wang < > >> > > wangg...@gmail.com > >> > > > > > >> > > > > >>>> wrote: > >> > > > > >>>>> > >> > > > > >>>>>> 1) Got it, could you list this class along with all its > >> > > functions > >> > > > in > >> > > > > >>> the > >> > > > > >>>>>> proposed public APIs as well? > >> > > > > >>>>>> > >> > > > > >>>>>> 2) Ack, thanks! > >> > > > > >>>>>> > >> > > > > >>>>>> On Thu, Jun 20, 2019 at 11:27 PM Jukka Karvanen < > >> > > > > >>>>>> jukka.karva...@jukinimi.com> > >> > > > > >>>>>> wrote: > >> > > > > >>>>>> > >> > > > > >>>>>>> Hi Guozhang, > >> > > > > >>>>>>> > >> > > > > >>>>>>> 1) This TestRecord is new class in my proposal. So it is a > >> > > > > >> simplified > >> > > > > >>>>>>> version of ProducerRecord and ConsumerRecord containing > >> only > >> > > the > >> > > > > >>> fields > >> > > > > >>>>>>> needed to test record content. > >> > > > > >>>>>>> > >> > > > > >>>>>>> 2) > >> > > > > >>>>>>> public final <K, V> TestInputTopic<K, V> > >> > createInputTopic(final > >> > > > > >>> String > >> > > > > >>>>>>> topicName, final Serde<K> keySerde, final Serde<V> > >> > valueSerde); > >> > > > > >>>>>>> public final <K, V> TestOutputTopic<K, V> > >> > > createOutputTopic(final > >> > > > > >>>> String > >> > > > > >>>>>>> topicName, final Serde<K> keySerde, final Serde<V> > >> > valueSerde); > >> > > > > >>>>>>> The purpose is to create separate object for each input > >> and > >> > > > output > >> > > > > >>>> topic > >> > > > > >>>>>>> you are using. The topic name is given to > >> > > createInput/OutputTopic > >> > > > > >>> when > >> > > > > >>>>>>> initialize topic object. > >> > > > > >>>>>>> > >> > > > > >>>>>>> For example: > >> > > > > >>>>>>> > >> > > > > >>>>>>> final TestInputTopic<Long, String> inputTopic1 = > >> > > > > >>>>>>> testDriver.createInputTopic( INPUT_TOPIC, longSerde, > >> > > > stringSerde); > >> > > > > >>>>>>> final TestInputTopic<Long, String> inputTopic2 = > >> > > > > >>>>>>> testDriver.createInputTopic( INPUT_TOPIC_MAP, longSerde, > >> > > > > >>> stringSerde); > >> > > > > >>>>>>> final TestOutputTopic<Long, String> outputTopic1 = > >> > > > > >>>>>>> testDriver.createOutputTopic(OUTPUT_TOPIC, longSerde, > >> > > > stringSerde); > >> > > > > >>>>>>> final TestOutputTopic<String, Long> outputTopic2 = > >> > > > > >>>>>>> testDriver.createOutputTopic(OUTPUT_TOPIC_MAP, > >> stringSerde, > >> > > > > >>>>>>> longSerde); > >> > > > > >>>>>>> inputTopic1.pipeInput(1L, "Hello"); > >> > > > > >>>>>>> assertThat(outputTopic1.readKeyValue(), equalTo(new > >> > > > KeyValue<>(1L, > >> > > > > >>>>>>> "Hello"))); > >> > > > > >>>>>>> assertThat(outputTopic2.readKeyValue(), equalTo(new > >> > > > > >>> KeyValue<>("Hello", > >> > > > > >>>>>>> 1L))); > >> > > > > >>>>>>> inputTopic2.pipeInput(1L, "Hello"); > >> > > > > >>>>>>> > >> > > > > >>>>>>> > >> > > > > >>>>>>> Jukka > >> > > > > >>>>>>> > >> > > > > >>>>>>> to 20. kesäk. 2019 klo 23.52 Guozhang Wang ( > >> > wangg...@gmail.com > >> > > ) > >> > > > > >>>>>> kirjoitti: > >> > > > > >>>>>>> > >> > > > > >>>>>>>> Hello Jukka, > >> > > > > >>>>>>>> > >> > > > > >>>>>>>> Thanks for writing the KIP, I have a couple of quick > >> > > questions: > >> > > > > >>>>>>>> > >> > > > > >>>>>>>> 1) Is "TestRecord" an existing class that you propose to > >> > > > > >> piggy-back > >> > > > > >>>> on? > >> > > > > >>>>>>>> Right now we have a scala TestRecord case class but I > >> doubt > >> > > that > >> > > > > >> was > >> > > > > >>>>>> your > >> > > > > >>>>>>>> proposal, or are you proposing to add a new Java class? > >> > > > > >>>>>>>> > >> > > > > >>>>>>>> 2) Would the new API only allow a single input / output > >> > topic > >> > > > with > >> > > > > >>>>>>>> `createInput/OutputTopic`? If not, when we call pipeInput > >> > how > >> > > to > >> > > > > >>>>>>> determine > >> > > > > >>>>>>>> which topic this record should be pipe to? > >> > > > > >>>>>>>> > >> > > > > >>>>>>>> > >> > > > > >>>>>>>> Guozhang > >> > > > > >>>>>>>> > >> > > > > >>>>>>>> On Mon, Jun 17, 2019 at 1:34 PM John Roesler < > >> > > j...@confluent.io > >> > > > > > >> > > > > >>>>>> wrote: > >> > > > > >>>>>>>> > >> > > > > >>>>>>>>> Woah, I wasn't aware of that Hamcrest test style. > >> Awesome! > >> > > > > >>>>>>>>> > >> > > > > >>>>>>>>> Thanks for the updates. I look forward to hearing what > >> > others > >> > > > > >>> think. > >> > > > > >>>>>>>>> > >> > > > > >>>>>>>>> -John > >> > > > > >>>>>>>>> > >> > > > > >>>>>>>>> On Mon, Jun 17, 2019 at 4:12 AM Jukka Karvanen > >> > > > > >>>>>>>>> <jukka.karva...@jukinimi.com> wrote: > >> > > > > >>>>>>>>>> > >> > > > > >>>>>>>>>> Wiki page updated: > >> > > > > >>>>>>>>>> > >> > > > > >>>>>>>>> > >> > > > > >>>>>>>> > >> > > > > >>>>>>> > >> > > > > >>>>>> > >> > > > > >>>> > >> > > > > >>> > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-470%3A+TopologyTestDriver+test+input+and+output+usability+improvements > >> > > > > >>>>>>>>>> > >> > > > > >>>>>>>>>> > >> > > > > >>>>>>>>>> ClientRecord removed and replaced with TestRecord in > >> > method > >> > > > > >> calls. > >> > > > > >>>>>>>>>> TestRecordFactory removed (time tracking functionality > >> to > >> > be > >> > > > > >>>>>> included > >> > > > > >>>>>>>> to > >> > > > > >>>>>>>>>> TestInputTopic) > >> > > > > >>>>>>>>>> OutputVerifier deprecated > >> > > > > >>>>>>>>>> TestRecord topic removed and getters added > >> > > > > >>>>>>>>>> > >> > > > > >>>>>>>>>> Getters in TestRecord enable writing test ignoring > >> > selected > >> > > > > >> fields > >> > > > > >>>>>>> with > >> > > > > >>>>>>>>>> hamcrest like this: > >> > > > > >>>>>>>>>> > >> > > > > >>>>>>>>>> assertThat(outputTopic.readRecord(), allOf( > >> > > > > >>>>>>>>>> hasProperty("key", equalTo(1L)), > >> > > > > >>>>>>>>>> hasProperty("value", equalTo("Hello")), > >> > > > > >>>>>>>>>> hasProperty("headers", equalTo(headers)))); > >> > > > > >>>>>>>>>> > >> > > > > >>>>>>>>>> > >> > > > > >>>>>>>>>> Jukka > >> > > > > >>>>>>>>>> > >> > > > > >>>>>>>>>> la 15. kesäk. 2019 klo 1.10 John Roesler ( > >> > j...@confluent.io > >> > > ) > >> > > > > >>>>>>>> kirjoitti: > >> > > > > >>>>>>>>>> > >> > > > > >>>>>>>>>>> Sounds good. Thanks as always for considering my > >> > feedback! > >> > > > > >>>>>>>>>>> -John > >> > > > > >>>>>>>>>>> > >> > > > > >>>>>>>>>>> On Fri, Jun 14, 2019 at 12:12 PM Jukka Karvanen > >> > > > > >>>>>>>>>>> <jukka.karva...@jukinimi.com> wrote: > >> > > > > >>>>>>>>>>>> > >> > > > > >>>>>>>>>>>> Ok, I will modify KIP Public Interface in a wiki > >> based > >> > on > >> > > > the > >> > > > > >>>>>>>>> feedback. > >> > > > > >>>>>>>>>>>> > >> > > > > >>>>>>>>>>>> TestRecordFactory / ConsumerRecordFactory was used by > >> > > > > >>>>>>>> TestInputTopic > >> > > > > >>>>>>>>> with > >> > > > > >>>>>>>>>>>> the version I had with KIP456, but maybe I can merge > >> > That > >> > > > > >>>>>>>>> functionality > >> > > > > >>>>>>>>>>> to > >> > > > > >>>>>>>>>>>> InputTopic or TestRecordFactory can kept non > >> public > >> > > maybe > >> > > > > >>>>>>> moving > >> > > > > >>>>>>>>> it to > >> > > > > >>>>>>>>>>>> internals package. > >> > > > > >>>>>>>>>>>> > >> > > > > >>>>>>>>>>>> I will make the proposal with a slim down interface. > >> > > > > >>>>>>>>>>>> I don't want to go to so slim as you proposed with > >> only > >> > > > > >>>>>>> TestRecord > >> > > > > >>>>>>>> or > >> > > > > >>>>>>>>>>>> List<TestRecord>, because you then still end up doing > >> > > helper > >> > > > > >>>>>>>> methods > >> > > > > >>>>>>>>> to > >> > > > > >>>>>>>>>>>> construct List of TestRecord. > >> > > > > >>>>>>>>>>>> The list of values is easier to write and clearer to > >> > read > >> > > > than > >> > > > > >>>>>> if > >> > > > > >>>>>>>> you > >> > > > > >>>>>>>>>>> need > >> > > > > >>>>>>>>>>>> to contruct list of TestRecords. > >> > > > > >>>>>>>>>>>> > >> > > > > >>>>>>>>>>>> For example: > >> > > > > >>>>>>>>>>>> > >> > > > > >>>>>>>>>>>> final List<String> inputValues = Arrays.asList( > >> > > > > >>>>>>>>>>>> "Apache Kafka Streams Example", > >> > > > > >>>>>>>>>>>> "Using Kafka Streams Test Utils", > >> > > > > >>>>>>>>>>>> "Reading and Writing Kafka Topic" > >> > > > > >>>>>>>>>>>> ); > >> > > > > >>>>>>>>>>>> inputTopic.pipeValueList(inputValues); > >> > > > > >>>>>>>>>>>> > >> > > > > >>>>>>>>>>>> > >> > > > > >>>>>>>>>>>> Let's check after the next iteration is it still > >> worth > >> > > > > >> reducing > >> > > > > >>>>>>> the > >> > > > > >>>>>>>>>>> methods. > >> > > > > >>>>>>>>>>>> > >> > > > > >>>>>>>>>>>> > >> > > > > >>>>>>>>>>>> Jukka > >> > > > > >>>>>>>>>>>> > >> > > > > >>>>>>>>>>>> > >> > > > > >>>>>>>>>>>> pe 14. kesäk. 2019 klo 18.27 John Roesler ( > >> > > > j...@confluent.io) > >> > > > > >>>>>>>>> kirjoitti: > >> > > > > >>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>> Thanks, Jukka, > >> > > > > >>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>> Ok, I buy this reasoning. > >> > > > > >>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>> Just to echo what I think I read, you would just > >> drop > >> > > > > >>>>>>>> ClientRecord > >> > > > > >>>>>>>>>>>>> from the proposal, and TestRecord would stand on its > >> > own, > >> > > > > >>>>>> with > >> > > > > >>>>>>>> the > >> > > > > >>>>>>>>>>>>> same methods and properties you proposed, and the > >> > "input > >> > > > > >>>>>> topic" > >> > > > > >>>>>>>>> would > >> > > > > >>>>>>>>>>>>> accept TestRecord, and the "output topic" would > >> produce > >> > > > > >>>>>>>> TestRecord? > >> > > > > >>>>>>>>>>>>> Further, the "input and output topic" classes would > >> > > > > >>>>>> internally > >> > > > > >>>>>>>>> handle > >> > > > > >>>>>>>>>>>>> the conversion to and from ConsumerRecord and > >> > > > ProducerRecord > >> > > > > >>>>>> to > >> > > > > >>>>>>>>> pass > >> > > > > >>>>>>>>>>>>> to and from the TopologyTestDriver? > >> > > > > >>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>> This seems good to me. > >> > > > > >>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>> Since the object coming out of the "output topic" is > >> > much > >> > > > > >>>>>> more > >> > > > > >>>>>>>>>>>>> ergonomic, I suspect we won't need the > >> OutputVerifier > >> > at > >> > > > all. > >> > > > > >>>>>>> It > >> > > > > >>>>>>>>> was > >> > > > > >>>>>>>>>>>>> mostly needed because of all the extra junk in > >> > > > ProducerRecord > >> > > > > >>>>>>> you > >> > > > > >>>>>>>>> need > >> > > > > >>>>>>>>>>>>> to ignore. It seems better to just deprecate it. If > >> in > >> > > the > >> > > > > >>>>>>> future > >> > > > > >>>>>>>>> it > >> > > > > >>>>>>>>>>>>> turns out there is some actual use case for a > >> verifier, > >> > > we > >> > > > > >>>>>> can > >> > > > > >>>>>>>>> have a > >> > > > > >>>>>>>>>>>>> very small KIP to add one. But reading your > >> response, I > >> > > > > >>>>>> suspect > >> > > > > >>>>>>>>> that > >> > > > > >>>>>>>>>>>>> existing test verification libraries would be > >> > sufficient > >> > > on > >> > > > > >>>>>>> their > >> > > > > >>>>>>>>> own. > >> > > > > >>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>> Similarly, it seems like we can shrink the total > >> > > interface > >> > > > by > >> > > > > >>>>>>>>> removing > >> > > > > >>>>>>>>>>>>> the TestRecordFactory from the proposal. If > >> TestRecord > >> > > > > >>>>>> already > >> > > > > >>>>>>>>> offers > >> > > > > >>>>>>>>>>>>> all the constructors you'd want, then the only > >> benefit > >> > of > >> > > > the > >> > > > > >>>>>>>>> factory > >> > > > > >>>>>>>>>>>>> is to auto-increment the timestamps, but then again, > >> > the > >> > > > > >>>>>> "input > >> > > > > >>>>>>>>> topic" > >> > > > > >>>>>>>>>>>>> can already do that (e.g., it can do it if the > >> record > >> > > > > >>>>>> timestamp > >> > > > > >>>>>>>> is > >> > > > > >>>>>>>>> not > >> > > > > >>>>>>>>>>>>> set). > >> > > > > >>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>> Likewise, if the TestRecords are easy to create, > >> then > >> > we > >> > > > > >>>>>> don't > >> > > > > >>>>>>>> need > >> > > > > >>>>>>>>>>>>> all the redundant methods in "input topic" to pipe > >> > > values, > >> > > > or > >> > > > > >>>>>>>>>>>>> key/values, or key/value/timestamp, etc. We can do > >> with > >> > > > just > >> > > > > >>>>>>> two > >> > > > > >>>>>>>>>>>>> methods, one for a single TestRecord and one for a > >> > > > collection > >> > > > > >>>>>>> of > >> > > > > >>>>>>>>> them. > >> > > > > >>>>>>>>>>>>> This reduces API ambiguity and also dramatically > >> > > decreases > >> > > > > >>>>>> the > >> > > > > >>>>>>>>> surface > >> > > > > >>>>>>>>>>>>> area of the interface, which ultimately makes it > >> much > >> > > > easier > >> > > > > >>>>>> to > >> > > > > >>>>>>>>> use. > >> > > > > >>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>> It's best to start with the smallest interface that > >> > will > >> > > do > >> > > > > >>>>>> the > >> > > > > >>>>>>>> job > >> > > > > >>>>>>>>>>>>> and expand it upon request, rather than throwing in > >> > > > > >>>>>> everything > >> > > > > >>>>>>>> you > >> > > > > >>>>>>>>> can > >> > > > > >>>>>>>>>>>>> think of up front. The extra stuff would be > >> confusing > >> > to > >> > > > > >>>>>> people > >> > > > > >>>>>>>>> facing > >> > > > > >>>>>>>>>>>>> two practically identical paths to accomplish the > >> same > >> > > > goal, > >> > > > > >>>>>>> and > >> > > > > >>>>>>>>> it's > >> > > > > >>>>>>>>>>>>> very difficult to slim the interface down later, > >> > because > >> > > we > >> > > > > >>>>>>> don't > >> > > > > >>>>>>>>>>>>> really know which parts are more popular (i.e., no > >> one > >> > > > > >>>>>> submits > >> > > > > >>>>>>>>>>>>> "feature requests" to _remove_ stuff they don't > >> need, > >> > > only > >> > > > to > >> > > > > >>>>>>>> _add_ > >> > > > > >>>>>>>>>>>>> stuff that they need. > >> > > > > >>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>> But overall, I really like the structure of this > >> > design. > >> > > > I'm > >> > > > > >>>>>>>> super > >> > > > > >>>>>>>>>>>>> excited about this KIP. > >> > > > > >>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>> Thanks, > >> > > > > >>>>>>>>>>>>> -John > >> > > > > >>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>> On Fri, Jun 14, 2019 at 2:55 AM Jukka Karvanen > >> > > > > >>>>>>>>>>>>> <jukka.karva...@jukinimi.com> wrote: > >> > > > > >>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>> Hi, > >> > > > > >>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>> I am not a fan of swapping only ProducerRecord and > >> > > > > >>>>>>>>> ConsumerRecord. > >> > > > > >>>>>>>>>>>>>> As a test writer point of view I do not want to > >> care > >> > > about > >> > > > > >>>>>>> the > >> > > > > >>>>>>>>>>> difference > >> > > > > >>>>>>>>>>>>>> of those and > >> > > > > >>>>>>>>>>>>>> that way I like to have object type which can be > >> used > >> > to > >> > > > > >>>>>> pipe > >> > > > > >>>>>>>>>>> records in > >> > > > > >>>>>>>>>>>>>> and compare outputs. > >> > > > > >>>>>>>>>>>>>> That way avoid unnecessary conversions between > >> > > > > >>>>>> ProducerRecord > >> > > > > >>>>>>>> and > >> > > > > >>>>>>>>>>>>>> ConsumerRecord. > >> > > > > >>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>> My initial assumption was that ProducerRecord and > >> > > > > >>>>>>>>>>> ConsumerRecord.could > >> > > > > >>>>>>>>>>>>>> implement the same ClientRecord > >> > > > > >>>>>>>>>>>>>> and that way test writer could have used either of > >> > > those, > >> > > > > >>>>>> but > >> > > > > >>>>>>>>> seems > >> > > > > >>>>>>>>>>> that > >> > > > > >>>>>>>>>>>>>> return type of timestamp method long vs Long is not > >> > > > > >>>>>>> compatible. > >> > > > > >>>>>>>>>>>>>> Now the main advantage of ClientRecord is no need > >> to > >> > > > > >>>>>>> duplicate > >> > > > > >>>>>>>>>>>>>> OutputVerifier when it is modified from > >> ProducerRecord > >> > > to > >> > > > > >>>>>>>>>>> ClientRecord. > >> > > > > >>>>>>>>>>>>>> Generally is there need for OutputVerifier. Can we > >> > > > > >>>>>> deprecate > >> > > > > >>>>>>>> the > >> > > > > >>>>>>>>>>> existing > >> > > > > >>>>>>>>>>>>>> and use standard assertion libraries for new test. > >> > > > > >>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>> If you use hamcrest, assert-j or any other > >> assertion > >> > > > > >>>>>> library > >> > > > > >>>>>>>>> for the > >> > > > > >>>>>>>>>>>>> rest > >> > > > > >>>>>>>>>>>>>> of the test, why not use it with these also. > >> > > > > >>>>>>>>>>>>>> When we have these methods to access only needed > >> > fields > >> > > it > >> > > > > >>>>>> is > >> > > > > >>>>>>>>> easier > >> > > > > >>>>>>>>>>> to > >> > > > > >>>>>>>>>>>>>> write test like this: > >> > > > > >>>>>>>>>>>>>> > >> > assertThat(outputTopic.readValue()).isEqualTo("Hello"); > >> > > > > >>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>> or > >> > > > > >>>>>>>>>>>>>> > >> > > > > >>>>>>> > >> > assertThat(outputTopic.readRecord()).isEqualTo(expectedRecord); > >> > > > > >>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>> Only value for new OutputVerifier is when needing > >> to > >> > > > ignore > >> > > > > >>>>>>>> some > >> > > > > >>>>>>>>>>> fields > >> > > > > >>>>>>>>>>>>>> ClientRecord actual = outputTopic.readRecord(); > >> > > > > >>>>>>>>>>>>>> assertThat(actual.value()).isEqualTo("Hello"); > >> > > > > >>>>>>>>>>>>>> assertThat(actual.key()).isEqualTo(expectedKey); > >> > > > > >>>>>>>>>>>>>> > >> > > > > >>>>>> > >> assertThat(actual.timestamp()).isEqualTo(expectedTimestamp); > >> > > > > >>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>> So if want to leave client package untouched, I > >> would > >> > > > > >>>>>> modify > >> > > > > >>>>>>>> the > >> > > > > >>>>>>>>>>> methods > >> > > > > >>>>>>>>>>>>>> with ClientRecord now in InputTopic and > >> OutputTopic to > >> > > > pass > >> > > > > >>>>>>> in > >> > > > > >>>>>>>>> and > >> > > > > >>>>>>>>>>> out > >> > > > > >>>>>>>>>>>>> this > >> > > > > >>>>>>>>>>>>>> TestRecord instead. > >> > > > > >>>>>>>>>>>>>> In that case there would be possibility to add > >> methods > >> > > to > >> > > > > >>>>>>>>> TestRecord > >> > > > > >>>>>>>>>>> to > >> > > > > >>>>>>>>>>>>>> help ignore some fields in assertions like: > >> > > > > >>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>> > >> > > > > >>>>>>>>>>> > >> > > > > >>>>>>>>> > >> > > > > >>>>>>>> > >> > > > > >>>>>>> > >> > > > > >>>>>> > >> > > > > >>>> > >> > > > > >>> > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> assertThat(outputTopic.readRecord().getValueTimestamp()).isEqualTo(expectedRecord.get > >> > > > > >>>>>>>>>>>>>> ValueTimestamp()); > >> > > > > >>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>> How about this alternative? > >> > > > > >>>>>>>>>>>>>> If this way sounds better I will modify KIP page in > >> > > wiki. > >> > > > > >>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>> Jukka > >> > > > > >>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>> to 13. kesäk. 2019 klo 18.30 John Roesler ( > >> > > > > >>>>>> j...@confluent.io > >> > > > > >>>>>>> ) > >> > > > > >>>>>>>>>>> kirjoitti: > >> > > > > >>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>>> Hey, all, maybe we can jump-start this discussion. > >> > > > > >>>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>>> I think this approach would be very ergonomic for > >> > > > > >>>>>> testing, > >> > > > > >>>>>>>> and > >> > > > > >>>>>>>>>>> would > >> > > > > >>>>>>>>>>>>>>> help reduce boilerplate in tests. > >> > > > > >>>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>>> The think I like most about it is that it mirrors > >> the > >> > > > > >>>>>>> mental > >> > > > > >>>>>>>>> model > >> > > > > >>>>>>>>>>>>>>> that people already have from Kafka Streams, in > >> which > >> > > you > >> > > > > >>>>>>>>> write to > >> > > > > >>>>>>>>>>> an > >> > > > > >>>>>>>>>>>>>>> "input topic" and then get your results from an > >> > "output > >> > > > > >>>>>>>>> topic". As > >> > > > > >>>>>>>>>>> a > >> > > > > >>>>>>>>>>>>>>> side benefit, the input and output topics in the > >> > > proposal > >> > > > > >>>>>>>> also > >> > > > > >>>>>>>>>>> close > >> > > > > >>>>>>>>>>>>>>> over the serdes, which makes it much less > >> boilerplate > >> > > for > >> > > > > >>>>>>>> test > >> > > > > >>>>>>>>>>> code. > >> > > > > >>>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>>> If I can offer one suggestion for change, I'm not > >> > sure > >> > > > > >>>>>> I'm > >> > > > > >>>>>>>>> totally > >> > > > > >>>>>>>>>>>>>>> sold on the need for a new abstraction > >> "ClientRecord" > >> > > > > >>>>>> with > >> > > > > >>>>>>> an > >> > > > > >>>>>>>>>>>>>>> implementation for tests "TestRecord". It seems > >> like > >> > > this > >> > > > > >>>>>>>>>>>>>>> unnecessarily complicates the main (non-testing) > >> data > >> > > > > >>>>>>> model. > >> > > > > >>>>>>>> It > >> > > > > >>>>>>>>>>> seems > >> > > > > >>>>>>>>>>>>>>> like it would be sufficient, and just as > >> ergonomic, > >> > to > >> > > > > >>>>>> have > >> > > > > >>>>>>>> the > >> > > > > >>>>>>>>>>> input > >> > > > > >>>>>>>>>>>>>>> topic accept ProducerRecords and the output topic > >> > > return > >> > > > > >>>>>>>>>>>>>>> ConsumerRecords. I'm open to discussion on this > >> > point, > >> > > > > >>>>>>>> though. > >> > > > > >>>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>>> Thanks for the proposal, Jukka! > >> > > > > >>>>>>>>>>>>>>> -John > >> > > > > >>>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>>> On Mon, May 20, 2019 at 7:59 AM Jukka Karvanen > >> > > > > >>>>>>>>>>>>>>> <jukka.karva...@jukinimi.com> wrote: > >> > > > > >>>>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>>>> Hi All, > >> > > > > >>>>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>>>> I would like to start the discussion on KIP-470: > >> > > > > >>>>>>>>>>> TopologyTestDriver > >> > > > > >>>>>>>>>>>>> test > >> > > > > >>>>>>>>>>>>>>>> input and output usability improvements: > >> > > > > >>>>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>> > >> > > > > >>>>>>>>>>> > >> > > > > >>>>>>>>> > >> > > > > >>>>>>>> > >> > > > > >>>>>>> > >> > > > > >>>>>> > >> > > > > >>>> > >> > > > > >>> > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-470%3A+TopologyTestDriver+test+input+and+output+usability+improvements > >> > > > > >>>>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>>>> This KIP is inspired by the Discussion in > >> KIP-456: > >> > > > > >>>>>> Helper > >> > > > > >>>>>>>>>>> classes to > >> > > > > >>>>>>>>>>>>> make > >> > > > > >>>>>>>>>>>>>>>> it simpler to write test logic with > >> > > TopologyTestDriver: > >> > > > > >>>>>>>>>>>>>>>> > >> > > > > >>>>>>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-456 > >> > > > > >>>>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>> > >> > > > > >>>>>>>>>>> > >> > > > > >>>>>>>>> > >> > > > > >>>>>>>> > >> > > > > >>>>>>> > >> > > > > >>>>>> > >> > > > > >>>> > >> > > > > >>> > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> %3A+Helper+classes+to+make+it+simpler+to+write+test+logic+with+TopologyTestDriver > >> > > > > >>>>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>>>> The proposal in KIP-456 > >> > > > > >>>>>>>>>>>>>>>> < > >> > > > > >>>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>> > >> > > > > >>>>>>>>>>> > >> > > > > >>>>>>>>> > >> > > > > >>>>>>>> > >> > > > > >>>>>>> > >> > > > > >>>>>> > >> > > > > >>>> > >> > > > > >>> > >> > > > > >> > >> > > > > > >> > > > > >> > > > >> > > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP-456%3A+Helper+classes+to+make+it+simpler+to+write+test+logic+with+TopologyTestDriver > >> > > > > >>>>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>>>> was > >> > > > > >>>>>>>>>>>>>>>> to add alternate way to input and output topic, > >> but > >> > > > > >>>>>> this > >> > > > > >>>>>>>> KIP > >> > > > > >>>>>>>>>>> enhance > >> > > > > >>>>>>>>>>>>>>> those > >> > > > > >>>>>>>>>>>>>>>> classes and deprecate old functionality to make > >> > clear > >> > > > > >>>>>>>>> interface > >> > > > > >>>>>>>>>>> for > >> > > > > >>>>>>>>>>>>> test > >> > > > > >>>>>>>>>>>>>>>> writer to use. > >> > > > > >>>>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>>>> In current KIP-470 proposal, topic objects are > >> > created > >> > > > > >>>>>>> with > >> > > > > >>>>>>>>>>>>> topicName and > >> > > > > >>>>>>>>>>>>>>>> related serders. > >> > > > > >>>>>>>>>>>>>>>> public final <K, V> TestInputTopic<K, V> > >> > > > > >>>>>>>>>>> createInputTopic(final > >> > > > > >>>>>>>>>>>>>>> String > >> > > > > >>>>>>>>>>>>>>>> topicName, final Serde<K> keySerde, final > >> Serde<V> > >> > > > > >>>>>>>>> valueSerde); > >> > > > > >>>>>>>>>>>>>>>> public final <K, V> TestOutputTopic<K, V> > >> > > > > >>>>>>>>>>> createOutputTopic(final > >> > > > > >>>>>>>>>>>>>>> String > >> > > > > >>>>>>>>>>>>>>>> topicName, final Serde<K> keySerde, final > >> Serde<V> > >> > > > > >>>>>>>>> valueSerde); > >> > > > > >>>>>>>>>>>>>>>> One thing I wondered if there way to find out > >> topic > >> > > > > >>>>>>> related > >> > > > > >>>>>>>>> serde > >> > > > > >>>>>>>>>>>>> from > >> > > > > >>>>>>>>>>>>>>>> TopologyTestDriver topology, it would simply > >> > creation > >> > > > > >>>>>> of > >> > > > > >>>>>>>>> these > >> > > > > >>>>>>>>>>> Topic > >> > > > > >>>>>>>>>>>>>>>> objects: > >> > > > > >>>>>>>>>>>>>>>> public final <K, V> TestInputTopic<K, V> > >> > > > > >>>>>>>>>>> createInputTopic(final > >> > > > > >>>>>>>>>>>>>>> String > >> > > > > >>>>>>>>>>>>>>>> topicName); > >> > > > > >>>>>>>>>>>>>>>> public final <K, V> TestOutputTopic<K, V> > >> > > > > >>>>>>>>>>> createOutputTopic(final > >> > > > > >>>>>>>>>>>>>>> String > >> > > > > >>>>>>>>>>>>>>>> topicName); > >> > > > > >>>>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>>>> KIP-456 can be discarded if this KIP get > >> accepted. > >> > > > > >>>>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>>>>> Best Regards, > >> > > > > >>>>>>>>>>>>>>>> Jukka > >> > > > > >>>>>>>>>>>>>>> > >> > > > > >>>>>>>>>>>>> > >> > > > > >>>>>>>>>>> > >> > > > > >>>>>>>>> > >> > > > > >>>>>>>> > >> > > > > >>>>>>>> > >> > > > > >>>>>>>> -- > >> > > > > >>>>>>>> -- Guozhang > >> > > > > >>>>>>>> > >> > > > > >>>>>>> > >> > > > > >>>>>> > >> > > > > >>>>>> > >> > > > > >>>>>> -- > >> > > > > >>>>>> -- Guozhang > >> > > > > >>>>>> > >> > > > > >>>>> > >> > > > > >>>> > >> > > > > >>>> > >> > > > > >>> > >> > > > > >> > >> > > > > > > >> > > > > > >> > > > > > >> > > > > >> > > > >> > > >> > >> > >> -- > >> > >> Antony Stubbs > >> > >> Solutions Architect | Confluent > >> > >> +44 (0)7491 833 814 <+447491833814> > >> Follow us: Twitter <https://twitter.com/ConfluentInc> | blog > >> <http://www.confluent.io/blog> > >> > >
