Re: Topology with loops (intermediate topics) and potential bug in TopologyTestDriver
Hi John, Thanks for explanation. Yes, indeed, this is happening in my app. I also checked my state store behind the output topic and there is always the latest value. Is it possible that TestTopologyDriver can fully mimic Kafka Streams? Maybe somehow records in the output topic can be ordered by offsets? Regards, Adam. On Wed, 11 Sep 2019 at 18:09, John Roesler wrote: > Hi Adam, > > I haven't looked into your project yet, but just wanted to mention > this to see if it explains your observations. > > TopologyTestDriver processes every input record fully and > synchronously. So, when you pipe an input record in, it traverses all > internal processing, including intermediate topics, until all results > are written to sink topics (or the end of the topology is reached). > > This may indeed result in a different order of output records than > you'd observe with Kafka Streams, which would instead write to the > intermediate topic in one thread and then process it in a separate > thread. > > Does that add up with what you're seeing? > > Thanks, > -John > > On Wed, Sep 11, 2019 at 6:40 AM Adam Domanski wrote: > > > > Hi, > > > > I probably found a bug in TopologyTestDriver for quite non-trivial Kafka > > Streams topology. > > > > The streaming logic is the following: There is a concept of children and > > parents. Children are aggregated under parent. Some children of master > > parent can send poison pills to other parents. Such parents die but > already > > aggregated kids should go then to the master. Application remembers > killed > > parents in a local store. If a new kid comes, the store is checked and > then > > kid is rerouted to the master. > > > > I'm using intermediate topics in my topology as well as sometimes > > tombstones. > > > > Here is Gradle (Java 11 + Kotlin) based project which demonstrate the > > issue: https://github.com/czterocyty/kafka_streams_test_case > > > > You can see that few tests fail there as I expect that latest output > Parent > > records have the latest generation. > > > > So seems that TopologyTestDriver does not care about the order of output > > records. > > > > Best regards, > > Adam Domański >
Re: Topology with loops (intermediate topics) and potential bug in TopologyTestDriver
Hi Adam, I haven't looked into your project yet, but just wanted to mention this to see if it explains your observations. TopologyTestDriver processes every input record fully and synchronously. So, when you pipe an input record in, it traverses all internal processing, including intermediate topics, until all results are written to sink topics (or the end of the topology is reached). This may indeed result in a different order of output records than you'd observe with Kafka Streams, which would instead write to the intermediate topic in one thread and then process it in a separate thread. Does that add up with what you're seeing? Thanks, -John On Wed, Sep 11, 2019 at 6:40 AM Adam Domanski wrote: > > Hi, > > I probably found a bug in TopologyTestDriver for quite non-trivial Kafka > Streams topology. > > The streaming logic is the following: There is a concept of children and > parents. Children are aggregated under parent. Some children of master > parent can send poison pills to other parents. Such parents die but already > aggregated kids should go then to the master. Application remembers killed > parents in a local store. If a new kid comes, the store is checked and then > kid is rerouted to the master. > > I'm using intermediate topics in my topology as well as sometimes > tombstones. > > Here is Gradle (Java 11 + Kotlin) based project which demonstrate the > issue: https://github.com/czterocyty/kafka_streams_test_case > > You can see that few tests fail there as I expect that latest output Parent > records have the latest generation. > > So seems that TopologyTestDriver does not care about the order of output > records. > > Best regards, > Adam Domański
Topology with loops (intermediate topics) and potential bug in TopologyTestDriver
Hi, I probably found a bug in TopologyTestDriver for quite non-trivial Kafka Streams topology. The streaming logic is the following: There is a concept of children and parents. Children are aggregated under parent. Some children of master parent can send poison pills to other parents. Such parents die but already aggregated kids should go then to the master. Application remembers killed parents in a local store. If a new kid comes, the store is checked and then kid is rerouted to the master. I'm using intermediate topics in my topology as well as sometimes tombstones. Here is Gradle (Java 11 + Kotlin) based project which demonstrate the issue: https://github.com/czterocyty/kafka_streams_test_case You can see that few tests fail there as I expect that latest output Parent records have the latest generation. So seems that TopologyTestDriver does not care about the order of output records. Best regards, Adam Domański
