Re: Topology with loops (intermediate topics) and potential bug in TopologyTestDriver

2019-09-12 Thread Adam Domanski
Hi John,

Thanks for explanation.

Yes, indeed, this is happening in my app. I also checked my state store
behind the output topic and there is always the latest value.

Is it possible that TestTopologyDriver can fully mimic Kafka Streams? Maybe
somehow records in the output topic can be ordered by offsets?

Regards,
Adam.


On Wed, 11 Sep 2019 at 18:09, John Roesler  wrote:

> Hi Adam,
>
> I haven't looked into your project yet, but just wanted to mention
> this to see if it explains your observations.
>
> TopologyTestDriver processes every input record fully and
> synchronously. So, when you pipe an input record in, it traverses all
> internal processing, including intermediate topics, until all results
> are written to sink topics (or the end of the topology is reached).
>
> This may indeed result in a different order of output records than
> you'd observe with Kafka Streams, which would instead write to the
> intermediate topic in one thread and then process it in a separate
> thread.
>
> Does that add up with what you're seeing?
>
> Thanks,
> -John
>
> On Wed, Sep 11, 2019 at 6:40 AM Adam Domanski  wrote:
> >
> > Hi,
> >
> > I probably found a bug in TopologyTestDriver for quite non-trivial Kafka
> > Streams topology.
> >
> > The streaming logic is the following: There is a concept of children and
> > parents. Children are aggregated under parent. Some children of master
> > parent can send poison pills to other parents. Such parents die but
> already
> > aggregated kids should go then to the master. Application remembers
> killed
> > parents in a local store. If a new kid comes, the store is checked and
> then
> > kid is rerouted to the master.
> >
> > I'm using intermediate topics in my topology as well as sometimes
> > tombstones.
> >
> > Here is Gradle (Java 11 + Kotlin) based project which demonstrate the
> > issue: https://github.com/czterocyty/kafka_streams_test_case
> >
> > You can see that few tests fail there as I expect that latest output
> Parent
> > records have the latest generation.
> >
> > So seems that TopologyTestDriver does not care about the order of output
> > records.
> >
> > Best regards,
> > Adam Domański
>


Re: Topology with loops (intermediate topics) and potential bug in TopologyTestDriver

2019-09-11 Thread John Roesler
Hi Adam,

I haven't looked into your project yet, but just wanted to mention
this to see if it explains your observations.

TopologyTestDriver processes every input record fully and
synchronously. So, when you pipe an input record in, it traverses all
internal processing, including intermediate topics, until all results
are written to sink topics (or the end of the topology is reached).

This may indeed result in a different order of output records than
you'd observe with Kafka Streams, which would instead write to the
intermediate topic in one thread and then process it in a separate
thread.

Does that add up with what you're seeing?

Thanks,
-John

On Wed, Sep 11, 2019 at 6:40 AM Adam Domanski  wrote:
>
> Hi,
>
> I probably found a bug in TopologyTestDriver for quite non-trivial Kafka
> Streams topology.
>
> The streaming logic is the following: There is a concept of children and
> parents. Children are aggregated under parent. Some children of master
> parent can send poison pills to other parents. Such parents die but already
> aggregated kids should go then to the master. Application remembers killed
> parents in a local store. If a new kid comes, the store is checked and then
> kid is rerouted to the master.
>
> I'm using intermediate topics in my topology as well as sometimes
> tombstones.
>
> Here is Gradle (Java 11 + Kotlin) based project which demonstrate the
> issue: https://github.com/czterocyty/kafka_streams_test_case
>
> You can see that few tests fail there as I expect that latest output Parent
> records have the latest generation.
>
> So seems that TopologyTestDriver does not care about the order of output
> records.
>
> Best regards,
> Adam Domański


Topology with loops (intermediate topics) and potential bug in TopologyTestDriver

2019-09-11 Thread Adam Domanski
Hi,

I probably found a bug in TopologyTestDriver for quite non-trivial Kafka
Streams topology.

The streaming logic is the following: There is a concept of children and
parents. Children are aggregated under parent. Some children of master
parent can send poison pills to other parents. Such parents die but already
aggregated kids should go then to the master. Application remembers killed
parents in a local store. If a new kid comes, the store is checked and then
kid is rerouted to the master.

I'm using intermediate topics in my topology as well as sometimes
tombstones.

Here is Gradle (Java 11 + Kotlin) based project which demonstrate the
issue: https://github.com/czterocyty/kafka_streams_test_case

You can see that few tests fail there as I expect that latest output Parent
records have the latest generation.

So seems that TopologyTestDriver does not care about the order of output
records.

Best regards,
Adam Domański