Hi Guozhang, I tested your backport branch, and it looks like it works fine. The same code that was producing the "Invalid topology building" error is working correctly against your branch, just like it is with trunk.
Mathieu On Sun, Aug 14, 2016 at 11:25 PM, Guozhang Wang <wangg...@gmail.com> wrote: > Hi Mathieu, > > I have a PR against 0.10.0 branch to backport the bug fix plus some > refactoring, feel free to try it out: > > > https://github.com/apache/kafka/pull/1735 > > > Guozhang > > On Wed, Aug 10, 2016 at 2:28 PM, Mathieu Fenniak < > mathieu.fenn...@replicon.com> wrote: > > > Hi Guozhang, > > > > Yes, it does seem to be fixed in trunk. Thanks. I should have tried > that, > > but I assumed that the recently released 0.10.0.1 would be pretty close > to > > trunk. I can see where that was mistaken, since 0.10.0 is quite > divergent > > from trunk. > > > > Mathieu > > > > > > On Wed, Aug 10, 2016 at 2:39 PM, Guozhang Wang <wangg...@gmail.com> > wrote: > > > > > Hello Mathieu, > > > > > > I think this issue is fixed in trunk but may get missed in the 0.10.0 > > > branch, could you try running your program from trunk to verify if it > is > > > the case? If yes we can consider backportting the hotfix from trunk to > > > 0.10.0 and have another bug fix release. > > > > > > > > > Guozhang > > > > > > On Wed, Aug 10, 2016 at 7:32 AM, Mathieu Fenniak < > > > mathieu.fenn...@replicon.com> wrote: > > > > > > > Hey there, Kafka Users, > > > > > > > > I'm trying to join two topics with Kafka Streams. The first topic > is a > > > > changelog of one object, and the second is a changelog of a related > > > > object. In order to join these tables, I'm grouping the second table > > by > > > a > > > > piece of data in it that indicates what record it is related to in > the > > > > first table. But I'm getting an unexpected error related to the > > > > repartitioning topic for the aggregated table: > > > > > > > > org.apache.kafka.streams.errors.TopologyBuilderException: Invalid > > > topology > > > > building: External source topic not found: > > > > *TableNumber2Aggregated-repartition* > > > > at > > > > org.apache.kafka.streams.processor.internals. > StreamPartitionAssignor. > > > > ensureCopartitioning(StreamPartitionAssignor.java:452) > > > > at > > > > org.apache.kafka.streams.processor.internals. > StreamPartitionAssignor. > > > > ensureCopartitioning(StreamPartitionAssignor.java:440) > > > > > > > > > > > > (Full exception: > > > > https://gist.github.com/mfenniak/11ca081191932fbb33a0c3cc32ad1686) > > > > > > > > It appears that the "TableNumber2Aggregated-repartition" topic *is* > > > > created > > > > in Kafka by the streams application, but the Kafka topic has a prefix > > > that > > > > matches my application id (timesheet-status). Perhaps something is > > > > prefixing the topic name, but it isn't being applied everywhere? > > > > > > > > $ ./kafka-topics.sh --zookeeper localhost --list > > > > TableNumber1 > > > > TableNumber2 > > > > __consumer_offsets > > > > timesheet-status-TableNumber2Aggregated-repartition > > > > > > > > > > > > Here's a sample that reproduces the issue (note, I've cut out all the > > > > actual mapping, grouping, and aggregating logic, but, this still > > > reproduces > > > > the error): > > > > > > > > public static TopologyBuilder createTopology() { > > > > KStreamBuilder builder = new KStreamBuilder(); > > > > > > > > KTable table1Mapped = builder.table(Serdes.String(), new > > > > JsonSerde(Map.class), "TableNumber1") > > > > .mapValues((value) -> null); > > > > > > > > KTable table2Aggregated = builder.table(Serdes.String(), new > > > > JsonSerde(Map.class), "TableNumber2") > > > > .groupBy((key, value) -> null) > > > > .aggregate(() -> null, (k, v, t) -> null, (k, v, t) -> > > null, > > > > new JsonSerde(Map.class), "TableNumber2Aggregated"); > > > > > > > > table1Mapped.join(table2Aggregated, (left, right) -> { > > > > LOG.debug("join"); > > > > return null; > > > > }); > > > > > > > > return builder; > > > > } > > > > > > > > I'm using the latest Kafka Streams release, 0.10.0.1. Any thoughts > on > > > how > > > > I could proceed to debug or workaround this? > > > > > > > > Thanks all, > > > > > > > > Mathieu > > > > > > > > > > > > > > > > -- > > > -- Guozhang > > > > > > > > > -- > -- Guozhang >
