Hi
I have a KafkaStreams application that's pretty simple, and acts as a
repartitioner... It reads from input topics and send to output topics,
based on a input-to-output topics map. It has a custom Repartitioner that
will be responsible for assigning new partitions for the data in the output
topics.
The topology is simple:
protected Topology buildTopology(Map<String, String> topics) {
StreamsBuilder builder = new StreamsBuilder();
for (Entry<String, String> entry : topics.entrySet()) {
builder
.stream(entry.getKey(), Consumed.with(Serdes.ByteArray(),
Serdes.ByteArray()))
.to(entry.getValue(), Produced.with(Serdes.ByteArray(),
Serdes.ByteArray()));
}
return builder.build();
}
The resulting topolgy will look like this:
Sub-topology: 0
Source: KSTREAM-SOURCE-0000000000 (topics: [kafka.inputA])
--> KSTREAM-SINK-0000000001
Sink: KSTREAM-SINK-0000000001 (topic: kafka.inputA-repartitioned)
<-- KSTREAM-SOURCE-0000000000
Sub-topology: 1
Source: KSTREAM-SOURCE-0000000002 (topics: [kafka.inputB])
--> KSTREAM-SINK-0000000003
Sink: KSTREAM-SINK-0000000003 (topic: kafka.inputB-repartitioned)
<-- KSTREAM-SOURCE-0000000002
Sub-topology: 2
Source: KSTREAM-SOURCE-0000000004 (topics: [kafka.inputC])
--> KSTREAM-SINK-0000000005
Sink: KSTREAM-SINK-0000000005 (topic: kafka.inputC-repartitioned)
<-- KSTREAM-SOURCE-0000000004
... (about 25 diffent sub-topologies in total)
I am facing some inconsistent NPE in some environments, that I'll send in
the next e-mail not to polute this one anymore.
But I am wondering if I can really do that, and what would happen if I:
1) add a new sub-topolgy;
2) adding the new sub-topology breaks the order of the iteration;
Will any of the above operations require a reset to the KafkaStreams
application (or new app.id)?
Thanks
Murilo