Hi
I have a KafkaStreams application that's pretty simple, and acts as a
repartitioner... It reads from input topics and send to output topics,
based on a input-to-output topics map. It has a custom Repartitioner that
will be responsible for assigning new partitions for the data in the output
topics.

The topology is simple:
  protected Topology buildTopology(Map<String, String> topics) {
    StreamsBuilder builder = new StreamsBuilder();
    for (Entry<String, String> entry : topics.entrySet()) {
      builder
          .stream(entry.getKey(), Consumed.with(Serdes.ByteArray(),
Serdes.ByteArray()))
          .to(entry.getValue(), Produced.with(Serdes.ByteArray(),
Serdes.ByteArray()));
    }
    return builder.build();
  }

The resulting topolgy will look like this:
Sub-topology: 0
 Source: KSTREAM-SOURCE-0000000000 (topics: [kafka.inputA])
   --> KSTREAM-SINK-0000000001
 Sink: KSTREAM-SINK-0000000001 (topic: kafka.inputA-repartitioned)
   <-- KSTREAM-SOURCE-0000000000

Sub-topology: 1
 Source: KSTREAM-SOURCE-0000000002 (topics: [kafka.inputB])
   --> KSTREAM-SINK-0000000003
 Sink: KSTREAM-SINK-0000000003 (topic: kafka.inputB-repartitioned)
   <-- KSTREAM-SOURCE-0000000002

Sub-topology: 2
 Source: KSTREAM-SOURCE-0000000004 (topics: [kafka.inputC])
   --> KSTREAM-SINK-0000000005
 Sink: KSTREAM-SINK-0000000005 (topic: kafka.inputC-repartitioned)
   <-- KSTREAM-SOURCE-0000000004
... (about 25 diffent sub-topologies in total)

I am facing some inconsistent NPE in some environments, that I'll send in
the next e-mail not to polute this one anymore.

But I am wondering if I can really do that, and what would happen if I:
1) add a new sub-topolgy;
2) adding the new sub-topology breaks the order of the iteration;

Will any of the above operations require a reset to the KafkaStreams
application (or new app.id)?

Thanks
Murilo

Reply via email to