I haven't done this experimentally before, so take this with a grain of salt, but...
Kafka Group Ids are essentially used to track where a logical (aka application level, not thread/machine level) producer / consumer is at. As such, I think it would be fine to use just one group id, even when writing to multiple topics On Thu, Nov 10, 2022 at 6:30 PM Evan Galpin <egal...@apache.org> wrote: > Hey folks, > > I can see in the docs for "withEOS"[1] in the KafkaIO#Write section that > the sinkGroupId is recommended to be unique per job. I'm wondering about a > case where a single job outputs to multiple topics. Would it be advisable > to have a unique sinkGroupId per instance of KafkaIO#Write transform, or > still only per job even if the job has multiple KafkaIO#Write? > > Thanks in advance! > > [1] > https://beam.apache.org/releases/javadoc/2.41.0/org/apache/beam/sdk/io/kafka/KafkaIO.WriteRecords.html#withEOS-int-java.lang.String- >
