The Kafka consumer offset key <https://github.com/apache/kafka/blob/trunk/core/src/main/resources/common/message/OffsetCommitKey.json> is (group, topic, partition)
On Fri, Nov 11, 2022 at 7:58 AM John Casey via user <user@beam.apache.org> wrote: > I haven't done this experimentally before, so take this with a grain of > salt, but... > > Kafka Group Ids are essentially used to track where a logical (aka > application level, not thread/machine level) producer / consumer is at. As > such, I think it would be fine to use just one group id, even when writing > to multiple topics > > On Thu, Nov 10, 2022 at 6:30 PM Evan Galpin <egal...@apache.org> wrote: > >> Hey folks, >> >> I can see in the docs for "withEOS"[1] in the KafkaIO#Write section that >> the sinkGroupId is recommended to be unique per job. I'm wondering about a >> case where a single job outputs to multiple topics. Would it be advisable >> to have a unique sinkGroupId per instance of KafkaIO#Write transform, or >> still only per job even if the job has multiple KafkaIO#Write? >> >> Thanks in advance! >> >> [1] >> https://beam.apache.org/releases/javadoc/2.41.0/org/apache/beam/sdk/io/kafka/KafkaIO.WriteRecords.html#withEOS-int-java.lang.String- >> >
