Hi Juan, Under normal usage, Kafka will maintain ordering within a partition without any extra work by you.
For 2, you can use .commitOffsetsInFinalize to only commit back to the source topic once the pipeline has persisted the message, at which point it may not be fully processed, but it is guaranteed that it will be processed. For 1, I would generally recommend against trying to write to a specific partition. This undermines the load balancing that Kafka enables with partitioning. I'd recommend separate topics instead. We don't support writing to specific partitions in the current IO. John On Thu, Apr 13, 2023 at 2:15 PM Juan Romero <[email protected]> wrote: > Hi Apache Beam team. I have been working on a POC for the company i'm > working for and Im using apache beam kafka connector to read from kafka > topic and write into other kafka topic. The source and target topic have 3 > partitions and is compulsory keep ordering by certain message keys. > Regarding it I have two questions: > > 1. How can I write an specific message to an specific kafka partition. > 2. How can we commit the message to the source topic only and only when > the pipeline had processed the message. > > I looking forward and hope you can help me with these doubts. >
