Hi! > On 27 Dec 2016, at 19:35, radai <radai.rosenbl...@gmail.com> wrote: > > IIUC if you replicate from a single source cluster to a single target > cluster, the topic has the same number of partitions on both, and no one > writes directly to the target cluster (so master --> slave) the offsets > would be preserved. >
Yes, exactly. When you 1) create topic with the same number of partitions on both master and slave clusters 2) write only to master 3) replicate partition to partition from master to slave - in this case the offsets will be preserved. However, you usually already have cluster that works and want to replicate some topics to another one. IMHO, in this scenario there should be a way to make message offsets equal on both clusters. > but in the general case - how would you handle the case where multiple > producers "claim" the same offset ? The same way as Kafka handles concurrent produce requests for the same partition - produce requests for partition are serialized. If the next produce request “overlaps” with previous one, it fails. > > > On Mon, Dec 26, 2016 at 4:52 AM, Andrey L. Neporada < > anepor...@yandex-team.ru> wrote: > >> Hi all! >> >> Suppose you have two Kafka clusters and want to replicate topics from >> primary cluster to secondary one. >> It would be very convenient for readers if the message offsets for >> replicated topics would be the same as for primary topics. >> >> As far as I know, currently there is no way to achieve this. >> I wonder is it possible/reasonable to add message offset to ProduceRequest? >> >> >> — >> Andrey Neporada >> >> >> >>
