Kafka transaction between 2 kafka clusters
Hello, I would like to know if there is a Java client that would allow me to consume from topics on a cluster A and produce to topics on a cluster B with exactly-once semantics. My understanding of the Kafka transactions is that on the paper it could work, but the kafka java client assumes both are running on the same cluster. Is it something do-able? If it is and nothing exists yet, would it be an interesting contribution?
Re: Kafka transaction between 2 kafka clusters
Hello Emmanuel, Yes I think it is do-able technically. Note that it means the offsets of cluster A would be stored on cluster B and hence upon restarting one need to talk to cluster B in order to get the committed position in cluster A. Guozhang On Thu, May 9, 2019 at 11:58 AM Emmanuel wrote: > Hello, > > I would like to know if there is a Java client that would allow me to > consume from topics on a cluster A and produce to topics on a cluster B > with exactly-once semantics. My understanding of the Kafka transactions is > that on the paper it could work, but the kafka java client assumes both are > running on the same cluster. Is it something do-able? If it is and nothing > exists yet, would it be an interesting contribution? > -- -- Guozhang
Re: Kafka transaction between 2 kafka clusters
Thanks! I wonder if this is a bit far-fetched as no one seems to do this at the moment. On Fri, May 10, 2019 at 12:50 AM Guozhang Wang wrote: > Hello Emmanuel, > > Yes I think it is do-able technically. Note that it means the offsets of > cluster A would be stored on cluster B and hence upon restarting one need > to talk to cluster B in order to get the committed position in cluster A. > > > Guozhang > > > On Thu, May 9, 2019 at 11:58 AM Emmanuel wrote: > > > Hello, > > > > I would like to know if there is a Java client that would allow me to > > consume from topics on a cluster A and produce to topics on a cluster B > > with exactly-once semantics. My understanding of the Kafka transactions > is > > that on the paper it could work, but the kafka java client assumes both > are > > running on the same cluster. Is it something do-able? If it is and > nothing > > exists yet, would it be an interesting contribution? > > > > > -- > -- Guozhang >
Re: Kafka transaction between 2 kafka clusters
MirrorMaker 2.0 stores the offsets of one cluster in another. So, you can read the offsets from the same cluster once this KIP is implemented. https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0#KIP-382:MirrorMaker2.0-RemoteClusterUtils On Fri, May 10, 2019 at 12:29 PM Emmanuel wrote: > Thanks! I wonder if this is a bit far-fetched as no one seems to do this at > the moment. > > On Fri, May 10, 2019 at 12:50 AM Guozhang Wang wrote: > > > Hello Emmanuel, > > > > Yes I think it is do-able technically. Note that it means the offsets of > > cluster A would be stored on cluster B and hence upon restarting one need > > to talk to cluster B in order to get the committed position in cluster A. > > > > > > Guozhang > > > > > > On Thu, May 9, 2019 at 11:58 AM Emmanuel wrote: > > > > > Hello, > > > > > > I would like to know if there is a Java client that would allow me to > > > consume from topics on a cluster A and produce to topics on a cluster B > > > with exactly-once semantics. My understanding of the Kafka transactions > > is > > > that on the paper it could work, but the kafka java client assumes both > > are > > > running on the same cluster. Is it something do-able? If it is and > > nothing > > > exists yet, would it be an interesting contribution? > > > > > > > > > -- > > -- Guozhang > > >
