Will and Garvit, you can use a load balancer with health checks for this purpose.
Ryanne On Wed, Aug 28, 2019, 6:09 PM Will Weber <rwawe...@gmail.com> wrote: > Apologies for piggybacking on a thread, figured the discussion was pretty > relevant to a thought I had kicking around my brain. > > In the event of complete failure or sustained loss of connectivity of the > first cluster, could the secondary cluster act as a failover for a given > configuration? > > Assuming the respective clusters are set up like the following: > > Cluster A > brokerA01:9092 > brokerA02:9092 > brokerA03:9092 > > Cluster B > brokerB01:9092 > brokerB02:9092 > brokerB03:9092 > > And the corresponding producer's properties line contained something like: > > > bootstrap.servers="brokerA01:9092,brokerA02:9092,brokerA03:9092,brokerB01:9092,brokerB02:9092,brokerB03:9092" > > My initial theory is that a failover action would respond like: > 1. Attempting to reach brokerA01:9092 with a metadata fetch, cannot reach, > attempt to reach next broker > 2. Attempting to reach brokerA02:9092 with a metadata fetch, cannot reach, > attempt to reach next broker > 3. Attempting to reach brokerA03:9092 with a metadata fetch, cannot reach, > attempt to reach next broker > 4. Attempting to reach brokerB01:9092 with a metadata fetch, able to reach, > begin exchange with reachable cluster > > I have a feeling that this would not work, but I don't really have any > evidence to substantiate it at this point. > > Anyone have any experience on this sort of a situation? > The intended goal is more to ensure that the messages end up on *some > *cluster, not > necessarily a particular cluster. > > Best, > > On Wed, Aug 21, 2019 at 3:28 AM David Jacot <dja...@confluent.io> wrote: > > > Hello, > > > > As of today, the producer is able to talk to only one Kafka cluster. > > *bootstrap.servers* is to provide a list of brokers belonging to the same > > cluster to ensure at least one is available. > > > > In your case, you have to replicate the data from the first cluster to > the > > second cluster with mirror maker for instance. > > > > Best, > > David > > > > On Mon, Aug 12, 2019 at 11:13 AM lsroudi abdel <lsro...@gmail.com> > wrote: > > > > > I guess it more about replication, you can push your data in one > choosen > > > cluster and replicate your data to the second one by using mirror maker > > or > > > confluent replicator or the new open sourced project by LinkedIn, > > > "Brooklin" > > > > > > Le lun. 12 août 2019 à 10:19, Garvit Sharma <eng.gar...@gmail.com> a > > > écrit : > > > > > > > Thanks, Isroudi. But in my usecase there are two separate zookeeper > as > > > > well. Let me know how would it handle. > > > > > > > > On Mon, Aug 12, 2019 at 1:45 PM lsroudi abdel <lsro...@gmail.com> > > wrote: > > > > > > > > > If it is two cluster it mean you have two zookeeper, I guess you > > can't > > > do > > > > > that, I f you have one zookeeper it will be ok, in the case you > have > > > one > > > > > zookeeper it's just an architecture with replica across two data > > > center, > > > > I > > > > > hope it clear for you > > > > > > > > > > Le lun. 12 août 2019 à 09:22, Garvit Sharma <eng.gar...@gmail.com> > a > > > > > écrit : > > > > > > > > > > > Hi, > > > > > > > > > > > > I have 2 kafka clusters in different data-centers so can I > provide > > > the > > > > > dns > > > > > > hostnames of both the clusters separated by comma in > > > > *bootstrap.servers* > > > > > > key in producer config ? > > > > > > > > > > > > If I provide two different clusters in *bootstrap.servers *then > how > > > > would > > > > > > the events get published ? > > > > > > Would events get published to both the clusters or either of them > > > > > randomly > > > > > > ? > > > > > > > > > > > > Looking forward to hearing from you. > > > > > > > > > > > > Thanks > > > > > > > > > > > > > > > > > > > > >
