You may already know this but the producer doesn't require a complete list of brokers in its config, it just requires the connection info for one active broker which it uses to discover the rest of the brokers. We allow you to specify multiple urls here for failover in cases where you aren't using a vip. So if you can put three brokers into the VIP for metadata bootstrapping you can still scale up and down the rest of the cluster.
-Jay On Sun, Feb 1, 2015 at 12:17 AM, Alex The Rocker <alex.m3...@gmail.com> wrote: > Jun: > > You raise a very good question: let me explain why we use > Broker.getConnectionString(), so may be we'll get a supported way to > answer our need. > > We use Broker.getConnectionString() because we deploy "Kafka services" > in Amazon EC2 with the following architecture: > * Three VMs dedicated to Zookeeper processes > * At least two VMs with Kafka broker, but depending on load it can be > scaled to more broker VMs. Brokers self-register their address in > Zookeeper by serializing Broker objects in Zk. > > The VMs with Zookeeper have Elastic IPs = stable public IPs, > > These public IPs are fed to the various "Application services" which > rely on Kafka to stream their logs & monitoring data to our central > Hadoop system. > > Using zkclient and the above mentionned public zookeeper IPs, we get > the list of brokers registrered to a given "Kafka service": this is > where we unserializer Broker objects and then use > getConnectionString() to discover the brokers' addresses. Then, > brokers addresses are used to initialize the Kafka producer(s). > > The whole trick is that we cannot use Elastic IP (=stable IPs) for > Kafka VMs, because of their 'elastic" nature : we want to be able to > scale up / down the number of VMs with Kafka brokers. > > Now, we understand that using non public Kafka API is bad : we've been > broken when moving to 0.8.1.1, then again when moving to 0.8.2.0... > > So it's time to raise the right question: what would be the supported > way to configure our producers given our dynamic-IP-for-brokers > context? > > Thanks, > Alex. > > 2015-02-01 8:55 GMT+01:00 VERMEERBERGEN Alexandre > <alexandre.vermeerber...@3ds.com>: > > > > -----Original Message----- > > From: Jun Rao [mailto:j...@confluent.io] > > Sent: Sunday, February 01, 2015 3:03 > > To: us...@kafka.apache.org; kafka-clie...@googlegroups.com > > Cc: dev@kafka.apache.org > > Subject: Re: [VOTE] 0.8.2.0 Candidate 3 > > > > Hi, Alex, > > > > Thanks for testing RC3. > > > > Broker.connectionString() is actually not part of the public api for the > producer. Is there a particular reason that you need to use this api? > > > > Thanks, > > > > Jun > > > > On Sat, Jan 31, 2015 at 1:53 PM, Alex The Rocker <alex.m3...@gmail.com> > > wrote: > > > >> Hello, > >> > >> I have read Broker.scala source code, and I found the answer: > >> - With Kafka 0.8.1.1 we used Broker.getConnectionString() in our Java > >> code. > >> - With Kafka 0.8.2.0, this method has been replaced by a 0-arity > >> method without the "get" prefix, so we have to change our Java code to > >> call > >> Broker.connectionString() > >> > >> So despite binary compatibility is broken, we have a by-pass. > >> I hope this will help other people relying on this API... > >> > >> and I'm going to continue tests with 0.8.2 rc3.. > >> > >> Alex > >> > >> 2015-01-31 21:23 GMT+01:00 Alex The Rocker <alex.m3...@gmail.com>: > >> > >> > Hello, > >> > > >> > I ran my own tests made with kafka_2.10-0.8.1.1.tgz binaries with > >> > our > >> > application: > >> > > >> > 1st test: > >> > ====== > >> > replace all kafka .jar files in our application on consumming side > >> > (without recompiling anything) > >> > => tests passed, OK > >> > > >> > 2nd test: > >> > ======= > >> > replace all kafka .jar files in our application on producubg side > >> > (without recompiling anything) > >> > => KO, we get this error: > >> > > >> > 2015-01-31 20:54:00,094 [Timer-2] ERROR c.d.i.t.StdOutErrRedirect - > >> > Exception in thread "Timer-2" > >> > 2015-01-31 20:54:00,111 [Timer-2] ERROR c.d.i.t.StdOutErrRedirect - > >> > java.lang.NoSuchMethodError: > >> > kafka.cluster.Broker.getConnectionString()Ljava/lang/String; > >> > > >> > Which means that binary compatibility with 0.8.1.1 version has been > >> broken. > >> > We use getConnectionString() to get Broker's zookeepers adresses, > >> > see > >> this > >> > answer from Neha: > >> > > >> > > >> > > >> http://mail-archives.apache.org/mod_mbox/kafka-users/201404.mbox/%3CCA > >> OG_4QYnWrB=tmrtcryf8-pdagy_cgfe_cxotqbclrkj2+x...@mail.gmail.com%3E > >> > > >> > If the kafka.cluster.Broker.getConnectionString() method has been > >> > removed with Kafka 0.8.2.0, then what is the suitable replacement for > it ? > >> > > >> > Thanks > >> > Alex > >> > > >> > > >> >> -----Original Message----- > >> >> From: Jun Rao [mailto:j...@confluent.io] > >> >> Sent: Thursday, January 29, 2015 6:22 > >> >> To: dev@kafka.apache.org; us...@kafka.apache.org; > >> >> kafka-clie...@googlegroups.com > >> >> Subject: [VOTE] 0.8.2.0 Candidate 3 > >> >> > >> >> This is the third candidate for release of Apache Kafka 0.8.2.0. > >> >> > >> >> Release Notes for the 0.8.2.0 release > >> >> > >> >> > >> https://people.apache.org/~junrao/kafka-0.8.2.0-candidate3/RELEASE_NOT > >> ES.html > >> >> > >> >> *** Please download, test and vote by Saturday, Jan 31, 11:30pm PT > >> >> > >> >> Kafka's KEYS file containing PGP keys we use to sign the release: > >> >> http://kafka.apache.org/KEYS in addition to the md5, sha1 and sha2 > >> >> (SHA256) checksum. > >> >> > >> >> * Release artifacts to be voted upon (source and binary): > >> >> https://people.apache.org/~junrao/kafka-0.8.2.0-candidate3/ > >> >> > >> >> * Maven artifacts to be voted upon prior to release: > >> >> https://repository.apache.org/content/groups/staging/ > >> >> > >> >> * scala-doc > >> >> https://people.apache.org/~junrao/kafka-0.8.2.0-candidate3/scaladoc > >> >> / > >> >> > >> >> * java-doc > >> >> https://people.apache.org/~junrao/kafka-0.8.2.0-candidate3/javadoc/ > >> >> > >> >> * The tag to be voted upon (off the 0.8.2 branch) is the 0.8.2.0 > >> >> tag > >> >> > >> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=223ac42a7a > >> 2a0dab378cc411f4938a9cea1eb7ea > >> >> (commit 7130da90a9ee9e6fb4beb2a2a6ab05c06c9bfac4) > >> >> > >> >> /******************************************* > >> >> > >> >> Thanks, > >> >> > >> >> Jun > >> >> >
