+1 (binding). Verified quickstart and unit tests ran ok. On Sun, Feb 1, 2015 at 9:11 AM, Steven Wu <stevenz...@gmail.com> wrote:
> In Netflix, we have been using route53 DNS name as bootstrap servers in AWS > env. Basically, when a kafka broker start, we add it to route53 DNS name > for the cluster. this is like the VIP that Jay suggested. > > But we are also moving toward to use Eureka service registry for > bootstrapping. We are worried that if DNS name happens to resolve to a bad > broker. it might impact the bootstrap process/resiliency. We want to get a > list of brokers from Eureka to pass in as "bootstrap.servers". > > > > On Sun, Feb 1, 2015 at 5:30 AM, Jay Kreps <jay.kr...@gmail.com> wrote: > > > You may already know this but the producer doesn't require a complete > list > > of brokers in its config, it just requires the connection info for one > > active broker which it uses to discover the rest of the brokers. We allow > > you to specify multiple urls here for failover in cases where you aren't > > using a vip. So if you can put three brokers into the VIP for metadata > > bootstrapping you can still scale up and down the rest of the cluster. > > > > -Jay > > > > On Sun, Feb 1, 2015 at 12:17 AM, Alex The Rocker <alex.m3...@gmail.com> > > wrote: > > > > > Jun: > > > > > > You raise a very good question: let me explain why we use > > > Broker.getConnectionString(), so may be we'll get a supported way to > > > answer our need. > > > > > > We use Broker.getConnectionString() because we deploy "Kafka services" > > > in Amazon EC2 with the following architecture: > > > * Three VMs dedicated to Zookeeper processes > > > * At least two VMs with Kafka broker, but depending on load it can be > > > scaled to more broker VMs. Brokers self-register their address in > > > Zookeeper by serializing Broker objects in Zk. > > > > > > The VMs with Zookeeper have Elastic IPs = stable public IPs, > > > > > > These public IPs are fed to the various "Application services" which > > > rely on Kafka to stream their logs & monitoring data to our central > > > Hadoop system. > > > > > > Using zkclient and the above mentionned public zookeeper IPs, we get > > > the list of brokers registrered to a given "Kafka service": this is > > > where we unserializer Broker objects and then use > > > getConnectionString() to discover the brokers' addresses. Then, > > > brokers addresses are used to initialize the Kafka producer(s). > > > > > > The whole trick is that we cannot use Elastic IP (=stable IPs) for > > > Kafka VMs, because of their 'elastic" nature : we want to be able to > > > scale up / down the number of VMs with Kafka brokers. > > > > > > Now, we understand that using non public Kafka API is bad : we've been > > > broken when moving to 0.8.1.1, then again when moving to 0.8.2.0... > > > > > > So it's time to raise the right question: what would be the supported > > > way to configure our producers given our dynamic-IP-for-brokers > > > context? > > > > > > Thanks, > > > Alex. > > > > > > 2015-02-01 8:55 GMT+01:00 VERMEERBERGEN Alexandre > > > <alexandre.vermeerber...@3ds.com>: > > > > > > > > -----Original Message----- > > > > From: Jun Rao [mailto:j...@confluent.io] > > > > Sent: Sunday, February 01, 2015 3:03 > > > > To: us...@kafka.apache.org; kafka-clie...@googlegroups.com > > > > Cc: dev@kafka.apache.org > > > > Subject: Re: [VOTE] 0.8.2.0 Candidate 3 > > > > > > > > Hi, Alex, > > > > > > > > Thanks for testing RC3. > > > > > > > > Broker.connectionString() is actually not part of the public api for > > the > > > producer. Is there a particular reason that you need to use this api? > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > On Sat, Jan 31, 2015 at 1:53 PM, Alex The Rocker < > alex.m3...@gmail.com > > > > > > > wrote: > > > > > > > >> Hello, > > > >> > > > >> I have read Broker.scala source code, and I found the answer: > > > >> - With Kafka 0.8.1.1 we used Broker.getConnectionString() in our > Java > > > >> code. > > > >> - With Kafka 0.8.2.0, this method has been replaced by a 0-arity > > > >> method without the "get" prefix, so we have to change our Java code > to > > > >> call > > > >> Broker.connectionString() > > > >> > > > >> So despite binary compatibility is broken, we have a by-pass. > > > >> I hope this will help other people relying on this API... > > > >> > > > >> and I'm going to continue tests with 0.8.2 rc3.. > > > >> > > > >> Alex > > > >> > > > >> 2015-01-31 21:23 GMT+01:00 Alex The Rocker <alex.m3...@gmail.com>: > > > >> > > > >> > Hello, > > > >> > > > > >> > I ran my own tests made with kafka_2.10-0.8.1.1.tgz binaries with > > > >> > our > > > >> > application: > > > >> > > > > >> > 1st test: > > > >> > ====== > > > >> > replace all kafka .jar files in our application on consumming > side > > > >> > (without recompiling anything) > > > >> > => tests passed, OK > > > >> > > > > >> > 2nd test: > > > >> > ======= > > > >> > replace all kafka .jar files in our application on producubg > side > > > >> > (without recompiling anything) > > > >> > => KO, we get this error: > > > >> > > > > >> > 2015-01-31 20:54:00,094 [Timer-2] ERROR c.d.i.t.StdOutErrRedirect > - > > > >> > Exception in thread "Timer-2" > > > >> > 2015-01-31 20:54:00,111 [Timer-2] ERROR c.d.i.t.StdOutErrRedirect > - > > > >> > java.lang.NoSuchMethodError: > > > >> > kafka.cluster.Broker.getConnectionString()Ljava/lang/String; > > > >> > > > > >> > Which means that binary compatibility with 0.8.1.1 version has > been > > > >> broken. > > > >> > We use getConnectionString() to get Broker's zookeepers adresses, > > > >> > see > > > >> this > > > >> > answer from Neha: > > > >> > > > > >> > > > > >> > > > > >> > > http://mail-archives.apache.org/mod_mbox/kafka-users/201404.mbox/%3CCA > > > >> OG_4QYnWrB=tmrtcryf8-pdagy_cgfe_cxotqbclrkj2+x...@mail.gmail.com%3E > > > >> > > > > >> > If the kafka.cluster.Broker.getConnectionString() method has been > > > >> > removed with Kafka 0.8.2.0, then what is the suitable replacement > > for > > > it ? > > > >> > > > > >> > Thanks > > > >> > Alex > > > >> > > > > >> > > > > >> >> -----Original Message----- > > > >> >> From: Jun Rao [mailto:j...@confluent.io] > > > >> >> Sent: Thursday, January 29, 2015 6:22 > > > >> >> To: dev@kafka.apache.org; us...@kafka.apache.org; > > > >> >> kafka-clie...@googlegroups.com > > > >> >> Subject: [VOTE] 0.8.2.0 Candidate 3 > > > >> >> > > > >> >> This is the third candidate for release of Apache Kafka 0.8.2.0. > > > >> >> > > > >> >> Release Notes for the 0.8.2.0 release > > > >> >> > > > >> >> > > > >> > > https://people.apache.org/~junrao/kafka-0.8.2.0-candidate3/RELEASE_NOT > > > >> ES.html > > > >> >> > > > >> >> *** Please download, test and vote by Saturday, Jan 31, 11:30pm > PT > > > >> >> > > > >> >> Kafka's KEYS file containing PGP keys we use to sign the release: > > > >> >> http://kafka.apache.org/KEYS in addition to the md5, sha1 and > sha2 > > > >> >> (SHA256) checksum. > > > >> >> > > > >> >> * Release artifacts to be voted upon (source and binary): > > > >> >> https://people.apache.org/~junrao/kafka-0.8.2.0-candidate3/ > > > >> >> > > > >> >> * Maven artifacts to be voted upon prior to release: > > > >> >> https://repository.apache.org/content/groups/staging/ > > > >> >> > > > >> >> * scala-doc > > > >> >> > > https://people.apache.org/~junrao/kafka-0.8.2.0-candidate3/scaladoc > > > >> >> / > > > >> >> > > > >> >> * java-doc > > > >> >> > > https://people.apache.org/~junrao/kafka-0.8.2.0-candidate3/javadoc/ > > > >> >> > > > >> >> * The tag to be voted upon (off the 0.8.2 branch) is the 0.8.2.0 > > > >> >> tag > > > >> >> > > > >> > > https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=223ac42a7a > > > >> 2a0dab378cc411f4938a9cea1eb7ea > > > >> >> (commit 7130da90a9ee9e6fb4beb2a2a6ab05c06c9bfac4) > > > >> >> > > > >> >> /******************************************* > > > >> >> > > > >> >> Thanks, > > > >> >> > > > >> >> Jun > > > >> >> > > > > > > -- Thanks, Neha
