Mark, ZK-based producer is intended to support adding brokers dynamically. However, because of a bug, in 0.7.x, you will need to make sure that a topic already exists on a new broker.
Thanks, Jun On Thu, Oct 4, 2012 at 10:17 AM, Mark Grabois <[email protected]>wrote: > Okay, I was able to produce messages directly to a broker which was part of > the quorum list in zookeeper.properties. Then, once I restart the ZK-based > producer, it successfully produces to that broker. However, I can't seem to > add a kafka broker to a quorum if no topics are registered on it. > > Should I be able to add a kafka node to a zk-based quorum and produce > messages to it immediately, without first producing directly to that node > and then restarting ZK-based producer? > > > On Wed, Oct 3, 2012 at 12:14 PM, Jun Rao <[email protected]> wrote: > > > Mark, > > > > If you want to use all brokers, you need to specify all brokers in > > broker.list? You may need to make sure that the topic is already created > on > > each broker first (by sending at least one message to it directly). > > > > Thanks, > > > > Jun > > > > On Tue, Oct 2, 2012 at 8:45 AM, Mark Grabois <[email protected] > > >wrote: > > > > > Hi Jun, sorry for the delay, I'd like to revisit the issue I was having > > > earlier > > > > > > using the Kafka Java API, I initialized a KafkaProducer with following > > > parameters: > > > > > > props.put("serializer.class", "kafka.serializer.StringEncoder"); > > > props.put("producer.type", "async"); > > > props.put("compression.codec", "1"); > > > > > > props.put("broker.list", "0:<host_ip>:9092"); //instead of zk.connect > > > string > > > producer = new Producer<Integer, String>(new ProducerConfig(props)); > > > producer.send(new ProducerData<Integer, String>(topicId, message)); > > > -- where topicId is a new topic, not an existing one > > > > > > this did not seem to produce a message to the third node, which is not > > > listening to my producer on 9092 (or any other port) > > > > > > However, I also tried to produce to my active brokers which are > correctly > > > receiving messages from other producers. This didn't work by just > > > specifying the host ip in the broker.list param of a newly initialized > > > Producer. Is there something else I need to configure for this to work? > > > > > > Thank you, > > > Mark > > > > > > On Wed, Sep 12, 2012 at 12:18 AM, Jun Rao <[email protected]> wrote: > > > > > > > This may be due to a bug in 0.7.1 with respect to ZK based producer. > > > Could > > > > you try sending 1 produce request to the new broker directly first > and > > > then > > > > try the ZK based producer? > > > > > > > > Thanks, > > > > > > > > Jun > > > > > > > > On Tue, Sep 11, 2012 at 9:41 AM, Mark Grabois < > > [email protected] > > > > >wrote: > > > > > > > > > I think it's a broker issue. The producers are connected to the > > > zookeeper > > > > > running on the node, and currently they are producing messages > > > correctly > > > > to > > > > > the other nodes. > > > > > > > > > > On Mon, Sep 10, 2012 at 11:32 PM, Jun Rao <[email protected]> > wrote: > > > > > > > > > > > Mark, > > > > > > > > > > > > Is this a broker side issue or a producer side issue? > > > > > > > > > > > > Thanks, > > > > > > > > > > > > Jun > > > > > > > > > > > > On Mon, Sep 10, 2012 at 9:23 AM, Mark Grabois < > > > > [email protected] > > > > > > >wrote: > > > > > > > > > > > > > Hi All, > > > > > > > > > > > > > > Currently I have two nodes with both zookeeper and kafka > > instances > > > > > > running > > > > > > > successfully, processing messages with connections to both > > producer > > > > and > > > > > > > consumer clients. However, when I try to add a third kafka+zk > > node > > > to > > > > > the > > > > > > > quorum, configured in the exact same way and having specified > all > > > > > broker > > > > > > > ids and server ips properly, kafka is unable to establish a > > > > connection > > > > > to > > > > > > > my producers on the default listening port (9092). Zookeeper > > > appears > > > > to > > > > > > be > > > > > > > connecting properly to the cluster, but not Kafka. > > > > > > > > > > > > > > Any recommendations on how to troubleshoot this? All nodes are > > > > running > > > > > > > kafka-0.7.1. > > > > > > > > > > > > > > Thank you, > > > > > > > Mark > > > > > > > > > > > > > > > > > > > > > > > > > > > >
