I'm assuming for this auto-detection of partitions is only with the ZookeeperConsumerConnector and not the ConsumerConnector?
On Aug 29, 2013, at 1:04 PM, Philip O'Toole <phi...@loggly.com> wrote: > On Thu, Aug 29, 2013 at 11:09 AM, Mark <static.void....@gmail.com> wrote: > >> 1) Should a producer be aware of which broker to write to or is this >> somehow managed by Kafka itself. For example, If I have 2 brokers with a >> configured partition size of 1 will my messages be written in a round-robin >> type of fashion to each broker or is this completed dependent on which >> broker the consume connects to? >> > > Sounds like a question regarding the Producer, which I am not familiar > with. We use our own Producers here, written in C++. > > >> >> 2) Similar question goes for a writing to a topic that has multiple >> partitions. Are the writes randomly load-balanced between partitions or is >> this something the client needs to do? >> > > It depends. Producers can specify the partition, or use a partition of > '-1', which instructs Kafka to randomly choose a partition. > > >> >> 3) If I have 1 topic with 1 partition and 2 brokers how does consuming >> messages work. For example, if my consumer connects to broker 1 how does it >> consume messages written to broker 2? >> > > Don't think of partitions on one broker and partitions on another broker. > Simply think of the aggregate partition count across the cluster. The > high-level consumer that comes with Kafka will learn where all the > partitions are for a given topic via Zookeeper, create connections to the > Kafka brokers as needed, and drain all the partitions. > > >> >> Thanks >> >>
