Hi, I have one more question related to this setup. If we don't use zookeeper in single Kafka broker setup, what will be the loss in terms of functionality? I noticed in ProducerConfig.scala file, that if brokerlist is configured, then it ignore partitioner class configuration.
Thanks Subhash Agrawal -----Original Message----- From: Jamie Wang [mailto:[email protected]] Sent: Wednesday, December 12, 2012 2:26 PM To: [email protected] Subject: Design questions and your opinion and suggestions are appreciated Hi, We are incorporating Kafka as part of our centralized log aggregation service for a clustered server system. Our current thinking on the design as follows: Each of our clustered server will produce 8 different logs. A typical cluster will have about 4 or 5 nodes right now. Our design is to create a topic per each type of log and the number of partitions within each topic will equal to the number of nodes in the cluster. Therefore with a cluster of 5 nodes, we will have 8 topics and 5 partitions within each topic. On each of the node, we have a producer that has 8 threads. Each thread will always write to the same topic and partition. On the consumer side, we plan to have an aggregation client that has 8 threads. Each thread will pull from the same topic using round robin fashion to cycle through each partition within the topic. Couple of design questions we would like to hear your opinion and any suggestions are appreciated! 1) If we dynamically adds another node into the cluster, base on our design above, we would want to create a new partition and add to each of the 8 topics. Is this doable and how do I do that? 2) Can a producer send message to a specific partition? Base on our limited understanding so far, in a clustered kafka system, a producer sends message to the cluster, how does the messages being distributed among different brokers/partitions? 3) In our system, we also have a c++ process manager that's monitoring over kafka broker process. is there a heartbeat in the broker that we can ping? Or where can I find the C++ API? 4) I saw in kafka.server.KafkaServer has a shutdown method. Is there a command port that I can send a shutdown command or how to trigger this shutdown method from an external process such as our process monitor. 5) We are thinking if in the future our cluster grow much larger. We possibly have to add additional kafka broker. What are the parameters we should look for to determine at that point we should add another broker? Thanks in advance for your time and help. Jamie
