Hi there, Is there any news on this issue?
Thanks, ~Thai On Fri, Mar 16, 2012 at 8:04 AM, Matt (Updated) (JIRA) <j...@apache.org>wrote: > > [ > https://issues.apache.org/jira/browse/KAFKA-278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel] > > Matt updated KAFKA-278: > ----------------------- > > Description: > There are two cases where correct partitioning fails for a new topic. > > Case 1: Topic exists on current Kafka cluster. A new broker is added to > the cluster. The new broker will never host partitions for the existing > topic. > > To reproduce: > 1) Create a cluster of brokers along with a ZooKeeper ensemble. > 2) Send messages for a topic to the cluster. > 3) Add a new broker to the cluster. > 4) New broker will never see the existing topic. > > Case 2: Topic does not exist on current Kafka cluster. Producer sends > messages to a new topic that did not previously exist in the cluster. If, > during the producer session, one or more partitions are not created on a > broker, the broker will never host those partitions. > > To reproduce: > 1) Create a cluster of brokers along with a ZooKeeper ensemble. > 2) Send messages to a new topic. > 3) Shut down the producer before the topic is created on at least one > broker. > 4) The broker that did not allocate the topic will never host the topic. > > My guess(!) here is that when a new producer is created, it gets a list of > topics and partitions based on the current state of the brokers in the > cluster. Since some brokers are missing the topic, the producer will never > send messages to that broker and partitions will never be created. > > > Work around: > Manually create the topic/partition directories in the kafka logs > directory and reboot kafka. It will register the topic/partitions in > ZooKeeper. > > was: > There are two cases where correct partitioning fails for a new topic. > > Case 1: Topic exists on current Kafka cluster. A new broker is added to > the cluster. The new broker will never host partitions for the existing > topic. > > To reproduce: > 1) Create a cluster of brokers along with a ZooKeeper ensemble. > 2) Send messages for a topic to the cluster. > 3) Add a new broker to the cluster. > 4) New broker will never see the existing topic. > > Case 2: Topic does not exist on current Kafka cluster. Producer sends > messages to a new topic that did not previously exist in the cluster. If, > during the producer session, one or more partitions are not created on a > broker, the broker will never host those partitions. > > To reproduce: > 1) Create a cluster of brokers along with a ZooKeeper ensemble. > 2) Send messages to a new topic. > 3) Shut down the producer before the topic is created on at least one > broker. > 4) The broker that did not allocate the topic will never host the topic. > > My guess(!) here is that when a new producer is created, it gets a list of > topics and partitions based on the current state of the brokers in the > cluster. Since some brokers are missing the topic, the producer will never > send messages to that broker and partitions will never be created. > > > Added a work around. > > > Issues partitioning a new topic > > ------------------------------- > > > > Key: KAFKA-278 > > URL: https://issues.apache.org/jira/browse/KAFKA-278 > > Project: Kafka > > Issue Type: Bug > > Components: core > > Affects Versions: 0.7 > > Reporter: Matt > > Priority: Minor > > > > There are two cases where correct partitioning fails for a new topic. > > Case 1: Topic exists on current Kafka cluster. A new broker is added to > the cluster. The new broker will never host partitions for the existing > topic. > > To reproduce: > > 1) Create a cluster of brokers along with a ZooKeeper ensemble. > > 2) Send messages for a topic to the cluster. > > 3) Add a new broker to the cluster. > > 4) New broker will never see the existing topic. > > Case 2: Topic does not exist on current Kafka cluster. Producer sends > messages to a new topic that did not previously exist in the cluster. If, > during the producer session, one or more partitions are not created on a > broker, the broker will never host those partitions. > > To reproduce: > > 1) Create a cluster of brokers along with a ZooKeeper ensemble. > > 2) Send messages to a new topic. > > 3) Shut down the producer before the topic is created on at least one > broker. > > 4) The broker that did not allocate the topic will never host the topic. > > My guess(!) here is that when a new producer is created, it gets a list > of topics and partitions based on the current state of the brokers in the > cluster. Since some brokers are missing the topic, the producer will never > send messages to that broker and partitions will never be created. > > Work around: > > Manually create the topic/partition directories in the kafka logs > directory and reboot kafka. It will register the topic/partitions in > ZooKeeper. > > -- > This message is automatically generated by JIRA. > If you think it was sent incorrectly, please contact your JIRA > administrators: > https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa > For more information on JIRA, see: http://www.atlassian.com/software/jira > > >
