I can take a look at it this weekend :) - Prashanth
On Fri, Apr 20, 2012 at 10:17 AM, Jun Rao <jun...@gmail.com> wrote: > This jira may not be hard to fix since the required changes are likely > isolated in ZKBrokerPartitionInfo.getBrokerPartitionInfo(). Basically, we > just need to bootstrap a broker if it doesn't have a topic already. Anyone > interested in taking a shot at it? > > Thanks, > > Jun > > On Thu, Apr 19, 2012 at 7:17 PM, Bao Thai Ngo <baothai...@gmail.com> > wrote: > > > Hi there, > > > > Is there any news on this issue? > > > > Thanks, > > ~Thai > > > > On Fri, Mar 16, 2012 at 8:04 AM, Matt (Updated) (JIRA) <j...@apache.org > > >wrote: > > > > > > > > [ > > > > > > https://issues.apache.org/jira/browse/KAFKA-278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel > > ] > > > > > > Matt updated KAFKA-278: > > > ----------------------- > > > > > > Description: > > > There are two cases where correct partitioning fails for a new topic. > > > > > > Case 1: Topic exists on current Kafka cluster. A new broker is added to > > > the cluster. The new broker will never host partitions for the existing > > > topic. > > > > > > To reproduce: > > > 1) Create a cluster of brokers along with a ZooKeeper ensemble. > > > 2) Send messages for a topic to the cluster. > > > 3) Add a new broker to the cluster. > > > 4) New broker will never see the existing topic. > > > > > > Case 2: Topic does not exist on current Kafka cluster. Producer sends > > > messages to a new topic that did not previously exist in the cluster. > If, > > > during the producer session, one or more partitions are not created on > a > > > broker, the broker will never host those partitions. > > > > > > To reproduce: > > > 1) Create a cluster of brokers along with a ZooKeeper ensemble. > > > 2) Send messages to a new topic. > > > 3) Shut down the producer before the topic is created on at least one > > > broker. > > > 4) The broker that did not allocate the topic will never host the > topic. > > > > > > My guess(!) here is that when a new producer is created, it gets a list > > of > > > topics and partitions based on the current state of the brokers in the > > > cluster. Since some brokers are missing the topic, the producer will > > never > > > send messages to that broker and partitions will never be created. > > > > > > > > > Work around: > > > Manually create the topic/partition directories in the kafka logs > > > directory and reboot kafka. It will register the topic/partitions in > > > ZooKeeper. > > > > > > was: > > > There are two cases where correct partitioning fails for a new topic. > > > > > > Case 1: Topic exists on current Kafka cluster. A new broker is added to > > > the cluster. The new broker will never host partitions for the existing > > > topic. > > > > > > To reproduce: > > > 1) Create a cluster of brokers along with a ZooKeeper ensemble. > > > 2) Send messages for a topic to the cluster. > > > 3) Add a new broker to the cluster. > > > 4) New broker will never see the existing topic. > > > > > > Case 2: Topic does not exist on current Kafka cluster. Producer sends > > > messages to a new topic that did not previously exist in the cluster. > If, > > > during the producer session, one or more partitions are not created on > a > > > broker, the broker will never host those partitions. > > > > > > To reproduce: > > > 1) Create a cluster of brokers along with a ZooKeeper ensemble. > > > 2) Send messages to a new topic. > > > 3) Shut down the producer before the topic is created on at least one > > > broker. > > > 4) The broker that did not allocate the topic will never host the > topic. > > > > > > My guess(!) here is that when a new producer is created, it gets a list > > of > > > topics and partitions based on the current state of the brokers in the > > > cluster. Since some brokers are missing the topic, the producer will > > never > > > send messages to that broker and partitions will never be created. > > > > > > > > > Added a work around. > > > > > > > Issues partitioning a new topic > > > > ------------------------------- > > > > > > > > Key: KAFKA-278 > > > > URL: https://issues.apache.org/jira/browse/KAFKA-278 > > > > Project: Kafka > > > > Issue Type: Bug > > > > Components: core > > > > Affects Versions: 0.7 > > > > Reporter: Matt > > > > Priority: Minor > > > > > > > > There are two cases where correct partitioning fails for a new topic. > > > > Case 1: Topic exists on current Kafka cluster. A new broker is added > to > > > the cluster. The new broker will never host partitions for the existing > > > topic. > > > > To reproduce: > > > > 1) Create a cluster of brokers along with a ZooKeeper ensemble. > > > > 2) Send messages for a topic to the cluster. > > > > 3) Add a new broker to the cluster. > > > > 4) New broker will never see the existing topic. > > > > Case 2: Topic does not exist on current Kafka cluster. Producer sends > > > messages to a new topic that did not previously exist in the cluster. > If, > > > during the producer session, one or more partitions are not created on > a > > > broker, the broker will never host those partitions. > > > > To reproduce: > > > > 1) Create a cluster of brokers along with a ZooKeeper ensemble. > > > > 2) Send messages to a new topic. > > > > 3) Shut down the producer before the topic is created on at least one > > > broker. > > > > 4) The broker that did not allocate the topic will never host the > > topic. > > > > My guess(!) here is that when a new producer is created, it gets a > list > > > of topics and partitions based on the current state of the brokers in > the > > > cluster. Since some brokers are missing the topic, the producer will > > never > > > send messages to that broker and partitions will never be created. > > > > Work around: > > > > Manually create the topic/partition directories in the kafka logs > > > directory and reboot kafka. It will register the topic/partitions in > > > ZooKeeper. > > > > > > -- > > > This message is automatically generated by JIRA. > > > If you think it was sent incorrectly, please contact your JIRA > > > administrators: > > > > https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa > > > For more information on JIRA, see: > > http://www.atlassian.com/software/jira > > > > > > > > > > > >
