With topic.metadata.refresh.interval.ms=1000, the producer should refresh metadata and pick up the new partitions after 1 sec. Do you see metadata being refreshed? You may have to turn on the debug level logging.
Thanks, Jun On Mon, Jun 16, 2014 at 3:18 PM, Prakash Gowri Shankor < prakash.shan...@gmail.com> wrote: > Hi, > > I used the add partition functionality in create-topics to alter a previous > topic and increase the partitions. I noticed that after the new partitions > were added, they dont receive data immediately from the producer unless a > new producer is started up or the old producer is restarted. > Here is the sequence of steps: > 1) start cluster with 2 brokers. > 2) create topic with 2 partitions and replication factor 2. > 3) start 4 consumers in a group. 2 consumers are redundant. > 4) Start the performance producer to send messages. > 5) Now alter the topic to have 4 partitions. > 6) zookeeper immediately shows the new partitions > 7) On watching the partitions through Kafka Offset Monitor, I dont see any > messages going into the 2 new partitions. > 8) However, if i start a new performance producer or restart the one in 4) > I see that all the partitions are written to. > > Why does 7) happen ? Is this again related to the sticky partitioning > behavior change in 0.8.11 ? > My producer.properties has topic.metadata.refresh.interval.ms=1000 > > Thanks, > Prakash >
