This could be related to https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-IamusingtheZKbasedproducerin0.7andIseedataonlyproducedonsomeofthebrokers%2Cbutnotall%2Cwhy%3F
Thanks, Jun On Fri, Sep 20, 2013 at 11:06 AM, Suchi Amalapurapu <[email protected]>wrote: > Thx for your replies. Can u please clarify this further for me? > > In 0.7, if the partition count is per broker, shouldn't each broker have a > log directory for every topic? > I don't see that though... Only some topics have logs in different brokers. > > How is the consumer expected to handle this? My start sequence is as > follows > - start zookeeper > - start producer > - start broker1 > - start broker2 > - send message to topic1 > - send message to topic 2 > > log directory for topic1 is created on broker1 and broker2 > log directory for topic2 is created only on broker1 > > The consumer(kafka-storm spout) with a parallelism of one should ideally > discover the brokers via zookeeper and in this case should read the > messages in the correct sequence from both? > Suchi > > > On Fri, Sep 20, 2013 at 11:09 PM, Suchi Amalapurapu <[email protected] > >wrote: > > > Yes I understand that. I am letting the producer/consumer use zookeeper > to > > discover brokers. > > I can clearly see in the logs(brokers) that both the brokers create a new > > topic log for the same topic. > > > > The brokers are in different availability zones. Does that matter? > > Suchi > > > > > > On Fri, Sep 20, 2013 at 4:20 PM, Philip O'Toole <[email protected]> > wrote: > > > >> Seems to me you are confusing partitions and brokers. Partition count > has > >> nothing to do with the number of brokers to which a message a sent -- > just > >> the number of partitions into which that message will be split when it > >> gets > >> to a broker. > >> > >> You need to explicitly set the destination brokers in the Producer, not > >> via > >> setting "partition" count. > >> > >> Philip > >> > >> > >> On Fri, Sep 20, 2013 at 1:53 AM, Suchi Amalapurapu < > [email protected] > >> >wrote: > >> > >> > My current setup has 2 brokers, multiple topics, each topic with 1 > >> > partition, kafka 0.7.1 > >> > Some topics are logged onto multiple brokers inspite of partitions set > >> to > >> > 1. Kafka 0.7.1 does not have replication turned on. > >> > Why are the messages being logged onto multiple brokers when the > number > >> of > >> > partitions is set to 1? > >> > I have a consumer with parallelism set to 1 and its missing some of > >> these > >> > messages. > >> > Suchi > >> > > >> > > > > >
