OK, if you want each consumer to process the same data, then simply point each consumer at your Kafka cluster and have each Consumer consume all data. There is no synchronization required between those two consumers.
In other words, what you want to do is fine. Please read the Kafka design doc if you have not done so: http://kafka.apache.org/design.html Philip On Sun, Apr 21, 2013 at 9:16 AM, Oleg Ruchovets <oruchov...@gmail.com> wrote: > I am on POC stage , so I can configure the producer to write in different > partitions. > But how it will help me to process the same data with two consumers. > > I try to get such effect: > I got the data and store it to Kafka. > > I have 2 consumers: > 1) for real time which consumes the data for example every 10 seconds. > 2) for move data to hdfs - for example every 1 hour. > > But in case I'll use 2 partitions , each consumer process part of the data > (50%). Does it correct? > I need that 2 consumers will produce 100% of the data. > > Please advice. > > > On Sun, Apr 21, 2013 at 12:00 PM, Philip O'Toole <phi...@loggly.com> wrote: > >> On Sun, Apr 21, 2013 at 8:53 AM, Oleg Ruchovets <oruchov...@gmail.com> >> wrote: >> > Hi Philip. >> > Does it mean to store the same data twice - each time to different >> > partition? I tried to save data only one time. Using two partitions means >> > to store data twice? >> >> No, I mean spreading the data across the two partitions, so 50% goes >> in one, and 50% goes in the other. Have your Producer always write to >> partition "-1", which will tell Kafka to select a partition at random >> for each message. >> >> Then one of the Consumers will consume partition 0, the other partition 1. >> >> > By the way I am using kafka 0.7.2. >> > >> > Thanks >> > Oleg. >> > >> > >> > On Sun, Apr 21, 2013 at 11:30 AM, Philip O'Toole <phi...@loggly.com> >> wrote: >> > >> >> Read the design doc on the Kafka site. >> >> >> >> The short answer is to use two partitions for your topic. >> >> >> >> Philip >> >> >> >> On Apr 21, 2013, at 12:37 AM, Oleg Ruchovets <oruchov...@gmail.com> >> wrote: >> >> >> >> > Hi, >> >> > I have one producer for kafka and have 2 consumers. >> >> > I want to consume produced events to hdfs and storm. Copy to hdfs I >> will >> >> do >> >> > every hour but to storm every 10 seconds. >> >> > >> >> > Question: Is it supported by kafka? Where can I read how to organize 1 >> >> > producer and 2 consumers? >> >> > >> >> > Thanks >> >> > Oleg. >> >> >>
