I am on POC stage , so I can configure the producer to write in different partitions. But how it will help me to process the same data with two consumers.
I try to get such effect: I got the data and store it to Kafka. I have 2 consumers: 1) for real time which consumes the data for example every 10 seconds. 2) for move data to hdfs - for example every 1 hour. But in case I'll use 2 partitions , each consumer process part of the data (50%). Does it correct? I need that 2 consumers will produce 100% of the data. Please advice. On Sun, Apr 21, 2013 at 12:00 PM, Philip O'Toole <phi...@loggly.com> wrote: > On Sun, Apr 21, 2013 at 8:53 AM, Oleg Ruchovets <oruchov...@gmail.com> > wrote: > > Hi Philip. > > Does it mean to store the same data twice - each time to different > > partition? I tried to save data only one time. Using two partitions means > > to store data twice? > > No, I mean spreading the data across the two partitions, so 50% goes > in one, and 50% goes in the other. Have your Producer always write to > partition "-1", which will tell Kafka to select a partition at random > for each message. > > Then one of the Consumers will consume partition 0, the other partition 1. > > > By the way I am using kafka 0.7.2. > > > > Thanks > > Oleg. > > > > > > On Sun, Apr 21, 2013 at 11:30 AM, Philip O'Toole <phi...@loggly.com> > wrote: > > > >> Read the design doc on the Kafka site. > >> > >> The short answer is to use two partitions for your topic. > >> > >> Philip > >> > >> On Apr 21, 2013, at 12:37 AM, Oleg Ruchovets <oruchov...@gmail.com> > wrote: > >> > >> > Hi, > >> > I have one producer for kafka and have 2 consumers. > >> > I want to consume produced events to hdfs and storm. Copy to hdfs I > will > >> do > >> > every hour but to storm every 10 seconds. > >> > > >> > Question: Is it supported by kafka? Where can I read how to organize 1 > >> > producer and 2 consumers? > >> > > >> > Thanks > >> > Oleg. > >> >
