Redis supports check and set operations, and also has some functions for primitives like increment which could help solve your issue.
On Wed, Jul 27, 2016 at 1:06 PM, amir bahmanyari <[email protected]> wrote: > Hi Raghu, > Kafka is ok sending one record at a time. I proved it. > The issue is maintaining concurrency in Redis maps thats being > accessed+modified by parallel Pipelines at runtime. > And the variability of the apparent duplication is due to unpredictable > number of the parallel pipelines accessing a shared Redis map during each > separate run. > Thats my challenge at the moment. > Thanks for your help. > Amir > > > ------------------------------ > *From:* Raghu Angadi <[email protected]> > *To:* [email protected]; amir bahmanyari <[email protected]> > > *Sent:* Tuesday, July 26, 2016 2:34 PM > *Subject:* Re: Avoid reading duplicates from KafkaIO!!! > > > On Tue, Jul 26, 2016 at 2:29 PM, amir bahmanyari <[email protected]> > wrote: > > I know this is not Kafka forum ,but could Kafka be sending redundant > records? > > > no, equally unlikely. You could verify if you pushed duplicates to Kafka > by reading directly from kafka console consumer. > > e.g. : bin/kafka-console-consumer.sh --zookeeper localhost:2181 --topic > my_topi --from-beginning | grep .... > > >
