What the throughput of processing and for how long do you need to remember duplicates?
You can take all the events, put them in an RDD, group by the key, and then process each key only once. But if you have a long running application where you want to check that you didn't see the same value before, and check that for every value, you probably need a key-value store, not RDD. On Sun, Jul 26, 2015 at 7:38 PM Shushant Arora <shushantaror...@gmail.com> wrote: > Hi > > I have a requirement for processing large events but ignoring duplicate at > the same time. > > Events are consumed from kafka and each event has a eventid. It may happen > that an event is already processed and came again at some other offset. > > 1.Can I use Spark RDD to persist processed events and then lookup with > this rdd (How to do lookup inside a RDD ?I have a > JavaPairRDD<eventid,timestamp> ) > while processing new events and if event is present in persisted rdd > ignore it , else process the even. Does rdd.lookup(key) on billion of > events will be efficient ? > > 2. update the rdd (Since RDD is immutable how to update it)? > > Thanks > >
