Requirements – then see my abstracted interpretation – what else do you need in terms of Requirements …:
“Suppose I have an RDD that is loaded from some file and then I also have a DStream that has data coming from some stream. I want to keep union some of the tuples from the DStream into my RDD. For this I can use something like this:” A formal requirements spec derived from the above - I think the actual requirement here is picking up and adding Specific (filtered) Messages from EVERY DStream RDD to the Batch RDD rather than “preserving” (on top of that all) messages from sliding window and adding them to the Batch RDD. Such requiremet should be defined as the Frequency of Updates to the Batch RDD and what these updates are e.g. specific filtered messages and then using dstream.window() can be made equal to that frequency Essentialy the update frequency can range from the filtered messages of Every Single DStream RDD to the filetered messages of a SLIDING WINDOW Secondly what do you call “mutable uniniong” That was his initial code var myRDD: RDD[(String, Long)] = sc.fromText... dstream.foreachRDD{ rdd => myRDD = myRDD.union(rdd.filter(myfilter)) } Here is how it looks when Persisting the result from evet union – supposed to produce NEW PERSTINET IMMUTABLE Batch RDD – why is that supposed to be less “stable/reliable” – what are the exact tectnical reasons for that var myRDD: RDD[(String, Long)] = sc.fromText... dstream.foreachRDD{ rdd => myRDD = myRDD.union(rdd.filter(myfilter)).cashe() } From: Gerard Maas [mailto:gerard.m...@gmail.com] Sent: Tuesday, July 7, 2015 1:55 PM To: Evo Eftimov Cc: Anand Nalya; spark users Subject: Re: Evo, I'd let the OP clarify the question. I'm not in position of clarifying his requirements beyond what's written on the question. Regarding window vs mutable union: window is a well-supported feature that accumulates messages over time. The mutable unioning of RDDs is bound to operational trouble as there're no warranties tied to data preservation and it's unclear how one can produce 'cuts' of that union ready to be served for some process/computation. Intuitively, it will 'explode' at some point. -kr, Gerard. On Tue, Jul 7, 2015 at 2:06 PM, Evo Eftimov <evo.efti...@isecc.com> wrote: spark.streaming.unpersist = false // in order for SStreaming to not drop the raw RDD data spark.cleaner.ttl = <some reasonable value in seconds> why is the above suggested provided the persist/vache operation on the constantly unioniuzed Batch RDD will have to be invoked anyway (after every union with DStream RDD), besides it will result in DStraeam RDDs accumulating in RAM unncesesarily for the duration of TTL re “A more reliable way would be to do dstream.window(...) for the length of time you want to keep the data and then union that data with your RDD for further processing using transform.” I think the actual requirement here is picking up and adding Specific Messages from EVERY DStream RDD to the Batch RDD rather than “preserving” messages from specific sliding window and adding them to the Batch RDD This should be defined as the Frequency of Updates to the Batch RDD and then using dstream.window() equal to that frequency Can you also elaborate why you consider the dstream.window approach more “reliable” From: Gerard Maas [mailto:gerard.m...@gmail.com] Sent: Tuesday, July 7, 2015 12:56 PM To: Anand Nalya Cc: spark users Subject: Re: Anand, AFAIK, you will need to change two settings: spark.streaming.unpersist = false // in order for SStreaming to not drop the raw RDD data spark.cleaner.ttl = <some reasonable value in seconds> Also be aware that the lineage of your union RDD will grow with each batch interval. You will need to break lineage often with cache(), and rely on the ttl for clean up. You will probably be in some tricky ground with this approach. A more reliable way would be to do dstream.window(...) for the length of time you want to keep the data and then union that data with your RDD for further processing using transform. Something like: dstream.window(Seconds(900), Seconds(900)).transform(rdd => rdd union otherRdd)... If you need an unbound amount of dstream batch intervals, considering writing the data to secondary storage instead. -kr, Gerard. On Tue, Jul 7, 2015 at 1:34 PM, Anand Nalya <anand.na...@gmail.com> wrote: Hi, Suppose I have an RDD that is loaded from some file and then I also have a DStream that has data coming from some stream. I want to keep union some of the tuples from the DStream into my RDD. For this I can use something like this: var myRDD: RDD[(String, Long)] = sc.fromText... dstream.foreachRDD{ rdd => myRDD = myRDD.union(rdd.filter(myfilter)) } My questions is that for how long spark will keep RDDs underlying the dstream around? Is there some configuratoin knob that can control that? Regards, Anand