Hi Fernando, the "persistence" of a DStream is defined depending of the StorageLevel.
Window is not related to persistence: it's the processing of multiple DStream in one, a kind of "gather of DStreams". The transformation is applied on a "slide window". For instance, you define a window of 3 DStreams, and you apply the transformation on this window.
I hope it helps. Regards JB On 11/16/2015 07:38 PM, Fernando O. wrote:
Hi all, I was wondering if someone could give me a brief explanation or point me in the right direction in the code for where DStream persistence is donde. I'm looking at DStream.java but all it does is setting the StorageLevel, and neither WindowedDStream or ReducedWindowedDStream seem to change that behaviour a lot: WindowedDStream delegates on the parent and ReducedWindowedDStream calls super and delegates on an aggregated DStream So it seems like persist will only change the StorageLevel and then you need to compute the DStream to persist the inner RDDs So basically what I'm trying to verify is: persist is just lazy persistence method that sets the Storage level and the actual persistence takes place when you compute the DStream right?
-- Jean-Baptiste Onofré jbono...@apache.org http://blog.nanthrax.net Talend - http://www.talend.com --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org