You could enforce the evaluation of the transformed DStream by putting a
dummy output operation on it, and then do the windowing.
transformedDStream.foreachRDD { _.count() } // to enforce evaluation of
the trnasformation
transformedDStream.window(...).foreachRDD( rdd => ... }
On Thu, Dec 31,
Yeah it's awkward, the transforms being done are fairly time sensitive, so I
don't want them to wait 60 seconds or more.
I might have to move the code from a transform into a custom receiver instead,
so they'll be processed outside the window length. A buffered writer is a good
idea too,
Hi Ewan,Transforms are definitions of what needs to be done - they don't
execute until and action is triggered. For what you want, I think you might
need to have an action that writes out rdds to some sort of buffered writer.
-Ashic.
From: ewan.le...@realitymine.com
To: user@spark.apache.org
