Hi all, I'm sure this must have been solved already, but I can't see anything obvious.
Using Spark Streaming, I'm trying to execute a transform function on a DStream at short batch intervals (e.g. 1 second), but only write the resulting data to disk using saveAsTextFiles in a larger batch after a longer delay (say 60 seconds). I thought the ReceiverInputDStream window function might be a good help here, but instead, applying it to a transformed DStream causes the transform function to only execute at the end of the window too. Has anyone got a solution to this? Thanks, Ewan
