Here's what I understand:
batchDuration: How often should the streaming context update? how many seconds of data should each dstream contain? windowDuration: What size windows are you looking for from this dstream? slideDuration: Once I've given you that slice, how many units forward do you want me to move to give you the next one? On Wed, Jul 16, 2014 at 11:28 AM, hsy...@gmail.com <hsy...@gmail.com> wrote: > When I'm reading the API of spark streaming, I'm confused by the 3 > different durations > > StreamingContext(conf: SparkConf > <http://spark.apache.org/docs/latest/api/scala/org/apache/spark/SparkConf.html> > , batchDuration: Duration > <http://spark.apache.org/docs/latest/api/scala/org/apache/spark/streaming/Duration.html> > ) > > DStream window(windowDuration: Duration > <http://spark.apache.org/docs/latest/api/scala/org/apache/spark/streaming/Duration.html> > , slideDuration: Duration > <http://spark.apache.org/docs/latest/api/scala/org/apache/spark/streaming/Duration.html> > ): DStream > <http://spark.apache.org/docs/latest/api/scala/org/apache/spark/streaming/dstream/DStream.html> > [T] > > > Can anyone please explain these 3 different durations > > > Best, > Siyuan >
