If you read the streaming programming guide, you'll notice that Spark does
not do real streaming but emulates it with a so-called mini-batching
approach. Let's say you want to work with a continuous stream of incoming
events from a computing centre:
Batch interval:
That's the basic heartbeat of your streaming application. If you set this
to 1 second, Spark will create a RDD every second containing the events of
that second. That's your mini-batch of data.
Windowing:
That's a way to do aggregations on your streaming data. Let's say you want
to have a summary of how many warnings your system produced in the last
hour. Then you would use a windowed reduce with a window size of 1h.
Sliding:
This tells Spark how often to perform your windowed operation. If you would
set this to 1h as well, you would aggregate your data stream to consecutive
1h windows of data - no overlap. You could also tell spark to create your
1h aggregation 2 times a day only by setting the sliding interval to 12h.
Or you could tell Spark to create a 1h aggregation every 30 min. Then each
data window overlaps with the previous window of course.
I recommend to carefully read the programming guide- it explains these
concepts pretty well.
https://spark.apache.org/docs/latest/streaming-programming-guide.html
Regards,
Jeff
2015-02-26 18:51 GMT+01:00 Hafiz Mujadid hafizmujadi...@gmail.com:
Can somebody explain the difference between
batchinterval,windowinterval and window sliding interval with example.
If there is any real time use case of using these parameters?
Thanks
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/spark-streaming-batchinterval-windowinterval-and-window-sliding-interval-difference-tp21829.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
