Hi, Just wanted opinions on this.
In Spark streaming the parameter val ssc = new StreamingContext(sparkConf, Seconds(n)) defines the batch or sample interval for the incoming streams In addition there is windows Length // window length - The duration of the window below that must be multiple of batch interval n in = > StreamingContext(sparkConf, Seconds(n)) val windowLength = L And fibally the sliding interval // sliding interval - The interval at which the window operation is performed val slidingInterval = I OK so as given the windowLength L = multiples of n and the slidingInterval has to be consistent to ensure that we can the head and tail of the window. So as a heuristic approach for a batch interval of say 10 seconds, I put the windows length at 3 times that = 30 seconds and make the slidinginterval = batch interval = 10. Obviously these are subjective depending on what is being measured. However, I believe having slidinginterval = batch interval makes sense? Appreciate any views on this. Thanks, Dr Mich Talebzadeh LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com
