Any ideas/experience on this? Dr Mich Talebzadeh
LinkedIn * https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* http://talebzadehmich.wordpress.com On 4 May 2016 at 21:45, Mich Talebzadeh <[email protected]> wrote: > Hi, > > Just wanted opinions on this. > > In Spark streaming the parameter > > val ssc = new StreamingContext(sparkConf, Seconds(n)) > > defines the batch or sample interval for the incoming streams > > In addition there is windows Length > > // window length - The duration of the window below that must be multiple > of batch interval n in = > StreamingContext(sparkConf, Seconds(n)) > > val windowLength = L > > And fibally the sliding interval > // sliding interval - The interval at which the window operation is > performed > > val slidingInterval = I > > OK so as given the windowLength L = multiples of n and the > slidingInterval has to be consistent to ensure that we can the head and > tail of the window. > > So as a heuristic approach for a batch interval of say 10 seconds, I put > the windows length at 3 times that = 30 seconds and make the > slidinginterval = batch interval = 10. > > Obviously these are subjective depending on what is being measured. > However, I believe having slidinginterval = batch interval makes sense? > > Appreciate any views on this. > > Thanks, > > Dr Mich Talebzadeh > > > > LinkedIn * > https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw > <https://www.linkedin.com/profile/view?id=AAEAAAAWh2gBxianrbJd6zP6AcPCCdOABUrV8Pw>* > > > > http://talebzadehmich.wordpress.com > > >
