@Nirmalya: Yes, this is right if you temperatures don’t have any other field on which you could partition them.
@Stefano: Under some circumstances it would be possible to use a a combiner (I’m using the name as Hadoop MapReduce would use it, here). When the assignment of elements to windows happens based on the timestamp in the elements and window triggering happens based on watermark it is possible to combine locally. The reason is that the elements will end up in the same windows regardless of the time at which the window is processed so it can be done in two steps. Does that make sense? It’s a very ad-hoc description and I could make up a drawing or something if that helped. :D > On 18 Feb 2016, at 10:04, Stefano Baghino <stefano.bagh...@radicalbit.io> > wrote: > > Thanks, Aljosha, for the explanation. Isn't there a way to apply the concept > of the combiner to a streaming process? > > On Thu, Feb 18, 2016 at 3:56 AM, Nirmalya Sengupta > <sengupta.nirma...@gmail.com> wrote: > Hello Aljoscha <aljos...@apache.org> > > Thanks very much for clarifying the role of Pre-Aggregation (rather, > Incr-Aggregation, now that I understand the intention). It helps me to > understand. Thanks to Setfano too, for keeping at the original question of > mine. > > My current understanding is that if I have to compute the average of a > streaming set of _temperatures_ then the *best* way to accomplish this, is by > employing one node (or thread, on my laptop), losing speed but gaining > deterministic behaviour in the process. I can decide to capture the average > either by grouping the temperatures by count or by time. Because I am sliding > the window anyway, I don't run the risk of accumulation of elements in the > window and buffer overrun. > > Could you please confirm if my understanding is correct? I feel happy if I > 'understand' the basis of a design well! :-) > > -- Nirmalya > -- > Software Technologist > http://www.linkedin.com/in/nirmalyasengupta > "If you have built castles in the air, your work need not be lost. That is > where they should be. > Now put the foundation under them." > > > > -- > BR, > Stefano Baghino > > Software Engineer @ Radicalbit
