I think combiners are pretty awesome for certain cases to minimize network
usage (the average use case seems to fit perfectly), maybe it would be
worthwhile adding a detailed description of the approach to the docs?

On Thu, Feb 18, 2016 at 10:47 AM, Aljoscha Krettek <aljos...@apache.org>
wrote:

> @Nirmalya: Yes, this is right if you temperatures don’t have any other
> field on which you could partition them.
>
> @Stefano: Under some circumstances it would be possible to use a a
> combiner (I’m using the name as Hadoop MapReduce would use it, here). When
> the assignment of elements to windows happens based on the timestamp in the
> elements and window triggering happens based on watermark it is possible to
> combine locally. The reason is that the elements will end up in the same
> windows regardless of the time at which the window is processed so it can
> be done in two steps. Does that make sense? It’s a very ad-hoc description
> and I could make up a drawing or something if that helped. :D
>
>
> > On 18 Feb 2016, at 10:04, Stefano Baghino <stefano.bagh...@radicalbit.io>
> wrote:
> >
> > Thanks, Aljosha, for the explanation. Isn't there a way to apply the
> concept of the combiner to a streaming process?
> >
> > On Thu, Feb 18, 2016 at 3:56 AM, Nirmalya Sengupta <
> sengupta.nirma...@gmail.com> wrote:
> > Hello Aljoscha  <aljos...@apache.org>
> >
> > Thanks very much for clarifying the role of  Pre-Aggregation (rather,
> Incr-Aggregation, now that I understand the intention). It helps me to
> understand. Thanks to Setfano too, for keeping at the original question of
> mine.
> >
> > My current understanding is that if I have to compute the average of a
> streaming set of _temperatures_ then the *best* way to accomplish this, is
> by employing one node (or thread, on my laptop), losing speed but gaining
> deterministic behaviour in the process. I can decide to capture the average
> either by grouping the temperatures by count or by time. Because I am
> sliding the window anyway, I don't run the risk of accumulation of elements
> in the window and buffer overrun.
> >
> > Could  you please confirm if my understanding is correct? I feel happy
> if I 'understand' the basis of a design well! :-)
> >
> > --  Nirmalya
> > --
> > Software Technologist
> > http://www.linkedin.com/in/nirmalyasengupta
> > "If you have built castles in the air, your work need not be lost. That
> is where they should be.
> > Now put the foundation under them."
> >
> >
> >
> > --
> > BR,
> > Stefano Baghino
> >
> > Software Engineer @ Radicalbit
>
>


-- 
BR,
Stefano Baghino

Software Engineer @ Radicalbit

Reply via email to