-----BEGIN PGP SIGNED MESSAGE----- Hash: SHA1 Hi Gyula,
I read your comments of your PR. I have a question to this comment: "It only allows aggregations so we dont need to keep the full history in a buffer." What if the user implements an aggregation function like a median? For a median you need the full history, don't you? Am I missing something? Cheers, Bruno On 21.04.2015 14:31, Gyula Fóra wrote: > I have opened a PR for this feature: > > https://github.com/apache/flink/pull/614 > > Cheers, Gyula > > On Tue, Apr 21, 2015 at 1:10 PM, Gyula Fóra <gyula.f...@gmail.com> > wrote: > >> Thats a good idea, I will modify my PR to that :) >> >> Gyula >> >> On Tue, Apr 21, 2015 at 12:09 PM, Fabian Hueske >> <fhue...@gmail.com> wrote: >> >>> Is it possible to switch the order of the statements, i.e., >>> >>> dataStream.every(Time.of(4,sec)).reduce(...) instead of >>> dataStream.reduce(...).every(Time.of(4,sec)) >>> >>> I think that would be more consistent with the structure of the >>> remaining API. >>> >>> Cheers, Fabian >>> >>> 2015-04-21 10:57 GMT+02:00 Gyula Fóra <gyf...@apache.org>: >>> >>>> Hi Bruno, >>>> >>>> Of course you can do that as well. (That's the good part :p >>>> ) >>>> >>>> I will open a PR soon with the proposed changes (first >>>> without breaking >>> the >>>> current Api) and I will post it here. >>>> >>>> Cheers, Gyula >>>> >>>> On Tuesday, April 21, 2015, Bruno Cadonna < >>> cado...@informatik.hu-berlin.de >>>>> >>>> wrote: >>>> > Hi Gyula, > > I have a question regarding your suggestion. > > Can the current continuous aggregation be also specified with your > proposed periodic aggregation? > > I am thinking about something like > > dataStream.reduce(...).every(Count.of(1)) > > Cheers, Bruno > > On 20.04.2015 22:32, Gyula Fóra wrote: >>>>>>> Hey all, >>>>>>> >>>>>>> I think we are missing a quite useful feature that >>>>>>> could be implemented (with some slight modifications) >>>>>>> on top of the current windowing api. >>>>>>> >>>>>>> We currently provide 2 ways of aggregating (or >>>>>>> reducing) over streams: doing a continuous aggregation >>>>>>> and always output the aggregated value (which cannot be >>>>>>> done properly in parallel) or doing aggregation in a >>>>>>> window periodically. >>>>>>> >>>>>>> What we don't have at the moment is periodic >>>>>>> aggregations on the whole stream. I would even go as >>>>>>> far as to remove the continuous outputting >>>>>>> reduce/aggregate it and replace it with this version >>>>>>> as this in return can be done properly in parallel. >>>>>>> >>>>>>> My suggestion would be that a call: >>>>>>> >>>>>>> dataStream.reduce(..) dataStream.sum(..) >>>>>>> >>>>>>> would return a windowed data stream where the window is >>>>>>> the whole record history, and the user would need to >>>>>>> define a trigger to get the actual reduced values >>>>>>> like: >>>>>>> >>>>>>> dataStream.reduce(...).every(Time.of(4,sec)) to get the >>>>>>> actual reduced results. dataStream.sum(...).every(...) >>>>>>> >>>>>>> I think the current data stream reduce/aggregation is >>>>>>> very confusing without being practical for any normal >>>>>>> use-case. >>>>>>> >>>>>>> Also this would be a very api breaking change (but I >>>>>>> would still make this change as it is much more >>>>>>> intuitive than the current behaviour) so I would try to >>>>>>> push it before the release if we can agree. >>>>>>> >>>>>>> Cheers, Gyula >>>>>>> > >>>>> >>>> >>> >> >> > - -- ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ Dr. Bruno Cadonna Postdoctoral Researcher Databases and Information Systems Department of Computer Science Humboldt-Universität zu Berlin http://www.informatik.hu-berlin.de/~cadonnab ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -----BEGIN PGP SIGNATURE----- Version: GnuPG v1.4.11 (GNU/Linux) iQEcBAEBAgAGBQJVNkgYAAoJEKdCIJx7flKw7rUIAMmu80ZuMvfA/BvQemkEo7As bU3iWre+e3OUWNRLuf2JfG9CHMKFSjBJG6Jax/pWZBXTYh8oaYDrYixq7e+vljqf P9ypurhd1h8In71aSUyUPIsrTg6aJ5xo/beUxA6LFbB2LpVqawNDe0gjn3ZRMobM zmn962kqp0oHAVipYI2mzEU6RNl1Kh0PoaLaZRLRh+dlgKofqDFcBiB3hhG/VEoF sCsCAsC1bXtpToPRZ29cRcEfpHcnE3zCgivPeG83JsWYr4mIEj7gp+smFUz0PjoI 1wHv/pnZJS4Onk38HH1GcP95/uYpqm4gz3OBCuE7v+3b1bI852bIvnUZrCGLOew= =u1R0 -----END PGP SIGNATURE-----
