Hi Krzysztof, Thanks for sharing your solution! ProcessFunctions are the Swiss army knife of Flink :-)
Cheers, Fabian 2018-02-22 19:55 GMT+01:00 Krzysztof Białek <krzysiek.bia...@gmail.com>: > Hi Fabian, > > Thank you for your suggestion. In the meantime I rethought this problem > and implemented alternative solution without using windows at all. > I used plain ProcessFunction with > 1. Keyed state (by companyId) - to keep ratings per key > 2. EventTime timers - to remove outdated ratings from state and emit > recalculated score immediately > > This solution gives results in real-time, windows would delay the results > by 1 day I think. > > Regards, > Krzysztof > > > On Thu, Feb 22, 2018 at 9:44 AM, Fabian Hueske <fhue...@gmail.com> wrote: > >> Hi Krzysztof, >> >> you could compute the stats in two stages: >> >> 1) compute an daily window. You should use a ReduceFunction or >> AggreagteFunction here if possible to perform the computation eagerly. >> 2) compute a sliding window of 90 days with a 1 day hop (or 90 rows with >> a 1 row hop). >> >> That will crunch down the data in the first window and the second window >> will combine the pre-aggregated results. >> >> Hope this helps, >> Fabian >> >> 2018-02-19 16:36 GMT+01:00 Krzysztof Białek <krzysiek.bia...@gmail.com>: >> >>> Hi, >>> >>> My app is calculating Companies scores from Ratings given by users. >>> Only ratings from last 90 days should be considered. >>> >>> 1. Is it possible to construct window processing ratings from last 90 >>> days? >>> I've started with *misusing* countWindow but this solution looks ugly >>> for me. >>> >>> ratingStream >>> .filter(new OutdatedRatingsFilter(maxRatingAge)) >>> .keyBy(_.companyId) >>> .countWindow(0L).trigger(new OnEventTrigger).evictor(new >>> OutdatedRatingsEvictor(maxRatingAge)) >>> .process(ratingFunction) >>> >>> >>> 2. How to recalculate score once the rating expires (after 90 days)? >>> I don't want to put artificial ratings into the stream to trigger the >>> recalculation. >>> >>> Any idea how can I do it better? >>> >>> Regards, >>> Krzysztof >>> >>> >>> >> >
