Hi Grainier, indeed a very nice extension to the toolbox.
What I was thinking of is the following. Since you basically implemented window-semantic in plain java this is something we could need for various pipeline elements, e.g. a lightweight aggregation component in Java other than in Flink. So maybe it is a good idea to add such capabilities straight to the SDK/engine so that in case it is needed, pipeline element developers can leverage it. What do you guys think? Patrick > Am 03.06.2020 um 14:14 schrieb Philipp Zehnder <[email protected]>: > > Hi Grainier, > > very cool component, especially the cron window is a very good idea. > I have to admit, I have to get used to the syntax ;) but the feature is super > useful and this is a good way to make this component very flexible. > > By the way, in case you didn’t see it so far. In the extension project there > is a module 'streampipes-pipeline-elements-all' and there I also registered > your new components. > We use this project for the StreamPipes ‘lite’ version. All single host > pipeline elements are registered there. Then we just have to start a single > container and reduce the memory footprint for StreamPipes. > > Philipp > > >> On 1. Jun 2020, at 14:59, Grainier Perera <[email protected]> wrote: >> >> Thanks Dominik. I've also added Cron window support with RateLimiting. So >> it gives the flexibility of having various time triggers instead of fixed >> intervals. What do you think? >> >> Issue: https://issues.apache.org/jira/browse/STREAMPIPES-151 >> PR: https://github.com/apache/incubator-streampipes-extensions/pull/20 >> >> Grainier Perera. >> >> >> On Mon, 1 Jun 2020 at 03:11, Dominik Riemer <[email protected]> wrote: >> >>> This is super cool! >>> That's a component we should definitely highlight in the next release as >>> it can be used together with many other pipeline elements. >>> >>> Btw, I also moved the pipeline element overview from the documentation to >>> the main website, so that it's easier for users to discover available >>> pipeline elements and made the list searchable: >>> https://streampipes.apache.org/pipeline-elements.html >>> >>> Can't wait to add the new elements to this list after the next release 😊 >>> >>> Dominik >>> >>> -----Original Message----- >>> From: Grainier Perera <[email protected]> >>> Sent: Sunday, May 31, 2020 4:48 PM >>> To: [email protected] >>> Subject: PE to rate-limit events >>> >>> Hi all, >>> >>> As per the discussion on [1], I've implemented a PE to rate-limit events. >>> For now, it supports the following configurations, and I believe it covers >>> most rate-limit use cases. What do you think? >>> >>> - window type: time/length >>> - window size. >>> - event selection: first/last/all >>> - event grouping: by field (for each unique field value, it'll create a >>> separate window) >>> >>> Issue: https://issues.apache.org/jira/browse/STREAMPIPES-147 >>> PR: https://github.com/apache/incubator-streampipes-extensions/pull/18 >>> [1] >>> >>> https://lists.apache.org/thread.html/r836afda725bab338d81174c3f37b425f3cdd95c7b8699b1841d93170%40%3Cdev.streampipes.apache.org%3E >>> >>> Grainier Perera. >>> >>> > >
