RE: [EXT] Sliding windows

2019-06-04 Thread Peter Wicks (pwicks)
Craig, If you have a timestamp set as an attribute on the processor, then this is kind of possible. Have a regular MergeContent processor, with "Maximum Group Size" set to 1 mb, set "Max Bin Age" to 3 min; you may need to tweak settings to get the right cadence, but these are generally the set

Re: [EXT] Sliding windows

2019-06-04 Thread Joe Witt
...from the description it isn't clear what you're trying to achieve so lets first try to expand the detail on the use case. We should distinguish whether you're wanting to 'combine various objects in a data stream together on some time bound' from 'processing various objects in a data stream to m

Re: [EXT] Sliding windows

2019-06-04 Thread Craig Knell
Joe Thanks for the options. Kafka Flink option is interesting but to big at the moment. I’ll build a custom processor for this occasion. Much appreciated! Craig Knell > On 4 Jun 2019, at 21:56, Joe Witt wrote: > > ...from the description it isn't clear what you're trying to achieve so

Re: [EXT] Sliding windows

2019-06-05 Thread Michal Klempa
Hi Craig, I was doing something similar couple of years ago. Built custom processor and found out, that the problem is much bigger then I thought. In custom processor, the onTrigger is called from different threads, in parallel. So you have to maintain synchronized or concurrent data structures in