Re: Samza Throttling clarification

2016-02-05 Thread Benjamin Edwards
Recommend token bukets! Here is a solid implementation: https://github.com/bbeck/token-bucket On Fri, 5 Feb 2016 at 03:33 Jagadish Venkatraman wrote: > Hi Ramesh, > > I understand that you want to limit the calls to the destination service. > Currently, there's no framework level support for thr

Re: Truncating rocks db

2015-02-17 Thread Benjamin Edwards
0 > minutes in the RocksDB store (though, a minute later Kafka will drop it > from the changelog). If you don't need EXACT TTL guarantees, then this > should be fine. If you do need exact, then .all() is probably the way to > go. > > Cheers, > Chris > > On Mon, Feb 16,

Re: Truncating rocks db

2015-02-16 Thread Benjamin Edwards
re you > running with a changelog? > > Cheers, > Chris > > On Sun, Feb 15, 2015 at 10:41 AM, Benjamin Edwards > > wrote: > > > Hi, > > > > I am trialling samza for some windowed stream processing. Typically I > want > > to aggregate a bunch of sta

Re: Windowing Guarantees in samza

2015-02-16 Thread Benjamin Edwards
ur, based on the time stamp of the data, and other intervals. > Punctuation will allow timely emission even if there is no data flowing. > > Julian > > > On Feb 15, 2015, at 10:51, Benjamin Edwards > wrote: > > > > Hi > > > > Based on what I can see in the run loop c

Windowing Guarantees in samza

2015-02-15 Thread Benjamin Edwards
Hi Based on what I can see in the run loop class, there are a few things that seem a little problematic for windowed processing with respect to time: 1) No ability to schedule *when* on an interval you might start. For instance, if you wanted to process a window on the hour, every hour, there is

Truncating rocks db

2015-02-15 Thread Benjamin Edwards
Hi, I am trialling samza for some windowed stream processing. Typically I want to aggregate a bunch of state over some window of messages, process the data, then drop the current state. The only way that I can see to do that at the moment is to delete every key. This seems expensive. Is there no w