WindowOperator initial watermark

2016-04-14 Thread Michael Radford
This is a really minor issue, but it confused me during testing. The WindowOperator initial watermark is -1: https://github.com/apache/flink/blob/master/flink-streaming-java/src/main/java/org/apache/flink/streaming/runtime/operators/windowing/WindowOperator.java#L136 Whereas TimestampsAndPunctua

FoldFunction accumulator checkpointing

2016-04-12 Thread Michael Radford
I'm wondering whether the accumulator value maintained by a FoldFunction is automatically checkpointed? In general, but specifically when using the WindowedStream.apply variant that takes a FoldFunction: public DataStream apply(R initialValue, FoldFunction foldFunction,

Re: Windows, watermarks, and late data

2016-03-03 Thread Michael Radford
Ha, never mind, I realized I can just put the unique key into the aggregate object maintained by the FoldFunction. I'm still curious why RichWindowFunction (and RichFoldFunction) aren't supported for Scala WindowedStream.apply. Mike On Thu, Mar 3, 2016 at 4:50 PM, Michael Radf

Re: Windows, watermarks, and late data

2016-03-03 Thread Michael Radford
flink-streaming-java/src/main/java/org/apache/flink/streaming/api/windowing/triggers/EventTimeTrigger.java > > Kostas Kloudas worked on extending it, so maybe he could share his version > of the trigger as well. > > Cheers, > Aljoscha > > On 01 Mar 2016, at 18:35, Michael Radford

Windows, watermarks, and late data

2016-03-01 Thread Michael Radford
I'm evaluating Flink for a reporting application that will keep various aggregates updated in a database. It will be consuming from Kafka queues that are replicated from remote data centers, so in case there is a long outage in replication, I need to decide what to do about windowing and late data.