Re: Generate watermarks per key in a KeyedStream

2017-11-09 Thread Derek VerLee
We are contending with the same issue, as it happens.  We have dozens, and potentially down the line, may need to deal with thousands of different "time systems" as you put it, and may not be know at compile time or job start time.  In a practical sense, how

Re: Generate watermarks per key in a KeyedStream

2017-11-09 Thread Shailesh Jain
Thanks for your reply, Xingcan. On Wed, Nov 8, 2017 at 10:42 PM, Xingcan Cui wrote: > Hi Shailesh, > > actually, the watermarks are generated per partition, but all of them will > be forcibly aligned to the minimum one during processing. That is decided > by the semantics of

Re: Generate watermarks per key in a KeyedStream

2017-11-08 Thread Xingcan Cui
Hi Shailesh, actually, the watermarks are generated per partition, but all of them will be forcibly aligned to the minimum one during processing. That is decided by the semantics of watermark and KeyedStream, i.e., the watermarks belong to a whole stream and a stream is made up of different

Generate watermarks per key in a KeyedStream

2017-11-08 Thread Shailesh Jain
Hi, I'm working on implementing a use case wherein different physical devices are sending events, and due to network/power issues, there can be a delay in receiving events at Flink source. One of the operators within the flink job is the Pattern operator, and there are certain patterns which are