Re: Flink drops messages?

2017-11-13 Thread Fabian Hueske
Thanks for the correction! :-) 2017-11-13 13:05 GMT+01:00 Kien Truong : > Getting late elements from side-output is already available with Flink 1.3 > :) > > Regards, > > Kien > On 11/13/2017 5:00 PM, Fabian Hueske wrote: > > Hi Andrea, > > you are right. Flink's window operators can drop message

Re: Flink drops messages?

2017-11-13 Thread Kien Truong
Getting late elements from side-output is already available with Flink 1.3 :) Regards, Kien On 11/13/2017 5:00 PM, Fabian Hueske wrote: Hi Andrea, you are right. Flink's window operators can drop messages which are too late, i.e., have a timestamp smaller than the last watermark. This is e

Re: Flink drops messages?

2017-11-13 Thread Fabian Hueske
Hi Andrea, you are right. Flink's window operators can drop messages which are too late, i.e., have a timestamp smaller than the last watermark. This is expected behavior and documented at several places [1] [2]. There are a couple of options how to deal with late elements: 1. Use more conservat

Flink drops messages?

2017-11-12 Thread AndreaKinn
Hi, I'm running a Flink application where data are retrieved from a Kafka broker and forwarded to a Cassandra sink. I've implemented the following watermark emitter: public class CustomTimestampExtractor implements AssignerWithPeriodicWatermarks>{ private final long maxOutOfOrderness = 800;