Hi Colin, Flink's SQL runner does not support handling of late data yet. At the moment, late events are simply dropped. We plan to add support for late data in a future release.
The "withIdleStateRetentionTime" parameter only applies to non-windowed aggregation functions and controls when they can evict state for inactive keys. Best, Fabian 2017-11-10 4:17 GMT+01:00 Colin Williams <colin.williams.seat...@gmail.com>: > Hello, > > I've been given some flink application code and asked to implement and > ensure that our query is updated for late arriving entries. We're currently > creating a table using a Tumbling SQL query similar to the first example in > > https://ci.apache.org/projects/flink/flink-docs- > release-1.3/dev/table/sql.html#group-windows > > We then turn the result table back into a datastream using toAppendStream, > and eventually add a derivative stream to a sink. We've configured > TimeCharacteristic to event-time processing. > > From reading the documentation I was trying to configure using > withIdleStateRetentionTime, with the expectation that this setting would > allow me to deal with late arrivals past a given watermark time, but within > the retention time. > > Then to test this I created a simple source which triggers the watermark, > so that I'd have next a late arrival. However so far the watermark seems to > cause something to discriminate the late arrival. Then in my test sink > where I'm trying to capture all emitted outputs, and hopefully the updated > value I don't find one. > > So it seems that my understanding of how to deal with late events, or my > test platform is wrong. Can anyone recognize what I'm doing wrong? > > > Best, > > Colin Williams > > > >
