Re: Dropping late date in Structured Streaming

Yes, a built-in mechanism is planned in future releases. You can also drop it using a filter for now but the stateful operators will still keep state for old windows. Matei > On Aug 6, 2016, at 9:40 AM, Amit Sela wrote: > > I've noticed that when using Structured

Dropping late date in Structured Streaming

I've noticed that when using Structured Streaming with event-time windows (fixed/sliding), all windows are retained. This is clearly how "late" data is handled, but I was wondering if there is some pruning mechanism that I might have missed ? or is this planned in future releases ? Thanks, Amit