Hi Joe, The main effect this should have is more state to be kept until the windows can be fired (and state purged). This would of course increase the time it takes to checkpoint the operator.
I'm not sure if there will be significant runtime per-record impact caused by how windows are bookkeeped in data structures in the WindowOperator, maybe Aljoscha (cc'ed) can chime in here for anything. If it is certain that these windows will never fire (until far into the future) because the event-timestamps are in the first place corrupted, then it might make sense to have a way to drop windows based on some criteria. I'm not sure if that is supported in any way without triggers (since you mentioned that those windows might not receive any data), again Aljoscha might be able to provide more info here. Cheers, Gordon On Thu, May 21, 2020 at 7:02 PM Joe Malt <jm...@yelp.com> wrote: > Hi all, > > I'm looking into what happens when messages are ingested with timestamps > far into the future (e.g. due to corruption or a wrong clock at the sender). > > I'm aware of the effect on watermarking, but another thing I'm concerned > about is the performance impact of the extra windows this will create. > > If a Flink operator has many (perhaps hundreds or thousands) of windows > open but not receiving any data (and never firing), will this degrade > performance? > > Thanks, > Joe >
