Hi, I’m designing a streaming job whose elements need to be windowed by event time across a large set of keys. All elements are read from the same source. Event time progresses independently across keys. Is it possible to assign timestamps, and thus generate independent watermarks, per keyed stream, so late arriving elements can be handled per keyed stream?
And in general, what’s the best approach to designing a job that needs to process different keyed streams whose event times do not relate to each other? My current approach generates timestamps at the source but never generates watermarks so no record is ever considered late. This has the unfortunate side effect of windows never closing. As a result, each event time window relies on a custom trigger which fires and purges the window after a given amount of processing time elapses during which no new records arrived. Thanks, Jordan
