Hi, I implemented a Cortex sink that takes care of the constraint that Cortex requires samples to arrive ordered by time. The implementation is based on ideas of Kenn Knowles that he shared at https://stackoverflow.com/a/45911664/1517328. The gist is to use a stateful transform in combination with the global window.
My question is how to garbage collect the (per key) state. That state does not expire because the global window does not expire. My use case allows for early as well as late data. The only way I can think of is first to set an event time timer for the watermark passing the "on time" time and then, when the first timer fires, to set a second timer, this time base on processing time, and waiting for the additional late processing time. When the second timer fires per key state can be garbage collected. Is there a simpler way? TIA Stefan
