Re: Keeping track of how long something has been in a queue

2020-09-07 Thread Jungtaek Lim
You may want to google around "session window" and "duration", and check whether the concept fits your requirements. Probably adding some custom logic on top of the session window would work for you, which requires you to implement a custom function for flatMapGroupsWithState. Hope this helps.

Re: Keeping track of how long something has been in a queue

2020-09-04 Thread Hamish Whittal
Sorry, I moved a paragraph, (2) If Ms green.th was first seen at 13:04:04, then at 13:04:05 and finally > at 13:04:17, she's been in the queue for 13 seconds (ignoring the ms). >

Keeping track of how long something has been in a queue

2020-09-04 Thread Hamish Whittal
Hi folks, I have a stream coming from Kafka. It has this schema: { "id": 4, "account_id": 1070998, "uid": "green.th", "last_activity_time": "2020-09-03 13:04:04.520129" } Another event arrives a few milliseconds/seconds later: { "id": 9, "account_id": 1070998,