Re: Spark can't identify the event time column being supplied to withWatermark()

2018-06-08 Thread frankdede
You are exactly right! A few hours ago, I tried many things and finally got the example working by defining event timestamp column before groupByKey, just like what you suggested, but I wasn't able to figure out the reasoning behind my fix. val sessionUpdates = events

Spark can't identify the event time column being supplied to withWatermark()

2018-06-08 Thread frankdede
I was trying to find a way to resessionize features in different events based on the event timestamps using Spark and I found a code example that uses mapGroupsWithStateto resessionize events using processing timestamps in their repo.