Re: apache-spark Structured Stateful Streaming with window / SPARK-21641

2019-10-16 Thread Jungtaek Lim
First of all, I guess you've asked for using both "arbitrary stateful operation" and "native support of windowing". (Even you don't deal with state directly, whenever you use stateful operations like streaming aggregation or stream-stream join, you use state.) In short, there's no native support

apache-spark Structured Stateful Streaming with window / SPARK-21641

2019-10-15 Thread Richard Reitmeyer
What’s the right way use Structured Streaming with both state and windows? Looking at the slides from https://www.slideshare.net/databricks/arbitrary-stateful-aggregations-using-structured-streaming-in-apache-spark slides 26 and 31, it looks like stateful processing events for every device