Re: Data Source - State (SPARK-28190)

2020-03-30 Thread Jungtaek Lim
Hi Bryan, Thanks for the interest! Unfortunately there's lack of support on committers for SPARK-28190 (I have been struggling with lack of support on structured streaming contributions). I hope things will get better, but in the meantime, could you please try out my own project instead?

Data Source - State (SPARK-28190)

2020-03-30 Thread Bryan Jeffrey
Hi, Jungtaek. We've been investigating the use of Spark Structured Streaming to replace our Spark Streaming operations. We have several cases where we're using mapWithState to maintain state across batches, often with high volumes of data. We took a look at the Structured Streaming stateful