subject:"Spark 2.2 structured streaming with mapGroupsWithState \+ window functions"

Re: Spark 2.2 structured streaming with mapGroupsWithState + window functions

2017-09-05 Thread kant kodali

Hi Daniel, I am thinking you could use groupByKey & mapGroupsWithState to send whatever updates ("updated state") you want and then use .groupBy(window). will that work as expected? Thanks, Kant On Mon, Aug 28, 2017 at 7:06 AM, daniel williams wrote: > Hi all, > >

Re: Spark 2.2 structured streaming with mapGroupsWithState + window functions

2017-08-30 Thread kant kodali

+1 Is this ticket related https://issues.apache.org/jira/browse/SPARK-21641 ? On Mon, Aug 28, 2017 at 7:06 AM, daniel williams wrote: > Hi all, > > I've been looking heavily into Spark 2.2 to solve a problem I have by > specifically using mapGroupsWithState. What

Spark 2.2 structured streaming with mapGroupsWithState + window functions

2017-08-28 Thread daniel williams

Hi all, I've been looking heavily into Spark 2.2 to solve a problem I have by specifically using mapGroupsWithState. What I've discovered is that a *groupBy(window(..))* does not work when being used with a subsequent *mapGroupsWithState* and produces an AnalysisException of :