Re: Multiple aggregations over streaming dataframes

2016-07-08 Thread Arnaud Bailly
q...@gmail.com> > Cc: Sivakumaran S <siva.kuma...@me.com>, "user @spark" < > user@spark.apache.org> > Subject: Re: Multiple aggregations over streaming dataframes > > We are planning to address this issue in the future. > > At a high level, we'll have to a

Re: Multiple aggregations over streaming dataframes

2016-07-07 Thread Andy Davidson
; Date: Thursday, July 7, 2016 at 2:31 PM To: Arnaud Bailly <arnaud.oq...@gmail.com> Cc: Sivakumaran S <siva.kuma...@me.com>, "user @spark" <user@spark.apache.org> Subject: Re: Multiple aggregations over streaming dataframes > We are planning to address this issu

Re: Multiple aggregations over streaming dataframes

2016-07-07 Thread Michael Armbrust
il. 2016 12:55, "Sivakumaran S" <siva.kuma...@me.com> a écrit : >> >>> Hi Arnauld, >>> >>> Sorry for the doubt, but what exactly is multiple aggregation? What is >>> the use case? >>> >>> Regards, >>> >>> Sivakumar

Re: Multiple aggregations over streaming dataframes

2016-07-07 Thread Arnaud Bailly
r the doubt, but what exactly is multiple aggregation? What is >> the use case? >> >> Regards, >> >> Sivakumaran >> >> >> On 07-Jul-2016, at 11:18 AM, Arnaud Bailly <arnaud.oq...@gmail.com> >> wrote: >> >> Hello, >> >&

Re: Multiple aggregations over streaming dataframes

2016-07-07 Thread Sivakumaran S
t;> a écrit : > Hi Arnauld, > > Sorry for the doubt, but what exactly is multiple aggregation? What is the > use case? > > Regards, > > Sivakumaran > > >> On 07-Jul-2016, at 11:18 AM, Arnaud Bailly <arnaud.oq...@gmail.com >> <mailto:arnaud.oq.

Re: Multiple aggregations over streaming dataframes

2016-07-07 Thread Arnaud Bailly
<arnaud.oq...@gmail.com> wrote: > > Hello, > > I understand multiple aggregations over streaming dataframes is not > currently supported in Spark 2.0. Is there a workaround? Out of the top of > my head I could think of having a two stage approach: > - first query wri

Re: Multiple aggregations over streaming dataframes

2016-07-07 Thread Sivakumaran S
Hi Arnauld, Sorry for the doubt, but what exactly is multiple aggregation? What is the use case? Regards, Sivakumaran > On 07-Jul-2016, at 11:18 AM, Arnaud Bailly <arnaud.oq...@gmail.com> wrote: > > Hello, > > I understand multiple aggregations over streaming datafr

Multiple aggregations over streaming dataframes

2016-07-07 Thread Arnaud Bailly
Hello, I understand multiple aggregations over streaming dataframes is not currently supported in Spark 2.0. Is there a workaround? Out of the top of my head I could think of having a two stage approach: - first query writes output to disk/memory using "complete" mode - second query