Re: can we use mapGroupsWithState in raw sql?

ael Armbrust <mich...@databricks.com>, Tathagata Das <tathagata.das1...@gmail.com>, "user @spark" <user@spark.apache.org> Subject: Re: can we use mapGroupsWithState in raw sql? This is cool! Looks to me this works too select data.* from (SELECT max(struct(my_timestamp,*)) as

Re: can we use mapGroupsWithState in raw sql?

t;) >> >> >> Thanks, >> Arun >> >> From: Jungtaek Lim <kabh...@gmail.com> >> Date: Wednesday, April 18, 2018 at 4:54 PM >> To: Michael Armbrust <mich...@databricks.com> >> Cc: kant kodali <kanth...@gmail.com>, Arun Iyer <ar

Re: can we use mapGroupsWithState in raw sql?

uot;, > $"my_timestamp")).as("data")).select($"id", $"data.*") > > > Thanks, > Arun > > From: Jungtaek Lim <kabh...@gmail.com> > Date: Wednesday, April 18, 2018 at 4:54 PM > To: Michael Armbrust <mich...@databricks.com> &

Re: can we use mapGroupsWithState in raw sql?

Michael Armbrust <mich...@databricks.com> Cc: kant kodali <kanth...@gmail.com>, Arun Iyer <ar...@apache.org>, Tathagata Das <tathagata.das1...@gmail.com>, "user @spark" <user@spark.apache.org> Subject: Re: can we use mapGroupsWithState in raw sql? Thanks M

Re: can we use mapGroupsWithState in raw sql?

tMode(“complete”/“update")…. >>> >>> Unless the “stream” is already a grouped stream, in which case the above >>> would not work since the support for multiple aggregate operations is not >>> there yet. >>> >>> Thanks, >>> Arun >

Re: can we use mapGroupsWithState in raw sql?

ot;amount").writeStream.outputMode(“ >> complete”/“update")…. >> >> Unless the “stream” is already a grouped stream, in which case the above >> would not work since the support for multiple aggregate operations is not >> there yet. >> >> Thanks, >>

Re: can we use mapGroupsWithState in raw sql?

ate operations is not > there yet. > > Thanks, > Arun > > From: kant kodali <kanth...@gmail.com> > Date: Tuesday, April 17, 2018 at 11:41 AM > To: Tathagata Das <tathagata.das1...@gmail.com> > Cc: "user @spark" <user@spark.apache.org> > Subject

Re: can we use mapGroupsWithState in raw sql?

erations is not there yet. Thanks, Arun From: kant kodali <kanth...@gmail.com> Date: Tuesday, April 17, 2018 at 11:41 AM To: Tathagata Das <tathagata.das1...@gmail.com> Cc: "user @spark" <user@spark.apache.org> Subject: Re: can we use mapGroupsWithState in raw sql?

Re: can we use mapGroupsWithState in raw sql?

;> wouldn't want to get different result according to the micro-batch, and >> then you always want to deal with event time window. >> >> Thanks, >> Jungtaek Lim (HeartSaVioR) >> >> 2018년 4월 18일 (수) 오전 3:42, kant kodali <kanth...@gmail.com>님이 작성: >> >>> Hi TD, >>> >>> Thanks for that. The only reason I ask is I don't see any alternative >>> solution to solve the problem below using raw sql. >>> >>> >>> How to select the max row for every group in spark structured streaming >>> 2.3.0 without using order by since it requires complete mode or >>> mapGroupWithState? >>> >>> *Input:* >>> >>> id | amount | my_timestamp >>> --- >>> 1 | 5 | 2018-04-01T01:00:00.000Z >>> 1 | 10 | 2018-04-01T01:10:00.000Z >>> 2 | 20 | 2018-04-01T01:20:00.000Z >>> 2 | 30 | 2018-04-01T01:25:00.000Z >>> 2 | 40 | 2018-04-01T01:30:00.000Z >>> >>> *Expected Output:* >>> >>> id | amount | my_timestamp >>> --- >>> 1 | 10 | 2018-04-01T01:10:00.000Z >>> 2 | 40 | 2018-04-01T01:30:00.000Z >>> >>> Looking for a streaming solution using either raw sql like >>> sparkSession.sql("sql >>> query") or similar to raw sql but not something like mapGroupWithState >>> >>> On Mon, Apr 16, 2018 at 8:32 PM, Tathagata Das < >>> tathagata.das1...@gmail.com> wrote: >>> >>>> Unfortunately no. Honestly it does not make sense as for type-aware >>>> operations like map, mapGroups, etc., you have to provide an actual JVM >>>> function. That does not fit in with the SQL language structure. >>>> >>>> On Mon, Apr 16, 2018 at 7:34 PM, kant kodali <kanth...@gmail.com> >>>> wrote: >>>> >>>>> Hi All, >>>>> >>>>> can we use mapGroupsWithState in raw SQL? or is it in the roadmap? >>>>> >>>>> Thanks! >>>>> >>>>> >>>>> >>>> >>>

Re: can we use mapGroupsWithState in raw sql?

>> 1 | 5 | 2018-04-01T01:00:00.000Z >> 1 | 10 | 2018-04-01T01:10:00.000Z >> 2 | 20 | 2018-04-01T01:20:00.000Z >> 2 | 30 | 2018-04-01T01:25:00.000Z >> 2 | 40 | 2018-04-01T01:30:00.000Z >> >> *Expected Output:* >> >> id | amount | my_timestamp >> --- >> 1 | 10 | 2018-04-01T01:10:00.000Z >> 2 | 40 | 2018-04-01T01:30:00.000Z >> >> Looking for a streaming solution using either raw sql like >> sparkSession.sql("sql >> query") or similar to raw sql but not something like mapGroupWithState >> >> On Mon, Apr 16, 2018 at 8:32 PM, Tathagata Das < >> tathagata.das1...@gmail.com> wrote: >> >>> Unfortunately no. Honestly it does not make sense as for type-aware >>> operations like map, mapGroups, etc., you have to provide an actual JVM >>> function. That does not fit in with the SQL language structure. >>> >>> On Mon, Apr 16, 2018 at 7:34 PM, kant kodali <kanth...@gmail.com> wrote: >>> >>>> Hi All, >>>> >>>> can we use mapGroupsWithState in raw SQL? or is it in the roadmap? >>>> >>>> Thanks! >>>> >>>> >>>> >>> >>

Re: can we use mapGroupsWithState in raw sql?

0:00.000Z > 2 | 40 | 2018-04-01T01:30:00.000Z > > Looking for a streaming solution using either raw sql like > sparkSession.sql("sql > query") or similar to raw sql but not something like mapGroupWithState > > On Mon, Apr 16, 2018 at 8:32 PM, Tathagata Das < > tathagata.das1...@gmail.com> wrote: > >> Unfortunately no. Honestly it does not make sense as for type-aware >> operations like map, mapGroups, etc., you have to provide an actual JVM >> function. That does not fit in with the SQL language structure. >> >> On Mon, Apr 16, 2018 at 7:34 PM, kant kodali <kanth...@gmail.com> wrote: >> >>> Hi All, >>> >>> can we use mapGroupsWithState in raw SQL? or is it in the roadmap? >>> >>> Thanks! >>> >>> >>> >> >

Re: can we use mapGroupsWithState in raw sql?

nction. That does not fit in with the SQL language structure. > > On Mon, Apr 16, 2018 at 7:34 PM, kant kodali <kanth...@gmail.com> wrote: > >> Hi All, >> >> can we use mapGroupsWithState in raw SQL? or is it in the roadmap? >> >> Thanks! >> >> >> >

Re: can we use mapGroupsWithState in raw sql?

, > > can we use mapGroupsWithState in raw SQL? or is it in the roadmap? > > Thanks! > > >

can we use mapGroupsWithState in raw sql?

Hi All, can we use mapGroupsWithState in raw SQL? or is it in the roadmap? Thanks!