I think Kant meant time windowing functions. You can use `window(TIMESTAMP, '24 hours', '24 hours')`
On Tue, Jul 25, 2017 at 9:26 AM, Keith Chapman <keithgchap...@gmail.com> wrote: > Here is an example of a window lead function, > > select *, lead(someColumn1) over ( partition by someColumn2 order by > someColumn13 asc nulls first) as someName from someTable > > Regards, > Keith. > > http://keith-chapman.com > > On Tue, Jul 25, 2017 at 9:15 AM, kant kodali <kanth...@gmail.com> wrote: > >> How do I Specify windowInterval and slideInteval using raw sql string? >> >> On Tue, Jul 25, 2017 at 8:52 AM, Keith Chapman <keithgchap...@gmail.com> >> wrote: >> >>> You could issue a raw sql query to spark, there is no particular >>> advantage or disadvantage of doing so. Spark would build a logical plan >>> from the raw sql (or DSL) and optimize on that. Ideally you would end up >>> with the same physical plan, irrespective of it been written in raw sql / >>> DSL. >>> >>> Regards, >>> Keith. >>> >>> http://keith-chapman.com >>> >>> On Tue, Jul 25, 2017 at 12:50 AM, kant kodali <kanth...@gmail.com> >>> wrote: >>> >>>> HI All, >>>> >>>> I just want to run some spark structured streaming Job similar to this >>>> >>>> DS.filter(col("name").equalTo("john")) >>>> .groupBy(functions.window(df1.col("TIMESTAMP"), "24 hours", "24 >>>> hours"), df1.col("hourlyPay")) >>>> .agg(sum("hourlyPay").as("total")); >>>> >>>> >>>> I am wondering if I can express the above query in raw sql string? >>>> >>>> If so how would that look like and what are some of the disadvantages of >>>> using raw sql query vs spark DSL? >>>> >>>> >>>> Thanks! >>>> >>>> >>> >> >
