SQL windows with the 'over' syntax does not work in Structured Streaming.
It is very hard to incrementalize that in the general case. Hence non-time
windows are not supported.
On Sat, Apr 20, 2019, 2:16 PM Stephen Boesch <java...@gmail.com> wrote:
> Consider the following *intended* sql:
>
> select row_number()
> over (partition by Origin order by OnTimeDepPct desc) OnTimeDepRank,*
> from flights
>
> This will *not* work in *structured streaming* : The culprit is:
>
> partition by Origin
>
> The requirement is to use a timestamp-typed field such as
>
> partition by flightTime
>
> Tathagata Das (core committer for *spark streaming*) - replies on that in
> a nabble thread:
>
> The traditional SQL windows with `over` is not supported in streaming.
> Only time-based windows, that is, `window("timestamp", "10 minutes")` is
> supported in streaming
>
> *W**hat then* for my query above - which *must* be based on the *Origin*
> field?
> What is the closest equivalent to that query? Or what would be a workaround
> or different approach to achieve same results?
>
