Re: How save streaming aggregations on 'Structured Streams' in parquet format ?

rk("timestamp", "1 hour") > .groupBy(window("timestamp", "30 seconds")) > .agg(...) > > Read more here - https://databricks.com/blog/2017/05/08/event-time- > aggregation-watermarking-apache-sparks-structured-streaming.html > > >

Re: How save streaming aggregations on 'Structured Streams' in parquet format ?

sing Spark SQL and Structured Streaming ? Thanks Kaniska On Mon, Jun 19, 2017 at 11:55 AM, Burak Yavuz <brk...@gmail.com> wrote: > Hi Kaniska, > > In order to use append mode with aggregations, you need to set an event > time watermark (using `withWatermark`). Otherwise, Sp

How save streaming aggregations on 'Structured Streams' in parquet format ?

Hi, My goal is to ~ (1) either chain streaming aggregations in a single query OR (2) run multiple streaming aggregations and save data in some meaningful format to execute low latency / failsafe OLAP queries So my first choice is parquet format , but I failed to make it work ! I am using

Re: unable to stream kafka messages

- On Mon, Mar 27, 2017 at 1:25 PM, Michael Armbrust <mich...@databricks.com> wrote: > You need to import col from org.apache.spark.sql.functions. > > On Mon, Mar 27, 2017 at 1:20 PM, kaniska Mandal <kaniska.man...@gmail.com> > wrote: &g

Re: unable to stream kafka messages

N, > I think the code I provided is a good option, but if you are using a > different encoding you may need to write a UDF. > > On Fri, Mar 24, 2017 at 4:58 PM, kaniska Mandal <kaniska.man...@gmail.com> > wrote: > >> Hi Michael, >> >> Thanks much for the sug

Re: unable to stream kafka messages

Hi Michael, Thanks much for the suggestion. I was wondering - whats the best way to deserialize the 'value' field On Fri, Mar 24, 2017 at 11:47 AM, Michael Armbrust wrote: > Encoders can only map data into an object if those columns already exist. > When we are