Re: Structured Streaming Parquet Sink

2016-07-31 Thread Tathagata Das
> > >>> > As per documentation, parquet is the only available file sink. > >>> > >>> The following sinks are currently available in Spark: > >>> > >>> * ConsoleSink for console format. > >>> * FileStreamSink for parquet

Re: Structured Streaming Parquet Sink

2016-07-31 Thread Jacek Laskowski
mplementing StreamSinkProvider. >>> >>> > I am getting an error like 'path' is not specified. >>> > Any idea how to write this to parquet file? >>> >>> There are two ways to specify "path": >>> &g

Re: Structured Streaming Parquet Sink

2016-07-30 Thread Arun Patel
d. >> > Any idea how to write this to parquet file? >> >> There are two ways to specify "path": >> >> 1. Using option method >> 2. start(path: String): StreamingQuery >> >> Pozdrawiam, >> Jacek Laskowski >> >> https:

Re: Structured Streaming Parquet Sink

2016-07-30 Thread Tathagata Das
option method > 2. start(path: String): StreamingQuery > > Pozdrawiam, > Jacek Laskowski > > https://medium.com/@jaceklaskowski/ > Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark > Follow me at https://twitter.com/jaceklaskowski > > > On Sat, Jul 30,

Re: Structured Streaming Parquet Sink

2016-07-30 Thread Jacek Laskowski
askowski https://medium.com/@jaceklaskowski/ Mastering Apache Spark 2.0 http://bit.ly/mastering-apache-spark Follow me at https://twitter.com/jaceklaskowski On Sat, Jul 30, 2016 at 2:50 PM, Arun Patel <arunp.bigd...@gmail.com> wrote: > I am trying out Structured streaming parquet sink. As

Structured Streaming Parquet Sink

2016-07-30 Thread Arun Patel
I am trying out Structured streaming parquet sink. As per documentation, parquet is the only available file sink. I am getting an error like 'path' is not specified. scala> val query = streamingCountsDF.writeStream.format("parquet").start() java.lang.IllegalArgumentExc