I am trying out Structured streaming parquet sink. As per documentation,
parquet is the only available file sink.
I am getting an error like 'path' is not specified.
scala> val query = streamingCountsDF.writeStream.format("parquet").start()
java.lang.IllegalArgumentException: 'path' is not specified
at
org.apache.spark.sql.execution.datasources.DataSource$$anonfun$9.apply(DataSource.scala:264)
at
org.apache.spark.sql.execution.datasources.DataSource$$anonfun$9.apply(DataSource.scala:264)
at scala.collection.MapLike$class.getOrElse(MapLike.scala:128)
at
org.apache.spark.sql.execution.datasources.CaseInsensitiveMap.getOrElse(ddl.scala:117)
at
org.apache.spark.sql.execution.datasources.DataSource.createSink(DataSource.scala:263)
at
org.apache.spark.sql.streaming.DataStreamWriter.start(DataStreamWriter.scala:291)
... 60 elided
But, I don't see path or parquet in DataStreamWriter.
scala> val query = streamingCountsDF.writeStream.
foreach format option options outputMode partitionBy queryName
start trigger
Any idea how to write this to parquet file?
- Arun
