Hello Everyone,
I am new to Spark, especially spark streaming.
I am trying to read an input stream from Kafka, perform windowed aggregations
in spark using structured streaming, and finally write aggregates to a sink.
- MySQL as an output sink doesn’t seem to be an option, because this
block of code throws an error
streamingDF.writeStream.format("jdbc").start("jdbc:mysql…”)
ava.lang.UnsupportedOperationException: Data source jdbc does not support
streamed writing
This is strange because,
this<http://rxin.github.io/talks/2016-02-18_spark_summit_streaming.pdf>
document shows that jdbc is supported as an output sink!
- Parquet doesn’t seem to be an option, because it doesn’t support
“complete” output mode, but “append” only. As I’m preforming windows
aggregations in spark streaming, the output mode has to be complete, and cannot
be “append”
- Memory and console sinks are good for debugging, but are not
suitable for production jobs.
So, please correct me if I’m missing something in my code to enable jdbc output
sink.
If jdbc output sink is not option, please suggest me an alternative output sink
that suits my needs better.
Or since structured streaming is still ‘alpha’, should I resort to spark
dstreams to achieve my use case described above.
Please suggest.
Thanks in advance,
Hemanth
