Re: Kafka 0.8.x / 0.9.x support in structured streaming

2017-05-15 Thread David Kaczynski
I haven't done Structured Streaming in Spark 2.1 with Kafka 0.9.x, but I did do stream processing with Spark 2.0.1 and Kafka 0.10. Here's the official documenation that I used for Spark Streaming with Kafka 0.10: https://spark.apache.org/docs/2.1.0/streaming-kafka-integration.html. It looks like

Re: running spark program on intellij connecting to remote master for cluster

2017-05-10 Thread David Kaczynski
Do you have Spark installed locally on your laptop with IntelliJ? Are you using the SparkLauncher class or your local spark-submit script? A while back, I was trying to submit a spark job from my local workstation to a remote cluster using the SparkLauncher class, but I didn't actually have