Re: Chaining Spark Streaming Jobs

Sorry Michael, I ended up using kafka and missed noticing your message. Yes, I did specify the schema with read.schema and thats when I got: at

Change the owner of hdfs file being saved

Hello Experts, I am required to use a specific user id to save files on a remote hdfs cluster. Remote in the sense, spark jobs run on EMR and write to a CDH cluster. Hence I cannot change the hdfs-site.xml etc to point to the destination cluster. As a result I am using webhdfs to save the files

Re: Getting Message From Structured Streaming Format Kafka

Hi Daniel, Several things: 1) Your error seems to suggest you're using a different version of Spark and a different version of the sql-kafka connector. Could you make sure they are on the same Spark version? 2) With Structured Streaming, you may remove everything related to a StreamingContext.

Getting Message From Structured Streaming Format Kafka

Hello, I'm trying to run the following code, var newContextCreated = false // Flag to detect whether new context was created or not val kafkaBrokers = "localhost:9092" // comma separated list of broker:host private val batchDuration: Duration = Seconds(3) private val master: String = "local[2]"

Re: share datasets across multiple spark-streaming applications for lookup

Or Databaricks Delta (announced at Spark Summit) or IBM Event Store depending on the use case. On Oct 31, 2017, at 14:30, Joseph Pride > wrote: Folks: SnappyData. I’m fairly new to working with it myself, but it looks pretty

Spark as ETL, was: Re: Dose pyspark supports python3.6？

Pros: No need for Scala skills, Java can be used. Other companies are already doing it. > Support Yarn execution But not only… Complex use-case for import can easily be done in Java (see https://spark-summit.org/eu-2017/events/extending-apache-sparks-ingestion-building-your-own-java-data-source/