Re: How to convert Spark Streaming to Static Dataframe on the fly and pass it to a ML Model as batch

Hi, or you could just use the structured streaming https://spark.apache.org/docs/latest/structured-streaming-programming-guide.html Regards, Gourav Sengupta On Tue, Aug 14, 2018 at 10:51 AM, Gerard Maas wrote: > Hi Aakash, > > In Spark Streaming, forEachRDD provides you access to the data in

Re: How to convert Spark Streaming to Static Dataframe on the fly and pass it to a ML Model as batch

Hi Aakash, In Spark Streaming, forEachRDD provides you access to the data in each micro batch. You can transform that RDD into a DataFrame and implement the flow you describe. eg.: var historyRDD:RDD[mytype] = sparkContext.emptyRDD // create Kafka Dstream ... dstream.foreachRDD{ rdd => val a

How to convert Spark Streaming to Static Dataframe on the fly and pass it to a ML Model as batch

Hi all, The requirement is, to process file using Spark Streaming fed from Kafka Topic and once all the transformations are done, make it a batch of static dataframe and pass it into a Spark ML Model tuning. As of now, I had been doing it in the below fashion - 1) Read the file using Kafka 2) Co