RE: correct use of DStream foreachRDD

I think what you’ll want is to carry out the .map functions before the foreachRDD, something like: val lines = ssc.textFileStream(/stream).map(Sensor.parseSensor).map(Sensor.convertToPut) lines.foreachRDD { rdd = // parse the line of data into sensor object

Re: correct use of DStream foreachRDD

Yes, for example val sensorRDD = rdd.map(Sensor.parseSensor) is a line of code executed on the driver; it's part the function you supplied to foreachRDD. However that line defines an operation on an RDD, and the map function you supplied (parseSensor) will ultimately be carried out on the cluster.

Re: correct use of DStream foreachRDD

Thanks, this looks better // parse the lines of data into sensor objects val sensorDStream = ssc.textFileStream(/stream). map(Sensor.parseSensor) sensorDStream.foreachRDD { rdd = // filter sensor data for low psi val alertRDD = rdd.filter(sensor = sensor.psi 5.0)