Spark streaming question - SPARK-13758 Need to use an external RDD inside DStream processing...Please help

[shyla deshpande]

I have a situation similar to the following and I get SPARK-13758
<https://issues.apache.org/jira/browse/SPARK-13758>.


I understand why I get this error, but I want to know what should be
the approach in dealing with these situations.


Thanks


>     var cached = ssc.sparkContext.parallelize(Seq("apple, banana"))
>     val words = ssc.socketTextStream(ip, port).flatMap(_.split(" "))
>     words.foreachRDD((rdd: RDD[String]) => {
>       val res = rdd.map(word => (word, word.length)).collect()
>       println("words: " + res.mkString(", "))
>       cached = cached.union(rdd)
>       cached.checkpoint()
>       println("cached words: " + cached.collect.mkString(", "))
>     })