If I save an RDD as a sequence file such as:
val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _)
wordCounts.foreachRDD( d => {
d.saveAsSequenceFile("tachyon://localhost:19998/files/WordCounts-" +
(new SimpleDateFormat("yyyyMMdd-HHmmss") format
Calendar.getInstance.getTime).toString)
})
How can I use these results in another Spark app since there is no
StreamingContext.sequenceFileStream()?
Or,
What is the best way to save RDDs of objects to files in one streaming app
so that another app can stream those files in? Basically, reuse partially
reduced RDDs for further processing so that it doesn't have to be done more
than once.
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/streaming-sequence-files-tp10557.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
