Can you just call fileStream or textFileStream in the second app, to consume files that appear in HDFS / Tachyon from the first job?
On Thu, Jul 24, 2014 at 2:43 AM, Barnaby <[email protected]> wrote: > If I save an RDD as a sequence file such as: > > val wordCounts = words.map(x => (x, 1)).reduceByKey(_ + _) > wordCounts.foreachRDD( d => { > d.saveAsSequenceFile("tachyon://localhost:19998/files/WordCounts-" + > (new SimpleDateFormat("yyyyMMdd-HHmmss") format > Calendar.getInstance.getTime).toString) > }) > > How can I use these results in another Spark app since there is no > StreamingContext.sequenceFileStream()? > > Or, > > What is the best way to save RDDs of objects to files in one streaming app > so that another app can stream those files in? Basically, reuse partially > reduced RDDs for further processing so that it doesn't have to be done more > than once. > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/streaming-sequence-files-tp10557.html > Sent from the Apache Spark User List mailing list archive at Nabble.com.
