Heres something similar which i used to do:
unionDStream.foreachRDD(rdd => { val events = rdd.count() println("Received
Events : " + rdd.count()) if(events > 0 ){ val fw = new
FileWriter("events", true) fw.write(Calendar.getInstance().getTime + "," +
events + "\n") fw.close() } })
Sending from cellphone, not sure how the code snippet will look. :)
On 26 Mar 2015 01:20, "Adrian Mocanu" <[email protected]> wrote:
> Hi
>
> Is there a way to write all RDDs in a DStream to the same file?
>
> I tried this and got an empty file. I think it’s bc the file is not closed
> i.e. ESMinibatchFunctions.writer.close() executes before the stream is
> created.
>
>
>
> Here’s my code
>
> myStream.foreachRDD(rdd => {
>
> rdd.foreach(x => {
>
> ESMinibatchFunctions.writer.append(rdd.collect()(0).toString()+"
> the data ") })
>
> // localRdd = localRdd.union(rdd)
>
> // localArray = localArray ++ rdd.collect()
>
> } )
>
>
>
> ESMinibatchFunctions.writer.close()
>
>
>
> object ESMinibatchFunctions {
>
> val writer = new PrintWriter("c:/delme/exxx.txt")
>
> }
>
