To have a single text file output for each batch you can repartition it to 1 and then call the saveAsTextFiles
stream.repartition(1).saveAsTextFiles(location) On 21 Nov 2014 11:28, <jishnu.prat...@wipro.com> wrote: > Hi I am also having similar problem.. any fix suggested.. > > > > *Originally Posted by GaganBM* > > Hi, > > I am trying to persist the DStreams to text files. When I use the inbuilt > API 'saveAsTextFiles' as : > > stream.saveAsTextFiles(resultDirectory) > > this creates a number of subdirectories, for each batch, and within each > sub directory, it creates bunch of text files for each RDD (I assume). > > I am wondering if I can have single text files for each batch. Is there > any API for that ? Or else, a single output file for the entire stream ? > > I tried to manually write from each RDD stream to a text file as : > > stream.foreachRDD(rdd =>{ > rdd.foreach(element => { > fileWriter.write(element) > }) > }) > > where 'fileWriter' simply makes use of a Java BufferedWriter to write > strings to a file. However, this fails with exception : > > DStreamCheckpointData.writeObject used > java.io.BufferedWriter > java.io.NotSerializableException: java.io.BufferedWriter > at > java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183) > at > java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547) > ..... > > Any help on how to proceed with this ? > > The information contained in this electronic message and any attachments > to this message are intended for the exclusive use of the addressee(s) and > may contain proprietary, confidential or privileged information. If you are > not the intended recipient, you should not disseminate, distribute or copy > this e-mail. Please notify the sender immediately and destroy all copies of > this message and any attachments. > > WARNING: Computer viruses can be transmitted via email. The recipient > should check this email and any attachments for the presence of viruses. > The company accepts no liability for any damage caused by any virus > transmitted by this email. > > www.wipro.com >