To have a single text file output for each batch you can repartition it to
1 and then call the saveAsTextFiles
stream.repartition(1).saveAsTextFiles(location)
On 21 Nov 2014 11:28, <jishnu.prat...@wipro.com> wrote:
> Hi I am also having similar problem.. any fix suggested..
>
>
>
> *Originally Posted by GaganBM*
>
> Hi,
>
> I am trying to persist the DStreams to text files. When I use the inbuilt
> API 'saveAsTextFiles' as :
>
> stream.saveAsTextFiles(resultDirectory)
>
> this creates a number of subdirectories, for each batch, and within each
> sub directory, it creates bunch of text files for each RDD (I assume).
>
> I am wondering if I can have single text files for each batch. Is there
> any API for that ? Or else, a single output file for the entire stream ?
>
> I tried to manually write from each RDD stream to a text file as :
>
> stream.foreachRDD(rdd =>{
> rdd.foreach(element => {
> fileWriter.write(element)
> })
> })
>
> where 'fileWriter' simply makes use of a Java BufferedWriter to write
> strings to a file. However, this fails with exception :
>
> DStreamCheckpointData.writeObject used
> java.io.BufferedWriter
> java.io.NotSerializableException: java.io.BufferedWriter
> at
> java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183)
> at
> java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
> .....
>
> Any help on how to proceed with this ?
>
> The information contained in this electronic message and any attachments
> to this message are intended for the exclusive use of the addressee(s) and
> may contain proprietary, confidential or privileged information. If you are
> not the intended recipient, you should not disseminate, distribute or copy
> this e-mail. Please notify the sender immediately and destroy all copies of
> this message and any attachments.
>
> WARNING: Computer viruses can be transmitted via email. The recipient
> should check this email and any attachments for the presence of viruses.
> The company accepts no liability for any damage caused by any virus
> transmitted by this email.
>
> www.wipro.com
>
