like this?
myDStream.foreachRDD(rdd => rdd.saveAsTextFile("/sigmoid/", codec ))
Thanks
Best Regards
On Mon, Jun 8, 2015 at 8:06 PM, Bob Corsaro <rcors...@gmail.com> wrote:
> It looks like saveAsTextFiles doesn't support the compression parameter of
> RDD.saveAsTextFile. Is there a way to add the functionality in my client
> code without patching Spark? I tried making my own saveFunc function and
> calling DStream.foreachRDD but ran into trouble with invoking rddToFileName
> and making the RDD type parameter work properly. It's probably just do to
> my lack of Scala knowledge. Can anyone give me a hand?
>
> def saveAsTextFiles(prefix: String, suffix: String = ""): Unit =
> ssc.withScope {
> val saveFunc = (rdd: RDD[T], time: Time) => {
> val file = rddToFileName(prefix, suffix, time)
> rdd.saveAsTextFile(file)
> }
> this.foreachRDD(saveFunc)
> }
>
>
