[
https://issues.apache.org/jira/browse/SPARK-15919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15327229#comment-15327229
]
Aamir Abbas commented on SPARK-15919:
-------------------------------------
ForeachRDD is fine in case you want to save individual RDDs separately. I need
to do this for entire batch of stream. Could you please share the relevant link
to the documentation that can help me save the entire batch of the stream like
this?
> DStream "saveAsTextFile" doesn't update the prefix after each checkpoint
> ------------------------------------------------------------------------
>
> Key: SPARK-15919
> URL: https://issues.apache.org/jira/browse/SPARK-15919
> Project: Spark
> Issue Type: Bug
> Components: Java API
> Affects Versions: 1.6.1
> Environment: Amazon EMR
> Reporter: Aamir Abbas
>
> I have a Spark streaming job that reads a data stream, and saves it as a text
> file after a predefined time interval. In the function
> stream.dstream().repartition(1).saveAsTextFiles(getOutputPath(), "");
> The function getOutputPath() generates a new path every time the function is
> called, depending on the current system time.
> However, the output path prefix remains the same for all the batches, which
> effectively means that function is not called again for the next batch of the
> stream, although the files are being saved after each checkpoint interval.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
