[
https://issues.apache.org/jira/browse/SPARK-15919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15329294#comment-15329294
]
Sean Owen commented on SPARK-15919:
-----------------------------------
Every RDD has to be saved to a different location. One RDD here is one batch,
so I don't see the distinction you're making. If you mean all data from the
stream should be output into one dir, you can't do that. Really it doesn't
matter much, because you can read files over many directories at once in
subsequent operations. Or, if you want, you can later move or merge the files
elsewhere (with a rename).
> DStream "saveAsTextFile" doesn't update the prefix after each checkpoint
> ------------------------------------------------------------------------
>
> Key: SPARK-15919
> URL: https://issues.apache.org/jira/browse/SPARK-15919
> Project: Spark
> Issue Type: Bug
> Components: Java API
> Affects Versions: 1.6.1
> Environment: Amazon EMR
> Reporter: Aamir Abbas
>
> I have a Spark streaming job that reads a data stream, and saves it as a text
> file after a predefined time interval. In the function
> stream.dstream().repartition(1).saveAsTextFiles(getOutputPath(), "");
> The function getOutputPath() generates a new path every time the function is
> called, depending on the current system time.
> However, the output path prefix remains the same for all the batches, which
> effectively means that function is not called again for the next batch of the
> stream, although the files are being saved after each checkpoint interval.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
