Interestingly, I just faced with the same problem. By any change, do you want to process old files in the directory as well as new ones? It's my motivation and checkpointing my problem as well.
2017-02-08 22:02 GMT-08:00 Amit Sela <[email protected]>: > Not with checkpointing. > > On Thu, Feb 9, 2017, 04:58 Egor Pahomov <[email protected]> wrote: > >> Just guessing here, but would http://spark.apache.org/ >> docs/latest/streaming-programming-guide.html#basic-sources "*Queue of >> RDDs as a Stream*" work? Basically create DStream from your RDD and than >> union with other DStream. >> >> 2017-02-08 12:32 GMT-08:00 Amit Sela <[email protected]>: >> >> Hi all, >> >> I'm looking to union a DStream and RDD into a single stream. >> One important note is that the RDD has to be added to the DStream just >> once. >> >> Ideas ? >> >> Thanks, >> Amit >> >> >> >> >> >> -- >> >> >> *Sincerely yoursEgor Pakhomov* >> > -- *Sincerely yoursEgor Pakhomov*
