Bill, Spark Streaming's DStream provides overloaded methods for transform() and foreachRDD() that allow you to access the timestamp of a batch: http://spark.apache.org/docs/latest/api/scala/index.html#org.apache.spark.streaming.dstream.DStream
I think the timestamp is the end of the batch, not the beginning. For example, I compute runtime taking the difference between now() and the time I get as a parameter in foreachRDD(). Tobias On Thu, Jul 24, 2014 at 6:39 AM, Bill Jay <bill.jaypeter...@gmail.com> wrote: > Hi all, > > I have a question regarding Spark streaming. When we use the > saveAsTextFiles function and my batch is 60 seconds, Spark will generate a > series of files such as: > > result-1406148960000, result-1406148020000, result-1406148080000, etc. > > I think this is the timestamp for the beginning of each batch. How can we > extract the variable and use it in our code? Thanks! > > Bill >
