Starting from the checkpoint using getOrCreate should be sufficient if all you need is at-least-once semantics
http://spark.apache.org/docs/latest/streaming-programming-guide.html#checkpointing On Mon, Nov 30, 2015 at 9:38 AM, Guillermo Ortiz <konstt2...@gmail.com> wrote: > Hello, > > I have Spark and Kafka with directStream. I'm trying that if Spark dies it > could process all those messages when it starts. The offsets are stored in > chekpoints but I don't know how I could say to Spark to start in that point. > I saw that there's another createDirectStream method with a fromOffsets > parameter but, how could I access to the offsets? > > val ssc = new StreamingContext(sparkConf, Seconds(5)) > ssc.checkpoint(checkpoint) > val directKafkaStream = KafkaUtils.createDirectStream[String, String, > StringDecoder, StringDecoder](ssc, kafkaBrokers, topic) > >
