Im having some difficulty reliably restoring a streaming job from a checkpoint. When restoring a streaming job constructed from the following snippet, I receive NullPointerException's when `map` is called on the the restored RDD.
lazy val ssc = StreamingContext.getOrCreate(checkpointDir, createStreamingContext _) private def createStreamingContext: StreamingContext = { val ssc = new StreamingContext(spark.sparkContext, batchInterval) ssc.checkpoint(checkpointDir) consumeStreamingContext(ssc) ssc } def consumeStreamingContext(ssc: StreamingContext) = { //... create dstreams val dstream = KinesisUtil.createStream(.... ... dstream.checkpoint(batchInterval) dstream .foreachRDD(process) } def process(events: RDD[Event]) = { if (!events.isEmpty()) { logger.info("Transforming events for processing") //rdd seems to support some operations? logger.info(s"RDD LENGTH: ${events.count}") //nullpointer exception on call to .map val df = events.map(e => { ... } } } . . . . . . . . . . . . . . . . . . . . . . . . . . . Richard Moorhead Software Engineer richard.moorh...@c2fo.com<mailto:richard.moorh...@gmail.com> C2FO: The World's Market for Working CapitalĀ® [http://c2fo.com/wp-content/uploads/sites/1/2016/03/LinkedIN.png] <https://www.linkedin.com/company/c2fo?trk=vsrp_companies_res_name&trkInfo=VSRPsearchId%3A125658601427902817660%2CVSRPtargetId%3A1555109%2CVSRPcmpt%3Aprimary> [http://c2fo.com/wp-content/uploads/sites/1/2016/03/YouTube.png] <https://www.youtube.com/c/C2FOMarket> [http://c2fo.com/wp-content/uploads/sites/1/2016/03/Twitter.png] <https://twitter.com/C2FO> [http://c2fo.com/wp-content/uploads/sites/1/2016/03/Googleplus.png] <https://plus.google.com/+C2foMarket/posts> [http://c2fo.com/wp-content/uploads/sites/1/2016/03/Facebook.png] <https://www.facebook.com/C2FOMarketplace> [http://c2fo.com/wp-content/uploads/sites/1/2016/03/Forbes-Fintech-50.png] <https://c2fo.com/media-coverage/c2fo-included-forbes-fintech-50> The information contained in this message and any attachment may be privileged, confidential, and protected from disclosure. If you are not the intended recipient, or an employee, or agent responsible for delivering this message to the intended recipient, you are hereby notified that any dissemination, distribution, or copying of this communication is strictly prohibited. If you have received this communication in error, please notify us immediately by replying to the message and deleting from your computer.