Im having some difficulty reliably restoring a streaming job from a checkpoint.
When restoring a streaming job constructed from the following snippet, I
receive NullPointerException's when `map` is called on the the restored RDD.
lazy val ssc = StreamingContext.getOrCreate(checkpointDir,
createStreamingContext _)
private def createStreamingContext: StreamingContext = {
val ssc = new StreamingContext(spark.sparkContext, batchInterval)
ssc.checkpoint(checkpointDir)
consumeStreamingContext(ssc)
ssc
}
def consumeStreamingContext(ssc: StreamingContext) = {
//... create dstreams
val dstream = KinesisUtil.createStream(....
...
dstream.checkpoint(batchInterval)
dstream
.foreachRDD(process)
}
def process(events: RDD[Event]) = {
if (!events.isEmpty()) {
logger.info("Transforming events for processing")
//rdd seems to support some operations?
logger.info(s"RDD LENGTH: ${events.count}")
//nullpointer exception on call to .map
val df = events.map(e => {
...
}
}
}
. . . . . . . . . . . . . . . . . . . . . . . . . . .
Richard Moorhead
Software Engineer
richard.moorh...@c2fo.com<mailto:richard.moorh...@gmail.com>
C2FO: The World's Market for Working Capital®
[http://c2fo.com/wp-content/uploads/sites/1/2016/03/LinkedIN.png]
<https://www.linkedin.com/company/c2fo?trk=vsrp_companies_res_name&trkInfo=VSRPsearchId%3A125658601427902817660%2CVSRPtargetId%3A1555109%2CVSRPcmpt%3Aprimary>
[http://c2fo.com/wp-content/uploads/sites/1/2016/03/YouTube.png]
<https://www.youtube.com/c/C2FOMarket>
[http://c2fo.com/wp-content/uploads/sites/1/2016/03/Twitter.png]
<https://twitter.com/C2FO>
[http://c2fo.com/wp-content/uploads/sites/1/2016/03/Googleplus.png]
<https://plus.google.com/+C2foMarket/posts>
[http://c2fo.com/wp-content/uploads/sites/1/2016/03/Facebook.png]
<https://www.facebook.com/C2FOMarketplace>
[http://c2fo.com/wp-content/uploads/sites/1/2016/03/Forbes-Fintech-50.png]
<https://c2fo.com/media-coverage/c2fo-included-forbes-fintech-50>
The information contained in this message and any attachment may be privileged,
confidential, and protected from disclosure. If you are not the intended
recipient, or an employee, or agent responsible for delivering this message to
the intended recipient, you are hereby notified that any dissemination,
distribution, or copying of this communication is strictly prohibited. If you
have received this communication in error, please notify us immediately by
replying to the message and deleting from your computer.
