Re: Spark 1.6 Streaming with Checkpointing

On Fri, Aug 26, 2016 at 10:54 PM, Benjamin Kim wrote: > // Create a text file stream on an S3 bucket > val csv = ssc.textFileStream("s3a://" + awsS3BucketName + "/") > > csv.foreachRDD(rdd => { > if (!rdd.partitions.isEmpty) { >

Spark 1.6 Streaming with Checkpointing

I am trying to implement checkpointing in my streaming application but I am getting a not serializable error. Has anyone encountered this? I am deploying this job in YARN clustered mode. Here is a snippet of the main parts of the code. object S3EventIngestion { //create and setup streaming