Hi all. I am writing a twitter connector using spark streaming. i have written the following code to maintain checkpoint.
val ssc=StreamingContext.getOrCreate("hdfs://192.168.23.109:9000/home/cloud9/twitterCheckpoint",()=> { managingContext() }) def managingContext():StreamingContext = { //making spark context val conf = new SparkConf().setMaster("local[*]").setAppName("twitterConnector") val ssc = new StreamingContext(conf, Seconds(1)) val sqlContext = new org.apache.spark.sql.SQLContext(ssc.sparkContext) import sqlContext.implicits._ //checkpointing /ssc.checkpoint("hdfs://192.168.23.109:9000/home/cloud9/twitterCheckpoint") ssc } but it gives the following error: java.lang.IllegalArgumentException: requirement failed: WindowedDStream has been marked for checkpointing but the storage level has not been set to enable persisting. Please use DStream.persist() to set the storage level to use memory for better checkpointing performance. I have also mentioned the storage level through the following code. TwitterUtils.createStream(ssc,None,null,StorageLevel.MEMORY_AND_DISK_2) Can anyone help me in this regard? Thanks :) -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Checkpoint-issue-in-spark-streaming-tp24031.html Sent from the Apache Spark User List mailing list archive at Nabble.com. --------------------------------------------------------------------- To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org