Can you show us your function(s) ? Thanks
On Wed, Jul 15, 2015 at 12:46 PM, Chen Song <chen.song...@gmail.com> wrote: > The streaming job has been running ok in 1.2 and 1.3. After I upgraded to > 1.4, I started seeing error as below. It appears that it fails in validate > method in StreamingContext. Is there anything changed on 1.4.0 w.r.t > DStream checkpointint? > > Detailed error from driver: > > 15/07/15 18:00:39 ERROR yarn.ApplicationMaster: User class threw > exception: *java.io.NotSerializableException: DStream checkpointing has > been enabled but the DStreams with their functions are not serializable* > Serialization stack: > > java.io.NotSerializableException: DStream checkpointing has been enabled > but the DStreams with their functions are not serializable > Serialization stack: > > at > org.apache.spark.streaming.StreamingContext.validate(StreamingContext.scala:550) > at > org.apache.spark.streaming.StreamingContext.liftedTree1$1(StreamingContext.scala:587) > at > org.apache.spark.streaming.StreamingContext.start(StreamingContext.scala:586) > > -- > Chen Song > >
