Spark 1.4.1 just got released! So just download that. Yay for timing. On Wed, Jul 15, 2015 at 2:47 PM, Ted Yu <yuzhih...@gmail.com> wrote:
> Should be this one: > [SPARK-7180] [SPARK-8090] [SPARK-8091] Fix a number of > SerializationDebugger bugs and limitations > ... > Closes #6625 from tdas/SPARK-7180 and squashes the following commits: > > On Wed, Jul 15, 2015 at 2:37 PM, Chen Song <chen.song...@gmail.com> wrote: > >> Thanks >> >> Can you point me to the patch to fix the serialization stack? Maybe I can >> pull it in and rerun my job. >> >> Chen >> >> On Wed, Jul 15, 2015 at 4:40 PM, Tathagata Das <t...@databricks.com> >> wrote: >> >>> Your streaming job may have been seemingly running ok, but the DStream >>> checkpointing must have been failing in the background. It would have been >>> visible in the log4j logs. In 1.4.0, we enabled fast-failure for that so >>> that checkpointing failures dont get hidden in the background. >>> >>> The fact that the serialization stack is not being shown correctly, is a >>> known bug in Spark 1.4.0, but is fixed in 1.4.1 about to come out in the >>> next couple of days. That should help you to narrow down the culprit >>> preventing serialization. >>> >>> On Wed, Jul 15, 2015 at 1:12 PM, Ted Yu <yuzhih...@gmail.com> wrote: >>> >>>> Can you show us your function(s) ? >>>> >>>> Thanks >>>> >>>> On Wed, Jul 15, 2015 at 12:46 PM, Chen Song <chen.song...@gmail.com> >>>> wrote: >>>> >>>>> The streaming job has been running ok in 1.2 and 1.3. After I upgraded >>>>> to 1.4, I started seeing error as below. It appears that it fails in >>>>> validate method in StreamingContext. Is there anything changed on 1.4.0 >>>>> w.r.t DStream checkpointint? >>>>> >>>>> Detailed error from driver: >>>>> >>>>> 15/07/15 18:00:39 ERROR yarn.ApplicationMaster: User class threw >>>>> exception: *java.io.NotSerializableException: DStream checkpointing >>>>> has been enabled but the DStreams with their functions are not >>>>> serializable* >>>>> Serialization stack: >>>>> >>>>> java.io.NotSerializableException: DStream checkpointing has been >>>>> enabled but the DStreams with their functions are not serializable >>>>> Serialization stack: >>>>> >>>>> at >>>>> org.apache.spark.streaming.StreamingContext.validate(StreamingContext.scala:550) >>>>> at >>>>> org.apache.spark.streaming.StreamingContext.liftedTree1$1(StreamingContext.scala:587) >>>>> at >>>>> org.apache.spark.streaming.StreamingContext.start(StreamingContext.scala:586) >>>>> >>>>> -- >>>>> Chen Song >>>>> >>>>> >>>> >>> >> >> >> -- >> Chen Song >> >> >
