Ah, cool. Thanks. On Wed, Jul 15, 2015 at 5:58 PM, Tathagata Das <t...@databricks.com> wrote:
> Spark 1.4.1 just got released! So just download that. Yay for timing. > > On Wed, Jul 15, 2015 at 2:47 PM, Ted Yu <yuzhih...@gmail.com> wrote: > >> Should be this one: >> [SPARK-7180] [SPARK-8090] [SPARK-8091] Fix a number of >> SerializationDebugger bugs and limitations >> ... >> Closes #6625 from tdas/SPARK-7180 and squashes the following commits: >> >> On Wed, Jul 15, 2015 at 2:37 PM, Chen Song <chen.song...@gmail.com> >> wrote: >> >>> Thanks >>> >>> Can you point me to the patch to fix the serialization stack? Maybe I >>> can pull it in and rerun my job. >>> >>> Chen >>> >>> On Wed, Jul 15, 2015 at 4:40 PM, Tathagata Das <t...@databricks.com> >>> wrote: >>> >>>> Your streaming job may have been seemingly running ok, but the DStream >>>> checkpointing must have been failing in the background. It would have been >>>> visible in the log4j logs. In 1.4.0, we enabled fast-failure for that so >>>> that checkpointing failures dont get hidden in the background. >>>> >>>> The fact that the serialization stack is not being shown correctly, is >>>> a known bug in Spark 1.4.0, but is fixed in 1.4.1 about to come out in the >>>> next couple of days. That should help you to narrow down the culprit >>>> preventing serialization. >>>> >>>> On Wed, Jul 15, 2015 at 1:12 PM, Ted Yu <yuzhih...@gmail.com> wrote: >>>> >>>>> Can you show us your function(s) ? >>>>> >>>>> Thanks >>>>> >>>>> On Wed, Jul 15, 2015 at 12:46 PM, Chen Song <chen.song...@gmail.com> >>>>> wrote: >>>>> >>>>>> The streaming job has been running ok in 1.2 and 1.3. After I >>>>>> upgraded to 1.4, I started seeing error as below. It appears that it >>>>>> fails >>>>>> in validate method in StreamingContext. Is there anything changed on >>>>>> 1.4.0 >>>>>> w.r.t DStream checkpointint? >>>>>> >>>>>> Detailed error from driver: >>>>>> >>>>>> 15/07/15 18:00:39 ERROR yarn.ApplicationMaster: User class threw >>>>>> exception: *java.io.NotSerializableException: DStream checkpointing >>>>>> has been enabled but the DStreams with their functions are not >>>>>> serializable* >>>>>> Serialization stack: >>>>>> >>>>>> java.io.NotSerializableException: DStream checkpointing has been >>>>>> enabled but the DStreams with their functions are not serializable >>>>>> Serialization stack: >>>>>> >>>>>> at >>>>>> org.apache.spark.streaming.StreamingContext.validate(StreamingContext.scala:550) >>>>>> at >>>>>> org.apache.spark.streaming.StreamingContext.liftedTree1$1(StreamingContext.scala:587) >>>>>> at >>>>>> org.apache.spark.streaming.StreamingContext.start(StreamingContext.scala:586) >>>>>> >>>>>> -- >>>>>> Chen Song >>>>>> >>>>>> >>>>> >>>> >>> >>> >>> -- >>> Chen Song >>> >>> >> > -- Chen Song
