Re: Spark streaming: java.lang.ClassCastException: org.apache.spark.util.SerializableConfiguration ... on restart from checkpoint

Considering the @transient annotations and the work done in the instance initializer, not much state is really be broadcast to the executors. It might be simpler to just create these instances on the executors, rather than trying to broadcast them? -- View this message in context:

Re: Spark streaming: java.lang.ClassCastException: org.apache.spark.util.SerializableConfiguration ... on restart from checkpoint

I prepared simple example helping in reproducing problem: https://github.com/alberskib/spark-streaming-broadcast-issue I think that in that way it will be easier for you to understand problem and find solution (if any exists) Thanks Bartek 2015-12-16 23:34 GMT+01:00 Bartłomiej Alberski

Re: Spark streaming: java.lang.ClassCastException: org.apache.spark.util.SerializableConfiguration ... on restart from checkpoint

Streaming checkpoint doesn't support Accumulator or Broadcast. See https://issues.apache.org/jira/browse/SPARK-5206 Here is a workaround: https://issues.apache.org/jira/browse/SPARK-5206?focusedCommentId=14506806=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14506806

Re: Spark streaming: java.lang.ClassCastException: org.apache.spark.util.SerializableConfiguration ... on restart from checkpoint

First of all , thanks @tdas for looking into my problem. Yes, I checked it seperately and it is working fine. For below piece of code there is no single exception and values are sent correctly. val reporter = new MyClassReporter(...) reporter.send(...) val out = new

Spark streaming: java.lang.ClassCastException: org.apache.spark.util.SerializableConfiguration ... on restart from checkpoint

Hey all, When my streaming application is restarting from failure (from checkpoint) I am receiving strange error: java.lang.ClassCastException: org.apache.spark.util.SerializableConfiguration cannot be cast to com.example.sender.MyClassReporter. Instance of B class is created on driver side

Re: Spark streaming: java.lang.ClassCastException: org.apache.spark.util.SerializableConfiguration ... on restart from checkpoint

Can you show the complete stack trace for the ClassCastException ? Please see the following thread: http://search-hadoop.com/m/q3RTtgEUHVmJA1T1 Cheers On Mon, Dec 14, 2015 at 7:33 AM, alberskib wrote: > Hey all, > > When my streaming application is restarting from failure

Re: Spark streaming: java.lang.ClassCastException: org.apache.spark.util.SerializableConfiguration ... on restart from checkpoint

Below is the full stacktrace(real names of my classes were changed) with short description of entries from my code: rdd.mapPartitions{ partition => //this is the line to which second stacktrace entry is pointing val sender = broadcastedValue.value // this is the maing place to which first