I guess he used client model and the local Spark version is 1.5.2 but the standalone Spark version is 1.5.1. In other words, he used a 1.5.2 driver to talk with 1.5.1 executors.
On Mon, Feb 1, 2016 at 2:08 PM, Holden Karau <hol...@pigscanfly.ca> wrote: > So I'm a little confused to exactly how this might have happened - but one > quick guess is that maybe you've built an assembly jar with Spark core, can > you mark it is a provided and or post your build file? > > On Fri, Jan 29, 2016 at 7:35 AM, Ted Yu <yuzhih...@gmail.com> wrote: > >> I logged SPARK-13084 >> >> For the moment, please consider running with 1.5.2 on all the nodes. >> >> On Fri, Jan 29, 2016 at 5:29 AM, Jason Plurad <plur...@gmail.com> wrote: >> >>> I agree with you, Ted, if RDD had a serial version UID this might not be >>> an issue. So that could be a JIRA to submit to help avoid version >>> mismatches in future Spark versions, but that doesn't help my current >>> situation between 1.5.1 and 1.5.2. >>> >>> Any other ideas? Thanks. >>> On Thu, Jan 28, 2016 at 5:06 PM Ted Yu <yuzhih...@gmail.com> wrote: >>> >>>> I am not Scala expert. >>>> >>>> RDD extends Serializable but doesn't have @SerialVersionUID() >>>> annotation. >>>> This may explain what you described. >>>> >>>> One approach is to add @SerialVersionUID so that RDD's have stable >>>> serial version UID. >>>> >>>> Cheers >>>> >>>> On Thu, Jan 28, 2016 at 1:38 PM, Jason Plurad <plur...@gmail.com> >>>> wrote: >>>> >>>>> I've searched through the mailing list archive. It seems that if you >>>>> try to run, for example, a Spark 1.5.2 program against a Spark 1.5.1 >>>>> standalone server, you will run into an exception like this: >>>>> >>>>> WARN org.apache.spark.scheduler.TaskSetManager - Lost task 0.0 in >>>>> stage 0.0 (TID 0, 192.168.14.103): java.io.InvalidClassException: >>>>> org.apache.spark.rdd.RDD; local class incompatible: stream classdesc >>>>> serialVersionUID = -3343649307726848892, local class serialVersionUID = >>>>> -3996494161745401652 >>>>> >>>>> If my application is using a library that builds against Spark 1.5.2, >>>>> does that mean that my application is now tied to that same Spark >>>>> standalone server version? >>>>> >>>>> Is there a recommended way for that library to have a Spark dependency >>>>> but keep it compatible against a wider set of versions, i.e. any version >>>>> 1.5.x? >>>>> >>>>> Thanks! >>>>> >>>> >>>> >> > > > -- > Cell : 425-233-8271 > Twitter: https://twitter.com/holdenkarau >
