you'll see errors like this... "java.lang.RuntimeException: java.io.InvalidClassException: org.apache.spark.rpc.netty.RequestMessage; local class incompatible: stream classdesc serialVersionUID = -2221986757032131007, local class serialVersionUID = -5447855329526097695"
...when mixing versions of spark. i'm actually seeing this right now while testing across Spark 1.6.1 and Spark 2.0.1 for my all-in-one, hybrid cloud/on-premise Spark + Zeppelin + Kafka + Kubernetes + Docker + One-Click Spark ML Model Production Deployments initiative documented here: https://github.com/fluxcapacitor/pipeline/wiki/Kubernetes-Docker-Spark-ML and check out my upcoming meetup on this effort either in-person or online: http://www.meetup.com/Advanced-Spark-and-TensorFlow-Meetup/events/233978839/ we're throwing in some GPU/CUDA just to sweeten the offering! :) On Sat, Sep 10, 2016 at 2:57 PM, Holden Karau <hol...@pigscanfly.ca> wrote: > I don't think a 2.0 uber jar will play nicely on a 1.5 standalone cluster. > > > On Saturday, September 10, 2016, Felix Cheung <felixcheun...@hotmail.com> > wrote: > >> You should be able to get it to work with 2.0 as uber jar. >> >> What type cluster you are running on? YARN? And what distribution? >> >> >> >> >> >> On Sun, Sep 4, 2016 at 8:48 PM -0700, "Holden Karau" < >> hol...@pigscanfly.ca> wrote: >> >> You really shouldn't mix different versions of Spark between the master >> and worker nodes, if your going to upgrade - upgrade all of them. Otherwise >> you may get very confusing failures. >> >> On Monday, September 5, 2016, Rex X <dnsr...@gmail.com> wrote: >> >>> Wish to use the Pivot Table feature of data frame which is available >>> since Spark 1.6. But the spark of current cluster is version 1.5. Can we >>> install Spark 2.0 on the master node to work around this? >>> >>> Thanks! >>> >> >> >> -- >> Cell : 425-233-8271 >> Twitter: https://twitter.com/holdenkarau >> >> > > -- > Cell : 425-233-8271 > Twitter: https://twitter.com/holdenkarau > > -- *Chris Fregly* Research Scientist @ *PipelineIO* <http://pipeline.io> *Advanced Spark and TensorFlow Meetup* <http://www.meetup.com/Advanced-Spark-and-TensorFlow-Meetup/> *San Francisco* | *Chicago* | *Washington DC*
