Well, uber jar works in YARN, but not with standalone ;)
On Sun, Sep 18, 2016 at 12:44 PM -0700, "Chris Fregly" <ch...@fregly.com<mailto:ch...@fregly.com>> wrote: you'll see errors like this... "java.lang.RuntimeException: java.io.InvalidClassException: org.apache.spark.rpc.netty.RequestMessage; local class incompatible: stream classdesc serialVersionUID = -2221986757032131007, local class serialVersionUID = -5447855329526097695" ...when mixing versions of spark. i'm actually seeing this right now while testing across Spark 1.6.1 and Spark 2.0.1 for my all-in-one, hybrid cloud/on-premise Spark + Zeppelin + Kafka + Kubernetes + Docker + One-Click Spark ML Model Production Deployments initiative documented here: https://github.com/fluxcapacitor/pipeline/wiki/Kubernetes-Docker-Spark-ML and check out my upcoming meetup on this effort either in-person or online: http://www.meetup.com/Advanced-Spark-and-TensorFlow-Meetup/events/233978839/ we're throwing in some GPU/CUDA just to sweeten the offering! :) On Sat, Sep 10, 2016 at 2:57 PM, Holden Karau <hol...@pigscanfly.ca<mailto:hol...@pigscanfly.ca>> wrote: I don't think a 2.0 uber jar will play nicely on a 1.5 standalone cluster. On Saturday, September 10, 2016, Felix Cheung <felixcheun...@hotmail.com<mailto:felixcheun...@hotmail.com>> wrote: You should be able to get it to work with 2.0 as uber jar. What type cluster you are running on? YARN? And what distribution? On Sun, Sep 4, 2016 at 8:48 PM -0700, "Holden Karau" <hol...@pigscanfly.ca> wrote: You really shouldn't mix different versions of Spark between the master and worker nodes, if your going to upgrade - upgrade all of them. Otherwise you may get very confusing failures. On Monday, September 5, 2016, Rex X <dnsr...@gmail.com> wrote: Wish to use the Pivot Table feature of data frame which is available since Spark 1.6. But the spark of current cluster is version 1.5. Can we install Spark 2.0 on the master node to work around this? Thanks! -- Cell : 425-233-8271<tel:425-233-8271> Twitter: https://twitter.com/holdenkarau -- Cell : 425-233-8271<tel:425-233-8271> Twitter: https://twitter.com/holdenkarau -- Chris Fregly Research Scientist @ PipelineIO<http://pipeline.io> Advanced Spark and TensorFlow Meetup<http://www.meetup.com/Advanced-Spark-and-TensorFlow-Meetup/> San Francisco | Chicago | Washington DC
