it is also easy to launch many different spark versions on yarn by simply having them installed side-by-side.
1) build spark for your cdh version. for example for cdh 5 i do: $ git checkout v2.0.0 $ dev/make-distribution.sh --name cdh5.4-hive --tgz -Phadoop-2.6 -Dhadoop.version=2.6.0-cdh5.4.4 -Pyarn -Phive -Psparkr 2) scp to and untar the spark tar.gz on the server you want to launch from 3) modify the new spark's conf/spark-env.sh so it has this: export HADOOP_CONF_DIR=/etc/hadoop/conf 4) modify the new spark's conf/spark-defaults.conf so it has this: spark.master yarn 5) now launch your application with the bin/spark-submit script from the new spark distro On Mon, Sep 26, 2016 at 11:48 AM, Rex X <dnsr...@gmail.com> wrote: > Yes, I have a cloudera cluster with Yarn. Any more details on how to work > out with uber jar? > > Thank you. > > > On Sun, Sep 18, 2016 at 2:13 PM, Felix Cheung <felixcheun...@hotmail.com> > wrote: > >> Well, uber jar works in YARN, but not with standalone ;) >> >> >> >> >> >> On Sun, Sep 18, 2016 at 12:44 PM -0700, "Chris Fregly" <ch...@fregly.com> >> wrote: >> >> you'll see errors like this... >> >> "java.lang.RuntimeException: java.io.InvalidClassException: >> org.apache.spark.rpc.netty.RequestMessage; local class incompatible: >> stream classdesc serialVersionUID = -2221986757032131007, local class >> serialVersionUID = -5447855329526097695" >> >> ...when mixing versions of spark. >> >> i'm actually seeing this right now while testing across Spark 1.6.1 and >> Spark 2.0.1 for my all-in-one, hybrid cloud/on-premise Spark + Zeppelin + >> Kafka + Kubernetes + Docker + One-Click Spark ML Model Production >> Deployments initiative documented here: >> >> https://github.com/fluxcapacitor/pipeline/wiki/Kubernetes-Docker-Spark-ML >> >> and check out my upcoming meetup on this effort either in-person or >> online: >> >> http://www.meetup.com/Advanced-Spark-and-TensorFlow-Meetup/ >> events/233978839/ >> >> we're throwing in some GPU/CUDA just to sweeten the offering! :) >> >> On Sat, Sep 10, 2016 at 2:57 PM, Holden Karau <hol...@pigscanfly.ca> >> wrote: >> >>> I don't think a 2.0 uber jar will play nicely on a 1.5 standalone >>> cluster. >>> >>> >>> On Saturday, September 10, 2016, Felix Cheung <felixcheun...@hotmail.com> >>> wrote: >>> >>>> You should be able to get it to work with 2.0 as uber jar. >>>> >>>> What type cluster you are running on? YARN? And what distribution? >>>> >>>> >>>> >>>> >>>> >>>> On Sun, Sep 4, 2016 at 8:48 PM -0700, "Holden Karau" < >>>> hol...@pigscanfly.ca> wrote: >>>> >>>> You really shouldn't mix different versions of Spark between the master >>>> and worker nodes, if your going to upgrade - upgrade all of them. Otherwise >>>> you may get very confusing failures. >>>> >>>> On Monday, September 5, 2016, Rex X <dnsr...@gmail.com> wrote: >>>> >>>>> Wish to use the Pivot Table feature of data frame which is available >>>>> since Spark 1.6. But the spark of current cluster is version 1.5. Can we >>>>> install Spark 2.0 on the master node to work around this? >>>>> >>>>> Thanks! >>>>> >>>> >>>> >>>> -- >>>> Cell : 425-233-8271 >>>> Twitter: https://twitter.com/holdenkarau >>>> >>>> >>> >>> -- >>> Cell : 425-233-8271 >>> Twitter: https://twitter.com/holdenkarau >>> >>> >> >> >> -- >> *Chris Fregly* >> Research Scientist @ *PipelineIO* <http://pipeline.io> >> *Advanced Spark and TensorFlow Meetup* >> <http://www.meetup.com/Advanced-Spark-and-TensorFlow-Meetup/> >> *San Francisco* | *Chicago* | *Washington DC* >> >> >> >> >> >
