Re: run new spark version on old spark cluster ?

2019-05-21 Thread Nicolas Paris
Hi I finally got all working. Here are the steps (for information, I am on HDP 2.6.5): - copy the old hive-site.xml into the new spark conf folder - (optional?) donwload the jersey-bundle-1.8.jar and put it into the jars folder - build a tar gz from all the jars and copy that archive to hdfs

Re: run new spark version on old spark cluster ?

2019-05-20 Thread Koert Kuipers
most likely have to set something in spark-defaults.conf like spark.master yarn spark.submit.deployMode client On Mon, May 20, 2019 at 3:14 PM Nicolas Paris wrote: > Finally that was easy to connect to both hive/hdfs. I just had to copy > the hive-site.xml from the old spark version and that

Re: run new spark version on old spark cluster ?

2019-05-20 Thread Nicolas Paris
Finally that was easy to connect to both hive/hdfs. I just had to copy the hive-site.xml from the old spark version and that worked instantly after unzipping. Right now I am stuck on connecting to yarn. On Mon, May 20, 2019 at 02:50:44PM -0400, Koert Kuipers wrote: > we had very little issues

Re: run new spark version on old spark cluster ?

2019-05-20 Thread Koert Kuipers
we had very little issues with hdfs or hive, but then we use hive only for basic reading and writing of tables. depending on your vendor you might have to add a few settings to your spark-defaults.conf. i remember on hdp you had to set the hdp.version somehow. we prefer to build spark with hadoop

Re: run new spark version on old spark cluster ?

2019-05-20 Thread Nicolas Paris
> correct. note that you only need to install spark on the node you launch it > from. spark doesnt need to be installed on cluster itself. That sound reasonably doable for me. My guess is I will have some troubles to make that spark version work with both hive & hdfs installed on the cluster - or

Re: run new spark version on old spark cluster ?

2019-05-20 Thread Koert Kuipers
correct. note that you only need to install spark on the node you launch it from. spark doesnt need to be installed on cluster itself. the shared components between spark jobs on yarn are only really spark-shuffle-service in yarn and spark-history-server. i have found compatibility for these to

Re: run new spark version on old spark cluster ?

2019-05-20 Thread Pat Ferrel
@spark.apache.org Date: May 20, 2019 at 11:02:49 AM To: user@spark.apache.org Subject: Re: run new spark version on old spark cluster ? > you will need the spark version you intend to launch with on the machine you > launch from and point to the correct spark-submit does this mean to install a

Re: run new spark version on old spark cluster ?

2019-05-20 Thread Nicolas Paris
> you will need the spark version you intend to launch with on the machine you > launch from and point to the correct spark-submit does this mean to install a second spark version (2.4) on the cluster ? thanks On Mon, May 20, 2019 at 01:58:11PM -0400, Koert Kuipers wrote: > yarn can happily run

Re: run new spark version on old spark cluster ?

2019-05-20 Thread Koert Kuipers
yarn can happily run multiple spark versions side-by-side you will need the spark version you intend to launch with on the machine you launch from and point to the correct spark-submit On Mon, May 20, 2019 at 1:50 PM Nicolas Paris wrote: > Hi > > I am wondering whether that's feasible to: > -

run new spark version on old spark cluster ?

2019-05-20 Thread Nicolas Paris
Hi I am wondering whether that's feasible to: - build a spark application (with sbt/maven) based on spark2.4 - deploy that jar on yarn on a spark2.3 based installation thanks by advance, -- nicolas - To unsubscribe e-mail: