Hi
I finally got all working. Here are the steps (for information, I am on HDP
2.6.5):
- copy the old hive-site.xml into the new spark conf folder
- (optional?) donwload the jersey-bundle-1.8.jar and put it into the jars folder
- build a tar gz from all the jars and copy that archive to hdfs
most likely have to set something in spark-defaults.conf like
spark.master yarn
spark.submit.deployMode client
On Mon, May 20, 2019 at 3:14 PM Nicolas Paris
wrote:
> Finally that was easy to connect to both hive/hdfs. I just had to copy
> the hive-site.xml from the old spark version and that
Finally that was easy to connect to both hive/hdfs. I just had to copy
the hive-site.xml from the old spark version and that worked instantly
after unzipping.
Right now I am stuck on connecting to yarn.
On Mon, May 20, 2019 at 02:50:44PM -0400, Koert Kuipers wrote:
> we had very little issues
we had very little issues with hdfs or hive, but then we use hive only for
basic reading and writing of tables.
depending on your vendor you might have to add a few settings to your
spark-defaults.conf. i remember on hdp you had to set the hdp.version
somehow.
we prefer to build spark with hadoop
> correct. note that you only need to install spark on the node you launch it
> from. spark doesnt need to be installed on cluster itself.
That sound reasonably doable for me. My guess is I will have some
troubles to make that spark version work with both hive & hdfs installed
on the cluster - or
correct. note that you only need to install spark on the node you launch it
from. spark doesnt need to be installed on cluster itself.
the shared components between spark jobs on yarn are only really
spark-shuffle-service in yarn and spark-history-server. i have found
compatibility for these to
@spark.apache.org
Date: May 20, 2019 at 11:02:49 AM
To: user@spark.apache.org
Subject: Re: run new spark version on old spark cluster ?
> you will need the spark version you intend to launch with on the machine
you
> launch from and point to the correct spark-submit
does this mean to install a
> you will need the spark version you intend to launch with on the machine you
> launch from and point to the correct spark-submit
does this mean to install a second spark version (2.4) on the cluster ?
thanks
On Mon, May 20, 2019 at 01:58:11PM -0400, Koert Kuipers wrote:
> yarn can happily run
yarn can happily run multiple spark versions side-by-side
you will need the spark version you intend to launch with on the machine
you launch from and point to the correct spark-submit
On Mon, May 20, 2019 at 1:50 PM Nicolas Paris
wrote:
> Hi
>
> I am wondering whether that's feasible to:
> -
Hi
I am wondering whether that's feasible to:
- build a spark application (with sbt/maven) based on spark2.4
- deploy that jar on yarn on a spark2.3 based installation
thanks by advance,
--
nicolas
-
To unsubscribe e-mail:
10 matches
Mail list logo