I guess this is not related to SparkR. It seems that Spark can’t pick
hostname/IP address of RM.
Make sure you have correctly set YARN_CONF_DIR env var and have configured
address of RM in yarn-site.xml.
From: Amit Behera [mailto:amit.bd...@gmail.com]
Sent: Friday, November 13, 2015 9:38 PM
To: Sun, Rui; user@spark.apache.org
Subject: Re: Connecting SparkR through Yarn
Hi Sun,
Thank you for reply.
I did the same, but now I am getting another issue.
i.e: Not able to connect to ResourceManager after submitting the job
the Error message showing something like this
Connecting to ResourceManager at /0.0.0.0:8032<http://0.0.0.0:8032>
- INFO ipc.Client: Retrying connect to server:
0.0.0.0/0.0.0.0:8032<http://0.0.0.0/0.0.0.0:8032>. Already tried 0 time(s);
-INFO ipc.Client: Retrying connect to server:
0.0.0.0/0.0.0.0:8032<http://0.0.0.0/0.0.0.0:8032>. Already tried 1 time(s)
I tried resolving this , but not able to do.
On Wed, Nov 11, 2015 at 12:02 PM, Sun, Rui
<rui@intel.com<mailto:rui@intel.com>> wrote:
Amit,
You can simply set “MASTER” as “yarn-client” before calling sparkR.init().
Sys.setenv("MASTER"="yarn-client")
I assume that you have set “YARN_CONF_DIR” env variable required for running
Spark on YARN.
If you want to set more YARN specific configurations, you can for example
Sys.setenv ("SPARKR_SUBMIT_ARGS", " --master yarn-client --num-executors 4
sparkr-shell"
Before calling sparkR.init().
From: Amit Behera [mailto:amit.bd...@gmail.com<mailto:amit.bd...@gmail.com>]
Sent: Monday, November 9, 2015 2:36 AM
To: user@spark.apache.org<mailto:user@spark.apache.org>
Subject: Connecting SparkR through Yarn
Hi All,
Spark Version = 1.5.1
Hadoop Version = 2.6.0
I set up the cluster in Amazon EC2 machines (1+5)
I am able create a SparkContext object using init method from RStudio.
But I do not know how can I create a SparkContext object in yarn mode.
I got the below link to run on yarn. but in this document its given for Spark
version >= 0.9.0 and <= 1.2.
https://github.com/amplab-extras/SparkR-pkg/blob/master/README.md#running-on-yarn
Please help me how can I connect SparkR on Yarn.
Thanks,
Amit.