Thank you, Marcelo,
so it is possible to simply copy the YARN configuration from the remote
cluster to the local machine (assuming, the local machine can resolve
the YARN host etc.) and just letting Spark do the rest?
This would actually be great!
Our "local" machine is just another virtual machine running in the same
environment, connected to the cluster via a virtual network.
Cheers
Florian
Am 14.10.2015 um 19:13 schrieb Marcelo Vanzin:
On Wed, Oct 14, 2015 at 10:01 AM, Florian Kaspar
<florian.kas...@onelogic.de> wrote:
we are working on a project running on Spark. Currently we connect to a remote
Spark-Cluster in Standalone mode to obtain the SparkContext using
new JavaSparkContext(new
SparkConf().setAppName("<AppName>").setMaster("spark://<remoteClusterAddress>:7077"));
Can anyone tell me how to create a Spark context programmatically connecting to
a remote YARN cluster?
You should be able to replace the standalone URL with "yarn-client",
and it should work, assuming you have the HADOOP_CONF_DIR (or
YARN_CONF_DIR) env variable pointing at a valid YARN configuration.
Note that if the machine running this code is far from the cluster
performance might not be that great.
--
Florian Kaspar
ONE LOGIC
Dr. Hans-Kapfinger-Str. 3, DE 94032 Passau
T +49 851 22590 25
florian.kas...@onelogic.de
www.onelogic.de
ONE LOGIC GmbH, HRB 7780 Amtsgericht Passau
Geschäftsführung Andreas Böhm, Prof. Dr. Andreas Pfeifer
