Harika,
I think you can modify existing spark on ec2 cluster to run Yarn mapreduce,
not sure if this is what you are looking for.
To try:
1) logon to master
2) go into either ephemeral-hdfs/conf/ or persistent-hdfs/conf/
and add this to mapred-site.xml :
property
namemapreduce.framework.name/name
valueyarn/value
/property
3) use copy-dir to copy this file over to the slaves (don't know if this
step is necessary)
eg.
~/spark-ec2/copy-dir.sh ~/ephemeral-hdfs/conf/mapred-site.xml
4) stop and restart hdfs (for pesistent-hdfs it wasn't started to begin
with)
ephemeral-hdfs]$ ./sbin/stop-all.sh
ephemeral-hdfs]$ ./sbin/start-all.sh
HTH
Deb
On Wed, Feb 25, 2015 at 11:46 PM, Harika matha.har...@gmail.com wrote:
Hi,
I want to setup a Spark cluster with YARN dependency on Amazon EC2. I was
reading this https://spark.apache.org/docs/1.2.0/running-on-yarn.html
document and I understand that Hadoop has to be setup for running Spark
with
YARN. My questions -
1. Do we have to setup Hadoop cluster on EC2 and then build Spark on it?
2. Is there a way to modify the existing Spark cluster to work with YARN?
Thanks in advance.
Harika
--
View this message in context:
http://apache-spark-user-list.1001560.n3.nabble.com/Setting-up-Spark-with-YARN-on-EC2-cluster-tp21818.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
-
To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
For additional commands, e-mail: user-h...@spark.apache.org
