Harika,
I think you can modify existing spark on ec2 cluster to run Yarn mapreduce,
not sure if this is what you are looking for.
To try:
1) logon to master
2) go into either ephemeral-hdfs/conf/ or persistent-hdfs/conf/
and add this to mapred-site.xml :
<property>
<name>mapreduce.framework.name</name>
<value>yarn</value>
</property>
3) use copy-dir to copy this file over to the slaves (don't know if this
step is necessary)
eg.
~/spark-ec2/copy-dir.sh ~/ephemeral-hdfs/conf/mapred-site.xml
4) stop and restart hdfs (for pesistent-hdfs it wasn't started to begin
with)
ephemeral-hdfs]$ ./sbin/stop-all.sh
ephemeral-hdfs]$ ./sbin/start-all.sh
HTH
Deb
On Wed, Feb 25, 2015 at 11:46 PM, Harika <matha.har...@gmail.com> wrote:
> Hi,
>
> I want to setup a Spark cluster with YARN dependency on Amazon EC2. I was
> reading this <https://spark.apache.org/docs/1.2.0/running-on-yarn.html>
> document and I understand that Hadoop has to be setup for running Spark
> with
> YARN. My questions -
>
> 1. Do we have to setup Hadoop cluster on EC2 and then build Spark on it?
> 2. Is there a way to modify the existing Spark cluster to work with YARN?
>
> Thanks in advance.
>
> Harika
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Setting-up-Spark-with-YARN-on-EC2-cluster-tp21818.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
> For additional commands, e-mail: user-h...@spark.apache.org
>
>
