Re: How to create spark AMI in AWS

2015-02-09 Thread Guodong Wang
but so far I can bring up the Master and a bunch of Workers inside and ASG and run spark code successfully. cheers On Mon, Feb 9, 2015 at 10:06 PM, Guodong Wang wangg...@gmail.com wrote: Hi guys, I want to launch spark cluster in AWS. And I know there is a spark_ec2.py script. I am

How to create spark AMI in AWS

2015-02-09 Thread Guodong Wang
Hi guys, I want to launch spark cluster in AWS. And I know there is a spark_ec2.py script. I am using the AWS service in China. But I can not find the AMI in the region of China. So, I have to build one. My question is 1. Where is the bootstrap script to create the Spark AMI? Is it here(

Re: Why spark on yarn applicationmaster cannot get a proper resourcemanager address from yarnconfiguration?

2014-09-03 Thread Guodong Wang
Did you follow the exact step in this page https://spark.apache.org/docs/1.0.2/running-on-yarn.html ? Please be sure that HADOOP_CONF_DIR or YARN_CONF_DIR points to the directory which contains the (client side) configuration files for the Hadoop cluster. Guodong On Thu, Sep 4, 2014 at 10:15