Re: How to create spark AMI in AWS
OK, good luck! On Mon Feb 09 2015 at 6:41:14 PM Guodong Wang wrote: > Hi Nicholas, > > Thanks for your quick reply. > > I'd like to try to build a image with create_image.sh. Then let's see how > we can launch spark cluster in region cn-north-1. > > > > Guodong > > On Tue, Feb 10, 2015 at 3:59 AM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> Guodong, >> >> spark-ec2 does not currently support the cn-north-1 region, but you can >> follow [SPARK-4241](https://issues.apache.org/jira/browse/SPARK-4241) to >> find out when it does. >> >> The base AMI used to generate the current Spark AMIs is very old. I'm not >> sure anyone knows what it is anymore. What I know is that it is an Amazon >> Linux AMI. >> >> Yes, the create_image.sh script is what is used to generate the current >> Spark AMI. >> >> Nick >> >> On Mon Feb 09 2015 at 3:27:13 AM Franc Carter < >> franc.car...@rozettatech.com> wrote: >> >>> >>> Hi, >>> >>> I'm very new to Spark, but experienced with AWS - so take that in to >>> account with my suggestions. >>> >>> I started with an AWS base image and then added the pre-built Spark-1.2. >>> I then added made a 'Master' version and a 'Worker' versions and then made >>> AMIs for them. >>> >>> The Master comes up with a static IP and the Worker image has this baked >>> in. I haven't completed everything I am planning to do but so far I can >>> bring up the Master and a bunch of Workers inside and ASG and run spark >>> code successfully. >>> >>> cheers >>> >>> >>> On Mon, Feb 9, 2015 at 10:06 PM, Guodong Wang >>> wrote: >>> Hi guys, I want to launch spark cluster in AWS. And I know there is a spark_ec2.py script. I am using the AWS service in China. But I can not find the AMI in the region of China. So, I have to build one. My question is 1. Where is the bootstrap script to create the Spark AMI? Is it here( https://github.com/mesos/spark-ec2/blob/branch-1.3/create_image.sh) ? 2. What is the base image of the Spark AMI? Eg, the base image of this ( https://github.com/mesos/spark-ec2/blob/branch-1.3/ami-list/us-west-1/hvm ) 3. Shall I install scala during building the AMI? Thanks. Guodong >>> >>> >>> >>> -- >>> >>> *Franc Carter* | Systems Architect | Rozetta Technology >>> >>> franc.car...@rozettatech.com | >>> www.rozettatechnology.com >>> >>> Tel: +61 2 8355 2515 >>> >>> Level 4, 55 Harrington St, The Rocks NSW 2000 >>> >>> PO Box H58, Australia Square, Sydney NSW 1215 >>> >>> AUSTRALIA >>> >>> >
Re: How to create spark AMI in AWS
Hi Nicholas, Thanks for your quick reply. I'd like to try to build a image with create_image.sh. Then let's see how we can launch spark cluster in region cn-north-1. Guodong On Tue, Feb 10, 2015 at 3:59 AM, Nicholas Chammas < nicholas.cham...@gmail.com> wrote: > Guodong, > > spark-ec2 does not currently support the cn-north-1 region, but you can > follow [SPARK-4241](https://issues.apache.org/jira/browse/SPARK-4241) to > find out when it does. > > The base AMI used to generate the current Spark AMIs is very old. I'm not > sure anyone knows what it is anymore. What I know is that it is an Amazon > Linux AMI. > > Yes, the create_image.sh script is what is used to generate the current > Spark AMI. > > Nick > > On Mon Feb 09 2015 at 3:27:13 AM Franc Carter < > franc.car...@rozettatech.com> wrote: > >> >> Hi, >> >> I'm very new to Spark, but experienced with AWS - so take that in to >> account with my suggestions. >> >> I started with an AWS base image and then added the pre-built Spark-1.2. >> I then added made a 'Master' version and a 'Worker' versions and then made >> AMIs for them. >> >> The Master comes up with a static IP and the Worker image has this baked >> in. I haven't completed everything I am planning to do but so far I can >> bring up the Master and a bunch of Workers inside and ASG and run spark >> code successfully. >> >> cheers >> >> >> On Mon, Feb 9, 2015 at 10:06 PM, Guodong Wang wrote: >> >>> Hi guys, >>> >>> I want to launch spark cluster in AWS. And I know there is a >>> spark_ec2.py script. >>> >>> I am using the AWS service in China. But I can not find the AMI in the >>> region of China. >>> >>> So, I have to build one. My question is >>> 1. Where is the bootstrap script to create the Spark AMI? Is it here( >>> https://github.com/mesos/spark-ec2/blob/branch-1.3/create_image.sh) ? >>> 2. What is the base image of the Spark AMI? Eg, the base image of this ( >>> https://github.com/mesos/spark-ec2/blob/branch-1.3/ami-list/us-west-1/hvm >>> ) >>> 3. Shall I install scala during building the AMI? >>> >>> >>> Thanks. >>> >>> Guodong >>> >> >> >> >> -- >> >> *Franc Carter* | Systems Architect | Rozetta Technology >> >> franc.car...@rozettatech.com | >> www.rozettatechnology.com >> >> Tel: +61 2 8355 2515 >> >> Level 4, 55 Harrington St, The Rocks NSW 2000 >> >> PO Box H58, Australia Square, Sydney NSW 1215 >> >> AUSTRALIA >> >>
Re: How to create spark AMI in AWS
Guodong, spark-ec2 does not currently support the cn-north-1 region, but you can follow [SPARK-4241](https://issues.apache.org/jira/browse/SPARK-4241) to find out when it does. The base AMI used to generate the current Spark AMIs is very old. I'm not sure anyone knows what it is anymore. What I know is that it is an Amazon Linux AMI. Yes, the create_image.sh script is what is used to generate the current Spark AMI. Nick On Mon Feb 09 2015 at 3:27:13 AM Franc Carter wrote: > > Hi, > > I'm very new to Spark, but experienced with AWS - so take that in to > account with my suggestions. > > I started with an AWS base image and then added the pre-built Spark-1.2. I > then added made a 'Master' version and a 'Worker' versions and then made > AMIs for them. > > The Master comes up with a static IP and the Worker image has this baked > in. I haven't completed everything I am planning to do but so far I can > bring up the Master and a bunch of Workers inside and ASG and run spark > code successfully. > > cheers > > > On Mon, Feb 9, 2015 at 10:06 PM, Guodong Wang wrote: > >> Hi guys, >> >> I want to launch spark cluster in AWS. And I know there is a spark_ec2.py >> script. >> >> I am using the AWS service in China. But I can not find the AMI in the >> region of China. >> >> So, I have to build one. My question is >> 1. Where is the bootstrap script to create the Spark AMI? Is it here( >> https://github.com/mesos/spark-ec2/blob/branch-1.3/create_image.sh) ? >> 2. What is the base image of the Spark AMI? Eg, the base image of this ( >> https://github.com/mesos/spark-ec2/blob/branch-1.3/ami-list/us-west-1/hvm >> ) >> 3. Shall I install scala during building the AMI? >> >> >> Thanks. >> >> Guodong >> > > > > -- > > *Franc Carter* | Systems Architect | Rozetta Technology > > franc.car...@rozettatech.com | > www.rozettatechnology.com > > Tel: +61 2 8355 2515 > > Level 4, 55 Harrington St, The Rocks NSW 2000 > > PO Box H58, Australia Square, Sydney NSW 1215 > > AUSTRALIA > >
Re: How to create spark AMI in AWS
Hi, I'm very new to Spark, but experienced with AWS - so take that in to account with my suggestions. I started with an AWS base image and then added the pre-built Spark-1.2. I then added made a 'Master' version and a 'Worker' versions and then made AMIs for them. The Master comes up with a static IP and the Worker image has this baked in. I haven't completed everything I am planning to do but so far I can bring up the Master and a bunch of Workers inside and ASG and run spark code successfully. cheers On Mon, Feb 9, 2015 at 10:06 PM, Guodong Wang wrote: > Hi guys, > > I want to launch spark cluster in AWS. And I know there is a spark_ec2.py > script. > > I am using the AWS service in China. But I can not find the AMI in the > region of China. > > So, I have to build one. My question is > 1. Where is the bootstrap script to create the Spark AMI? Is it here( > https://github.com/mesos/spark-ec2/blob/branch-1.3/create_image.sh) ? > 2. What is the base image of the Spark AMI? Eg, the base image of this ( > https://github.com/mesos/spark-ec2/blob/branch-1.3/ami-list/us-west-1/hvm) > 3. Shall I install scala during building the AMI? > > > Thanks. > > Guodong > -- *Franc Carter* | Systems Architect | Rozetta Technology franc.car...@rozettatech.com | www.rozettatechnology.com Tel: +61 2 8355 2515 Level 4, 55 Harrington St, The Rocks NSW 2000 PO Box H58, Australia Square, Sydney NSW 1215 AUSTRALIA
