The AMI should automatically switch between PVM and HVM based on the instance type you specify on the command line. For reference (note you don't need to specify this on the command line), the PVM ami id is ami-5bb18832 in us-east-1.
FWIW we maintain the list of AMI Ids (across regions and pvm, hvm) at https://github.com/mesos/spark-ec2/tree/v2/ami-list Thanks Shivaram On Wed, Apr 9, 2014 at 9:12 AM, Marco Costantini < silvio.costant...@granatads.com> wrote: > Ah, tried that. I believe this is an HVM AMI? We are exploring paravirtual > AMIs. > > > On Wed, Apr 9, 2014 at 11:17 AM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> And for the record, that AMI is ami-35b1885c. Again, you don't need to >> specify it explicitly; spark-ec2 will default to it. >> >> >> On Wed, Apr 9, 2014 at 11:08 AM, Nicholas Chammas < >> nicholas.cham...@gmail.com> wrote: >> >>> Marco, >>> >>> If you call spark-ec2 launch without specifying an AMI, it will default >>> to the Spark-provided AMI. >>> >>> Nick >>> >>> >>> On Wed, Apr 9, 2014 at 9:43 AM, Marco Costantini < >>> silvio.costant...@granatads.com> wrote: >>> >>>> Hi there, >>>> To answer your question; no there is no reason NOT to use an AMI that >>>> Spark has prepared. The reason we haven't is that we were not aware such >>>> AMIs existed. Would you kindly point us to the documentation where we can >>>> read about this further? >>>> >>>> Many many thanks, Shivaram. >>>> Marco. >>>> >>>> >>>> On Tue, Apr 8, 2014 at 4:42 PM, Shivaram Venkataraman < >>>> shiva...@eecs.berkeley.edu> wrote: >>>> >>>>> Is there any reason why you want to start with a vanilla amazon AMI >>>>> rather than the ones we build and provide as a part of Spark EC2 scripts ? >>>>> The AMIs we provide are close to the vanilla AMI but have the root account >>>>> setup properly and install packages like java that are used by Spark. >>>>> >>>>> If you wish to customize the AMI, you could always start with our AMI >>>>> and add more packages you like -- I have definitely done this recently and >>>>> it works with HVM and PVM as far as I can tell. >>>>> >>>>> Shivaram >>>>> >>>>> >>>>> On Tue, Apr 8, 2014 at 8:50 AM, Marco Costantini < >>>>> silvio.costant...@granatads.com> wrote: >>>>> >>>>>> I was able to keep the "workaround" ...around... by overwriting the >>>>>> generated '/root/.ssh/authorized_keys' file with a known good one, in the >>>>>> '/etc/rc.local' file >>>>>> >>>>>> >>>>>> On Tue, Apr 8, 2014 at 10:12 AM, Marco Costantini < >>>>>> silvio.costant...@granatads.com> wrote: >>>>>> >>>>>>> Another thing I didn't mention. The AMI and user used: naturally >>>>>>> I've created several of my own AMIs with the following characteristics. >>>>>>> None of which worked. >>>>>>> >>>>>>> 1) Enabling ssh as root as per this guide ( >>>>>>> http://blog.tiger-workshop.com/enable-root-access-on-amazon-ec2-instance/). >>>>>>> When doing this, I do not specify a user for the spark-ec2 script. What >>>>>>> happens is that, it works! But only while it's alive. If I stop the >>>>>>> instance, create an AMI, and launch a new instance based from the new >>>>>>> AMI, >>>>>>> the change I made in the '/root/.ssh/authorized_keys' file is >>>>>>> overwritten >>>>>>> >>>>>>> 2) adding the 'ec2-user' to the 'root' group. This means that the >>>>>>> ec2-user does not have to use sudo to perform any operations needing >>>>>>> root >>>>>>> privilidges. When doing this, I specify the user 'ec2-user' for the >>>>>>> spark-ec2 script. An error occurs: rsync fails with exit code 23. >>>>>>> >>>>>>> I believe HVMs still work. But it would be valuable to the community >>>>>>> to know that the root user work-around does/doesn't work any more for >>>>>>> paravirtual instances. >>>>>>> >>>>>>> Thanks, >>>>>>> Marco. >>>>>>> >>>>>>> >>>>>>> On Tue, Apr 8, 2014 at 9:51 AM, Marco Costantini < >>>>>>> silvio.costant...@granatads.com> wrote: >>>>>>> >>>>>>>> As requested, here is the script I am running. It is a simple shell >>>>>>>> script which calls spark-ec2 wrapper script. I execute it from the >>>>>>>> 'ec2' >>>>>>>> directory of spark, as usual. The AMI used is the raw one from the AWS >>>>>>>> Quick Start section. It is the first option (an Amazon Linux >>>>>>>> paravirtual >>>>>>>> image). Any ideas or confirmation would be GREATLY appreciated. Please >>>>>>>> and >>>>>>>> thank you. >>>>>>>> >>>>>>>> >>>>>>>> #!/bin/sh >>>>>>>> >>>>>>>> export AWS_ACCESS_KEY_ID=MyCensoredKey >>>>>>>> export AWS_SECRET_ACCESS_KEY=MyCensoredKey >>>>>>>> >>>>>>>> AMI_ID=ami-2f726546 >>>>>>>> >>>>>>>> ./spark-ec2 -k gds-generic -i ~/.ssh/gds-generic.pem -u ec2-user -s >>>>>>>> 10 -v 0.9.0 -w 300 --no-ganglia -a ${AMI_ID} -m m3.2xlarge -t >>>>>>>> m3.2xlarge >>>>>>>> launch marcotest >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> On Mon, Apr 7, 2014 at 6:21 PM, Shivaram Venkataraman < >>>>>>>> shivaram.venkatara...@gmail.com> wrote: >>>>>>>> >>>>>>>>> Hmm -- That is strange. Can you paste the command you are using to >>>>>>>>> launch the instances ? The typical workflow is to use the spark-ec2 >>>>>>>>> wrapper >>>>>>>>> script using the guidelines at >>>>>>>>> http://spark.apache.org/docs/latest/ec2-scripts.html >>>>>>>>> >>>>>>>>> Shivaram >>>>>>>>> >>>>>>>>> >>>>>>>>> On Mon, Apr 7, 2014 at 1:53 PM, Marco Costantini < >>>>>>>>> silvio.costant...@granatads.com> wrote: >>>>>>>>> >>>>>>>>>> Hi Shivaram, >>>>>>>>>> >>>>>>>>>> OK so let's assume the script CANNOT take a different user and >>>>>>>>>> that it must be 'root'. The typical workaround is as you said, allow >>>>>>>>>> the >>>>>>>>>> ssh with the root user. Now, don't laugh, but, this worked last >>>>>>>>>> Friday, but >>>>>>>>>> today (Monday) it no longer works. :D Why? ... >>>>>>>>>> >>>>>>>>>> ...It seems that NOW, when you launch a 'paravirtual' ami, the >>>>>>>>>> root user's 'authorized_keys' file is always overwritten. This means >>>>>>>>>> the >>>>>>>>>> workaround doesn't work anymore! I would LOVE for someone to verify >>>>>>>>>> this. >>>>>>>>>> >>>>>>>>>> Just to point out, I am trying to make this work with a >>>>>>>>>> paravirtual instance and not an HVM instance. >>>>>>>>>> >>>>>>>>>> Please and thanks, >>>>>>>>>> Marco. >>>>>>>>>> >>>>>>>>>> >>>>>>>>>> On Mon, Apr 7, 2014 at 4:40 PM, Shivaram Venkataraman < >>>>>>>>>> shivaram.venkatara...@gmail.com> wrote: >>>>>>>>>> >>>>>>>>>>> Right now the spark-ec2 scripts assume that you have root access >>>>>>>>>>> and a lot of internal scripts assume have the user's home directory >>>>>>>>>>> hard >>>>>>>>>>> coded as /root. However all the Spark AMIs we build should have >>>>>>>>>>> root ssh >>>>>>>>>>> access -- Do you find this not to be the case ? >>>>>>>>>>> >>>>>>>>>>> You can also enable root ssh access in a vanilla AMI by editing >>>>>>>>>>> /etc/ssh/sshd_config and setting "PermitRootLogin" to yes >>>>>>>>>>> >>>>>>>>>>> Thanks >>>>>>>>>>> Shivaram >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> On Mon, Apr 7, 2014 at 11:14 AM, Marco Costantini < >>>>>>>>>>> silvio.costant...@granatads.com> wrote: >>>>>>>>>>> >>>>>>>>>>>> Hi all, >>>>>>>>>>>> On the old Amazon Linux EC2 images, the user 'root' was enabled >>>>>>>>>>>> for ssh. Also, it is the default user for the Spark-EC2 script. >>>>>>>>>>>> >>>>>>>>>>>> Currently, the Amazon Linux images have an 'ec2-user' set up >>>>>>>>>>>> for ssh instead of 'root'. >>>>>>>>>>>> >>>>>>>>>>>> I can see that the Spark-EC2 script allows you to specify which >>>>>>>>>>>> user to log in with, but even when I change this, the script fails >>>>>>>>>>>> for >>>>>>>>>>>> various reasons. And the output SEEMS that the script is still >>>>>>>>>>>> based on the >>>>>>>>>>>> specified user's home directory being '/root'. >>>>>>>>>>>> >>>>>>>>>>>> Am I using this script wrong? >>>>>>>>>>>> Has anyone had success with this 'ec2-user' user? >>>>>>>>>>>> Any ideas? >>>>>>>>>>>> >>>>>>>>>>>> Please and thank you, >>>>>>>>>>>> Marco. >>>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
