spark-ec2 is the way to go however you may need to debug connectivity issues. For example do you know that the servers were correctly setup in AWS and can you access each node using ssh? If no then you need to work out why (it’s not a spark issue). If yes then you will need to work out why ssh via the spark-ec2 script is not working.
I’ve used spark-ec2 successfully many times but have never used the —vpc-id and —subnet-id options and that may be the source of your problems, especially since it appears to be a hostname resolution issue. If you could confirm the above questions then maybe someone on the list can help diagnose the specific problem. ------------------------------------------------------------------------------- Robin East Spark GraphX in Action Michael Malak and Robin East Manning Publications Co. http://www.manning.com/malak/ <http://www.manning.com/malak/> > On 24 Aug 2015, at 13:45, Garry Chen <g...@cornell.edu> wrote: > > So what is the best way to deploy spark cluster in EC2 environment any > suggestions? > > Garry > > From: Akhil Das [mailto:ak...@sigmoidanalytics.com > <mailto:ak...@sigmoidanalytics.com>] > Sent: Friday, August 21, 2015 4:27 PM > To: Garry Chen <g...@cornell.edu <mailto:g...@cornell.edu>> > Cc: user@spark.apache.org <mailto:user@spark.apache.org> > Subject: Re: Spark ec2 lunch problem > > It may happen that the version of spark-ec2 script you are using is buggy or > sometime AWS have problem provisioning machines. > > On Aug 21, 2015 7:56 AM, "Garry Chen" <g...@cornell.edu > <mailto:g...@cornell.edu>> wrote: > Hi All, > I am trying to lunch a spark ec2 cluster by running > spark-ec2 --key-pair=key --identity-file=my.pem --vpc-id=myvpc > --subnet-id=subnet-011 --spark-version=1.4.1 launch spark-cluster but getting > following message endless. Please help. > > > Warning: SSH connection error. (This could be temporary.) > Host: > SSH return code: 255 > SSH output: ssh: Could not resolve hostname : Name or service not known
