Re: Spark ec2 lunch problem
spark-ec2 is the way to go however you may need to debug connectivity issues. For example do you know that the servers were correctly setup in AWS and can you access each node using ssh? If no then you need to work out why (it’s not a spark issue). If yes then you will need to work out why ssh via the spark-ec2 script is not working. I’ve used spark-ec2 successfully many times but have never used the —vpc-id and —subnet-id options and that may be the source of your problems, especially since it appears to be a hostname resolution issue. If you could confirm the above questions then maybe someone on the list can help diagnose the specific problem. --- Robin East Spark GraphX in Action Michael Malak and Robin East Manning Publications Co. http://www.manning.com/malak/ http://www.manning.com/malak/ On 24 Aug 2015, at 13:45, Garry Chen g...@cornell.edu wrote: So what is the best way to deploy spark cluster in EC2 environment any suggestions? Garry From: Akhil Das [mailto:ak...@sigmoidanalytics.com mailto:ak...@sigmoidanalytics.com] Sent: Friday, August 21, 2015 4:27 PM To: Garry Chen g...@cornell.edu mailto:g...@cornell.edu Cc: user@spark.apache.org mailto:user@spark.apache.org Subject: Re: Spark ec2 lunch problem It may happen that the version of spark-ec2 script you are using is buggy or sometime AWS have problem provisioning machines. On Aug 21, 2015 7:56 AM, Garry Chen g...@cornell.edu mailto:g...@cornell.edu wrote: Hi All, I am trying to lunch a spark ec2 cluster by running spark-ec2 --key-pair=key --identity-file=my.pem --vpc-id=myvpc --subnet-id=subnet-011 --spark-version=1.4.1 launch spark-cluster but getting following message endless. Please help. Warning: SSH connection error. (This could be temporary.) Host: SSH return code: 255 SSH output: ssh: Could not resolve hostname : Name or service not known
RE: Spark ec2 lunch problem
So what is the best way to deploy spark cluster in EC2 environment any suggestions? Garry From: Akhil Das [mailto:ak...@sigmoidanalytics.com] Sent: Friday, August 21, 2015 4:27 PM To: Garry Chen g...@cornell.edu Cc: user@spark.apache.org Subject: Re: Spark ec2 lunch problem It may happen that the version of spark-ec2 script you are using is buggy or sometime AWS have problem provisioning machines. On Aug 21, 2015 7:56 AM, Garry Chen g...@cornell.edumailto:g...@cornell.edu wrote: Hi All, I am trying to lunch a spark ec2 cluster by running spark-ec2 --key-pair=key --identity-file=my.pem --vpc-id=myvpc --subnet-id=subnet-011 --spark-version=1.4.1 launch spark-cluster but getting following message endless. Please help. Warning: SSH connection error. (This could be temporary.) Host: SSH return code: 255 SSH output: ssh: Could not resolve hostname : Name or service not known
Re: Spark ec2 lunch problem
Hey Garry, Have you verified that your particular VPC and subnet are open to the world? In particular, have you verified the route table attached to your VPC / subnet contains an internet gateway open to the public? I've run into this issue myself recently and that was the problem for me. -Andrew 2015-08-24 5:58 GMT-07:00 Robin East robin.e...@xense.co.uk: spark-ec2 is the way to go however you may need to debug connectivity issues. For example do you know that the servers were correctly setup in AWS and can you access each node using ssh? If no then you need to work out why (it’s not a spark issue). If yes then you will need to work out why ssh via the spark-ec2 script is not working. I’ve used spark-ec2 successfully many times but have never used the —vpc-id and —subnet-id options and that may be the source of your problems, especially since it appears to be a hostname resolution issue. If you could confirm the above questions then maybe someone on the list can help diagnose the specific problem. --- Robin East *Spark GraphX in Action* Michael Malak and Robin East Manning Publications Co. http://www.manning.com/malak/ On 24 Aug 2015, at 13:45, Garry Chen g...@cornell.edu wrote: So what is the best way to deploy spark cluster in EC2 environment any suggestions? Garry *From:* Akhil Das [mailto:ak...@sigmoidanalytics.com ak...@sigmoidanalytics.com] *Sent:* Friday, August 21, 2015 4:27 PM *To:* Garry Chen g...@cornell.edu *Cc:* user@spark.apache.org *Subject:* Re: Spark ec2 lunch problem It may happen that the version of spark-ec2 script you are using is buggy or sometime AWS have problem provisioning machines. On Aug 21, 2015 7:56 AM, Garry Chen g...@cornell.edu wrote: Hi All, I am trying to lunch a spark ec2 cluster by running spark-ec2 --key-pair=key --identity-file=my.pem --vpc-id=myvpc --subnet-id=subnet-011 --spark-version=1.4.1 launch spark-cluster but getting following message endless. Please help. Warning: SSH connection error. (This could be temporary.) Host: SSH return code: 255 SSH output: ssh: Could not resolve hostname : Name or service not known
Spark ec2 lunch problem
Hi All, I am trying to lunch a spark ec2 cluster by running spark-ec2 --key-pair=key --identity-file=my.pem --vpc-id=myvpc --subnet-id=subnet-011 --spark-version=1.4.1 launch spark-cluster but getting following message endless. Please help. Warning: SSH connection error. (This could be temporary.) Host: SSH return code: 255 SSH output: ssh: Could not resolve hostname : Name or service not known
RE: Spark ec2 lunch problem
No, the message never end. I have to ctrl-c out of it. Garry From: shahid ashraf [mailto:sha...@trialx.com] Sent: Friday, August 21, 2015 11:13 AM To: Garry Chen g...@cornell.edu Cc: user@spark.apache.org Subject: Re: Spark ec2 lunch problem Does the cluster work at the end ? On Fri, Aug 21, 2015 at 8:25 PM, Garry Chen g...@cornell.edumailto:g...@cornell.edu wrote: Hi All, I am trying to lunch a spark ec2 cluster by running spark-ec2 --key-pair=key --identity-file=my.pem --vpc-id=myvpc --subnet-id=subnet-011 --spark-version=1.4.1 launch spark-cluster but getting following message endless. Please help. Warning: SSH connection error. (This could be temporary.) Host: SSH return code: 255 SSH output: ssh: Could not resolve hostname : Name or service not known -- with Regards Shahid Ashraf
Re: Spark ec2 lunch problem
Does the cluster work at the end ? On Fri, Aug 21, 2015 at 8:25 PM, Garry Chen g...@cornell.edu wrote: Hi All, I am trying to lunch a spark ec2 cluster by running spark-ec2 --key-pair=key --identity-file=my.pem --vpc-id=myvpc --subnet-id=subnet-011 --spark-version=1.4.1 launch spark-cluster but getting following message endless. Please help. Warning: SSH connection error. (This could be temporary.) Host: SSH return code: 255 SSH output: ssh: Could not resolve hostname : Name or service not known -- with Regards Shahid Ashraf
Re: Spark ec2 lunch problem
It may happen that the version of spark-ec2 script you are using is buggy or sometime AWS have problem provisioning machines. On Aug 21, 2015 7:56 AM, Garry Chen g...@cornell.edu wrote: Hi All, I am trying to lunch a spark ec2 cluster by running spark-ec2 --key-pair=key --identity-file=my.pem --vpc-id=myvpc --subnet-id=subnet-011 --spark-version=1.4.1 launch spark-cluster but getting following message endless. Please help. Warning: SSH connection error. (This could be temporary.) Host: SSH return code: 255 SSH output: ssh: Could not resolve hostname : Name or service not known
