Re: Launching Spark cluster on EC2 with Ubuntu AMI
I know that Spark EC2 scripts are not guaranteed to work with custom AMIs but still, it should work… Nope, it shouldn’t, unfortunately. The Spark base AMIs are custom-built for spark-ec2. No other AMI will work unless it was built with that goal in mind. Using a random AMI from the Amazon marketplace is unlikely to work because there are several tools and packages (e.g. like git) that need to be on the AMI. Furthermore, the spark-ec2 scripts all assume a yum-based Linux distribution, so you won’t be able to use Ubuntu (and apt-get-based distro) without some significant changes to the shell scripts used to build the AMI. There is some work ongoing as part of SPARK-3821 <https://issues.apache.org/jira/browse/SPARK-3821> to make it easier to generate AMIs that work with spark-ec2. Nick On Sun Feb 22 2015 at 7:42:52 PM Ted Yu wrote: > bq. bash: git: command not found > > Looks like the AMI doesn't have git pre-installed. > > Cheers > > On Sun, Feb 22, 2015 at 4:29 PM, olegshirokikh wrote: > >> I'm trying to launch Spark cluster on AWS EC2 with custom AMI (Ubuntu) >> using >> the following: >> >> ./ec2/spark-ec2 --key-pair=*** --identity-file='/home/***.pem' >> --region=us-west-2 --zone=us-west-2b --spark-version=1.2.1 --slaves=2 >> --instance-type=t2.micro --ami=ami-29ebb519 --user=ubuntu launch >> spark-ubuntu-cluster >> >> Everything starts OK and instances are launched: >> >> Found 1 master(s), 2 slaves >> Waiting for all instances in cluster to enter 'ssh-ready' state. >> Generating cluster's SSH key on master. >> >> But then I'm getting the following SSH errors until it stops trying and >> quits: >> >> bash: git: command not found >> Connection to ***.us-west-2.compute.amazonaws.com closed. >> Error executing remote command, retrying after 30 seconds: Command >> '['ssh', >> '-o', 'StrictHostKeyChecking=no', '-i', '/home/***t.pem', '-o', >> 'UserKnownHostsFile=/dev/null', '-t', '-t', >> u'ubuntu@***.us-west-2.compute.amazonaws.com', 'rm -rf spark-ec2 && git >> clone https://github.com/mesos/spark-ec2.git -b v4']' returned non-zero >> exit >> status 127 >> >> I know that Spark EC2 scripts are not guaranteed to work with custom AMIs >> but still, it should work... Any advice would be greatly appreciated! >> >> >> >> >> -- >> View this message in context: >> http://apache-spark-user-list.1001560.n3.nabble.com/Launching-Spark-cluster-on-EC2-with-Ubuntu-AMI-tp21757.html >> Sent from the Apache Spark User List mailing list archive at Nabble.com. >> >> - >> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org >> For additional commands, e-mail: user-h...@spark.apache.org >> >> >
Re: Launching Spark cluster on EC2 with Ubuntu AMI
bq. bash: git: command not found Looks like the AMI doesn't have git pre-installed. Cheers On Sun, Feb 22, 2015 at 4:29 PM, olegshirokikh wrote: > I'm trying to launch Spark cluster on AWS EC2 with custom AMI (Ubuntu) > using > the following: > > ./ec2/spark-ec2 --key-pair=*** --identity-file='/home/***.pem' > --region=us-west-2 --zone=us-west-2b --spark-version=1.2.1 --slaves=2 > --instance-type=t2.micro --ami=ami-29ebb519 --user=ubuntu launch > spark-ubuntu-cluster > > Everything starts OK and instances are launched: > > Found 1 master(s), 2 slaves > Waiting for all instances in cluster to enter 'ssh-ready' state. > Generating cluster's SSH key on master. > > But then I'm getting the following SSH errors until it stops trying and > quits: > > bash: git: command not found > Connection to ***.us-west-2.compute.amazonaws.com closed. > Error executing remote command, retrying after 30 seconds: Command '['ssh', > '-o', 'StrictHostKeyChecking=no', '-i', '/home/***t.pem', '-o', > 'UserKnownHostsFile=/dev/null', '-t', '-t', > u'ubuntu@***.us-west-2.compute.amazonaws.com', 'rm -rf spark-ec2 && git > clone https://github.com/mesos/spark-ec2.git -b v4']' returned non-zero > exit > status 127 > > I know that Spark EC2 scripts are not guaranteed to work with custom AMIs > but still, it should work... Any advice would be greatly appreciated! > > > > > -- > View this message in context: > http://apache-spark-user-list.1001560.n3.nabble.com/Launching-Spark-cluster-on-EC2-with-Ubuntu-AMI-tp21757.html > Sent from the Apache Spark User List mailing list archive at Nabble.com. > > - > To unsubscribe, e-mail: user-unsubscr...@spark.apache.org > For additional commands, e-mail: user-h...@spark.apache.org > >
Launching Spark cluster on EC2 with Ubuntu AMI
I'm trying to launch Spark cluster on AWS EC2 with custom AMI (Ubuntu) using the following: ./ec2/spark-ec2 --key-pair=*** --identity-file='/home/***.pem' --region=us-west-2 --zone=us-west-2b --spark-version=1.2.1 --slaves=2 --instance-type=t2.micro --ami=ami-29ebb519 --user=ubuntu launch spark-ubuntu-cluster Everything starts OK and instances are launched: Found 1 master(s), 2 slaves Waiting for all instances in cluster to enter 'ssh-ready' state. Generating cluster's SSH key on master. But then I'm getting the following SSH errors until it stops trying and quits: bash: git: command not found Connection to ***.us-west-2.compute.amazonaws.com closed. Error executing remote command, retrying after 30 seconds: Command '['ssh', '-o', 'StrictHostKeyChecking=no', '-i', '/home/***t.pem', '-o', 'UserKnownHostsFile=/dev/null', '-t', '-t', u'ubuntu@***.us-west-2.compute.amazonaws.com', 'rm -rf spark-ec2 && git clone https://github.com/mesos/spark-ec2.git -b v4']' returned non-zero exit status 127 I know that Spark EC2 scripts are not guaranteed to work with custom AMIs but still, it should work... Any advice would be greatly appreciated! -- View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Launching-Spark-cluster-on-EC2-with-Ubuntu-AMI-tp21757.html Sent from the Apache Spark User List mailing list archive at Nabble.com. - To unsubscribe, e-mail: user-unsubscr...@spark.apache.org For additional commands, e-mail: user-h...@spark.apache.org
