Hi Jeremy, That's interesting, I don't think anyone has ever reported an issue running these scripts due to Python incompatibility, but they may require Python 2.7+. I regularly run them from the AWS Ubuntu 12.04 AMI... that might be a good place to start. But if there is a straightforward way to make them compatible with 2.6 we should do that.
For r3.large, we can add that to the script. It's a newer type. Any interest in contributing this? - Patrick On May 30, 2014 5:08 AM, "Jeremy Lee" <unorthodox.engine...@gmail.com> wrote: > > Hi there! I'm relatively new to the list, so sorry if this is a repeat: > > I just wanted to mention there are still problems with the EC2 scripts. > Basically, they don't work. > > First, if you run the scripts on Amazon's own suggested version of linux, > they break because amazon installs Python2.6.9, and the scripts use a > couple of Python2.7 commands. I have to "sudo yum install python27", and > then edit the spark-ec2 shell script to use that specific version. > Annoying, but minor. > > (the base "python" command isn't upgraded to 2.7 on many systems, > apparently because it would break yum) > > The second minor problem is that the script doesn't know about the > "r3.large" servers... also easily fixed by adding to the spark_ec2.py > script. Minor, > > The big problem is that after the EC2 cluster is provisioned, installed, > set up, and everything, it fails to start up the webserver on the master. > Here's the tail of the log: > > Starting GANGLIA gmond: [ OK ] > Shutting down GANGLIA gmond: [FAILED] > Starting GANGLIA gmond: [ OK ] > Connection to ec2-54-183-82-48.us-west-1.compute.amazonaws.com closed. > Shutting down GANGLIA gmond: [FAILED] > Starting GANGLIA gmond: [ OK ] > Connection to ec2-54-183-82-24.us-west-1.compute.amazonaws.com closed. > Shutting down GANGLIA gmetad: [FAILED] > Starting GANGLIA gmetad: [ OK ] > Stopping httpd: [FAILED] > Starting httpd: httpd: Syntax error on line 153 of > /etc/httpd/conf/httpd.conf: Cannot load modules/mod_authn_alias.so into > server: /etc/httpd/modules/mod_authn_alias.so: cannot open shared object > file: No such file or directory > [FAILED] > > Basically, the AMI you have chosen does not seem to have a "full" install > of apache, and is missing several modules that are referred to in the > httpd.conf file that is installed. The full list of missing modules is: > > authn_alias_module modules/mod_authn_alias.so > authn_default_module modules/mod_authn_default.so > authz_default_module modules/mod_authz_default.so > ldap_module modules/mod_ldap.so > authnz_ldap_module modules/mod_authnz_ldap.so > disk_cache_module modules/mod_disk_cache.so > > Alas, even if these modules are commented out, the server still fails to > start. > > root@ip-172-31-11-193 ~]$ service httpd start > Starting httpd: AH00534: httpd: Configuration error: No MPM loaded. > > That means Spark 1.0.0 clusters on EC2 are Dead-On-Arrival when run > according to the instructions. Sorry. > > Any suggestions on how to proceed? I'll keep trying to fix the webserver, > but (a) changes to httpd.conf get blown away by "resume", and (b) anything > I do has to be redone every time I provision another cluster. Ugh. > > -- > Jeremy Lee BCompSci(Hons) > The Unorthodox Engineers >
