Hi there, Patrick. Thanks for the reply... It wouldn't surprise me that AWS Ubuntu has Python 2.7. Ubuntu is cool like that. :-)
Alas, the Amazon Linux AMI (2014.03.1) does not, and it's the very first one on the recommended instance list. (Ubuntu is #4, after Amazon, RedHat, SUSE) So, users such as myself who deliberately pick the "Most Amazon-ish obvious first choice" find they picked the wrong one. But that's trivial compared to the failure of the cluster to come up, apparently due to the master's http configuration. Any help on that would be much appreciated... it's giving me serious grief. On Sat, May 31, 2014 at 1:37 PM, Patrick Wendell <pwend...@gmail.com> wrote: > Hi Jeremy, > > That's interesting, I don't think anyone has ever reported an issue > running these scripts due to Python incompatibility, but they may require > Python 2.7+. I regularly run them from the AWS Ubuntu 12.04 AMI... that > might be a good place to start. But if there is a straightforward way to > make them compatible with 2.6 we should do that. > > For r3.large, we can add that to the script. It's a newer type. Any > interest in contributing this? > > - Patrick > > On May 30, 2014 5:08 AM, "Jeremy Lee" <unorthodox.engine...@gmail.com> > wrote: > >> >> Hi there! I'm relatively new to the list, so sorry if this is a repeat: >> >> I just wanted to mention there are still problems with the EC2 scripts. >> Basically, they don't work. >> >> First, if you run the scripts on Amazon's own suggested version of linux, >> they break because amazon installs Python2.6.9, and the scripts use a >> couple of Python2.7 commands. I have to "sudo yum install python27", and >> then edit the spark-ec2 shell script to use that specific version. >> Annoying, but minor. >> >> (the base "python" command isn't upgraded to 2.7 on many systems, >> apparently because it would break yum) >> >> The second minor problem is that the script doesn't know about the >> "r3.large" servers... also easily fixed by adding to the spark_ec2.py >> script. Minor, >> >> The big problem is that after the EC2 cluster is provisioned, installed, >> set up, and everything, it fails to start up the webserver on the master. >> Here's the tail of the log: >> >> Starting GANGLIA gmond: [ OK ] >> Shutting down GANGLIA gmond: [FAILED] >> Starting GANGLIA gmond: [ OK ] >> Connection to ec2-54-183-82-48.us-west-1.compute.amazonaws.com closed. >> Shutting down GANGLIA gmond: [FAILED] >> Starting GANGLIA gmond: [ OK ] >> Connection to ec2-54-183-82-24.us-west-1.compute.amazonaws.com closed. >> Shutting down GANGLIA gmetad: [FAILED] >> Starting GANGLIA gmetad: [ OK ] >> Stopping httpd: [FAILED] >> Starting httpd: httpd: Syntax error on line 153 of >> /etc/httpd/conf/httpd.conf: Cannot load modules/mod_authn_alias.so into >> server: /etc/httpd/modules/mod_authn_alias.so: cannot open shared object >> file: No such file or directory >> [FAILED] >> >> Basically, the AMI you have chosen does not seem to have a "full" install >> of apache, and is missing several modules that are referred to in the >> httpd.conf file that is installed. The full list of missing modules is: >> >> authn_alias_module modules/mod_authn_alias.so >> authn_default_module modules/mod_authn_default.so >> authz_default_module modules/mod_authz_default.so >> ldap_module modules/mod_ldap.so >> authnz_ldap_module modules/mod_authnz_ldap.so >> disk_cache_module modules/mod_disk_cache.so >> >> Alas, even if these modules are commented out, the server still fails to >> start. >> >> root@ip-172-31-11-193 ~]$ service httpd start >> Starting httpd: AH00534: httpd: Configuration error: No MPM loaded. >> >> That means Spark 1.0.0 clusters on EC2 are Dead-On-Arrival when run >> according to the instructions. Sorry. >> >> Any suggestions on how to proceed? I'll keep trying to fix the webserver, >> but (a) changes to httpd.conf get blown away by "resume", and (b) anything >> I do has to be redone every time I provision another cluster. Ugh. >> >> -- >> Jeremy Lee BCompSci(Hons) >> The Unorthodox Engineers >> > > > > > -- Jeremy Lee BCompSci(Hons) The Unorthodox Engineers
