Oh, sorry, I forgot to add: here are the extra lines in my spark_ec2.py
@205
"r3.large": "hvm",
"r3.xlarge": "hvm",
"r3.2xlarge": "hvm",
"r3.4xlarge": "hvm",
"r3.8xlarge": "hvm"
Clearly a masterpiece of hacking. :-) I haven't tested all of them. The r3
set seems to act like i2.
On Sun, Jun 1, 2014 at 12:45 AM, Jeremy Lee <unorthodox.engine...@gmail.com>
wrote:
> Hi there, Patrick. Thanks for the reply...
>
> It wouldn't surprise me that AWS Ubuntu has Python 2.7. Ubuntu is cool
> like that. :-)
>
> Alas, the Amazon Linux AMI (2014.03.1) does not, and it's the very first
> one on the recommended instance list. (Ubuntu is #4, after Amazon, RedHat,
> SUSE) So, users such as myself who deliberately pick the "Most Amazon-ish
> obvious first choice" find they picked the wrong one.
>
> But that's trivial compared to the failure of the cluster to come up,
> apparently due to the master's http configuration. Any help on that would
> be much appreciated... it's giving me serious grief.
>
>
>
> On Sat, May 31, 2014 at 1:37 PM, Patrick Wendell <pwend...@gmail.com>
> wrote:
>
>> Hi Jeremy,
>>
>> That's interesting, I don't think anyone has ever reported an issue
>> running these scripts due to Python incompatibility, but they may require
>> Python 2.7+. I regularly run them from the AWS Ubuntu 12.04 AMI... that
>> might be a good place to start. But if there is a straightforward way to
>> make them compatible with 2.6 we should do that.
>>
>> For r3.large, we can add that to the script. It's a newer type. Any
>> interest in contributing this?
>>
>> - Patrick
>>
>> On May 30, 2014 5:08 AM, "Jeremy Lee" <unorthodox.engine...@gmail.com>
>> wrote:
>>
>>>
>>> Hi there! I'm relatively new to the list, so sorry if this is a repeat:
>>>
>>> I just wanted to mention there are still problems with the EC2 scripts.
>>> Basically, they don't work.
>>>
>>> First, if you run the scripts on Amazon's own suggested version of
>>> linux, they break because amazon installs Python2.6.9, and the scripts use
>>> a couple of Python2.7 commands. I have to "sudo yum install python27", and
>>> then edit the spark-ec2 shell script to use that specific version.
>>> Annoying, but minor.
>>>
>>> (the base "python" command isn't upgraded to 2.7 on many systems,
>>> apparently because it would break yum)
>>>
>>> The second minor problem is that the script doesn't know about the
>>> "r3.large" servers... also easily fixed by adding to the spark_ec2.py
>>> script. Minor,
>>>
>>> The big problem is that after the EC2 cluster is provisioned, installed,
>>> set up, and everything, it fails to start up the webserver on the master.
>>> Here's the tail of the log:
>>>
>>> Starting GANGLIA gmond: [ OK ]
>>> Shutting down GANGLIA gmond: [FAILED]
>>> Starting GANGLIA gmond: [ OK ]
>>> Connection to ec2-54-183-82-48.us-west-1.compute.amazonaws.com closed.
>>> Shutting down GANGLIA gmond: [FAILED]
>>> Starting GANGLIA gmond: [ OK ]
>>> Connection to ec2-54-183-82-24.us-west-1.compute.amazonaws.com closed.
>>> Shutting down GANGLIA gmetad: [FAILED]
>>> Starting GANGLIA gmetad: [ OK ]
>>> Stopping httpd: [FAILED]
>>> Starting httpd: httpd: Syntax error on line 153 of
>>> /etc/httpd/conf/httpd.conf: Cannot load modules/mod_authn_alias.so into
>>> server: /etc/httpd/modules/mod_authn_alias.so: cannot open shared object
>>> file: No such file or directory
>>> [FAILED]
>>>
>>> Basically, the AMI you have chosen does not seem to have a "full"
>>> install of apache, and is missing several modules that are referred to in
>>> the httpd.conf file that is installed. The full list of missing modules is:
>>>
>>> authn_alias_module modules/mod_authn_alias.so
>>> authn_default_module modules/mod_authn_default.so
>>> authz_default_module modules/mod_authz_default.so
>>> ldap_module modules/mod_ldap.so
>>> authnz_ldap_module modules/mod_authnz_ldap.so
>>> disk_cache_module modules/mod_disk_cache.so
>>>
>>> Alas, even if these modules are commented out, the server still fails to
>>> start.
>>>
>>> root@ip-172-31-11-193 ~]$ service httpd start
>>> Starting httpd: AH00534: httpd: Configuration error: No MPM loaded.
>>>
>>> That means Spark 1.0.0 clusters on EC2 are Dead-On-Arrival when run
>>> according to the instructions. Sorry.
>>>
>>> Any suggestions on how to proceed? I'll keep trying to fix the
>>> webserver, but (a) changes to httpd.conf get blown away by "resume", and
>>> (b) anything I do has to be redone every time I provision another cluster.
>>> Ugh.
>>>
>>> --
>>> Jeremy Lee BCompSci(Hons)
>>> The Unorthodox Engineers
>>>
>>
>>
>>
>>
>>
>
>
> --
> Jeremy Lee BCompSci(Hons)
> The Unorthodox Engineers
>
--
Jeremy Lee BCompSci(Hons)
The Unorthodox Engineers
