[
https://issues.apache.org/jira/browse/WHIRR-455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrei Savu updated WHIRR-455:
------------------------------
Description:
Trying to use whirr to create clusters of 50+ nodes on ec2 using spot instances.
It will eventually fail trying to poll the security groups for the pending
instances, and eventually ec2 will block the requests due to the number of
requests.
The spot bids will be created, and instances will be started in them, but whirr
can't see them due to being locked out.
I will attach a log file, with jclouds ec2 logging turned up. This may be a
jclouds issue, but I figured I'd start here.
configuration file:
{noformat}
# Change the cluster name here
whirr.cluster-name=hadoop
# Change the number of machines in the cluster here
whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,50
hadoop-datanode+hadoop-tasktracker
# Uncomment out these lines to run CDH
whirr.hadoop.install-function=install_cdh_hadoop
whirr.hadoop.configure-function=configure_cdh_hadoop
# For EC2 set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
whirr.provider=aws-ec2
whirr.identity=
whirr.credential=
# The size of the instance to use. See http://aws.amazon.com/ec2/instance-types/
whirr.hardware-id=c1.xlarge
# Ubuntu 10.04 LTS Lucid. See http://alestic.com/
whirr.image-id=us-east-1/ami-da0cf8b3
# If you choose a different location, make sure whirr.image-id is updated too
whirr.location-id=us-east-1a
# By default use the user system SSH keys. Override them here.
whirr.private-key-file=${sys:user.home}/.ssh/id_rsa_hadoop
whirr.public-key-file=${whirr.private-key-file}.pub
# You can also specify the spot instance price
# http://aws.amazon.com/ec2/spot-instances/
whirr.aws-ec2-spot-price=0.68
{noformat}
was:
Trying to use whirr to create clusters of 50+ nodes on ec2 using spot instances.
It will eventually fail trying to poll the security groups for the pending
instances, and eventually ec2 will block the requests due to the number of
requests.
The spot bids will be created, and instances will be started in them, but whirr
can't see them due to being locked out.
I will attach a log file, with jclouds ec2 logging turned up. This may be a
jclouds issue, but I figured I'd start here.
configuration file:
# Change the cluster name here
whirr.cluster-name=hadoop
# Change the number of machines in the cluster here
whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,50
hadoop-datanode+hadoop-tasktracker
# Uncomment out these lines to run CDH
whirr.hadoop.install-function=install_cdh_hadoop
whirr.hadoop.configure-function=configure_cdh_hadoop
# For EC2 set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment variables.
whirr.provider=aws-ec2
whirr.identity=
whirr.credential=
# The size of the instance to use. See http://aws.amazon.com/ec2/instance-types/
whirr.hardware-id=c1.xlarge
# Ubuntu 10.04 LTS Lucid. See http://alestic.com/
whirr.image-id=us-east-1/ami-da0cf8b3
# If you choose a different location, make sure whirr.image-id is updated too
whirr.location-id=us-east-1a
# By default use the user system SSH keys. Override them here.
whirr.private-key-file=${sys:user.home}/.ssh/id_rsa_hadoop
whirr.public-key-file=${whirr.private-key-file}.pub
# You can also specify the spot instance price
# http://aws.amazon.com/ec2/spot-instances/
whirr.aws-ec2-spot-price=0.68
> Whirr cannot create large ec2 clusters -- too chatty -- RequestLimitExceeded
> ----------------------------------------------------------------------------
>
> Key: WHIRR-455
> URL: https://issues.apache.org/jira/browse/WHIRR-455
> Project: Whirr
> Issue Type: Bug
> Components: core
> Affects Versions: 0.7.0
> Environment: bash-3.2$ uname -a
> Darwin bcotton-mbp.f4tech.com 11.2.0 Darwin Kernel Version 11.2.0: Tue Aug 9
> 20:54:00 PDT 2011; root:xnu-1699.24.8~1/RELEASE_X86_64 x86_64
> bash-3.2$ java -version
> java version "1.6.0_29"
> Java(TM) SE Runtime Environment (build 1.6.0_29-b11-402-11M3527)
> Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02-402, mixed mode)
> Reporter: Bob Cotton
> Attachments: whirr.log.too-many-requests.gz
>
>
> Trying to use whirr to create clusters of 50+ nodes on ec2 using spot
> instances.
> It will eventually fail trying to poll the security groups for the pending
> instances, and eventually ec2 will block the requests due to the number of
> requests.
> The spot bids will be created, and instances will be started in them, but
> whirr can't see them due to being locked out.
> I will attach a log file, with jclouds ec2 logging turned up. This may be a
> jclouds issue, but I figured I'd start here.
> configuration file:
> {noformat}
> # Change the cluster name here
> whirr.cluster-name=hadoop
> # Change the number of machines in the cluster here
> whirr.instance-templates=1 hadoop-namenode+hadoop-jobtracker,50
> hadoop-datanode+hadoop-tasktracker
> # Uncomment out these lines to run CDH
> whirr.hadoop.install-function=install_cdh_hadoop
> whirr.hadoop.configure-function=configure_cdh_hadoop
> # For EC2 set AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY environment
> variables.
> whirr.provider=aws-ec2
> whirr.identity=
> whirr.credential=
> # The size of the instance to use. See
> http://aws.amazon.com/ec2/instance-types/
> whirr.hardware-id=c1.xlarge
> # Ubuntu 10.04 LTS Lucid. See http://alestic.com/
> whirr.image-id=us-east-1/ami-da0cf8b3
> # If you choose a different location, make sure whirr.image-id is updated too
> whirr.location-id=us-east-1a
> # By default use the user system SSH keys. Override them here.
> whirr.private-key-file=${sys:user.home}/.ssh/id_rsa_hadoop
> whirr.public-key-file=${whirr.private-key-file}.pub
> # You can also specify the spot instance price
> # http://aws.amazon.com/ec2/spot-instances/
> whirr.aws-ec2-spot-price=0.68
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
