[
https://issues.apache.org/jira/browse/BROOKLYN-394?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15681595#comment-15681595
]
Aled Sage commented on BROOKLYN-394:
------------------------------------
This is at one level surprising because jclouds does automatic retry with
exponential backoff. However, I guess if we're doing too many calls then even
with the backoff we might still get it repeatedly rejected.
My understanding is that AWS will rate-limit based on the nature (as well as
number) of API calls. For example, if creating/modifying security groups is a
more exensive operation (from AWS's perspective) than a simple poll for a
machine's state, then those requests would cause rate-limiting sooner.
Your list sounds good. A couple of other suggestions spring to mind:
1. We could also tweak {{jclouds.retries-delay-start}} and
{{jclouds.max-retries}}, which default to 50ms and 5 respectively. The max
period between retries is {{10 * delayStart}}, so would just be 500ms; the time
increases exponentially in powers of 2. See
https://github.com/jclouds/jclouds/blob/master/core/src/main/java/org/jclouds/http/handlers/BackoffLimitedRetryHandler.java#L116-L122
2. We should try to be smart about how we open ports in a security group, to
minimise the number of API calls. We should review the jclouds code more
closely to see how it behaves when using the standard jclouds {{inboundPorts:
...}} - including looking at the wire log for what actually happens. It looks
like it does make a single API call for aws-ec2 (see
https://github.com/jclouds/jclouds/blob/rel/jclouds-1.9.2/providers/aws-ec2/src/main/java/org/jclouds/aws/ec2/compute/loaders/AWSEC2CreateSecurityGroupIfNeeded.java#L89-L128);
it also looks at first blush that the use of {{RangeSet}} will squash together
contiguous ports into a minimum number of ranges.
Option (1) is probably worth adding to Brooklyn asap to override the default
(e.g. {{jclouds.retries-delay-start=500}}).
> "Request limit exceeded" on Amazon
> ----------------------------------
>
> Key: BROOKLYN-394
> URL: https://issues.apache.org/jira/browse/BROOKLYN-394
> Project: Brooklyn
> Issue Type: Bug
> Reporter: Svetoslav Neykov
>
> Any moderately sized blueprint could trigger {{Request limit exceeded}} on
> Amazon (say kubernetes). The only way users have control over the request
> rate is by setting {{maxConcurrentMachineCreations}} with the current
> recommended value of 3 (see clocker.io).
> It's bad user experience if one needs to adapt the location based on the
> blueprint.
> Possible steps to improve:
> * Add to troubleshooting documentation
> * Make maxConcurrentMachineCreations default to 3
> * Check are we polling for machine creation too often.
> * Check how many requests are we hitting Amazon with (per created machine)
> * The number of requests per machine could vary from blueprint to blueprint
> (say if the blueprint is creating security networks, using other amazon
> services). Is there a way to throttle our requests to amazon and stay below a
> certain limit per second?
> * I've hit the error during machine tear down as well, so
> {{maxConcurrentMachineCreations}} is not enough to work around
> Some docs on rate limits at
> http://docs.aws.amazon.com/general/latest/gr/aws_service_limits.html.
> Related: https://github.com/jclouds/legacy-jclouds/issues/1214
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
