[openstack-dev] [savanna] This request was rate-limited. (HTTP 413)
Hi, When I try to create a big hadoop cluster (21 nodes), sometimes I am getting this error: 2013-12-05 12:17:57.920 29553 ERROR savanna.context [-] Thread 'cluster-creating-8d093d9b-c675-4222-b53a-3319d54bc61f' fails with exception: 'This request was rate-limited. (HTTP 413)' 2013-12-05 12:17:57.920 29553 TRACE savanna.context Traceback (most recent call last): 2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/savanna/context.py", line 128, in wrapper 2013-12-05 12:17:57.920 29553 TRACE savanna.context func(*args, **kwargs) 2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/savanna/service/api.py", line 123, in _provision_cluster 2013-12-05 12:17:57.920 29553 TRACE savanna.context i.create_cluster(cluster) 2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/savanna/service/instances.py", line 56, in create_cluster 2013-12-05 12:17:57.920 29553 TRACE savanna.context _rollback_cluster_creation(cluster, ex) 2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__ 2013-12-05 12:17:57.920 29553 TRACE savanna.context self.gen.next() 2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/savanna/service/instances.py", line 36, in create_cluster 2013-12-05 12:17:57.920 29553 TRACE savanna.context _create_instances(cluster) 2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/savanna/service/instances.py", line 111, in _create_instances 2013-12-05 12:17:57.920 29553 TRACE savanna.context _run_instance(cluster, node_group, idx, aa_groups, userdata) 2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/savanna/service/instances.py", line 173, in _run_instance 2013-12-05 12:17:57.920 29553 TRACE savanna.context key_name=cluster.user_keypair_id) 2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/v1_1/servers.py", line 658, in create 2013-12-05 12:17:57.920 29553 TRACE savanna.context **boot_kwargs) 2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/base.py", line 402, in _boot 2013-12-05 12:17:57.920 29553 TRACE savanna.context return_raw=return_raw, **kwargs) 2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/base.py", line 145, in _create 2013-12-05 12:17:57.920 29553 TRACE savanna.context _resp, body = self.api.client.post(url, body=body) 2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/client.py", line 232, in post 2013-12-05 12:17:57.920 29553 TRACE savanna.context return self._cs_request(url, 'POST', **kwargs) 2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/client.py", line 213, in _cs_request 2013-12-05 12:17:57.920 29553 TRACE savanna.context **kwargs) 2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/client.py", line 195, in _time_request 2013-12-05 12:17:57.920 29553 TRACE savanna.context resp, body = self.request(url, method, **kwargs) 2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/client.py", line 189, in request 2013-12-05 12:17:57.920 29553 TRACE savanna.context raise exceptions.from_response(resp, body, url, method) 2013-12-05 12:17:57.920 29553 TRACE savanna.context OverLimit: This request was rate-limited. (HTTP 413) 2013-12-05 12:17:57.920 29553 TRACE savanna.context How I can prevent this to happen? Any help will be highly appreciated. Regards, Arindam ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [savanna] This request was rate-limited. (HTTP 413)
Hello Arindam, While deploying Hadoop cluster Savanna does pretty many API requests to Nova. Naturally, the number of requests is directly proportional to the size of the cluster. On the other hand Nova has a protection agains users abusing API with too many requests. It is called rate limiting. You need to set limits higher than they are right now if you want to spin-up a cluster of that size. You can find details in the Nova docs: http://docs.openstack.org/grizzly/openstack-compute/admin/content//configuring-compute-API.html Dmitry 2013/12/5 Arindam Choudhury > Hi, > > When I try to create a big hadoop cluster (21 nodes), sometimes I am > getting this error: > > 2013-12-05 12:17:57.920 29553 ERROR savanna.context [-] Thread > 'cluster-creating-8d093d9b-c675-4222-b53a-3319d54bc61f' fails with > exception: 'This request was rate-limited. (HTTP 413)' > 2013-12-05 12:17:57.920 29553 TRACE savanna.context Traceback (most recent > call last): > 2013-12-05 12:17:57.920 29553 TRACE savanna.context File > "/root/savanna/savanna/context.py", line 128, in wrapper > 2013-12-05 12:17:57.920 29553 TRACE savanna.context func(*args, > **kwargs) > 2013-12-05 12:17:57.920 29553 TRACE savanna.context File > "/root/savanna/savanna/service/api.py", line 123, in _provision_cluster > 2013-12-05 12:17:57.920 29553 TRACE savanna.context > i.create_cluster(cluster) > 2013-12-05 12:17:57.920 29553 TRACE savanna.context File > "/root/savanna/savanna/service/instances.py", line 56, in create_cluster > 2013-12-05 12:17:57.920 29553 TRACE savanna.context > _rollback_cluster_creation(cluster, ex) > 2013-12-05 12:17:57.920 29553 TRACE savanna.context File > "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__ > 2013-12-05 12:17:57.920 29553 TRACE savanna.context self.gen.next() > 2013-12-05 12:17:57.920 29553 TRACE savanna.context File > "/root/savanna/savanna/service/instances.py", line 36, in create_cluster > 2013-12-05 12:17:57.920 29553 TRACE savanna.context > _create_instances(cluster) > 2013-12-05 12:17:57.920 29553 TRACE savanna.context File > "/root/savanna/savanna/service/instances.py", line 111, in _create_instances > 2013-12-05 12:17:57.920 29553 TRACE savanna.context > _run_instance(cluster, node_group, idx, aa_groups, userdata) > 2013-12-05 12:17:57.920 29553 TRACE savanna.context File > "/root/savanna/savanna/service/instances.py", line 173, in _run_instance > 2013-12-05 12:17:57.920 29553 TRACE savanna.context > key_name=cluster.user_keypair_id) > 2013-12-05 12:17:57.920 29553 TRACE savanna.context File > "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/v1_1/servers.py", > line 658, in create > 2013-12-05 12:17:57.920 29553 TRACE savanna.context **boot_kwargs) > 2013-12-05 12:17:57.920 29553 TRACE savanna.context File > "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/base.py", > line 402, in _boot > 2013-12-05 12:17:57.920 29553 TRACE savanna.context > return_raw=return_raw, **kwargs) > 2013-12-05 12:17:57.920 29553 TRACE savanna.context File > "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/base.py", > line 145, in _create > 2013-12-05 12:17:57.920 29553 TRACE savanna.context _resp, body = > self.api.client.post(url, body=body) > 2013-12-05 12:17:57.920 29553 TRACE savanna.context File > "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/client.py", > line 232, in post > 2013-12-05 12:17:57.920 29553 TRACE savanna.context return > self._cs_request(url, 'POST', **kwargs) > 2013-12-05 12:17:57.920 29553 TRACE savanna.context File > "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/client.py", > line 213, in _cs_request > 2013-12-05 12:17:57.920 29553 TRACE savanna.context **kwargs) > 2013-12-05 12:17:57.920 29553 TRACE savanna.context File > "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/client.py", > line 195, in _time_request > 2013-12-05 12:17:57.920 29553 TRACE savanna.context resp, body = > self.request(url, method, **kwargs) > 2013-12-05 12:17:57.920 29553 TRACE savanna.context File > "/root/savanna/.tox/venv/lib/python2.6/site-packages/novaclient/client.py", > line 189, in request > 2013-12-05 12:17:57.920 29553 TRACE savanna.context raise > exceptions.from_response(resp, body, url, method) > 2013-12-05 12:17:57.920 29553 TRACE savanna.context OverLimit: This > request was rate-limited. (HTTP 413) > 2013-12-05 12:17:57.920 29553 TRACE savanna.context > > How I can prevent this to happen? Any help will be highly appreciated. > > Regards, > > Arindam > > ___ > OpenStack-dev mailing list > OpenStack-dev@lists.openstack.org > http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev > > ___ OpenStack-dev mailing list OpenStack-dev@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-dev
Re: [openstack-dev] [savanna] This request was rate-limited. (HTTP 413)
Hi, I introduced this in my nova api-paste.ini: [pipeline:openstack_compute_api_v2] pipeline = faultwrap authtoken keystonecontext ratelimit osapi_compute_app_v2 [pipeline:openstack_volume_api_v1] pipeline = faultwrap authtoken keystonecontext ratelimit osapi_volume_app_v1 [filter:ratelimit] paste.filter_factory = nova.api.openstack.compute.limits:RateLimitingMiddleware.factory limits =(POST, "*", .*, 100, MINUTE);(POST, "*/servers", ^/servers, 500, DAY);(PUT, "*", .*, 100, MINUTE);(GET, "*changes-since*", .*changes-since.*, 30, MINUTE);(DELETE, "*", .*, 100, MINUTE) I am getting this error when I try to restart openstack-nova-api: 2013-12-05 12:51:20.035 2350 ERROR nova.wsgi [-] Ambiguous section names ['composite:openstack_compute_api_v2', 'pipeline:openstack_compute_api_v2'] for section 'openstack_compute_api_v2' (prefixed by 'app' or 'application' or 'composite' or 'composit' or 'pipeline' or 'filter-app') found in config /etc/nova/api-paste.ini 2013-12-05 12:51:20.044 2549 INFO nova.ec2.wsgi.server [-] (2549) wsgi starting up on http://0.0.0.0:8773/ 2013-12-05 12:51:20.045 2350 CRITICAL nova [-] Could not load paste app 'osapi_compute' from /etc/nova/api-paste.ini 2013-12-05 12:51:20.045 2350 TRACE nova Traceback (most recent call last): 2013-12-05 12:51:20.045 2350 TRACE nova File "/usr/bin/nova-api", line 61, in 2013-12-05 12:51:20.045 2350 TRACE nova server = service.WSGIService(api, use_ssl=should_use_ssl) 2013-12-05 12:51:20.045 2350 TRACE nova File "/usr/lib/python2.6/site-packages/nova/service.py", line 598, in __init__ 2013-12-05 12:51:20.045 2350 TRACE nova self.app = self.loader.load_app(name) 2013-12-05 12:51:20.045 2350 TRACE nova File "/usr/lib/python2.6/site-packages/nova/wsgi.py", line 485, in load_app 2013-12-05 12:51:20.045 2350 TRACE nova raise exception.PasteAppNotFound(name=name, path=self.config_path) 2013-12-05 12:51:20.045 2350 TRACE nova PasteAppNotFound: Could not load paste app 'osapi_compute' from /etc/nova/api-paste.ini 2013-12-05 12:51:20.045 2350 TRACE nova 2013-12-05 12:51:20.407 2549 INFO nova.service [-] Parent process has died unexpectedly, exiting Regards, Arindam Date: Thu, 5 Dec 2013 15:37:21 +0400 From: dmescherya...@mirantis.com To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [savanna] This request was rate-limited. (HTTP 413) Hello Arindam, While deploying Hadoop cluster Savanna does pretty many API requests to Nova. Naturally, the number of requests is directly proportional to the size of the cluster. On the other hand Nova has a protection agains users abusing API with too many requests. It is called rate limiting. You need to set limits higher than they are right now if you want to spin-up a cluster of that size. You can find details in the Nova docs: http://docs.openstack.org/grizzly/openstack-compute/admin/content//configuring-compute-API.html Dmitry 2013/12/5 Arindam Choudhury Hi, When I try to create a big hadoop cluster (21 nodes), sometimes I am getting this error: 2013-12-05 12:17:57.920 29553 ERROR savanna.context [-] Thread 'cluster-creating-8d093d9b-c675-4222-b53a-3319d54bc61f' fails with exception: 'This request was rate-limited. (HTTP 413)' 2013-12-05 12:17:57.920 29553 TRACE savanna.context Traceback (most recent call last): 2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/savanna/context.py", line 128, in wrapper 2013-12-05 12:17:57.920 29553 TRACE savanna.context func(*args, **kwargs) 2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/savanna/service/api.py", line 123, in _provision_cluster 2013-12-05 12:17:57.920 29553 TRACE savanna.context i.create_cluster(cluster) 2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/savanna/service/instances.py", line 56, in create_cluster 2013-12-05 12:17:57.920 29553 TRACE savanna.context _rollback_cluster_creation(cluster, ex) 2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/usr/lib64/python2.6/contextlib.py", line 23, in __exit__ 2013-12-05 12:17:57.920 29553 TRACE savanna.context self.gen.next() 2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/savanna/service/instances.py", line 36, in create_cluster 2013-12-05 12:17:57.920 29553 TRACE savanna.context _create_instances(cluster) 2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/savanna/service/instances.py", line 111, in _create_instances 2013-12-05 12:17:57.920 29553 TRACE savanna.context _run_instance(cluster, node_group, idx, aa_groups, userdata) 2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savann
Re: [openstack-dev] [savanna] This request was rate-limited. (HTTP 413)
Hmm, not sure, I am not an expert in Nova. By the way the link I gave you is for Grizzly. If you are running a different release, take a look at that release doc as configuration might look different here. Dmitry 2013/12/5 Arindam Choudhury > Hi, > > I introduced this in my nova api-paste.ini: > > [pipeline:openstack_compute_api_v2] > pipeline = faultwrap authtoken keystonecontext ratelimit > osapi_compute_app_v2 > > [pipeline:openstack_volume_api_v1] > pipeline = faultwrap authtoken keystonecontext ratelimit > osapi_volume_app_v1 > > [filter:ratelimit] > paste.filter_factory = > nova.api.openstack.compute.limits:RateLimitingMiddleware.factory > limits =(POST, "*", .*, 100, MINUTE);(POST, "*/servers", ^/servers, 500, > DAY);(PUT, "*", .*, 100, MINUTE);(GET, "*changes-since*", > .*changes-since.*, 30, MINUTE);(DELETE, "*", .*, 100, MINUTE) > > I am getting this error when I try to restart openstack-nova-api: > > > 2013-12-05 12:51:20.035 2350 ERROR nova.wsgi [-] Ambiguous section names > ['composite:openstack_compute_api_v2', 'pipeline:openstack_compute_api_v2'] > for section 'openstack_compute_api_v2' (prefixed by 'app' or 'application' > or 'composite' or 'composit' or 'pipeline' or 'filter-app') found in config > /etc/nova/api-paste.ini > 2013-12-05 12:51:20.044 2549 INFO nova.ec2.wsgi.server [-] (2549) wsgi > starting up on http://0.0.0.0:8773/ > > 2013-12-05 12:51:20.045 2350 CRITICAL nova [-] Could not load paste app > 'osapi_compute' from /etc/nova/api-paste.ini > 2013-12-05 12:51:20.045 2350 TRACE nova Traceback (most recent call last): > 2013-12-05 12:51:20.045 2350 TRACE nova File "/usr/bin/nova-api", line > 61, in > 2013-12-05 12:51:20.045 2350 TRACE nova server = > service.WSGIService(api, use_ssl=should_use_ssl) > 2013-12-05 12:51:20.045 2350 TRACE nova File > "/usr/lib/python2.6/site-packages/nova/service.py", line 598, in __init__ > 2013-12-05 12:51:20.045 2350 TRACE nova self.app = > self.loader.load_app(name) > 2013-12-05 12:51:20.045 2350 TRACE nova File > "/usr/lib/python2.6/site-packages/nova/wsgi.py", line 485, in load_app > 2013-12-05 12:51:20.045 2350 TRACE nova raise > exception.PasteAppNotFound(name=name, path=self.config_path) > 2013-12-05 12:51:20.045 2350 TRACE nova PasteAppNotFound: Could not load > paste app 'osapi_compute' from /etc/nova/api-paste.ini > 2013-12-05 12:51:20.045 2350 TRACE nova > 2013-12-05 12:51:20.407 2549 INFO nova.service [-] Parent process has died > unexpectedly, exiting > > > Regards, > > Arindam > > -- > Date: Thu, 5 Dec 2013 15:37:21 +0400 > From: dmescherya...@mirantis.com > To: openstack-dev@lists.openstack.org > Subject: Re: [openstack-dev] [savanna] This request was rate-limited. > (HTTP 413) > > > Hello Arindam, > > While deploying Hadoop cluster Savanna does pretty many API requests to > Nova. Naturally, the number of requests is directly proportional to the > size of the cluster. > > On the other hand Nova has a protection agains users abusing API with too > many requests. It is called rate limiting. You need to set limits higher > than they are right now if you want to spin-up a cluster of that size. You > can find details in the Nova docs: > > http://docs.openstack.org/grizzly/openstack-compute/admin/content//configuring-compute-API.html > > Dmitry > > > 2013/12/5 Arindam Choudhury > > Hi, > > When I try to create a big hadoop cluster (21 nodes), sometimes I am > getting this error: > > 2013-12-05 12:17:57.920 29553 ERROR savanna.context [-] Thread > 'cluster-creating-8d093d9b-c675-4222-b53a-3319d54bc61f' fails with > exception: 'This request was rate-limited. (HTTP 413)' > 2013-12-05 12:17:57.920 29553 TRACE savanna.context Traceback (most recent > call last): > 2013-12-05 12:17:57.920 29553 TRACE savanna.context File > "/root/savanna/savanna/context.py", line 128, in wrapper > 2013-12-05 12:17:57.920 29553 TRACE savanna.context func(*args, > **kwargs) > 2013-12-05 12:17:57.920 29553 TRACE savanna.context File > "/root/savanna/savanna/service/api.py", line 123, in _provision_cluster > 2013-12-05 12:17:57.920 29553 TRACE savanna.context > i.create_cluster(cluster) > 2013-12-05 12:17:57.920 29553 TRACE savanna.context File > "/root/savanna/savanna/service/instances.py", line 56, in create_cluster > 2013-12-05 12:17:57.920 29553 TRACE savanna.context > _rollback_cluster_creation(cluster, ex) > 2013-12-05 12:17:57.920 29553 TRACE
Re: [openstack-dev] [savanna] This request was rate-limited. (HTTP 413)
Hi, I am using openstack grizzly. I just commented out [pipeline:openstack_compute_api_v2] pipeline = faultwrap authtoken keystonecontext ratelimit osapi_compute_app_v2 [pipeline:openstack_volume_api_v1] pipeline = faultwrap authtoken keystonecontext ratelimit osapi_volume_app_v1 and its working good. Thanks a lot. Regards, Arindam Date: Thu, 5 Dec 2013 16:34:17 +0400 From: dmescherya...@mirantis.com To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [savanna] This request was rate-limited. (HTTP 413) Hmm, not sure, I am not an expert in Nova. By the way the link I gave you is for Grizzly. If you are running a different release, take a look at that release doc as configuration might look different here. Dmitry 2013/12/5 Arindam Choudhury Hi, I introduced this in my nova api-paste.ini: [pipeline:openstack_compute_api_v2] pipeline = faultwrap authtoken keystonecontext ratelimit osapi_compute_app_v2 [pipeline:openstack_volume_api_v1] pipeline = faultwrap authtoken keystonecontext ratelimit osapi_volume_app_v1 [filter:ratelimit] paste.filter_factory = nova.api.openstack.compute.limits:RateLimitingMiddleware.factory limits =(POST, "*", .*, 100, MINUTE);(POST, "*/servers", ^/servers, 500, DAY);(PUT, "*", .*, 100, MINUTE);(GET, "*changes-since*", .*changes-since.*, 30, MINUTE);(DELETE, "*", .*, 100, MINUTE) I am getting this error when I try to restart openstack-nova-api: 2013-12-05 12:51:20.035 2350 ERROR nova.wsgi [-] Ambiguous section names ['composite:openstack_compute_api_v2', 'pipeline:openstack_compute_api_v2'] for section 'openstack_compute_api_v2' (prefixed by 'app' or 'application' or 'composite' or 'composit' or 'pipeline' or 'filter-app') found in config /etc/nova/api-paste.ini 2013-12-05 12:51:20.044 2549 INFO nova.ec2.wsgi.server [-] (2549) wsgi starting up on http://0.0.0.0:8773/ 2013-12-05 12:51:20.045 2350 CRITICAL nova [-] Could not load paste app 'osapi_compute' from /etc/nova/api-paste.ini 2013-12-05 12:51:20.045 2350 TRACE nova Traceback (most recent call last): 2013-12-05 12:51:20.045 2350 TRACE nova File "/usr/bin/nova-api", line 61, in 2013-12-05 12:51:20.045 2350 TRACE nova server = service.WSGIService(api, use_ssl=should_use_ssl) 2013-12-05 12:51:20.045 2350 TRACE nova File "/usr/lib/python2.6/site-packages/nova/service.py", line 598, in __init__ 2013-12-05 12:51:20.045 2350 TRACE nova self.app = self.loader.load_app(name) 2013-12-05 12:51:20.045 2350 TRACE nova File "/usr/lib/python2.6/site-packages/nova/wsgi.py", line 485, in load_app 2013-12-05 12:51:20.045 2350 TRACE nova raise exception.PasteAppNotFound(name=name, path=self.config_path) 2013-12-05 12:51:20.045 2350 TRACE nova PasteAppNotFound: Could not load paste app 'osapi_compute' from /etc/nova/api-paste.ini 2013-12-05 12:51:20.045 2350 TRACE nova 2013-12-05 12:51:20.407 2549 INFO nova.service [-] Parent process has died unexpectedly, exiting Regards, Arindam Date: Thu, 5 Dec 2013 15:37:21 +0400 From: dmescherya...@mirantis.com To: openstack-dev@lists.openstack.org Subject: Re: [openstack-dev] [savanna] This request was rate-limited. (HTTP 413) Hello Arindam, While deploying Hadoop cluster Savanna does pretty many API requests to Nova. Naturally, the number of requests is directly proportional to the size of the cluster. On the other hand Nova has a protection agains users abusing API with too many requests. It is called rate limiting. You need to set limits higher than they are right now if you want to spin-up a cluster of that size. You can find details in the Nova docs: http://docs.openstack.org/grizzly/openstack-compute/admin/content//configuring-compute-API.html Dmitry 2013/12/5 Arindam Choudhury Hi, When I try to create a big hadoop cluster (21 nodes), sometimes I am getting this error: 2013-12-05 12:17:57.920 29553 ERROR savanna.context [-] Thread 'cluster-creating-8d093d9b-c675-4222-b53a-3319d54bc61f' fails with exception: 'This request was rate-limited. (HTTP 413)' 2013-12-05 12:17:57.920 29553 TRACE savanna.context Traceback (most recent call last): 2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/savanna/context.py", line 128, in wrapper 2013-12-05 12:17:57.920 29553 TRACE savanna.context func(*args, **kwargs) 2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/savanna/service/api.py", line 123, in _provision_cluster 2013-12-05 12:17:57.920 29553 TRACE savanna.context i.create_cluster(cluster) 2013-12-05 12:17:57.920 29553 TRACE savanna.context File "/root/savanna/savanna/service/instances.py", line 56, in create_cluster 2013-12-05 12:17:57.920 29553 TRACE savanna.context _rollback
