Public bug reported: We upgraded environment to OpenStack Kilo 2015.1.2 and then we cannot use heat autoscaling feature anymore.
Keystone version 1:2015.1.2-0ubuntu2~cloud0 Heat version 2015.1.2-0ubuntu1~cloud0 - First we hit authentication failure on ec2tokens heat-api-cfn.log 2016-01-15 14:21:07.505 6590 INFO heat.api.aws.ec2token [-] Checking AWS credentials.. 2016-01-15 14:21:07.507 6590 INFO heat.api.aws.ec2token [-] AWS credentials found, checking against keystone. 2016-01-15 14:21:07.509 6590 INFO heat.api.aws.ec2token [-] Authenticating with http://10.200.26.10:5000/ec2tokens 2016-01-15 14:21:07.533 6590 INFO heat.api.aws.ec2token [-] AWS authentication failure. 2016-01-15 14:21:07.535 6590 INFO eventlet.wsgi.server [-] 10.200.26.11 - - [15/Jan/2016 14:21:07] "POST /v1/signal/arn%3Aopenstack%3Aheat%3A%3A4a28da719dc64deb9d1d5cfa770bbca2%3Astacks%2Fheat_autoscale_cpu_vpc20%2F3a268b22-d436-4cf8-9088-8418702be09a%2Fresources%2Fserver_scaleup_policy?Timestamp=2016-01-15T13%3A14%3A22Z&SignatureMethod=HmacSHA256&AWSAccessKeyId=f042045aabc74728a49059a4c52226bf&SignatureVersion=2&Signature=GomCM13riX1UYu3OpJZ3ERXpfekHkmeACA4DznTL6g8%3D HTTP/1.1" 403 301 0.031258 We applied patch to keystone https://bugs.launchpad.net/openstack- ansible/+bug/1515485 https://git.openstack.org/cgit/openstack/keystone/commit/?id=f66badae63b1544c3bcc83e0c1afc8e9cdc71bab Then we hit error on python-novaclient auth boot during autoscaling heat-engine.log 2016-01-15 06:43:05.836 11301 ERROR heat.engine.stack [req-fd1fd187-1be5-489a-803b-8c08e491ebd4 - 4a28da719dc64deb9d1d5cfa770bbca2] Exception: (NotFound(),) 2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack Traceback (most recent call last): 2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/heat/engine/stack.py", line 593, in validate 2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack result = res.validate() 2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/heat/engine/resources/openstack/nova/server.py", line 1254, in validate 2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack limits = self.client_plugin().absolute_limits() 2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/nova.py", line 439, in absolute_limits 2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack limits = self.client().limits.get() 2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/v2/limits.py", line 100, in get 2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack return self._get("/limits%s" % query_string, "limits") 2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 147, in _get 2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack _resp, body = self.api.client.get(url) 2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 440, in get 2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack return self._cs_request(url, 'GET', **kwargs) 2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 427, in _cs_request 2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack self.authenticate() 2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 569, in authenticate 2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack self._v2_auth(auth_url) 2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 634, in _v2_auth 2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack return self._authenticate(url, body) 2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 647, in _authenticate 2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack **kwargs) 2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 392, in _time_request 2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack resp, body = self.request(url, method, **kwargs) 2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 386, in request 2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack raise exceptions.from_response(resp, body, url, method) 2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack NotFound: The resource could not be found. (HTTP 404) 2016-01-15 06:43:05.836 11301 TRACE heat.engine.stack 2016-01-15 06:43:05.838 11301 INFO heat.engine.stack [req-fd1fd187-1be5-489a-803b-8c08e491ebd4 - 4a28da719dc64deb9d1d5cfa770bbca2] Failed to validate: The resource could not be found. (HTTP 404) It was caused by bad auth_url argument passed into novaclient - http://10.0.170.10:5000/v3/v2.0 We modified /usr/lib/python2.7/dist- packages/heat/engine/clients/os/nova.py to replace v3 to v2.0 args = { 'project_id': self.context.tenant, 'auth_url': self.context.auth_url.replace('/v3','/v2.0'), 'service_type': 'compute', 'username': None, 'api_key': None, 'extensions': extensions, 'endpoint_type': endpoint_type, 'http_log_debug': self._get_client_option('nova', 'http_log_debug'), 'cacert': self._get_client_option('nova', 'ca_file'), 'insecure': self._get_client_option('nova', 'insecure') } And then we got following error heat-engine.log 2016-01-15 14:36:05.464 8195 ERROR heat.engine.stack [req-b19963cf-81d3-482d-b31a-27d7b0862b89 - 4a28da719dc64deb9d1d5cfa770bbca2] Exception: (ClientException(),) 2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack Traceback (most recent call last): 2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/heat/engine/stack.py", line 593, in validate 2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack result = res.validate() 2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/heat/engine/resources/openstack/nova/server.py", line 1254, in validate 2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack limits = self.client_plugin().absolute_limits() 2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/heat/engine/clients/os/nova.py", line 439, in absolute_limits 2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack limits = self.client().limits.get() 2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/v2/limits.py", line 100, in get 2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack return self._get("/limits%s" % query_string, "limits") 2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/base.py", line 147, in _get 2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack _resp, body = self.api.client.get(url) 2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 439, in get 2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack return self._cs_request(url, 'GET', **kwargs) 2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 426, in _cs_request 2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack self.authenticate() 2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 561, in authenticate 2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack auth_url = self._v1_auth(auth_url) 2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack File "/usr/lib/python2.7/dist-packages/novaclient/client.py", line 609, in _v1_auth 2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack raise exceptions.from_response(resp, body, url) 2016-01-15 14:36:05.464 8195 TRACE heat.engine.stack ClientException: Unknown Error (HTTP 300) And then we have found that there is only auth_token and no username and password. It seems that python-novaclient cannot use v3 authentication token over v2.0. Has anybody tested heat autoscaling on latest Kilo update? ** Affects: heat Importance: Undecided Status: New ** Affects: keystone Importance: Undecided Status: New ** Affects: heat (Ubuntu) Importance: Undecided Status: Confirmed ** Tags: autoscaling heat in-stable-kilo keystone ** Also affects: keystone Importance: Undecided Status: New ** Also affects: heat (Ubuntu) Importance: Undecided Status: New -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to heat in Ubuntu. https://bugs.launchpad.net/bugs/1534655 Title: Autoscaling auth failure in OpenStack Kilo 2015.1.2 To manage notifications about this bug go to: https://bugs.launchpad.net/heat/+bug/1534655/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
