Hi,
I am using Kilo release on CentOS. We have recently enabled multiple regions
and it seems that Cinder have some problems with multiple endpoints.
Thinks are working fine with nova but cinder is behaving strange. Here are my
endpoints
————
[root@controller: ~] # openstack service list
+----------------------------------+----------------+--------------+
| ID | Name | Type |
+----------------------------------+----------------+--------------+
| 0a33e6f259794ff2a99e626be37c0c2b | cinderv2-hpc2n | volumev2 |
| 1fcae9bd76304853a3168c39c7fe8e6b | nova | compute |
| 2c7828120c294d3f82e3a17835babb85 | neutron | network |
| 3804fcd8f9494d30b589b55fe6abb811 | nova-hpc2n | compute |
| 478eff4e96464ae8a958ba29f750b14c | glance | image |
| 4a5a771d915e43c28e66538b8bc6e625 | cinder | volume |
| 72d1be82b2e5478dbf0f3fb9e7ba969d | cinderv2 | volumev2 |
| 97f977f8a7a04bae89da167fd25dc06c | glance-hpc2n | image |
| a985795b49e2440db82970b81248c86e | cinder-hpc2n | volume |
| dccd39b92ab547ddaf9047b38620145a | swift | object-store |
| ebb1660d1d9746759a48de921521bfad | keystone | identity |
+----------------------------------+----------------+--------------+
[root@controller: ~] # openstack endpoint show a985795b49e2440db82970b81248c86e
+--------------+--------------------------------------------------+
| Field | Value |
+--------------+--------------------------------------------------+
| adminurl | http://<URL>:8776/v1/%(tenant_id)s |
| enabled | True |
| id | d4003e91ddf24cfb9fa497da81b01a18 |
| internalurl | http://<URL>:8776/v1/%(tenant_id)s |
| publicurl | http://<URL>:8776/v1/%(tenant_id)s |
| region | HPC2N |
| service_id | a985795b49e2440db82970b81248c86e |
| service_name | cinder-hpc2n |
| service_type | volume |
+--------------+--------------------------------------------------+
[root@controller: ~] # openstack endpoint show 4a5a771d915e43c28e66538b8bc6e625
+--------------+------------------------------------------------+
| Field | Value |
+--------------+------------------------------------------------+
| adminurl | http://<URL>:8776/v1/%(tenant_id)s |
| enabled | True |
| id | 5f19c0b535674dbd9e318c7b6d61b3bc |
| internalurl | http://<URL>:8776/v1/%(tenant_id)s |
| publicurl | http://<URL>:8776/v1/%(tenant_id)s |
| region | regionOne |
| service_id | 4a5a771d915e43c28e66538b8bc6e625 |
| service_name | cinder |
| service_type | volume |
+--------------+------------------------------------------------+
And same for v2 endpoints
————
——— nova-api.log ———
achmentController object at 0x598a3d0>>, body: {"volumeAttachment": {"device":
"", "volumeId": "93d96eab-e3fd-4131-9549-ed51e7299da2"}} _process_stack
/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:780
2015-12-08 12:52:05.847 3376 INFO nova.api.openstack.compute.contrib.volumes
[req-d6e1380d-c6bc-4911-b2c6-251bc8b4c62c a62c20fdf99c443a924f0d50a51514b1
3c9d997982e04c6db0e02b82fa18fdd8 - - -] Attach volume
93d96eab-e3fd-4131-9549-ed51e7299da2 to instance
3a4c8722-52a7-48f2-beb7-db8938698a0d at
2015-12-08 12:52:05.847 3376 DEBUG nova.compute.api
[req-d6e1380d-c6bc-4911-b2c6-251bc8b4c62c a62c20fdf99c443a924f0d50a51514b1
3c9d997982e04c6db0e02b82fa18fdd8 - - -] [instance:
3a4c8722-52a7-48f2-beb7-db8938698a0d] Fetching instance by UUID get
/usr/lib/python2.7/site-packages/nova/compute/api.py:1911
2015-12-08 12:52:05.988 3376 ERROR nova.api.openstack
[req-d6e1380d-c6bc-4911-b2c6-251bc8b4c62c a62c20fdf99c443a924f0d50a51514b1
3c9d997982e04c6db0e02b82fa18fdd8 - - -] Caught error: internalURL endpoint for
volume service named cinder not found
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack Traceback (most recent
call last):
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/nova/api/openstack/__init__.py", line 125, in
__call__
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
req.get_response(self.application)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack application,
catch_exc_info=False)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in
call_application
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack app_iter =
application(self.environ, start_response)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return resp(environ,
start_response)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py",
line 634, in __call__
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
self._call_app(env, start_response)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py",
line 554, in _call_app
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return self._app(env,
_fake_start_response)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return resp(environ,
start_response)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return resp(environ,
start_response)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack response =
self.app(environ, start_response)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return resp(environ,
start_response)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack resp =
self.call_func(req, *args, **self.kwargs)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return self.func(req,
*args, **kwargs)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 756, in
__call__
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack content_type, body,
accept)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 821, in
_process_stack
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack action_result =
self.dispatch(meth, request, action_args)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 911, in
dispatch
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
method(req=request, **action_args)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/nova/api/openstack/compute/contrib/volumes.py",
line 305, in create
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack volume_id, device)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 230, in wrapped
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return func(self,
context, target, *args, **kwargs)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 219, in inner
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return function(self,
context, instance, *args, **kwargs)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 200, in inner
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return f(self,
context, instance, *args, **kw)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3035, in
attach_volume
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack disk_bus, device_type)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3016, in
_attach_volume
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack volume_bdm.destroy()
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack
six.reraise(self.type_, self.value, self.tb)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3009, in
_attach_volume
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack volume =
self.volume_api.get(context, volume_id)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 214, in wrapper
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack res = method(self,
ctx, volume_id, *args, **kwargs)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 286, in get
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack item =
cinderclient(context).volumes.get(volume_id)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 115, in
cinderclient
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack url =
_SESSION.get_endpoint(auth, **service_parameters)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 635, in
get_endpoint
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return
auth.get_endpoint(self, **kwargs)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line
219, in get_endpoint
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack
service_name=service_name)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 318, in inner
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return func(*args,
**kwargs)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File
"/usr/lib/python2.7/site-packages/keystoneclient/service_catalog.py", line 254,
in url_for
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack raise
exceptions.EndpointNotFound(msg)
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack EndpointNotFound:
internalURL endpoint for volume service named cinder not found
2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack
———
If I remove cinder-hpc2n and cinderv2-hpc2n from the service list, everything
started to work fine.
This is how I have created endpoints:
openstack endpoint create --publicurl http://<URL>:8776/v1/%\(tenant_id\)s
--internalurl http://<URL>:8776/v1/%\(tenant_id\)s --adminurl
http://<URL>:8776/v1/%\(tenant_id\)s --region regionOne
4a5a771d915e43c28e66538b8bc6e625
openstack endpoint create --publicurl http://<URL>:8776/v1/%\(tenant_id\)s
--internalurl http://<URL>:8776/v1/%\(tenant_id\)s --adminurl
http://<URL>:8776/v1/%\(tenant_id\)s --region HPC2N
a985795b49e2440db82970b81248c86e
using the service-id.
Anyone experiencing this behavior? or Any suggestion how to fix it? Again
regions are working fine with nova and glance.
Regards..
Salman.
PhD, Scientific Computing
Researcher, IT Department,
Uppsala University.
Senior Cloud Architect,
SNIC.
Cloud Application Expert,
UPPMAX.
Visiting Researcher,
Helsinki Institute of Physics (HIP).
salman.t...@it.uu.se<mailto:salman.t...@it.uu.se>
http://www.it.uu.se/katalog/salto690
_______________________________________________
OpenStack-operators mailing list
OpenStack-operators@lists.openstack.org
http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
