Hi, It is working by setting exact names of the services.
[root@smog: ~] # openstack service list +----------------------------------+--------------+--------------+ | ID | Name | Type | +----------------------------------+--------------+--------------+ | 1fcae9bd76304853a3168c39c7fe8e6b | nova | compute | | 2c7828120c294d3f82e3a17835babb85 | neutron | network | | 3804fcd8f9494d30b589b55fe6abb811 | nova-hpc2n | compute | | 478eff4e96464ae8a958ba29f750b14c | glance | image | | 61d8baeb4ee74c7798a60758b2f4171f | cinderv2 | volumev2 | | 75f89962c7864507b07055fbfc98053e | cinder | volume | | 7bd5d667ec4b4d65b6c1b0de8b303fe3 | cinder | volume | | 97f977f8a7a04bae89da167fd25dc06c | glance-hpc2n | image | | 9d2a7ef6b36c45b096e552bf73cb89ae | cinderv2 | volumev2 | | dccd39b92ab547ddaf9047b38620145a | swift | object-store | | ebb1660d1d9746759a48de921521bfad | keystone | identity | +----------------------------------+--------------+--------------+ [root@smog: ~] # openstack endpoint list +----------------------------------+-----------+--------------+--------------+ | ID | Region | Service Name | Service Type | +----------------------------------+-----------+--------------+--------------+ | 3000cb23c6ab4ee5b68876ee08257338 | regionOne | cinderv2 | volumev2 | | e21e935ed861484e976d1b93e0fda0f0 | regionOne | nova | compute | | cdfb008c47a0472ab3a93e6ee07e9ba4 | regionOne | neutron | network | | e2693bcaf3da4be2810f04acd7995d7f | regionOne | cinder | volume | | 1400c42cbf154f63b0c3e8d64352d1f2 | HPC2N | cinderv2 | volumev2 | | 04acd1666704433d991cf9a75957c815 | HPC2N | glance-hpc2n | image | | cfd1c766c56744309767cd84034f9bfb | regionOne | swift | object-store | | abe46f0d86064056927a2177a705787c | HPC2N | nova-hpc2n | compute | | 52bb09199ec84ef09642151348eab695 | HPC2N | cinder | volume | | ac522bcd576b4d2b9f5adfb5405730be | regionOne | keystone | identity | | bad524f18fd74ec3b7fb6647ea661686 | regionOne | glance | image | +----------------------------------+-----------+--------------+--------------+ But different names are working perfectly fine with glance and nova. Anyways thanks for all your time and efforts. If I manage to reproduce it on devstack, will report you back. Regards.. Salman ________________________________ From: Joe Topjian [j...@topjian.net] Sent: Friday, December 11, 2015 7:16 PM To: Salman Toor Cc: openstack-operators@lists.openstack.org Subject: Re: [Openstack-operators] Cinder API with multiple regions not working. Hi Salman, I'm stumped. I was able to confirm that Keystone acts a little differently when you have multiple regional endpoints connected to the same "service_name" and "service_type" than if you have multiple regional endpoints, each with their own "service_name" but same "service_type"'s. For example, you have: regionOne, cinder, volume regionOne, cinderv2, volumev2 HPC2N, cinderhpc2n, volume HPC2N, cinderv2hpc2n, volumev2 When you do "openstack endpoint show volume", you should not get any output. But if you do "openstack endpoint show cinderhpc2n", then you will see output. Swap the service_type and service_names and the pattern continues. However, I am still unable to reproduce your issue. My initial tests used the following format: regionOne, cinder, volume regionOne, cinderv2, volumev2 HPC2N, cinder, volume HPC2N, cinderv2, volumev2 With this format, you will run into the UUID ordering issue I described previously. But again, I'm not able to reproduce the error you're seeing. I really hate to give this answer, but at this point, I'd recommend setting up a devstack environment and comparing the configuration files under /etc/nova and /etc/cinder. This problem is not occurring for me in devstack, so there has to be some difference between the two that is causing this issue. Once you have devstack installed, you can create a fictional second region by doing: Your current style: openstack service create --name cinderhpc2n volume openstack service create --name cinderhpc2nv2 volumev2 openstack endpoint create --region HPC2N cinderhpc2n --publicurl http://10.1.0.113:8776/v1/%\(tenant_id\)s --internalurl http://10.1.0.113:8776/v1/%\(tenant_id\)s --adminurl http://10.1.0.113:8776/v1/%\(tenant_id\)s openstack endpoint create --region HPC2N cinderhpc2nv2 --publicurl http://10.1.0.113:8776/v2/%\(tenant_id\)s --internalurl http://10.1.0.113:8776/v2/%\(tenant_id\)s --adminurl http://10.1.0.113:8776/v2/%\(tenant_id\)s My style which I was originally testing with: openstack endpoint create --region HPC2N volume --publicurl http://10.1.0.113:8776/v1/%\(tenant_id\)s --internalurl http://10.1.0.113:8776/v1/%\(tenant_id\)s --adminurl http://10.1.0.113:8776/v1/%\(tenant_id\)s openstack endpoint create --region HPC2N volumev2 --publicurl http://10.1.0.113:8776/v2/%\(tenant_id\)s --internalurl http://10.1.0.113:8776/v2/%\(tenant_id\)s --adminurl http://10.1.0.113:8776/v2/%\(tenant_id\)s If the IP address is not in use, you will see commands hang when they try to reach the wrong region. I find this to be an easy troubleshooting method to discover where problems are being hit. If you figure out what the issue is, please let me know as I'm very curious. Joe On Fri, Dec 11, 2015 at 3:58 AM, Salman Toor <salman.t...@it.uu.se<mailto:salman.t...@it.uu.se>> wrote: Hi, Still no luck. I have remove the “-“ from the names. [root@controller: ~/openstack] # openstack service list +----------------------------------+---------------+--------------+ | ID | Name | Type | +----------------------------------+---------------+--------------+ | 0f6736013cba469b9ddf21988a071952 | cinderhpc2n | volume | | 1fcae9bd76304853a3168c39c7fe8e6b | nova | compute | | 2c7828120c294d3f82e3a17835babb85 | neutron | network | | 3804fcd8f9494d30b589b55fe6abb811 | nova-hpc2n | compute | | 478eff4e96464ae8a958ba29f750b14c | glance | image | | 61d8baeb4ee74c7798a60758b2f4171f | cinderv2 | volumev2 | | 7bd5d667ec4b4d65b6c1b0de8b303fe3 | cinder | volume | | 97f977f8a7a04bae89da167fd25dc06c | glance-hpc2n | image | | dccd39b92ab547ddaf9047b38620145a | swift | object-store | | dec231e13a114365b6e094421204255f | cinderhpc2nv2 | volumev2 | | ebb1660d1d9746759a48de921521bfad | keystone | identity | +----------------------------------+---------------+--------------+ [root@controller: ~/openstack] # openstack endpoint list +----------------------------------+-----------+---------------+--------------+ | ID | Region | Service Name | Service Type | +----------------------------------+-----------+---------------+--------------+ | 3000cb23c6ab4ee5b68876ee08257338 | regionOne | cinderv2 | volumev2 | | e21e935ed861484e976d1b93e0fda0f0 | regionOne | nova | compute | | 42211e6e977d43308a4e288e68b97263 | HPC2N | cinderhpc2nv2 | volumev2 | | cdfb008c47a0472ab3a93e6ee07e9ba4 | regionOne | neutron | network | | e2693bcaf3da4be2810f04acd7995d7f | regionOne | cinder | volume | | 04acd1666704433d991cf9a75957c815 | HPC2N | glance-hpc2n | image | | eb1a12119c5c4b2dba78927982121be7 | HPC2N | cinderhpc2n | volume | | cfd1c766c56744309767cd84034f9bfb | regionOne | swift | object-store | | abe46f0d86064056927a2177a705787c | HPC2N | nova-hpc2n | compute | | ac522bcd576b4d2b9f5adfb5405730be | regionOne | keystone | identity | | bad524f18fd74ec3b7fb6647ea661686 | regionOne | glance | image | +----------------------------------+-----------+---------------+--------------+ [root@controller: ~/openstack] # systemctl restart openstack-nova-api.service Even with all the services didn’t help. [root@controller: ~/openstack] # nova volume-attach 800d8ba0-cc17-4877-894a-89adecfb5eb7 85ab8b8a-c75c-45a1-9f51-44ed75ba3210 /dev/vdb ERROR (ClientException): The server has either erred or is incapable of performing the requested operation. (HTTP 500) (Request-ID: req-f0d9372d-863b-43f6-8281-5c9196a866af) [root@controller: ~/openstack] # openstack service delete cinderhpc2n [root@controller: ~/openstack] # openstack service delete cinderhpc2nv2 [root@controller: ~/openstack] # [root@controller: ~/openstack] # [root@controller: ~/openstack] # [root@controller: ~/openstack] # [root@controller: ~/openstack] # nova volume-attach 800d8ba0-cc17-4877-894a-89adecfb5eb7 85ab8b8a-c75c-45a1-9f51-44ed75ba3210 /dev/vdb +----------+--------------------------------------+ | Property | Value | +----------+--------------------------------------+ | device | /dev/vdb | | id | 85ab8b8a-c75c-45a1-9f51-44ed75ba3210 | | serverId | 800d8ba0-cc17-4877-894a-89adecfb5eb7 | | volumeId | 85ab8b8a-c75c-45a1-9f51-44ed75ba3210 | +----------+--------------------------------------+ —— nova-api.log —— 2015-12-11 09:43:08.534 16529 TRACE nova.api.openstack EndpointNotFound: internalURL endpoint for volume service named cinder in regionOne region not found —— ——nova.conf [cinder] os_region_name = regionOne —— —— cinder.conf [default] os_region_name = regionOne —— Regards.. Salman. PhD, Scientific Computing Researcher, IT Department, Uppsala University. Senior Cloud Architect, SNIC. Cloud Application Expert, UPPMAX. Visiting Researcher, Helsinki Institute of Physics (HIP). salman.t...@it.uu.se<mailto:salman.t...@it.uu.se> http://www.it.uu.se/katalog/salto690 On 10 Dec 2015, at 23:35, Joe Topjian <j...@topjian.net<mailto:j...@topjian.net>> wrote: Hi Salman, This has turned into a bit of fun -- I'm seeing a lot of wacky things. First, I'm pretty this issue is local to multi-regions and doesn't have to do with having both Cinder v1 and v2 in the Keystone catalog. I changed my catalog to only have Cinder v2 and I still see multi-region issues. If I had more time, I would figure out how to forcefully make nova-api use Cinder v1 to confirm it's not a v2 issue, but I'm pretty confident that it is not. Second, strange things happen depending on the UUID of the endpoints. Let's say I create two cinder v2 regions: openstack endpoint create --region RegionOne volumev2 --publicurl http://10.1.0.112:8776/v2/%\(tenant_id\)s<http://10.1.0.112:8776/v2/%/(tenant_id/)s> --internalurl http://10.1.0.112:8776/v2/%\(tenant_id\)s<http://10.1.0.112:8776/v2/%/(tenant_id/)s> --adminurl http://10.1.0.112:8776/v2/%\(tenant_id\)s<http://10.1.0.112:8776/v2/%/(tenant_id/)s> +--------------+-----------------------------------------+ | Field | Value | +--------------+-----------------------------------------+ | adminurl | http://10.1.0.112:8776/v2/%(tenant_id)s | | id | a46a5f86b0134944b66c25a7802f7b32 | | internalurl | http://10.1.0.112:8776/v2/%(tenant_id)s | | publicurl | http://10.1.0.112:8776/v2/%(tenant_id)s | | region | RegionOne | | service_id | 9cb6eeba4ae5484080ef1a5272b03367 | | service_name | cinderv2 | | service_type | volumev2 | +--------------+-----------------------------------------+ openstack endpoint create --region RegionTwo volumev2 --publicurl http://10.1.0.113:8776/v2/%\(tenant_id\)s<http://10.1.0.113:8776/v2/%/(tenant_id/)s> --internalurl http://10.1.0.113:8776/v2/%\(tenant_id\)s<http://10.1.0.113:8776/v2/%/(tenant_id/)s> --adminurl http://10.1.0.113:8776/v2/%\(tenant_id\)s<http://10.1.0.113:8776/v2/%/(tenant_id/)s> +--------------+-----------------------------------------+ | Field | Value | +--------------+-----------------------------------------+ | adminurl | http://10.1.0.113:8776/v2/%(tenant_id)s | | id | e9cd7a3fd8734b12a77154d73990261d | | internalurl | http://10.1.0.113:8776/v2/%(tenant_id)s | | publicurl | http://10.1.0.113:8776/v2/%(tenant_id)s | | region | RegionTwo | | service_id | 9cb6eeba4ae5484080ef1a5272b03367 | | service_name | cinderv2 | | service_type | volumev2 | +--------------+-----------------------------------------+ Note the IDs: RegionOne has a46... and RegionTwo has e9c... When I do "openstack endpoint show cinderv2", RegionTwo will be displayed. Why? Because "e" comes after "a". If I regenerate the endpoints until RegionOne has a UUID that is alphabetically later than RegionTwo, then RegionOne will be displayed. It does not matter what my OS_REGION_NAME environment variable is set to, either. Now, let's say my nova.conf file for the nova-api server in RegionOne does not have the [cinder] os_region_name = RegionOne setting I mentioned earlier (also, note that this needs to go in a "[cinder]" section). If RegionOne's UUID comes earlier in the alphabet (so RegionOne would NOT be displayed in "openstack endpoint show cinderv2"), then "nova volume-attach" works. nova-api will grab the correct endpoint for RegionOne. If RegionOne's UUID comes later in the alphabet, then "nova volume-attach" will try to contact RegionTwo. My guess is that the openstack client is reporting the last returned catalog entry while the nova-api service is using the first returned catalog entry. BUT, if I set [cinder] os_region_name = RegionOne then "nova volume-attach" works, no matter what order the entries are in the catalog. Given all of the above, here is what is working for me: 1. I have two services in my Keystone catalog: cinder: volume cinderv2: volumev2 It looks like you have 4: cinder: volume cinderv2: volumev2 cinder-hpc2n: volume cinderv2-hpc2n: volumev2 I don't know if that makes a difference, but maybe simplify the services to just the standard 2. 2. nova.conf on the nova-api server has the "os_region_name" setting set to the correct local region. I only had to restart the nova-api service to see changes take effect, but maybe restart nova-* -- who knows at this point! 3. Check the spelling of your regions. I noticed that you sometimes have "regionOne" and other times have "RegionOne" but maybe that is just a typo from sanitizing your data. Let me know if this helps :) Joe On Thu, Dec 10, 2015 at 2:31 AM, Salman Toor <salman.t...@it.uu.se<mailto:salman.t...@it.uu.se>> wrote: Hi Joe, Thanks for your reply, yes Me and Marco are working together. I have tried to used os_region_name = RegionOne both in cinder.conf and nova.conf but no luck so far. But the error message is a bit different. It seems like now the system is recognising the region. 2015-12-10 09:32:54.661 26792 TRACE nova.api.openstack EndpointNotFound: internalURL endpoint for volume service named cinder in regionOne region not found Following are my endpoints: —— [root@controller: ~] # openstack endpoint show cinder +--------------+------------------------------------------------+ | Field | Value | +--------------+------------------------------------------------+ | adminurl | http://cinder:8776/v1/%(tenant_id)s | | enabled | True | | id | e2693bcaf3da4be2810f04acd7995d7f | | internalurl | http://cinder:8776/v1/%(tenant_id)s | | publicurl | http://<http://smog.uppmax.uu.se:8776/v1/%(tenant_id)s>cinder<http://cinder:8776/v1/%(tenant_id)s>:8776/v1/%(tenant_id)s<http://smog.uppmax.uu.se:8776/v1/%(tenant_id)s> | | region | regionOne | | service_id | 7bd5d667ec4b4d65b6c1b0de8b303fe3 | | service_name | cinder | | service_type | volume | +--------------+------------------------------------------------+ [root@controller: ~] # openstack endpoint show cinderv2 +--------------+------------------------------------------------+ | Field | Value | +--------------+------------------------------------------------+ | adminurl | http://cinder:8776/v2/%(tenant_id)s | | enabled | True | | id | 3000cb23c6ab4ee5b68876ee08257338 | | internalurl | http://cinder:8776/v2/%(tenant_id)s | | publicurl | http://<http://smog.uppmax.uu.se:8776/v2/%(tenant_id)s>cinder<http://cinder:8776/v1/%(tenant_id)s>:8776/v2/%(tenant_id)s<http://smog.uppmax.uu.se:8776/v2/%(tenant_id)s> | | region | regionOne | | service_id | 61d8baeb4ee74c7798a60758b2f4171f | | service_name | cinderv2 | | service_type | volumev2 | +--------------+------------------------------------------------+ [root@controller: ~] # openstack endpoint show cinder-hpc2n +--------------+--------------------------------------------------+ | Field | Value | +--------------+--------------------------------------------------+ | adminurl | http://<http://u-mn01.hpc2n.umu.se:8776/v1/%(tenant_id)s>cinder<http://cinder:8776/v1/%(tenant_id)s>:8776/v1/%(tenant_id)s<http://u-mn01.hpc2n.umu.se:8776/v1/%(tenant_id)s> | | enabled | True | | id | c088f1f9ca0248a08eb60d4ac8492150 | | internalurl | http://<http://u-mn01.hpc2n.umu.se:8776/v1/%(tenant_id)s>cinder<http://cinder:8776/v1/%(tenant_id)s>:8776/v1/%(tenant_id)s<http://u-mn01.hpc2n.umu.se:8776/v1/%(tenant_id)s> | | publicurl | http://<http://u-mn01.hpc2n.umu.se:8776/v1/%(tenant_id)s>cinder<http://cinder:8776/v1/%(tenant_id)s>:8776/v1/%(tenant_id)s<http://u-mn01.hpc2n.umu.se:8776/v1/%(tenant_id)s> | | region | HPC2N | | service_id | 473f3eab043f4130b30d9ee64fe43703 | | service_name | cinder-hpc2n | | service_type | volume | +--------------+--------------------------------------------------+ [root@controller: ~] # openstack endpoint show cinderv2-hpc2n +--------------+--------------------------------------------------+ | Field | Value | +--------------+--------------------------------------------------+ | adminurl | http://<http://u-mn01.hpc2n.umu.se:8776/v2/%(tenant_id)s>cinder<http://cinder:8776/v1/%(tenant_id)s>:8776/v2/%(tenant_id)s<http://u-mn01.hpc2n.umu.se:8776/v2/%(tenant_id)s> | | enabled | True | | id | 985e5b8b068e4c3d99c12788236ee0fe | | internalurl | http://<http://u-mn01.hpc2n.umu.se:8776/v2/%(tenant_id)s>cinder<http://cinder:8776/v1/%(tenant_id)s>:8776/v2/%(tenant_id)s<http://u-mn01.hpc2n.umu.se:8776/v2/%(tenant_id)s> | | publicurl | http://<http://u-mn01.hpc2n.umu.se:8776/v2/%(tenant_id)s>cinder<http://cinder:8776/v1/%(tenant_id)s>:8776/v2/%(tenant_id)s<http://u-mn01.hpc2n.umu.se:8776/v2/%(tenant_id)s> | | region | HPC2N | | service_id | 089186fad4d7444b95e74ea9d7769423 | | service_name | cinderv2-hpc2n | | service_type | volumev2 | +--------------+--------------------------------------------------+ —— Again if I remove hpc2n everything started to work fine. Can you tell me what service I suppose to restart (if required)? Regards.. Salman PhD, Scientific Computing Researcher, IT Department, Uppsala University. Senior Cloud Architect, SNIC. Cloud Application Expert, UPPMAX. Visiting Researcher, Helsinki Institute of Physics (HIP). salman.t...@it.uu.se<mailto:salman.t...@it.uu.se> http://www.it.uu.se/katalog/salto690 On 09 Dec 2015, at 17:48, Joe Topjian <j...@topjian.net<mailto:j...@topjian.net>> wrote: Hi Salman, Someone mentioned this same issue yesterday in relation to Terraform (maybe a colleague of yours?), so given the two occurrences, I thought I'd look into this. I have a Liberty environment readily available, so I created a second set of volume and volumev2 endpoints for a fictional region. Everything worked as expected, so I started reviewing the config files and saw that /etc/cinder/cinder.conf had an option [DEFAULT] os_region_name = RegionOne I commented that out, but things still worked. Then in /etc/nova/nova.conf, I saw: [cinder] os_region_name = RegionOne commenting this out caused volume attachments to hang indefinitely because nova was trying to contact cinder at RegionTwo (I'm assuming this is the first catalog entry that was returned). Given this is a Liberty environment, it's not accurately reproducing your problem, but could you check and see if you have that option set in nova.conf? I have a Kilo environment in the process of building. Once it has finished, I'll see if I can reproduce your error there. Thanks, Joe On Wed, Dec 9, 2015 at 4:35 AM, Salman Toor <salman.t...@it.uu.se<mailto:salman.t...@it.uu.se>> wrote: Hi, I am using Kilo release on CentOS. We have recently enabled multiple regions and it seems that Cinder have some problems with multiple endpoints. Thinks are working fine with nova but cinder is behaving strange. Here are my endpoints ———— [root@controller: ~] # openstack service list +----------------------------------+----------------+--------------+ | ID | Name | Type | +----------------------------------+----------------+--------------+ | 0a33e6f259794ff2a99e626be37c0c2b | cinderv2-hpc2n | volumev2 | | 1fcae9bd76304853a3168c39c7fe8e6b | nova | compute | | 2c7828120c294d3f82e3a17835babb85 | neutron | network | | 3804fcd8f9494d30b589b55fe6abb811 | nova-hpc2n | compute | | 478eff4e96464ae8a958ba29f750b14c | glance | image | | 4a5a771d915e43c28e66538b8bc6e625 | cinder | volume | | 72d1be82b2e5478dbf0f3fb9e7ba969d | cinderv2 | volumev2 | | 97f977f8a7a04bae89da167fd25dc06c | glance-hpc2n | image | | a985795b49e2440db82970b81248c86e | cinder-hpc2n | volume | | dccd39b92ab547ddaf9047b38620145a | swift | object-store | | ebb1660d1d9746759a48de921521bfad | keystone | identity | +----------------------------------+----------------+--------------+ [root@controller: ~] # openstack endpoint show a985795b49e2440db82970b81248c86e +--------------+--------------------------------------------------+ | Field | Value | +--------------+--------------------------------------------------+ | adminurl | http://<URL>:8776/v1/%(tenant_id)s | | enabled | True | | id | d4003e91ddf24cfb9fa497da81b01a18 | | internalurl | http://<URL>:8776/v1/%(tenant_id)s | | publicurl | http://<URL>:8776/v1/%(tenant_id)s | | region | HPC2N | | service_id | a985795b49e2440db82970b81248c86e | | service_name | cinder-hpc2n | | service_type | volume | +--------------+--------------------------------------------------+ [root@controller: ~] # openstack endpoint show 4a5a771d915e43c28e66538b8bc6e625 +--------------+------------------------------------------------+ | Field | Value | +--------------+------------------------------------------------+ | adminurl | http://<URL>:8776/v1/%(tenant_id)s | | enabled | True | | id | 5f19c0b535674dbd9e318c7b6d61b3bc | | internalurl | http://<URL>:8776/v1/%(tenant_id)s | | publicurl | http://<URL>:8776/v1/%(tenant_id)s | | region | regionOne | | service_id | 4a5a771d915e43c28e66538b8bc6e625 | | service_name | cinder | | service_type | volume | +--------------+------------------------------------------------+ And same for v2 endpoints ———— ——— nova-api.log ——— achmentController object at 0x598a3d0>>, body: {"volumeAttachment": {"device": "", "volumeId": "93d96eab-e3fd-4131-9549-ed51e7299da2"}} _process_stack /usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py:780 2015-12-08 12:52:05.847 3376 INFO nova.api.openstack.compute.contrib.volumes [req-d6e1380d-c6bc-4911-b2c6-251bc8b4c62c a62c20fdf99c443a924f0d50a51514b1 3c9d997982e04c6db0e02b82fa18fdd8 - - -] Attach volume 93d96eab-e3fd-4131-9549-ed51e7299da2 to instance 3a4c8722-52a7-48f2-beb7-db8938698a0d at 2015-12-08 12:52:05.847 3376 DEBUG nova.compute.api [req-d6e1380d-c6bc-4911-b2c6-251bc8b4c62c a62c20fdf99c443a924f0d50a51514b1 3c9d997982e04c6db0e02b82fa18fdd8 - - -] [instance: 3a4c8722-52a7-48f2-beb7-db8938698a0d] Fetching instance by UUID get /usr/lib/python2.7/site-packages/nova/compute/api.py:1911 2015-12-08 12:52:05.988 3376 ERROR nova.api.openstack [req-d6e1380d-c6bc-4911-b2c6-251bc8b4c62c a62c20fdf99c443a924f0d50a51514b1 3c9d997982e04c6db0e02b82fa18fdd8 - - -] Caught error: internalURL endpoint for volume service named cinder not found 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack Traceback (most recent call last): 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/__init__.py", line 125, in __call__ 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return req.get_response(self.application) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/request.py", line 1296, in send 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack application, catch_exc_info=False) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/request.py", line 1260, in call_application 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack app_iter = application(self.environ, start_response) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return resp(environ, start_response) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 634, in __call__ 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return self._call_app(env, start_response) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/keystonemiddleware/auth_token/__init__.py", line 554, in _call_app 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return self._app(env, _fake_start_response) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return resp(environ, start_response) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return resp(environ, start_response) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/routes/middleware.py", line 131, in __call__ 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack response = self.app(environ, start_response) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 144, in __call__ 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return resp(environ, start_response) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 130, in __call__ 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack resp = self.call_func(req, *args, **self.kwargs) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/webob/dec.py", line 195, in call_func 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return self.func(req, *args, **kwargs) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 756, in __call__ 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack content_type, body, accept) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 821, in _process_stack 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack action_result = self.dispatch(meth, request, action_args) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/wsgi.py", line 911, in dispatch 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return method(req=request, **action_args) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/api/openstack/compute/contrib/volumes.py", line 305, in create 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack volume_id, device) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 230, in wrapped 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return func(self, context, target, *args, **kwargs) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 219, in inner 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return function(self, context, instance, *args, **kwargs) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 200, in inner 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return f(self, context, instance, *args, **kw) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3035, in attach_volume 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack disk_bus, device_type) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3016, in _attach_volume 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack volume_bdm.destroy() 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/oslo_utils/excutils.py", line 85, in __exit__ 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack six.reraise(self.type_, self.value, self.tb) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/compute/api.py", line 3009, in _attach_volume 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack volume = self.volume_api.get(context, volume_id) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 214, in wrapper 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack res = method(self, ctx, volume_id, *args, **kwargs) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 286, in get 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack item = cinderclient(context).volumes.get(volume_id) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/nova/volume/cinder.py", line 115, in cinderclient 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack url = _SESSION.get_endpoint(auth, **service_parameters) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/keystoneclient/session.py", line 635, in get_endpoint 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return auth.get_endpoint(self, **kwargs) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/keystoneclient/auth/identity/base.py", line 219, in get_endpoint 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack service_name=service_name) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/keystoneclient/utils.py", line 318, in inner 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack return func(*args, **kwargs) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack File "/usr/lib/python2.7/site-packages/keystoneclient/service_catalog.py", line 254, in url_for 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack raise exceptions.EndpointNotFound(msg) 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack EndpointNotFound: internalURL endpoint for volume service named cinder not found 2015-12-08 12:52:05.988 3376 TRACE nova.api.openstack ——— If I remove cinder-hpc2n and cinderv2-hpc2n from the service list, everything started to work fine. This is how I have created endpoints: openstack endpoint create --publicurl http://<URL>:8776/v1/%\(tenant_id\)s --internalurl http://<URL>:8776/v1/%\(tenant_id\)s --adminurl http://<URL>:8776/v1/%\(tenant_id\)s --region regionOne 4a5a771d915e43c28e66538b8bc6e625 openstack endpoint create --publicurl http://<URL>:8776/v1/%\(tenant_id\)s --internalurl http://<URL>:8776/v1/%\(tenant_id\)s --adminurl http://<URL>:8776/v1/%\(tenant_id\)s --region HPC2N a985795b49e2440db82970b81248c86e using the service-id. Anyone experiencing this behavior? or Any suggestion how to fix it? Again regions are working fine with nova and glance. Regards.. Salman. PhD, Scientific Computing Researcher, IT Department, Uppsala University. Senior Cloud Architect, SNIC. Cloud Application Expert, UPPMAX. Visiting Researcher, Helsinki Institute of Physics (HIP). salman.t...@it.uu.se<mailto:salman.t...@it.uu.se> http://www.it.uu.se/katalog/salto690 _______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org<mailto:OpenStack-operators@lists.openstack.org> http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators
_______________________________________________ OpenStack-operators mailing list OpenStack-operators@lists.openstack.org http://lists.openstack.org/cgi-bin/mailman/listinfo/openstack-operators