Public bug reported: I'm running stable/pike devstack, and I was playing around with what happens when there are many endpoints in multiple regions, and I stumbled over a scenario where the keystone authentication code hangs.
My original endpoint list looked like this: ubuntu@devstack:/opt/stack/devstack$ openstack endpoint list +----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+ | ID | Region | Service Name | Service Type | Enabled | Interface | URL | +----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+ | 0a9979ebfdbf48ce91ccf4e2dd952c1a | RegionOne | kingbird | synchronization | True | internal | http://127.0.0.1:8118/v1.0 | | 11d5507afe2a4eddb4f030695699114f | RegionOne | placement | placement | True | public | http://128.224.186.226/placement | | 1e42cf139398405188755b7e00aecb4d | RegionOne | keystone | identity | True | admin | http://128.224.186.226/identity | | 2daf99edecae4afba88bb58233595481 | RegionOne | glance | image | True | public | http://128.224.186.226/image | | 2ece52e8bbb34d47b9bd5611f5959385 | RegionOne | kingbird | synchronization | True | admin | http://127.0.0.1:8118/v1.0 | | 4835a089666a4b03bd2f499457ade6c2 | RegionOne | kingbird | synchronization | True | public | http://127.0.0.1:8118/v1.0 | | 78e9fbc0a47642268eda3e3576920f37 | RegionOne | nova | compute | True | public | http://128.224.186.226/compute/v2.1 | | 96a1e503dc0e4520a190b01f6a0cf79c | RegionOne | keystone | identity | True | public | http://128.224.186.226/identity | | a1887dbc8c5e4af5b4a6dc5ce224b8ff | RegionOne | cinderv2 | volumev2 | True | public | http://128.224.186.226/volume/v2/$(project_id)s | | b7d5938141694a4c87adaed5105ea3ab | RegionOne | cinder | volume | True | public | http://128.224.186.226/volume/v1/$(project_id)s | | bb169382cbea4715964e4652acd48070 | RegionOne | nova_legacy | compute_legacy | True | public | http://128.224.186.226/compute/v2/$(project_id)s | | e01c8d8e08874d61b9411045a99d4860 | RegionOne | neutron | network | True | public | http://128.224.186.226:9696/ | | f94c96ed474249a29a6c0a1bb2b2e500 | RegionOne | cinderv3 | volumev3 | True | public | http://128.224.186.226/volume/v3/$(project_id)s | +----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+ I was able to successfully run the following python code: from keystoneauth1 import loading from keystoneauth1 import loading from keystoneauth1 import session from keystoneclient.v3 import client loader = loading.get_plugin_loader("password") auth = loader.load_from_options(username='admin',password='secret',project_name='admin',auth_url='http://128.224.186.226/identity') sess = session.Session(auth=auth) keystone = client.Client(session=sess) keystone.services.list() I then duplicated all of the endpoints in a new region "region2", and was able to run the python code. When I duplicated all the endpoints again in a new region "region3" (for a total of 39 endpoints) the python code hung at the final line. Removing all the "region3" endpoints allowed the python code to work again. During all of this the command "openstack endpoint list" worked fine. Further testing seems to indicate that it is the third "admin" keystone endpoint that is causing the problem. I can add multiple "public" keystone endpoints, but three or more "admin" keystone endpoints cause the python code to hang. ** Affects: keystone Importance: Undecided Status: New ** Summary changed: - authentication code hangs when there are many endpoints + authentication code hangs when there are three or more admin keystone endpoints ** Description changed: I'm running stable/pike devstack, and I was playing around with what happens when there are many endpoints in multiple regions, and I stumbled over a scenario where the keystone authentication code hangs. My original endpoint list looked like this: ubuntu@devstack:/opt/stack/devstack$ openstack endpoint list +----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+ | ID | Region | Service Name | Service Type | Enabled | Interface | URL | +----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+ | 0a9979ebfdbf48ce91ccf4e2dd952c1a | RegionOne | kingbird | synchronization | True | internal | http://127.0.0.1:8118/v1.0 | | 11d5507afe2a4eddb4f030695699114f | RegionOne | placement | placement | True | public | http://128.224.186.226/placement | | 1e42cf139398405188755b7e00aecb4d | RegionOne | keystone | identity | True | admin | http://128.224.186.226/identity | | 2daf99edecae4afba88bb58233595481 | RegionOne | glance | image | True | public | http://128.224.186.226/image | | 2ece52e8bbb34d47b9bd5611f5959385 | RegionOne | kingbird | synchronization | True | admin | http://127.0.0.1:8118/v1.0 | | 4835a089666a4b03bd2f499457ade6c2 | RegionOne | kingbird | synchronization | True | public | http://127.0.0.1:8118/v1.0 | | 78e9fbc0a47642268eda3e3576920f37 | RegionOne | nova | compute | True | public | http://128.224.186.226/compute/v2.1 | | 96a1e503dc0e4520a190b01f6a0cf79c | RegionOne | keystone | identity | True | public | http://128.224.186.226/identity | | a1887dbc8c5e4af5b4a6dc5ce224b8ff | RegionOne | cinderv2 | volumev2 | True | public | http://128.224.186.226/volume/v2/$(project_id)s | | b7d5938141694a4c87adaed5105ea3ab | RegionOne | cinder | volume | True | public | http://128.224.186.226/volume/v1/$(project_id)s | | bb169382cbea4715964e4652acd48070 | RegionOne | nova_legacy | compute_legacy | True | public | http://128.224.186.226/compute/v2/$(project_id)s | | e01c8d8e08874d61b9411045a99d4860 | RegionOne | neutron | network | True | public | http://128.224.186.226:9696/ | | f94c96ed474249a29a6c0a1bb2b2e500 | RegionOne | cinderv3 | volumev3 | True | public | http://128.224.186.226/volume/v3/$(project_id)s | +----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+ I was able to successfully run the following python code: from keystoneauth1 import loading from keystoneauth1 import loading from keystoneauth1 import session from keystoneclient.v3 import client loader = loading.get_plugin_loader("password") auth = loader.load_from_options(username='admin',password='secret',project_name='admin',auth_url='http://128.224.186.226/identity') sess = session.Session(auth=auth) keystone = client.Client(session=sess) keystone.services.list() - - I then duplicated all of the endpoints in a new region "region2", and was able to run the python code. When I duplicated all the endpoints again in a new region "region3" (for a total of 39 endpoints) the python code hung at the final line. + I then duplicated all of the endpoints in a new region "region2", and + was able to run the python code. When I duplicated all the endpoints + again in a new region "region3" (for a total of 39 endpoints) the python + code hung at the final line. Removing all the "region3" endpoints allowed the python code to work again. During all of this the command "openstack endpoint list" worked fine. - Further testing seems to indicate that it is the third keystone endpoint - that is causing the problem. I can add all the others, but if I add the - third keystone endpoint (regardless of what order I add them in) then - the python code hangs. + Further testing seems to indicate that it is the third "admin" keystone + endpoint that is causing the problem. I can add all the others, but if + I add the third keystone endpoint (regardless of what order I add them + in) then the python code hangs. ** Description changed: I'm running stable/pike devstack, and I was playing around with what happens when there are many endpoints in multiple regions, and I stumbled over a scenario where the keystone authentication code hangs. My original endpoint list looked like this: ubuntu@devstack:/opt/stack/devstack$ openstack endpoint list +----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+ | ID | Region | Service Name | Service Type | Enabled | Interface | URL | +----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+ | 0a9979ebfdbf48ce91ccf4e2dd952c1a | RegionOne | kingbird | synchronization | True | internal | http://127.0.0.1:8118/v1.0 | | 11d5507afe2a4eddb4f030695699114f | RegionOne | placement | placement | True | public | http://128.224.186.226/placement | | 1e42cf139398405188755b7e00aecb4d | RegionOne | keystone | identity | True | admin | http://128.224.186.226/identity | | 2daf99edecae4afba88bb58233595481 | RegionOne | glance | image | True | public | http://128.224.186.226/image | | 2ece52e8bbb34d47b9bd5611f5959385 | RegionOne | kingbird | synchronization | True | admin | http://127.0.0.1:8118/v1.0 | | 4835a089666a4b03bd2f499457ade6c2 | RegionOne | kingbird | synchronization | True | public | http://127.0.0.1:8118/v1.0 | | 78e9fbc0a47642268eda3e3576920f37 | RegionOne | nova | compute | True | public | http://128.224.186.226/compute/v2.1 | | 96a1e503dc0e4520a190b01f6a0cf79c | RegionOne | keystone | identity | True | public | http://128.224.186.226/identity | | a1887dbc8c5e4af5b4a6dc5ce224b8ff | RegionOne | cinderv2 | volumev2 | True | public | http://128.224.186.226/volume/v2/$(project_id)s | | b7d5938141694a4c87adaed5105ea3ab | RegionOne | cinder | volume | True | public | http://128.224.186.226/volume/v1/$(project_id)s | | bb169382cbea4715964e4652acd48070 | RegionOne | nova_legacy | compute_legacy | True | public | http://128.224.186.226/compute/v2/$(project_id)s | | e01c8d8e08874d61b9411045a99d4860 | RegionOne | neutron | network | True | public | http://128.224.186.226:9696/ | | f94c96ed474249a29a6c0a1bb2b2e500 | RegionOne | cinderv3 | volumev3 | True | public | http://128.224.186.226/volume/v3/$(project_id)s | +----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+ I was able to successfully run the following python code: from keystoneauth1 import loading from keystoneauth1 import loading from keystoneauth1 import session from keystoneclient.v3 import client loader = loading.get_plugin_loader("password") auth = loader.load_from_options(username='admin',password='secret',project_name='admin',auth_url='http://128.224.186.226/identity') sess = session.Session(auth=auth) keystone = client.Client(session=sess) keystone.services.list() I then duplicated all of the endpoints in a new region "region2", and was able to run the python code. When I duplicated all the endpoints again in a new region "region3" (for a total of 39 endpoints) the python code hung at the final line. Removing all the "region3" endpoints allowed the python code to work again. During all of this the command "openstack endpoint list" worked fine. Further testing seems to indicate that it is the third "admin" keystone - endpoint that is causing the problem. I can add all the others, but if - I add the third keystone endpoint (regardless of what order I add them - in) then the python code hangs. + endpoint that is causing the problem. I can add multiple "public" + keystone endpoints, but three or more "admin" keystone endpoints cause + the python code to hang. -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1724686 Title: authentication code hangs when there are three or more admin keystone endpoints Status in OpenStack Identity (keystone): New Bug description: I'm running stable/pike devstack, and I was playing around with what happens when there are many endpoints in multiple regions, and I stumbled over a scenario where the keystone authentication code hangs. My original endpoint list looked like this: ubuntu@devstack:/opt/stack/devstack$ openstack endpoint list +----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+ | ID | Region | Service Name | Service Type | Enabled | Interface | URL | +----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+ | 0a9979ebfdbf48ce91ccf4e2dd952c1a | RegionOne | kingbird | synchronization | True | internal | http://127.0.0.1:8118/v1.0 | | 11d5507afe2a4eddb4f030695699114f | RegionOne | placement | placement | True | public | http://128.224.186.226/placement | | 1e42cf139398405188755b7e00aecb4d | RegionOne | keystone | identity | True | admin | http://128.224.186.226/identity | | 2daf99edecae4afba88bb58233595481 | RegionOne | glance | image | True | public | http://128.224.186.226/image | | 2ece52e8bbb34d47b9bd5611f5959385 | RegionOne | kingbird | synchronization | True | admin | http://127.0.0.1:8118/v1.0 | | 4835a089666a4b03bd2f499457ade6c2 | RegionOne | kingbird | synchronization | True | public | http://127.0.0.1:8118/v1.0 | | 78e9fbc0a47642268eda3e3576920f37 | RegionOne | nova | compute | True | public | http://128.224.186.226/compute/v2.1 | | 96a1e503dc0e4520a190b01f6a0cf79c | RegionOne | keystone | identity | True | public | http://128.224.186.226/identity | | a1887dbc8c5e4af5b4a6dc5ce224b8ff | RegionOne | cinderv2 | volumev2 | True | public | http://128.224.186.226/volume/v2/$(project_id)s | | b7d5938141694a4c87adaed5105ea3ab | RegionOne | cinder | volume | True | public | http://128.224.186.226/volume/v1/$(project_id)s | | bb169382cbea4715964e4652acd48070 | RegionOne | nova_legacy | compute_legacy | True | public | http://128.224.186.226/compute/v2/$(project_id)s | | e01c8d8e08874d61b9411045a99d4860 | RegionOne | neutron | network | True | public | http://128.224.186.226:9696/ | | f94c96ed474249a29a6c0a1bb2b2e500 | RegionOne | cinderv3 | volumev3 | True | public | http://128.224.186.226/volume/v3/$(project_id)s | +----------------------------------+-----------+--------------+-----------------+---------+-----------+--------------------------------------------------+ I was able to successfully run the following python code: from keystoneauth1 import loading from keystoneauth1 import loading from keystoneauth1 import session from keystoneclient.v3 import client loader = loading.get_plugin_loader("password") auth = loader.load_from_options(username='admin',password='secret',project_name='admin',auth_url='http://128.224.186.226/identity') sess = session.Session(auth=auth) keystone = client.Client(session=sess) keystone.services.list() I then duplicated all of the endpoints in a new region "region2", and was able to run the python code. When I duplicated all the endpoints again in a new region "region3" (for a total of 39 endpoints) the python code hung at the final line. Removing all the "region3" endpoints allowed the python code to work again. During all of this the command "openstack endpoint list" worked fine. Further testing seems to indicate that it is the third "admin" keystone endpoint that is causing the problem. I can add multiple "public" keystone endpoints, but three or more "admin" keystone endpoints cause the python code to hang. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1724686/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp