Public bug reported: In Openstack HA environment, sometimes Kesystone client is throwing the error "unable to establish connection error". This is happening once in fifteen times. The error and stack trace of the error log is given below.
Error: 1) keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://Controller-VIP:8004/v1/ca1d10786fdc41698e276a8f22d98741/stacks 2) ConnectionRefused: Unable to establish connection to http://Controller-VIP:9696/v2.0/ports.json?tenant_id=ca1d10786fdc41698e276a8f22d98741&device_id=183bea96-b0da-42a5-a731-713882598624 Stack trace of the error: NFO (connectionpool) Starting new HTTP connection (1): Controller-VIP Traceback (most recent call last): File "/usr/bin/heat", line 10, in <module> sys.exit(main()) File "/usr/lib/python2.7/dist-packages/heatclient/shell.py", line 706, in main HeatShell().main(args) File "/usr/lib/python2.7/dist-packages/heatclient/shell.py", line 656, in main args.func(client, args) File "/usr/lib/python2.7/dist-packages/heatclient/v1/shell.py", line 137, in do_stack_create hc.stacks.create(**fields) File "/usr/lib/python2.7/dist-packages/heatclient/v1/stacks.py", line 134, in create data=kwargs, headers=headers) File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 266, in json_request resp = self._http_request(url, method, **kwargs) File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 358, in _http_request raise_exc=False, **kwargs) File "/usr/local/lib/python2.7/dist-packages/keystoneclient/utils.py", line 337, in inner return func(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 390, in request resp = send(**kwargs) File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 434, in _send_request raise exceptions.ConnectionRefused(msg) keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://Controller-VIP:8004/v1/ca1d10786fdc41698e276a8f22d98741/stacks I think we should retry the API connection from keystone client whenever there is a failure of connection to API. When I added code to retry it is working fine. In the existing code Keystoneclient is trying only one time and returning an error if it is not able to connect to any other service API. ** Affects: keystone Importance: Undecided Assignee: Satyanarayana Patibandla (satya-patibandla) Status: Opinion ** Tags: identity openstack ** Changed in: keystone Assignee: (unassigned) => Satyanarayana Patibandla (satya-patibandla) ** Changed in: keystone Status: New => Opinion -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1535640 Title: Sometimes getting unable to establish connection error in Keystone client Status in OpenStack Identity (keystone): Opinion Bug description: In Openstack HA environment, sometimes Kesystone client is throwing the error "unable to establish connection error". This is happening once in fifteen times. The error and stack trace of the error log is given below. Error: 1) keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://Controller-VIP:8004/v1/ca1d10786fdc41698e276a8f22d98741/stacks 2) ConnectionRefused: Unable to establish connection to http://Controller-VIP:9696/v2.0/ports.json?tenant_id=ca1d10786fdc41698e276a8f22d98741&device_id=183bea96-b0da-42a5-a731-713882598624 Stack trace of the error: NFO (connectionpool) Starting new HTTP connection (1): Controller-VIP Traceback (most recent call last): File "/usr/bin/heat", line 10, in <module> sys.exit(main()) File "/usr/lib/python2.7/dist-packages/heatclient/shell.py", line 706, in main HeatShell().main(args) File "/usr/lib/python2.7/dist-packages/heatclient/shell.py", line 656, in main args.func(client, args) File "/usr/lib/python2.7/dist-packages/heatclient/v1/shell.py", line 137, in do_stack_create hc.stacks.create(**fields) File "/usr/lib/python2.7/dist-packages/heatclient/v1/stacks.py", line 134, in create data=kwargs, headers=headers) File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 266, in json_request resp = self._http_request(url, method, **kwargs) File "/usr/lib/python2.7/dist-packages/heatclient/common/http.py", line 358, in _http_request raise_exc=False, **kwargs) File "/usr/local/lib/python2.7/dist-packages/keystoneclient/utils.py", line 337, in inner return func(*args, **kwargs) File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 390, in request resp = send(**kwargs) File "/usr/local/lib/python2.7/dist-packages/keystoneclient/session.py", line 434, in _send_request raise exceptions.ConnectionRefused(msg) keystoneauth1.exceptions.connection.ConnectFailure: Unable to establish connection to http://Controller-VIP:8004/v1/ca1d10786fdc41698e276a8f22d98741/stacks I think we should retry the API connection from keystone client whenever there is a failure of connection to API. When I added code to retry it is working fine. In the existing code Keystoneclient is trying only one time and returning an error if it is not able to connect to any other service API. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1535640/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
