Happening in Ocata too: http://logs.openstack.org/86/474286/1/gate/gate- tempest-dsvm-neutron-dvr-ubuntu- xenial/2b247e0/logs/screen-q-svc.txt.gz#_2017-06-26_21_16_20_582
** Changed in: tempest Importance: Medium => High ** Also affects: devstack Importance: Undecided Status: New ** Also affects: keystone Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1681866 Title: Bad response code while validating token: 502 Status in devstack: New Status in OpenStack Identity (keystone): New Status in tempest: Confirmed Bug description: Found this while investigating a gate failure [1]. Tempest logs say "2017-04-11 10:07:02,765 23082 INFO [tempest.lib.common.rest_client] Request (TestSecurityGroupsBasicOps:_run_cleanups): 503 DELETE https://198.72.124.138:8774/v2.1/servers/f736a878-2ac4-4c37-b6a8-e5cd8df5a7fd 0.018s" That 503 looks suspicious. So I go to the nova-api logs. Which gives 2017-04-11 10:07:02.762 32191 ERROR keystonemiddleware.auth_token [...] Bad response code while validating token: 502 2017-04-11 10:07:02.763 32191 WARNING keystonemiddleware.auth_token [...] Identity response: <!DOCTYPE HTML PUBLIC "-//IETF//DTD HTML 2.0//EN"> <html><head> <title>502 Proxy Error</title> </head><body> <h1>Proxy Error</h1> <p>The proxy server received an invalid response from an upstream server.<br /> The proxy server could not handle the request <em><a href="/identity_admin/v3/auth/tokens">GET /identity_admin/v3/auth/tokens</a></em>.<p> Reason: <strong>Error reading from remote server</strong></p></p> <hr> <address>Apache/2.4.18 (Ubuntu) Server at 198.72.124.138 Port 443</address> </body></html> 2017-04-11 10:07:02.763 32191 CRITICAL keystonemiddleware.auth_token [...] Unable to validate token: Failed to fetch token data from identity server So Apache is complaining, some network connection issue, related to proxy-ing. So I open "logs/apache/tls-proxy_error.txt.gz" and find [Tue Apr 11 10:07:02.761420 2017] [proxy_http:error] [pid 7136:tid 140090189690624] (20014)Internal error (specific information not available): [client 198.72.124.138:38722] [frontend 198.72.124.138:443] AH01102: error reading status line from remote server 198.72.124.138:80 [Tue Apr 11 10:07:02.761454 2017] [proxy:error] [pid 7136:tid 140090189690624] [client 198.72.124.138:38722] [frontend 198.72.124.138:443] AH00898: Error reading from remote server returned by /identity_admin/v3/auth/tokens Interesting. Google says that adding "proxy-initial-not-pooled" to the apache2 vhost config could help. Anyway, a good elasticsearch query for this is message:"Bad response code while validating token: 502" 8 hits, no worries. [1] : http://logs.openstack.org/03/455303/2/check/gate-tempest-dsvm- neutron-full-ubuntu-xenial/aa8c7fd/console.html To manage notifications about this bug go to: https://bugs.launchpad.net/devstack/+bug/1681866/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp