Public bug reported: Description:
When a client, eg. the OpenStack dashboard, makes a nova service request of "http://controller:8774/v2.1"; (show controller versions), it receives a redirect to "http://controller:8774/v2.1/";. This is erroneous for at least the following two reasons: 1. If, for security reasons, you place the nova service behind an SSL termination proxy, the redirect generated as follows: from : https://controller:proxy_port/v2.1 to : http://controller:proxy_port/v2.1/ is invalid because the proxy_port requires encrypted traffic and the replacement URL is using the wrong protocol (http). The request fails on the client side with "Unable to establish connection to http://controller:proxy_port/v2.1/: ('Connection aborted.', BadStatusLine("''",))". 2. Even if we are not using a proxy server, the nova service is effectively complaining about a missing trailing forward slash ("/"), telling the client to reissue the same request but with the missing character. This creates unnecessary network traffic (the redirect plus a second request) and additional server load (two requests instead of one). It should be noted that "http://controller:8774/v2.1"; is the endpoint specification recommended in the OpenStack nova installation guides for the ocata, and pike releases. This will result in unnecessary traffic and load on many installations, which will go unnoticed because the request eventually works. Solution: Replace the first ROUTE_LIST entry (and associated comments) in nova.api.openstack.compute.routes, changing it from: # NOTE: This is a redirection from '' to '/'. The request to the '/v2.1' # or '/2.0' without the ending '/' will get a response with status code # '302' returned. ('', '/'), to: # The following 3 lines replaces a redirect specification that caused additional network traffic and load. See bug #xxxxx. ('', { 'GET': [version_controller, 'show'] I've applied/tested a fix/workaround here: https://github.com/hep- gc/nova/commit/b9c27bf29f7042cf637b58c87d6a9b2f3a9b78b6 To recreate: 1. Install Openstack (ocata/pike) as per https://docs.openstack.org/pike/install/ 2. Monitor network traffic (tcpdump) on client. 3. Login to the dashboard, and view compute->project->overview To see "Unable to establish connection to http://controller:proxy_port/v2.1/: ('Connection aborted.', BadStatusLine("''",))" error: 4. Install HAProxy. 5. Serve the nova public endpoint via the SSL termination proxy server. Our HAProxy configuration for this is as follows: frontend nova_public bind controller_fqdn:18774 ssl crt /etc/letsencrypt/live/controller_fqdn/web_crt_key.pem reqadd X-Forwarded-Proto:\ https default_backend nova_internal backend nova_internal redirect scheme https code 301 if !{ ssl_fc } server controller controller:8774 check 6. Redefine the nova public endpoint in the sql database: mysql -ukeystone -p connect keystone; update endpoint set url="https://otter.heprc.uvic.ca:18774/v2.1"; where id="xxxxxxx" 7. Dashboard will display "Unable to retrieve usage data" red flag each time the project overview page is displayed, and the http error log will report the connection failure. ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1728732 Title: OpenStack nova service responds with an erroneous httpd redirect to a "GET,version_controller,show" request. Status in OpenStack Compute (nova): New Bug description: Description: When a client, eg. the OpenStack dashboard, makes a nova service request of "http://controller:8774/v2.1"; (show controller versions), it receives a redirect to "http://controller:8774/v2.1/";. This is erroneous for at least the following two reasons: 1. If, for security reasons, you place the nova service behind an SSL termination proxy, the redirect generated as follows: from : https://controller:proxy_port/v2.1 to : http://controller:proxy_port/v2.1/ is invalid because the proxy_port requires encrypted traffic and the replacement URL is using the wrong protocol (http). The request fails on the client side with "Unable to establish connection to http://controller:proxy_port/v2.1/: ('Connection aborted.', BadStatusLine("''",))". 2. Even if we are not using a proxy server, the nova service is effectively complaining about a missing trailing forward slash ("/"), telling the client to reissue the same request but with the missing character. This creates unnecessary network traffic (the redirect plus a second request) and additional server load (two requests instead of one). It should be noted that "http://controller:8774/v2.1"; is the endpoint specification recommended in the OpenStack nova installation guides for the ocata, and pike releases. This will result in unnecessary traffic and load on many installations, which will go unnoticed because the request eventually works. Solution: Replace the first ROUTE_LIST entry (and associated comments) in nova.api.openstack.compute.routes, changing it from: # NOTE: This is a redirection from '' to '/'. The request to the '/v2.1' # or '/2.0' without the ending '/' will get a response with status code # '302' returned. ('', '/'), to: # The following 3 lines replaces a redirect specification that caused additional network traffic and load. See bug #xxxxx. ('', { 'GET': [version_controller, 'show'] I've applied/tested a fix/workaround here: https://github.com/hep- gc/nova/commit/b9c27bf29f7042cf637b58c87d6a9b2f3a9b78b6 To recreate: 1. Install Openstack (ocata/pike) as per https://docs.openstack.org/pike/install/ 2. Monitor network traffic (tcpdump) on client. 3. Login to the dashboard, and view compute->project->overview To see "Unable to establish connection to http://controller:proxy_port/v2.1/: ('Connection aborted.', BadStatusLine("''",))" error: 4. Install HAProxy. 5. Serve the nova public endpoint via the SSL termination proxy server. Our HAProxy configuration for this is as follows: frontend nova_public bind controller_fqdn:18774 ssl crt /etc/letsencrypt/live/controller_fqdn/web_crt_key.pem reqadd X-Forwarded-Proto:\ https default_backend nova_internal backend nova_internal redirect scheme https code 301 if !{ ssl_fc } server controller controller:8774 check 6. Redefine the nova public endpoint in the sql database: mysql -ukeystone -p connect keystone; update endpoint set url="https://otter.heprc.uvic.ca:18774/v2.1"; where id="xxxxxxx" 7. Dashboard will display "Unable to retrieve usage data" red flag each time the project overview page is displayed, and the http error log will report the connection failure. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1728732/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
