Public bug reported: Detailed bug description: Problem discovered on MOS 9.0 ISO #495 RC2
Problem was discovered during execution of NovaServers.resize_server rally test: AMQP server is unreachable 6 times from 97. Nova log file atached to this bug. Nova api log of this issue: 2016-07-07 13:09:44.688 19298 DEBUG keystoneauth.session [-] REQ: curl -g -i -X GET http://192.168.0.2:35357/v3/auth/tokens -H "X-Subject-Token: {SHA1}b4af682fec7d7d1d9eae361e225206444e9248c2" -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}63adf02cc36a824b4c6f0ee6ba6003f06ea88ae9" _http_log_request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:248 2016-07-07 13:09:48.363 19400 DEBUG nova.metadata.wsgi.server [req-17a7f302-f66c-432d-8ebf-ec180b6da5cc - - - - -] (19400) accepted ('240.0.0.2', 42160) server /usr/lib/python2.7/dist-packages/eventlet/wsgi.py:867 2016-07-07 13:09:48.366 19400 INFO nova.metadata.wsgi.server [-] 240.0.0.2 "OPTIONS / HTTP/1.0" status: 200 len: 234 time: 0.0007420 2016-07-07 13:09:48.811 19311 DEBUG nova.osapi_compute.wsgi.server [req-ea429186-3ca0-4e49-bd6b-fd333b1ba765 - - - - -] (19311) accepted ('240.0.0.2', 45402) server /usr/lib/python2.7/dist-packages/eventlet/wsgi.py:867 2016-07-07 13:09:48.813 19311 DEBUG nova.api.openstack.wsgi [-] Calling method '<bound method Versions.index of <nova.api.openstack.compute.versions.Versions object at 0x7f8517191690>>' _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:699 2016-07-07 13:09:48.814 19311 INFO nova.osapi_compute.wsgi.server [-] 240.0.0.2 "OPTIONS / HTTP/1.0" status: 200 len: 501 time: 0.0018778 2016-07-07 13:09:56.098 19271 DEBUG oslo.messaging._drivers.impl_rabbit [-] Received recoverable error from kombu: on_error /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py:683 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 436, in _ensured 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, **kwargs) 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 508, in __call__ 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, channel=channels[0], **kwargs), channels[0] 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 734, in execute_method 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit method() 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1028, in _consume 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit self.connection.drain_events(timeout=poll_timeout) 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 275, in drain_events 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs) 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 95, in drain_events 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit return connection.drain_events(**kwargs) 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 303, in drain_events 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit chanmap, None, timeout=timeout, 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 366, in _wait_multiple 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit channel, method_sig, args, content = read_timeout(timeout) 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 337, in read_timeout 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit return self.method_reader.read_method() 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 189, in read_method 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit raise m 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit error: [Errno 110] Connection timed out 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit 2016-07-07 13:09:56.100 19271 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.7:5673 is unreachable: [Errno 110] Connection timed out. Trying again in 1 seconds. Steps to reproduce: Start rally test NovaServers.resize_server during the load. Expected results: Test passed. Server flavor was chenged. Actual result: Test failed. Reproducibility: Iterations: 97, Failures: 6 Workaround: n/a Impact: unknown Description of the environment: * 10 baremetal nodes: - CPU: 12 x 2.10 GHz - Disks: 2 drives (SSD - 80 GB, HDD - 931.5 GB), 1006.0 GB total - Memory: 2 x 16.0 GB, 32.0 GB total - NUMA topology: 1 NUMA node * Node roles: - 3 controllers (1 was is offline because of disk problems) - 7 computes * Details: - OS on controllers: Mitaka on Ubuntu 14.04 - OS on computes: RHEL - Compute: KVM - Neutron with VLAN segmentation Additional information: Diagnostic snapshot download link: http://mos-scale-share.mirantis.com/fuel-snapshot-2016-07-11_08-57-28.tar.gz ** Affects: nova Importance: Undecided Status: New ** Tags: nova resize-server ** Attachment added: "nova-api.log.4.gz" https://bugs.launchpad.net/bugs/1602274/+attachment/4699447/+files/nova-api.log.4.gz -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1602274 Title: Connection timed out Status in OpenStack Compute (nova): New Bug description: Detailed bug description: Problem discovered on MOS 9.0 ISO #495 RC2 Problem was discovered during execution of NovaServers.resize_server rally test: AMQP server is unreachable 6 times from 97. Nova log file atached to this bug. Nova api log of this issue: 2016-07-07 13:09:44.688 19298 DEBUG keystoneauth.session [-] REQ: curl -g -i -X GET http://192.168.0.2:35357/v3/auth/tokens -H "X-Subject-Token: {SHA1}b4af682fec7d7d1d9eae361e225206444e9248c2" -H "User-Agent: python-keystoneclient" -H "Accept: application/json" -H "X-Auth-Token: {SHA1}63adf02cc36a824b4c6f0ee6ba6003f06ea88ae9" _http_log_request /usr/lib/python2.7/dist-packages/keystoneauth1/session.py:248 2016-07-07 13:09:48.363 19400 DEBUG nova.metadata.wsgi.server [req-17a7f302-f66c-432d-8ebf-ec180b6da5cc - - - - -] (19400) accepted ('240.0.0.2', 42160) server /usr/lib/python2.7/dist-packages/eventlet/wsgi.py:867 2016-07-07 13:09:48.366 19400 INFO nova.metadata.wsgi.server [-] 240.0.0.2 "OPTIONS / HTTP/1.0" status: 200 len: 234 time: 0.0007420 2016-07-07 13:09:48.811 19311 DEBUG nova.osapi_compute.wsgi.server [req-ea429186-3ca0-4e49-bd6b-fd333b1ba765 - - - - -] (19311) accepted ('240.0.0.2', 45402) server /usr/lib/python2.7/dist-packages/eventlet/wsgi.py:867 2016-07-07 13:09:48.813 19311 DEBUG nova.api.openstack.wsgi [-] Calling method '<bound method Versions.index of <nova.api.openstack.compute.versions.Versions object at 0x7f8517191690>>' _process_stack /usr/lib/python2.7/dist-packages/nova/api/openstack/wsgi.py:699 2016-07-07 13:09:48.814 19311 INFO nova.osapi_compute.wsgi.server [-] 240.0.0.2 "OPTIONS / HTTP/1.0" status: 200 len: 501 time: 0.0018778 2016-07-07 13:09:56.098 19271 DEBUG oslo.messaging._drivers.impl_rabbit [-] Received recoverable error from kombu: on_error /usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py:683 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit Traceback (most recent call last): 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 436, in _ensured 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, **kwargs) 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 508, in __call__ 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit return fun(*args, channel=channels[0], **kwargs), channels[0] 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 734, in execute_method 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit method() 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/oslo_messaging/_drivers/impl_rabbit.py", line 1028, in _consume 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit self.connection.drain_events(timeout=poll_timeout) 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/connection.py", line 275, in drain_events 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit return self.transport.drain_events(self.connection, **kwargs) 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/kombu/transport/pyamqp.py", line 95, in drain_events 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit return connection.drain_events(**kwargs) 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 303, in drain_events 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit chanmap, None, timeout=timeout, 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 366, in _wait_multiple 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit channel, method_sig, args, content = read_timeout(timeout) 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/connection.py", line 337, in read_timeout 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit return self.method_reader.read_method() 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit File "/usr/lib/python2.7/dist-packages/amqp/method_framing.py", line 189, in read_method 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit raise m 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit error: [Errno 110] Connection timed out 2016-07-07 13:09:56.098 19271 ERROR oslo.messaging._drivers.impl_rabbit 2016-07-07 13:09:56.100 19271 ERROR oslo.messaging._drivers.impl_rabbit [-] AMQP server on 192.168.0.7:5673 is unreachable: [Errno 110] Connection timed out. Trying again in 1 seconds. Steps to reproduce: Start rally test NovaServers.resize_server during the load. Expected results: Test passed. Server flavor was chenged. Actual result: Test failed. Reproducibility: Iterations: 97, Failures: 6 Workaround: n/a Impact: unknown Description of the environment: * 10 baremetal nodes: - CPU: 12 x 2.10 GHz - Disks: 2 drives (SSD - 80 GB, HDD - 931.5 GB), 1006.0 GB total - Memory: 2 x 16.0 GB, 32.0 GB total - NUMA topology: 1 NUMA node * Node roles: - 3 controllers (1 was is offline because of disk problems) - 7 computes * Details: - OS on controllers: Mitaka on Ubuntu 14.04 - OS on computes: RHEL - Compute: KVM - Neutron with VLAN segmentation Additional information: Diagnostic snapshot download link: http://mos-scale-share.mirantis.com/fuel-snapshot-2016-07-11_08-57-28.tar.gz To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1602274/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
