[Yahoo-eng-team] [Bug 1840094] [NEW] api-ref: layout of descriptions of host_status is broken
Public bug reported:
The layout of descriptions of 'host_status' is out of shape in the
following APIs in the compute API reference.
- PUT /servers/{server_id} (Update Server)
- POST /servers/{server_id}/action (Rebuild Server (rebuild Action))
** Affects: nova
Importance: Undecided
Assignee: Takashi NATSUME (natsume-takashi)
Status: New
** Tags: api-ref
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to OpenStack Compute (nova).
https://bugs.launchpad.net/bugs/1840094
Title:
api-ref: layout of descriptions of host_status is broken
Status in OpenStack Compute (nova):
New
Bug description:
The layout of descriptions of 'host_status' is out of shape in the
following APIs in the compute API reference.
- PUT /servers/{server_id} (Update Server)
- POST /servers/{server_id}/action (Rebuild Server (rebuild Action))
To manage notifications about this bug go to:
https://bugs.launchpad.net/nova/+bug/1840094/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1840090] [NEW] Cannot reparent projects
Public bug reported: For a variety of reasons Keystone needs the ability to 'safely' reparent a project (and children). This should be able to be done via and API, and should be able to be done on an enabled project. It should invalidate any tokens related to that project (and children) as well. ** Affects: keystone Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1840090 Title: Cannot reparent projects Status in OpenStack Identity (keystone): New Bug description: For a variety of reasons Keystone needs the ability to 'safely' reparent a project (and children). This should be able to be done via and API, and should be able to be done on an enabled project. It should invalidate any tokens related to that project (and children) as well. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1840090/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1839674] Re: ResourceTracker.compute_nodes won't try to create a ComputeNode a second time if the first create() fails
Reviewed: https://review.opendev.org/675704 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f578146f372386e1889561cba33e95495e66ce97 Submitter: Zuul Branch:master commit f578146f372386e1889561cba33e95495e66ce97 Author: Matt Riedemann Date: Fri Aug 9 17:17:45 2019 -0400 rt: only map compute node if we created it If ComputeNode.create() fails, the update_available_resource periodic will not try to create it again because it will be mapped in the compute_nodes dict and _init_compute_node will return early but trying to save changes to that ComputeNode object later will fail because there is no id on the object, since we failed to create it in the DB. This simply reverses the logic such that we only map the compute node if we successfully created it. Some existing _init_compute_node testing had to be changed since it relied on the order of when the ComputeNode object is created and put into the compute_nodes dict in order to pass the object along to some much lower-level PCI tracker code, which was arguably mocking too deep for a unit test. That is changed to avoid the low-level mocking and assertions and just assert that _setup_pci_tracker is called as expected. Change-Id: I9fa1d509a3de405d6246fb8670612c65c10cc93b Closes-Bug: #1839674 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1839674 Title: ResourceTracker.compute_nodes won't try to create a ComputeNode a second time if the first create() fails Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) ocata series: Triaged Status in OpenStack Compute (nova) pike series: Triaged Status in OpenStack Compute (nova) queens series: Triaged Status in OpenStack Compute (nova) rocky series: In Progress Status in OpenStack Compute (nova) stein series: In Progress Bug description: I found this while writing a functional recreate test for bug 1839560. As of this change in Ocata: https://github.com/openstack/nova/commit/1c967593fbb0ab8b9dc8b0b509e388591d32f537 The ResourceTracker.compute_nodes dict will store the ComputeNode object *before* trying to create it: https://github.com/openstack/nova/blob/6b7d0caad86fe32ffc49a8672de1eb7258f3b919/nova/compute/resource_tracker.py#L570-L571 The problem is if ComputeNode.create() fails for whatever reason, the next run through update_available_resource won't try to create the ComputeNode again because of this: https://github.com/openstack/nova/blob/6b7d0caad86fe32ffc49a8672de1eb7258f3b919/nova/compute/resource_tracker.py#L546 And eventually you get errors like this: b'2019-08-09 17:02:59,356 ERROR [nova.compute.manager] Error updating resources for node node2.' b'Traceback (most recent call last):' b' File "/home/osboxes/git/nova/nova/compute/manager.py", line 8250, in _update_available_resource_for_node' b'startup=startup)' b' File "/home/osboxes/git/nova/nova/compute/resource_tracker.py", line 715, in update_available_resource' b'self._update_available_resource(context, resources, startup=startup)' b' File "/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_concurrency/lockutils.py", line 328, in inner' b'return f(*args, **kwargs)' b' File "/home/osboxes/git/nova/nova/compute/resource_tracker.py", line 796, in _update_available_resource' b'self._update(context, cn, startup=startup)' b' File "/home/osboxes/git/nova/nova/compute/resource_tracker.py", line 1052, in _update' b'self.old_resources[nodename] = old_compute' b' File "/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_utils/excutils.py", line 220, in __exit__' b'self.force_reraise()' b' File "/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_utils/excutils.py", line 196, in force_reraise' b'six.reraise(self.type_, self.value, self.tb)' b' File "/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/six.py", line 693, in reraise' b'raise value' b' File "/home/osboxes/git/nova/nova/compute/resource_tracker.py", line 1046, in _update' b'compute_node.save()' b' File "/home/osboxes/git/nova/.tox/functional-py36/lib/python3.6/site-packages/oslo_versionedobjects/base.py", line 226, in wrapper' b'return fn(self, *args, **kwargs)' b' File "/home/osboxes/git/nova/nova/objects/compute_node.py", line 352, in save' b'db_compute = db.compute_node_update(self._context, self.id, updates)' b' File
[Yahoo-eng-team] [Bug 1839853] Re: Misuse of nova.objects.base.obj_equal_prims in tests
Reviewed: https://review.opendev.org/676030 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=5c1d9dcbb29f5a84d56c4e496b3111b4040cad13 Submitter: Zuul Branch:master commit 5c1d9dcbb29f5a84d56c4e496b3111b4040cad13 Author: Takashi NATSUME Date: Tue Aug 13 09:31:32 2019 +0900 Fix misuse of nova.objects.base.obj_equal_prims The nova.objects.base.obj_equal_prims returns True or False. It does not assert anything. So the return value should be asserted in tests. Add assertTrue where the nova.objects.base.obj_equal_prims is called. Change-Id: I49460ec3b572ee14b32229e771a5499ff91e8722 Closes-Bug: #1839853 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1839853 Title: Misuse of nova.objects.base.obj_equal_prims in tests Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) ocata series: Confirmed Status in OpenStack Compute (nova) pike series: Confirmed Status in OpenStack Compute (nova) queens series: Confirmed Status in OpenStack Compute (nova) rocky series: Confirmed Status in OpenStack Compute (nova) stein series: Confirmed Bug description: There are some tests, mostly related to BuildRequest objects, that are calling nova.objects.base.obj_equal_prims which does not assert anything, it only returns True or False - the test code itself must assert the expected result of the obj_equal_prims method. https://github.com/openstack/nova/blob/ab34c941be28f3486cd2699af8d9237e9edac351/nova/tests/functional/db/test_build_request.py https://github.com/openstack/nova/blob/d89579a66ac38fd1e30cea55306e6e7b69bab5b9/nova/tests/unit/objects/test_build_request.py To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1839853/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1784874] Re: ResourceTracker doesn't clean up compute_nodes or stats entries
** Also affects: nova/queens Importance: Undecided Status: New ** Changed in: nova/queens Importance: Undecided => Low ** Changed in: nova/queens Status: New => Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1784874 Title: ResourceTracker doesn't clean up compute_nodes or stats entries Status in OpenStack Compute (nova): Fix Released Status in OpenStack Compute (nova) queens series: Confirmed Bug description: This was noted in review: https://review.openstack.org/#/c/587636/4/nova/compute/resource_tracker.py@141 That the ResourceTracker.compute_nodes and ResourceTracker.stats (and old_resources) entries only grow and are never cleaned up as we rebalance nodes or nodes are deleted, which means it just takes up memory over time. When we cleanup compute nodes here: https://github.com/openstack/nova/blob/47ef500f4492c731ebfa33a12822ef6b5db4e7e2/nova/compute/manager.py#L7759 We should probably call a cleanup hook into the ResourceTracker to cleanup those entries as well. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1784874/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1840068] Re: (lxc) Instance failed to spawn: TypeError: object of type 'filter' has no len()
This filter code goes back to 2012 so we could backport the fix further (to pike and ocata) but no one is really using the libvirt+lxc code as far as I can tell, at least not with python3, so we can just backport to the non-extended-maintenance branches unless someone wants to backport them upstream to pike and ocata. ** Also affects: nova/stein Importance: Undecided Status: New ** Also affects: nova/queens Importance: Undecided Status: New ** Also affects: nova/rocky Importance: Undecided Status: New ** Summary changed: - (lxc) Instance failed to spawn: TypeError: object of type 'filter' has no len() + (lxc) Instance failed to spawn: TypeError: object of type 'filter' has no len() - python3 ** Changed in: nova/queens Importance: Undecided => Medium ** Changed in: nova Importance: High => Medium ** Changed in: nova/queens Status: New => Confirmed ** Changed in: nova/rocky Importance: Undecided => Medium ** Changed in: nova/stein Status: New => Confirmed ** Changed in: nova/stein Importance: Undecided => Medium ** Changed in: nova/rocky Status: New => Confirmed -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1840068 Title: (lxc) Instance failed to spawn: TypeError: object of type 'filter' has no len() - python3 Status in OpenStack Compute (nova): In Progress Status in OpenStack Compute (nova) queens series: Confirmed Status in OpenStack Compute (nova) rocky series: Confirmed Status in OpenStack Compute (nova) stein series: Confirmed Bug description: Seen in the nova-lxc CI job here: https://logs.opendev.org/24/676024/2/experimental/nova- lxc/f9a892c/controller/logs/screen-n-cpu.txt.gz#_Aug_12_23_31_05_043911 Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [None req-55d6dd1b-96ca-4afe-9a0c-cec902d3bd87 tempest-ServerAddressesTestJSON-1311986476 tempest-ServerAddressesTestJSON-1311986476] [instance: 842a9704-3700-42ef-adb3-b038ca525127] Instance failed to spawn: TypeError: object of type 'filter' has no len() Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] Traceback (most recent call last): Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/compute/manager.py", line 2495, in _build_resources Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] yield resources Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/compute/manager.py", line 2256, in _build_and_run_instance Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] block_device_info=block_device_info) Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3231, in spawn Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] destroy_disks_on_failure=True) Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5823, in _create_domain_and_network Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] destroy_disks_on_failure) Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] self.force_reraise() Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127]
[Yahoo-eng-team] [Bug 1839515] Re: Weird functional test failures hitting neutron API in unrelated resize flows since 8/5
Reviewed: https://review.opendev.org/675553 Committed: https://git.openstack.org/cgit/openstack/nova/commit/?id=f875c9d12fa03e1eb0d6ac0f3dd95f502ae1e6a1 Submitter: Zuul Branch:master commit f875c9d12fa03e1eb0d6ac0f3dd95f502ae1e6a1 Author: Balazs Gibizer Date: Fri Aug 9 09:53:45 2019 +0200 Prevent init_host test to interfere with other tests The test_migrate_disk_and_power_off_crash_finish_revert_migration test needs to simulate a compute host crash at a certain point. It stops the execution at a certain point by injecting a sleep then simulating a compute restart. However the sleep is just 30 seconds which allows the stopped function to return while other functional tests are running in the same test worker process making those tests fail in a weird way. One simple solution is to add a big enough sleep to the test that will never return before the whole functional test execution. This patch proposes a million seconds which is more than 277 hours. Similar to how the other test in this test package works. This solution is hacky but simple. A better solution would be to further enhance the capabilities of the functional test env supporting nova-compute service crash / kill + restart. Change-Id: Ib0d142806804e9113dd61d3a7ec15a98232775c8 Closes-Bug: #1839515 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1839515 Title: Weird functional test failures hitting neutron API in unrelated resize flows since 8/5 Status in OpenStack Compute (nova): Fix Released Bug description: Noticed here: https://logs.opendev.org/32/634832/43/check/nova-tox-functional- py36/d4f3be5/testr_results.html.gz With this test: nova.tests.functional.notification_sample_tests.test_service.TestServiceUpdateNotificationSampleLatest.test_service_disabled That's a simple test which disables a service and then asserts there is a service.update notification, but there is another notification happening as well: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/nova/nova/tests/functional/notification_sample_tests/test_service.py", line 122, in test_service_disabled 'uuid': self.service_uuid}) File "/home/zuul/src/opendev.org/openstack/nova/nova/tests/functional/notification_sample_tests/test_service.py", line 37, in _verify_notification base._verify_notification(sample_file_name, replacements, actual) File "/home/zuul/src/opendev.org/openstack/nova/nova/tests/functional/notification_sample_tests/notification_sample_base.py", line 148, in _verify_notification self.assertEqual(1, len(fake_notifier.VERSIONED_NOTIFICATIONS)) File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/testtools/testcase.py", line 411, in assertEqual self.assertThat(observed, matcher, message) File "/home/zuul/src/opendev.org/openstack/nova/.tox/functional-py36/lib/python3.6/site-packages/testtools/testcase.py", line 498, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: 1 != 2 And in the error output, we can see this weird traceback of a resize revert failure b/c the NeutronFixture isn't being used: 2019-08-07 23:22:23,621 ERROR [nova.network.neutronv2.api] The [neutron] section of your nova configuration file must be configured for authentication with the networking service endpoint. See the networking service install guide for details: https://docs.openstack.org/neutron/latest/install/ 2019-08-07 23:22:23,634 ERROR [nova.compute.manager] Setting instance vm_state to ERROR Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 8656, in _error_out_instance_on_exception yield File "/home/zuul/src/opendev.org/openstack/nova/nova/compute/manager.py", line 4830, in _resize_instance migration_p) File "/home/zuul/src/opendev.org/openstack/nova/nova/network/neutronv2/api.py", line 2697, in migrate_instance_start client = _get_ksa_client(context, admin=True) File "/home/zuul/src/opendev.org/openstack/nova/nova/network/neutronv2/api.py", line 215, in _get_ksa_client auth_plugin = _get_auth_plugin(context, admin=admin) File "/home/zuul/src/opendev.org/openstack/nova/nova/network/neutronv2/api.py", line 151, in _get_auth_plugin _ADMIN_AUTH = _load_auth_plugin(CONF) File "/home/zuul/src/opendev.org/openstack/nova/nova/network/neutronv2/api.py", line 82, in _load_auth_plugin raise neutron_client_exc.Unauthorized(message=err_msg) neutronclient.common.exceptions.Unauthorized: Unknown auth type: None According to logstash this started showing up around 8/5:
[Yahoo-eng-team] [Bug 1840068] [NEW] (lxc) Instance failed to spawn: TypeError: object of type 'filter' has no len()
Public bug reported: Seen in the nova-lxc CI job here: https://logs.opendev.org/24/676024/2/experimental/nova- lxc/f9a892c/controller/logs/screen-n-cpu.txt.gz#_Aug_12_23_31_05_043911 Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [None req-55d6dd1b-96ca-4afe-9a0c-cec902d3bd87 tempest-ServerAddressesTestJSON-1311986476 tempest-ServerAddressesTestJSON-1311986476] [instance: 842a9704-3700-42ef-adb3-b038ca525127] Instance failed to spawn: TypeError: object of type 'filter' has no len() Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] Traceback (most recent call last): Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/compute/manager.py", line 2495, in _build_resources Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] yield resources Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/compute/manager.py", line 2256, in _build_and_run_instance Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] block_device_info=block_device_info) Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 3231, in spawn Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] destroy_disks_on_failure=True) Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5823, in _create_domain_and_network Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] destroy_disks_on_failure) Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 220, in __exit__ Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] self.force_reraise() Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/usr/local/lib/python3.6/dist-packages/oslo_utils/excutils.py", line 196, in force_reraise Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] six.reraise(self.type_, self.value, self.tb) Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/usr/local/lib/python3.6/dist-packages/six.py", line 693, in reraise Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] raise value Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5789, in _create_domain_and_network Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] block_device_info): Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/usr/lib/python3.6/contextlib.py", line 81, in __enter__ Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] return next(self.gen) Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 5701, in _lxc_disk_handler Aug 12 23:31:05.043911 ubuntu-bionic-rax-ord-0010072710 nova-compute[27015]: ERROR nova.compute.manager [instance: 842a9704-3700-42ef-adb3-b038ca525127] block_device_info) Aug 12
[Yahoo-eng-team] [Bug 1839961] Re: Test tempest.api.compute.servers.test_novnc.NoVNCConsoleTestJSON.test_novnc failing often
*** This bug is a duplicate of bug 1669468 *** https://bugs.launchpad.net/bugs/1669468 ** This bug has been marked a duplicate of bug 1669468 tempest.api.compute.servers.test_novnc.NoVNCConsoleTestJSON.test_novnc fails intermittently in neutron multinode nv job -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1839961 Title: Test tempest.api.compute.servers.test_novnc.NoVNCConsoleTestJSON.test_novnc failing often Status in OpenStack Compute (nova): New Bug description: I see that Tempest API test tempest.api.compute.servers.test_novnc.NoVNCConsoleTestJSON.test_novnc is failing quite often on tempest-multinode-full and tempest-multinode-full-py3 jobs. Example: https://logs.opendev.org/12/672612/4/check/tempest-multinode-full-py3/72623e0/testr_results.html.gz Logstash query which I used to find other occurrences: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22AssertionError%3A%20True%20is%20not%20false%20%3A%20Token%20must%20be%20invalid%20because%20the%20connection%20closed.%5C%22 I found 61 entries in last 7 days. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1839961/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1839961] [NEW] Test tempest.api.compute.servers.test_novnc.NoVNCConsoleTestJSON.test_novnc failing often
Public bug reported: I see that Tempest API test tempest.api.compute.servers.test_novnc.NoVNCConsoleTestJSON.test_novnc is failing quite often on tempest-multinode-full and tempest-multinode-full-py3 jobs. Example: https://logs.opendev.org/12/672612/4/check/tempest-multinode-full-py3/72623e0/testr_results.html.gz Logstash query which I used to find other occurrences: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22AssertionError%3A%20True%20is%20not%20false%20%3A%20Token%20must%20be%20invalid%20because%20the%20connection%20closed.%5C%22 I found 61 entries in last 7 days. ** Affects: nova Importance: Undecided Status: New ** Tags: gate-failure -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1839961 Title: Test tempest.api.compute.servers.test_novnc.NoVNCConsoleTestJSON.test_novnc failing often Status in OpenStack Compute (nova): New Bug description: I see that Tempest API test tempest.api.compute.servers.test_novnc.NoVNCConsoleTestJSON.test_novnc is failing quite often on tempest-multinode-full and tempest-multinode-full-py3 jobs. Example: https://logs.opendev.org/12/672612/4/check/tempest-multinode-full-py3/72623e0/testr_results.html.gz Logstash query which I used to find other occurrences: http://logstash.openstack.org/#dashboard/file/logstash.json?query=message%3A%5C%22AssertionError%3A%20True%20is%20not%20false%20%3A%20Token%20must%20be%20invalid%20because%20the%20connection%20closed.%5C%22 I found 61 entries in last 7 days. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1839961/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1836618] Re: Due to case sensitivity of a user name compare in a keystone test, the test might fail
Reviewed: https://review.opendev.org/670610 Committed: https://git.openstack.org/cgit/openstack/tempest/commit/?id=4776b13babe5e5f836e1fac7793c066bc933fd71 Submitter: Zuul Branch:master commit 4776b13babe5e5f836e1fac7793c066bc933fd71 Author: manasa Date: Fri Jul 12 13:29:39 2019 -0400 Fix Upstream test cases to be case insensitive In any domain, due to case sensitivity, if a username for the project and the username in tempest.conf or accounts.yaml are not identical, then the test cases are failing. So, added .lower() method to change the username strings to be case insensitive. Closes-Bug: #1836618 Change-Id: Id7d079c881bbfae972d65ef6049f78da7f25fc1d ** Changed in: tempest Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1836618 Title: Due to case sensitivity of a user name compare in a keystone test, the test might fail Status in OpenStack Identity (keystone): Invalid Status in tempest: Fix Released Bug description: Due to case sensitivity of a user name compare in a keystone test, the test might return an error similar to the one below: Captured Traceback: Traceback (most recent call last): File "tempest/api/identity/v3/test_tokens.py", line 113, in test_token_auth_creation_existence_deletion self.assertEqual(token_details['user']['name'], user.username) File "/home/manasareddybethi/aqua-venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 411, in assertEqual self.assertThat(observed, matcher, message) File "/home/manasareddybethi/aqua-venv/local/lib/python2.7/site-packages/testtools/testcase.py", line 498, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: u'KellyTest' != u'kellytest' This is because it is comparing the name from keystone and the name from tempest.conf or accounts.yaml, when using preprovisioned creds. Here is he example that I have on our openstack instance. mismatch error from the stack trace, u'KellyTest' != u'kellytest' the username for the project and username on accounts.yaml or tempest.conf are not identical. To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1836618/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1839860] Re: cloud-init error while MAAS commissioning (PXE phase) P9 Witherspoon
** Also affects: maas Importance: Undecided Status: New ** Also affects: cloud-init Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1839860 Title: cloud-init error while MAAS commissioning (PXE phase) P9 Witherspoon Status in cloud-init: New Status in MAAS: New Status in The Ubuntu-power-systems project: New Bug description: While trying to commissioning bobone (P9 withersppon machine with OpenBMC in Server team's MAAS) the PXE phase ended with the following cloud-init error (shown at sol): Starting Wait until snapd is fully seeded... Ubuntu 18.04.3 LTS ubuntu hvc0 ubuntu login: [ 131.162174] cloud-init[5497]: Can not apply stage config, no datasource found! Likely bad things to come! [ 131.162320] cloud-init[5497]: [ 131.162414] cloud-init[5497]: Traceback (most recent call last): [ 131.162512] cloud-init[5497]: File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 485, in main_modules [ 131.162614] cloud-init[5497]: init.fetch(existing="trust") [ 131.162678] cloud-init[5497]: File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 351, in fetch [ 131.162776] cloud-init[5497]: return self._get_data_source(existing=existing) [ 131.162851] cloud-init[5497]: File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 261, in _get_data_source [ 131.162934] cloud-init[5497]: pkg_list, self.reporter) [ 131.163005] cloud-init[5497]: File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 741, in find_source [ 131.163104] cloud-init[5497]: raise DataSourceNotFoundException(msg) [ 131.163177] cloud-init[5497]: cloudinit.sources.DataSourceNotFoundException: Did not find any data source, searched classes: () [ 131.163269] cloud-init[5497]: [ 131.53] cloud-init[5551]: Can not apply stage final, no datasource found! Likely bad things to come! [ 131.566820] cloud-init[5551]: [ 131.566922] cloud-init[5551]: Traceback (most recent call last): [ 131.567004] cloud-init[5551]: File "/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 485, in main_modules [ 131.567116] cloud-init[5551]: init.fetch(existing="trust") [ 131.567193] cloud-init[5551]: File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 351, in fetch [ 131.567274] cloud-init[5551]: return self._get_data_source(existing=existing) [ 131.567348] cloud-init[5551]: File "/usr/lib/python3/dist-packages/cloudinit/stages.py", line 261, in _get_data_source [ 131.567438] cloud-init[5551]: pkg_list, self.reporter) [ 131.567508] cloud-init[5551]: File "/usr/lib/python3/dist-packages/cloudinit/sources/__init__.py", line 741, in find_source [ 131.567598] cloud-init[5551]: raise DataSourceNotFoundException(msg) [ 131.567679] cloud-init[5551]: cloudinit.sources.DataSourceNotFoundException: Did not find any data source, searched classes: () [ 131.567779] cloud-init[5551]: Ubuntu 18.04.3 LTS ubuntu hvc0 ubuntu login: MAAS log (from UI): TIMEEVENT Mon, 12 Aug. 2019 11:10:45 PXE Request - commissioning Mon, 12 Aug. 2019 11:08:43 Node powered on Mon, 12 Aug. 2019 11:08:14 Powering node on Mon, 12 Aug. 2019 11:08:14 Node - Started commissioning on 'bobone'. Mon, 12 Aug. 2019 11:08:14 Node changed status - From 'New' to 'Commissioning' Mon, 12 Aug. 2019 11:08:14 User starting node commissioning - (jfh) Mon, 12 Aug. 2019 11:07:04 Node powered off With that system cannot complete the Commissioning phase. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1839860/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1839920] [NEW] Macvtap CI fails on Train
Public bug reported: MacVtap CI[1] started to fail after merging commit[2] We think it related to this https://github.com/libvirt/libvirt/commit/b91a33638476cf57d910b6056a8fc11921edd029#diff-28bc83a0c3470bba712dfa6824a79c9d. So they change from setting the admin mac to the effective mac. The problem is that the sriov-nic agent relay on the admin mac to send rpc to the neutron server. If the mac and the pci slot don't much it ignores it and the VM stuck in spawn until timeout [1] https://wiki.openstack.org/wiki/ThirdPartySystems/Mellanox_CI [2] https://review.opendev.org/#/c/31/ ** Affects: nova Importance: Undecided Assignee: Lenny (lennyb) Status: In Progress -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1839920 Title: Macvtap CI fails on Train Status in OpenStack Compute (nova): In Progress Bug description: MacVtap CI[1] started to fail after merging commit[2] We think it related to this https://github.com/libvirt/libvirt/commit/b91a33638476cf57d910b6056a8fc11921edd029#diff-28bc83a0c3470bba712dfa6824a79c9d. So they change from setting the admin mac to the effective mac. The problem is that the sriov-nic agent relay on the admin mac to send rpc to the neutron server. If the mac and the pci slot don't much it ignores it and the VM stuck in spawn until timeout [1] https://wiki.openstack.org/wiki/ThirdPartySystems/Mellanox_CI [2] https://review.opendev.org/#/c/31/ To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1839920/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
