[Yahoo-eng-team] [Bug 1936667] Re: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3
Reviewed: https://review.opendev.org/c/openstack/zaqar/+/801080 Committed: https://opendev.org/openstack/zaqar/commit/24538768a6ff007ca37a0be2fbddb3086eb01694 Submitter: "Zuul (22348)" Branch:master commit 24538768a6ff007ca37a0be2fbddb3086eb01694 Author: Takashi Kajinami Date: Sat Jul 17 00:20:28 2021 +0900 Replace deprecated import of ABCs from collections ABCs in collections should be imported from collections.abc and direct import from collections is deprecated since Python 3.3. Closes-Bug: #1936667 Change-Id: I60679f9e78e91bce1145d899332e871dd62d3a9c ** Changed in: zaqar Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Identity (keystone). https://bugs.launchpad.net/bugs/1936667 Title: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3 Status in OpenStack Identity (keystone): In Progress Status in OpenStack Shared File Systems Service (Manila): Fix Released Status in Mistral: In Progress Status in neutron: Fix Released Status in OpenStack Object Storage (swift): Fix Released Status in tacker: In Progress Status in taskflow: Fix Released Status in tempest: Fix Released Status in zaqar: Fix Released Bug description: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3. For example: >>> import collections >>> collections.Iterable :1: DeprecationWarning: Using or importing the ABCs from 'collections' instead of from 'collections.abc' is deprecated since Python 3.3, and in 3.10 it will stop working >>> from collections import abc >>> abc.Iterable To manage notifications about this bug go to: https://bugs.launchpad.net/keystone/+bug/1936667/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1933502] Re: L3 DB FloatingIP callbacks improvements
Reviewed: https://review.opendev.org/c/openstack/neutron/+/801453 Committed: https://opendev.org/openstack/neutron/commit/b488fb8e223498703eed158e741c72d7ce03fe16 Submitter: "Zuul (22348)" Branch:master commit b488fb8e223498703eed158e741c72d7ce03fe16 Author: Szymon Wroblewski Date: Mon Jul 19 12:25:32 2021 +0200 Use payloads for FloatingIP AFTER callbacks Change FloatingIP AFTER_ notifications in l3_db to use publish calls. Move association_event field into Payload metadata. Closes-Bug: #1933502 Change-Id: Ie4c0f4a63a87c32026c49b03068e5f461deb38b6 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1933502 Title: L3 DB FloatingIP callbacks improvements Status in neutron: Fix Released Bug description: FloatingIP change notifications in l3_db use obsolete notify calls (instead of publish) and form that isn't compatible with OVOServerRpcInterface, which makes usage of RemoteResourceCache for FloatingIP by agents quite difficult. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1933502/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1905944] Re: live-migration job not aborted when live_monitor thread fails
Reviewed: https://review.opendev.org/c/openstack/nova/+/764435 Committed: https://opendev.org/openstack/nova/commit/39f0af5d18d6bea34fa15b8f7778115b25432749 Submitter: "Zuul (22348)" Branch:master commit 39f0af5d18d6bea34fa15b8f7778115b25432749 Author: Alexandre Arents Date: Thu Nov 26 15:24:19 2020 + libvirt: Abort live-migration job when monitoring fails During live migration process, a _live_migration_monitor thread checks progress of migration on source host, if for any reason we hit infrastructure issue involving a DB/RPC/libvirt-timeout failure, an Exception is raised to the nova-compute service and instance/migration is set to ERROR state. The issue is that we may let live-migration job running out of nova control. At the end of job, guest is resumed on target host while nova still reports it on source host, this may lead to a split-brain situation if instance is restarted. This change proposes to abort live-migration job if issue occurs during _live_migration_monitor. Change-Id: Ia593b500425c81e54eb401e38264db5cc5fc1f93 Closes-Bug: #1905944 ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1905944 Title: live-migration job not aborted when live_monitor thread fails Status in OpenStack Compute (nova): Fix Released Bug description: Description === During live migration, a monitoring thread poll each 0.5s libvirt job progress and update db with with jobs stats. If there control pane issue like DB/RPC or libvirt unexpected Exception (timeout) exception handling do not properly interrupt libvirt job. Steps to reproduce == On a multinode devstack master. #spawn instance on source_host 1) openstack server create --flavor m1.small --image cirros-0.5.1-x86_64-disk \ --nic net-id=private inst #ignite live block migration on dest_host, wait a bit( to be in monitoring thread), # and trigger an issue on DB for ex. 2) nova live-migration inst ; sleep 6 ; sudo service mysql restart 3) On source host you can survey libvirt job progess until it complete and disappear because libvirt resume guest on target host(starting writting data on target disk) source_host$ watch -n 1 virsh domjobinfo instance-000d 4) on dest host you will find instance active dest_host$ virsh list Id NameState --- 20 instance-000d running 5) nova show inst show instance still on source host. $nova show inst | grep host | OS-EXT-SRV-ATTR:host | source_host if admin try to recover the instance on source on as it in on nova DB, we can fall in split-brain where 2 qemu running on two different disks on two host (true story..) Expected result === If issue happen we must at least ensure that libvirt job is interrupted, avoiding the guest resume on target host. Actual result = If issue happen libvirt job continue and bring up guest on target host, nova still consider it on source. To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/1905944/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1940460] [NEW] ORM fixes broke opportunistic testing on py36
Public bug reported: The patch 9e002a77f2131d3594a2a4029a147beaf37f5b17 which is aimed at fixing things in advance of SQLAlchemy 2.0 seems to have broken our opportunistic testing of DB migrations on py36 only. This manifests as a total lockup of one worker during functional tests, which fails to report anything and eventually times out the job. ** Affects: glance Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to Glance. https://bugs.launchpad.net/bugs/1940460 Title: ORM fixes broke opportunistic testing on py36 Status in Glance: New Bug description: The patch 9e002a77f2131d3594a2a4029a147beaf37f5b17 which is aimed at fixing things in advance of SQLAlchemy 2.0 seems to have broken our opportunistic testing of DB migrations on py36 only. This manifests as a total lockup of one worker during functional tests, which fails to report anything and eventually times out the job. To manage notifications about this bug go to: https://bugs.launchpad.net/glance/+bug/1940460/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1940425] [NEW] test_live_migration_with_trunk tempest test fails due to port remains in down state
Public bug reported: Example failure is in [1]: 2021-08-18 10:40:52,334 124842 DEBUG[tempest.lib.common.utils.test_utils] Call _is_port_status_active returns false in 60.00 seconds }}} Traceback (most recent call last): File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 89, in wrapper return func(*func_args, **func_kwargs) File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 70, in wrapper return f(*func_args, **func_kwargs) File "/opt/stack/tempest/tempest/api/compute/admin/test_live_migration.py", line 281, in test_live_migration_with_trunk self.assertTrue( File "/usr/lib/python3.8/unittest/case.py", line 765, in assertTrue raise self.failureException(msg) AssertionError: False is not true Please note that a similar bug was reported and fixed previously https://bugs.launchpad.net/tempest/+bug/1924258 It seems that fix did not fully solved the issue. It is not super frequent I saw 4 occasions in the last 30 days [2]. [1] https://zuul.opendev.org/t/openstack/build/fdbda223dc10456db58f922b6435f680/logs [2] https://paste.opendev.org/show/808166/ ** Affects: neutron Importance: Undecided Status: New ** Tags: tempest trunk ** Tags added: tempest trunk -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1940425 Title: test_live_migration_with_trunk tempest test fails due to port remains in down state Status in neutron: New Bug description: Example failure is in [1]: 2021-08-18 10:40:52,334 124842 DEBUG[tempest.lib.common.utils.test_utils] Call _is_port_status_active returns false in 60.00 seconds }}} Traceback (most recent call last): File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 89, in wrapper return func(*func_args, **func_kwargs) File "/opt/stack/tempest/tempest/common/utils/__init__.py", line 70, in wrapper return f(*func_args, **func_kwargs) File "/opt/stack/tempest/tempest/api/compute/admin/test_live_migration.py", line 281, in test_live_migration_with_trunk self.assertTrue( File "/usr/lib/python3.8/unittest/case.py", line 765, in assertTrue raise self.failureException(msg) AssertionError: False is not true Please note that a similar bug was reported and fixed previously https://bugs.launchpad.net/tempest/+bug/1924258 It seems that fix did not fully solved the issue. It is not super frequent I saw 4 occasions in the last 30 days [2]. [1] https://zuul.opendev.org/t/openstack/build/fdbda223dc10456db58f922b6435f680/logs [2] https://paste.opendev.org/show/808166/ To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1940425/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1940009] Re: Some options are missing from neutron.conf generated by "tox -e genconfig"
Reviewed: https://review.opendev.org/c/openstack/neutron/+/804571 Committed: https://opendev.org/openstack/neutron/commit/c75df16e5021e28aaa9961ed0977f30fa278a1dc Submitter: "Zuul (22348)" Branch:master commit c75df16e5021e28aaa9961ed0977f30fa278a1dc Author: Takashi Kajinami Date: Sat Aug 14 23:29:49 2021 +0900 Add missing options to generated neutron.conf This change adds missing oslo.config.opts endpoints to the config file to generated neutron.conf, so that the following options are included in the neutron.conf generated by the oslo-config-generator command. 1) Some options of the oslo.service library, related to periodic tasks and base service implementation 2) Options of the oslo.reports library 3) Options of the healthcheck middleware from oslo.middleware Closes-Bug: #1940009 Change-Id: I9f4763efc02e5cacca3a00e9f101f83d3f3fb1e7 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1940009 Title: Some options are missing from neutron.conf generated by "tox -e genconfig" Status in neutron: Fix Released Bug description: Looking at neutron.conf generated by "tox -e genconfig" I noticed the following items are missing from the generated neutron.conf. 1) Some options of the oslo.service library like run_external_periodic_tasks 2) Options of the oslo.reports library 3) Options of the healthcheck middleware was added to api pipeline during Victoria cycle. https://review.opendev.org/c/openstack/neutron/+/724676 This is because there are some oslo.config.opts endpoints missing from the config file passed to the oslo-config-generator command. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1940009/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1940399] [NEW] vmware driver fails when setting vmware.maximum_objects to small value
Public bug reported: Description === There are two functions in the code, which call (indirectly) RetrievePropertyEx with at most vmware.maximum_objects per result, but do not iterate over the results if there are more, nor does it By default the value is 100, which is usually sufficient, but if someone should set the value lower, it will cause the following problems. Both are in the module nova.virt.vmwarapi.vm_util 1. get_all_cluster_mors The function is called to find the cluster reference for named cluster. If there are more clusters than maximum_objects, then there is a chance that the cluster won't be found by the driver despite being configured in the VCenter. 2. get_stats_from_cluster The function gets the statistics for all the hosts in a cluster. Since VSphere 7.0U1, the limit is 96 hosts per cluster, and quite likely the limit will increase past the default of 100, leaving the stats inaccurate. On top of it, the function does not call `cancel_retrieval` causing a leak in the VCenter. Steps to reproduce == Take any VCenter version with more than one cluster and more than one host (I would suggest three or more), and any release of Nova configured to run against the VCenter. I am not 100% according what the order is by which the clusters are returned, possibly in the order they have been created, or alphabetically. I would suggest to create first a cluster-a, and then a cluster-b, and add the hosts to cluster-b. That reflects how our clusters are created (chronologically alphabetically sorted). * Configure additionally [vmware] maximum_objects=1 * Try to start nova-compute Expected result === The nova-compute service would start up and get the stats from all the hosts in the configured cluster. Actual result = nova-compute fails to start with the error message: > The specified cluster '' was not found in vCenter If you have to clusters, and three hosts, then increasing maximum_objects to two will get you around that failure, and will trigger the second problem. You can verify that by checking the resources of the nova-compute node, which will report only the resources (CPUs,RAM...) of two of the ESXi- hosts in the cluster. Environment === 1. Exact version of OpenStack you are running. 370830e944 Merge "libvirt: Enable 'vmcoreinfo' feature by default" As far as I can see, all Nova releases are affected by this behavior. 2. Which hypervisor did you use? VMware VSphere What's the version of that? 7.0.1-17327586 3. Which networking type did you use? Neutron with NSX-T (https://github.com/sapcc/networking-nsx-t) ** Affects: nova Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1940399 Title: vmware driver fails when setting vmware.maximum_objects to small value Status in OpenStack Compute (nova): New Bug description: Description === There are two functions in the code, which call (indirectly) RetrievePropertyEx with at most vmware.maximum_objects per result, but do not iterate over the results if there are more, nor does it By default the value is 100, which is usually sufficient, but if someone should set the value lower, it will cause the following problems. Both are in the module nova.virt.vmwarapi.vm_util 1. get_all_cluster_mors The function is called to find the cluster reference for named cluster. If there are more clusters than maximum_objects, then there is a chance that the cluster won't be found by the driver despite being configured in the VCenter. 2. get_stats_from_cluster The function gets the statistics for all the hosts in a cluster. Since VSphere 7.0U1, the limit is 96 hosts per cluster, and quite likely the limit will increase past the default of 100, leaving the stats inaccurate. On top of it, the function does not call `cancel_retrieval` causing a leak in the VCenter. Steps to reproduce == Take any VCenter version with more than one cluster and more than one host (I would suggest three or more), and any release of Nova configured to run against the VCenter. I am not 100% according what the order is by which the clusters are returned, possibly in the order they have been created, or alphabetically. I would suggest to create first a cluster-a, and then a cluster-b, and add the hosts to cluster-b. That reflects how our clusters are created (chronologically alphabetically sorted). * Configure additionally [vmware] maximum_objects=1 * Try to start nova-compute Expected result === The nova-compute service would start up and get the stats from all the hosts in the configured cluster. Actual result = nova-compute fails to start with the error message: > The specif
[Yahoo-eng-team] [Bug 1940224] Re: UT neutron.tests.unit.services.ovn_l3.test_plugin.OVNL3ExtrarouteTests failing with neutron-lib from master
Reviewed: https://review.opendev.org/c/openstack/neutron-lib/+/804894 Committed: https://opendev.org/openstack/neutron-lib/commit/c6f576e66b577b9929f080f1ffd83cd3783794a4 Submitter: "Zuul (22348)" Branch:master commit c6f576e66b577b9929f080f1ffd83cd3783794a4 Author: Rodolfo Alonso Hernandez Date: Tue Aug 17 16:39:38 2021 + Use explicit "dict_or_nodata" validation for "external_gateway_info" That API definition will be used during the policy check to create the needed rule for each internal GW info element, specifically with the external network fixed IPs. Only the owner of the external network or the administrator can create fixed IPs on the external network. Change-Id: Ia70485ed7713436dd6433ed39ae721500dbed7dd Closes-Bug: #1940224 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1940224 Title: UT neutron.tests.unit.services.ovn_l3.test_plugin.OVNL3ExtrarouteTests failing with neutron-lib from master Status in neutron: Fix Released Bug description: Since 13.08.2021 one of our UT neutron.tests.unit.services.ovn_l3.test_plugin.OVNL3ExtrarouteTests.test_router_create_with_gwinfo_ext_ip_non_admin is failing with neutron-lib from master. Failure example: https://5095d1cf5e3173e1d222-5acdef5dc10478cee5291df1596ec66a.ssl.cf5.rackcdn.com/periodic/opendev.org/openstack/neutron/master/openstack- tox-py36-with-neutron-lib-master/292883e/testr_results.html Stacktrace: ft1.182: neutron.tests.unit.services.ovn_l3.test_plugin.OVNL3ExtrarouteTests.test_router_create_with_gwinfo_ext_ip_non_admintesttools.testresult.real._StringException: Traceback (most recent call last): File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 183, in func return f(self, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/base.py", line 183, in func return f(self, *args, **kwargs) File "/home/zuul/src/opendev.org/openstack/neutron/neutron/tests/unit/extensions/test_l3.py", line 749, in test_router_create_with_gwinfo_ext_ip_non_admin self.assertEqual(exc.HTTPForbidden.code, res.status_int) File "/home/zuul/src/opendev.org/openstack/neutron/.tox/py36/lib/python3.6/site-packages/testtools/testcase.py", line 393, in assertEqual self.assertThat(observed, matcher, message) File "/home/zuul/src/opendev.org/openstack/neutron/.tox/py36/lib/python3.6/site-packages/testtools/testcase.py", line 480, in assertThat raise mismatch_error testtools.matchers._impl.MismatchError: 403 != 201 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1940224/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
