[Yahoo-eng-team] [Bug 1961803] Re: netplan renaming does not work on maas deployed machines on reboot
Based on your comment, it sounds like this is a configuration issue, so I'm going to set this as Invalid for cloud-init. If that's incorrect, please do set it back to New. ** Changed in: cloud-init Status: Confirmed => Invalid -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1961803 Title: netplan renaming does not work on maas deployed machines on reboot Status in cloud-init: Invalid Status in netplan: Confirmed Bug description: Netplan renaming does not work on maas deployed machines when rebooting. I believe this is because of a race with curtin configs. I have spawned a focal VM on maas and I want to rename the interface. I create the following config file : # cat /etc/netplan/76-awaya.yaml network: version: 2 ethernets: enp5s0: set-name: awaya match: macaddress: 00:16:3e:e3:a0:bf When I reboot, the interface does not get renamed : # reboot # ip a 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: enp5s0: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:16:3e:e3:a0:bf brd ff:ff:ff:ff:ff:ff inet6 fd42:568f:86da:6c4a:216:3eff:fee3:a0bf/64 scope global dynamic mngtmpaddr noprefixroute valid_lft 3519sec preferred_lft 3519sec inet6 fe80::216:3eff:fee3:a0bf/64 scope link valid_lft forever preferred_lft forever If I remove the curtin network config file renaming works. # rm /etc/cloud/cloud.cfg.d/50-curtin-networking.cfg # reboot # ip a 1: lo: mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00 inet 127.0.0.1/8 scope host lo valid_lft forever preferred_lft forever inet6 ::1/128 scope host valid_lft forever preferred_lft forever 2: awaya: mtu 1500 qdisc mq state UP group default qlen 1000 link/ether 00:16:3e:e3:a0:bf brd ff:ff:ff:ff:ff:ff inet 10.93.227.2/24 brd 10.93.227.255 scope global awaya valid_lft forever preferred_lft forever inet6 fd42:568f:86da:6c4a:216:3eff:fee3:a0bf/64 scope global dynamic mngtmpaddr noprefixroute valid_lft 3596sec preferred_lft 3596sec inet6 fd42:568f:86da:6c4a:0:1::/64 scope global valid_lft forever preferred_lft forever inet6 fe80::216:3eff:fee3:a0bf/64 scope link valid_lft forever preferred_lft forever Contents of 50-curtin-networking.cfg # cat /etc/cloud/cloud.cfg.d/50-curtin-networking.cfg network: ethernets: enp5s0: addresses: - 10.93.227.2/24 - fd42:568f:86da:6c4a:0:1::/64 gateway4: 10.93.227.1 match: macaddress: 00:16:3e:e3:a0:bf mtu: 1500 nameservers: addresses: - 10.93.227.10 search: - maas set-name: enp5s0 version: 2 To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1961803/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1961874] Re: [ovn-octavia-provider] OVN NB DB timeouts
** Also affects: kolla/yoga
Importance: Undecided
Status: In Progress
** Also affects: kolla/victoria
Importance: Undecided
Status: New
** Also affects: kolla/wallaby
Importance: Undecided
Status: New
** Also affects: kolla/xena
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1961874
Title:
[ovn-octavia-provider] OVN NB DB timeouts
Status in kolla:
In Progress
Status in kolla victoria series:
New
Status in kolla wallaby series:
New
Status in kolla xena series:
New
Status in kolla yoga series:
In Progress
Status in neutron:
New
Bug description:
Hello,
During adding members to an LB/pool - following error shows up after
180 seconds (ovsdb_timeout)
2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn
[req-ef0be432-7fea-4015-bcfc-713a2bd9f492 - c80fde4636ec4656a531de5bfeac5541 -
d5125dbc6522477ebad3d78e8eeaa692 d5125dbc6522477ebad3d78e8eeaa692] OVS database
connection to OVN_Northbound failed with error: 'Timeout'. Verify that the OVS
and OVN services are available and that the 'ovn_nb_connection' and
'ovn_sb_connection' configuration options are correct.: Exception: Timeout
2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn
Traceback (most recent call last):
2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovn_octavia_provider/ovsdb/impl_idl_ovn.py",
line 62, in start_connection
2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn
self.ovsdb_connection.start()
2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py",
line 84, in start
2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn
idlutils.wait_for_change(self.idl, self.timeout)
2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py",
line 244, in wait_for_change
2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn
raise Exception("Timeout") # TODO(twilson) use TimeoutException?
2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn
Exception: Timeout
2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn
2022-02-22 17:00:22.792 24 ERROR octavia.api.drivers.driver_factory
[req-ef0be432-7fea-4015-bcfc-713a2bd9f492 - c80fde4636ec4656a531de5bfeac5541 -
d5125dbc6522477ebad3d78e8eeaa692 d5125dbc6522477ebad3d78e8eeaa692] Unable to
load provider driver ovn due to: OVS database connection to OVN_Northbound
failed with error: 'Timeout'. Verify that the OVS and OVN services are
available and that the 'ovn_nb_connection' and 'ovn_sb_connection'
configuration options are correct.:
ovn_octavia_provider.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS
database connection to OVN_Northbound failed with error: 'Timeout'. Verify that
the OVS and OVN services are available and that the 'ovn_nb_connection' and
'ovn_sb_connection' configuration options are correct.
2022-02-22 17:00:22.794 24 ERROR wsme.api
[req-ef0be432-7fea-4015-bcfc-713a2bd9f492 - c80fde4636ec4656a531de5bfeac5541 -
d5125dbc6522477ebad3d78e8eeaa692 d5125dbc6522477ebad3d78e8eeaa692] Server-side
error: "Provider 'ovn' was not found.". Detail:
Traceback (most recent call last):
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovn_octavia_provider/ovsdb/impl_idl_ovn.py",
line 62, in start_connection
self.ovsdb_connection.start()
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py",
line 84, in start
idlutils.wait_for_change(self.idl, self.timeout)
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py",
line 244, in wait_for_change
raise Exception("Timeout") # TODO(twilson) use TimeoutException?
Exception: Timeout
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/octavia/api/drivers/driver_factory.py",
line 44, in get_driver
invoke_on_load=True).driver
File "/var/lib/kolla/venv/lib/python3.6/site-packages/stevedore/driver.py",
line 62, in __init__
warn_on_missing_entrypoint=warn_on_missing_entrypoint
File "/var/lib/kolla/venv/lib/python3.6/site-packages/stevedore/named.py",
line 81, in __init__
verify_requirements)
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/stevedore/extension.py", line
233, in _load_plugins
self._on_load_failure_callback(self, ep, err)
File
[Yahoo-eng-team] [Bug 1961874] Re: [ovn-octavia-provider] OVN NB DB timeouts
** Also affects: kolla
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1961874
Title:
[ovn-octavia-provider] OVN NB DB timeouts
Status in kolla:
In Progress
Status in neutron:
New
Bug description:
Hello,
During adding members to an LB/pool - following error shows up after
180 seconds (ovsdb_timeout)
2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn
[req-ef0be432-7fea-4015-bcfc-713a2bd9f492 - c80fde4636ec4656a531de5bfeac5541 -
d5125dbc6522477ebad3d78e8eeaa692 d5125dbc6522477ebad3d78e8eeaa692] OVS database
connection to OVN_Northbound failed with error: 'Timeout'. Verify that the OVS
and OVN services are available and that the 'ovn_nb_connection' and
'ovn_sb_connection' configuration options are correct.: Exception: Timeout
2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn
Traceback (most recent call last):
2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovn_octavia_provider/ovsdb/impl_idl_ovn.py",
line 62, in start_connection
2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn
self.ovsdb_connection.start()
2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py",
line 84, in start
2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn
idlutils.wait_for_change(self.idl, self.timeout)
2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py",
line 244, in wait_for_change
2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn
raise Exception("Timeout") # TODO(twilson) use TimeoutException?
2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn
Exception: Timeout
2022-02-22 17:00:22.790 24 ERROR ovn_octavia_provider.ovsdb.impl_idl_ovn
2022-02-22 17:00:22.792 24 ERROR octavia.api.drivers.driver_factory
[req-ef0be432-7fea-4015-bcfc-713a2bd9f492 - c80fde4636ec4656a531de5bfeac5541 -
d5125dbc6522477ebad3d78e8eeaa692 d5125dbc6522477ebad3d78e8eeaa692] Unable to
load provider driver ovn due to: OVS database connection to OVN_Northbound
failed with error: 'Timeout'. Verify that the OVS and OVN services are
available and that the 'ovn_nb_connection' and 'ovn_sb_connection'
configuration options are correct.:
ovn_octavia_provider.ovsdb.impl_idl_ovn.OvsdbConnectionUnavailable: OVS
database connection to OVN_Northbound failed with error: 'Timeout'. Verify that
the OVS and OVN services are available and that the 'ovn_nb_connection' and
'ovn_sb_connection' configuration options are correct.
2022-02-22 17:00:22.794 24 ERROR wsme.api
[req-ef0be432-7fea-4015-bcfc-713a2bd9f492 - c80fde4636ec4656a531de5bfeac5541 -
d5125dbc6522477ebad3d78e8eeaa692 d5125dbc6522477ebad3d78e8eeaa692] Server-side
error: "Provider 'ovn' was not found.". Detail:
Traceback (most recent call last):
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovn_octavia_provider/ovsdb/impl_idl_ovn.py",
line 62, in start_connection
self.ovsdb_connection.start()
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/connection.py",
line 84, in start
idlutils.wait_for_change(self.idl, self.timeout)
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/ovsdbapp/backend/ovs_idl/idlutils.py",
line 244, in wait_for_change
raise Exception("Timeout") # TODO(twilson) use TimeoutException?
Exception: Timeout
The above exception was the direct cause of the following exception:
Traceback (most recent call last):
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/octavia/api/drivers/driver_factory.py",
line 44, in get_driver
invoke_on_load=True).driver
File "/var/lib/kolla/venv/lib/python3.6/site-packages/stevedore/driver.py",
line 62, in __init__
warn_on_missing_entrypoint=warn_on_missing_entrypoint
File "/var/lib/kolla/venv/lib/python3.6/site-packages/stevedore/named.py",
line 81, in __init__
verify_requirements)
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/stevedore/extension.py", line
233, in _load_plugins
self._on_load_failure_callback(self, ep, err)
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/stevedore/extension.py", line
225, in _load_plugins
verify_requirements,
File "/var/lib/kolla/venv/lib/python3.6/site-packages/stevedore/named.py",
line 158, in _load_one_plugin
verify_requirements,
File
"/var/lib/kolla/venv/lib/python3.6/site-packages/stevedore/extension.py", line
257, in _load_one_plugin
obj = plugin(*invoke_args, **invoke_kwds)
[Yahoo-eng-team] [Bug 1939169] Re: glance md-tag-create-multiple overwrites existing tags
** Changed in: glance/xena
Status: In Progress => Won't Fix
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to Glance.
https://bugs.launchpad.net/bugs/1939169
Title:
glance md-tag-create-multiple overwrites existing tags
Status in Glance:
Fix Released
Status in Glance xena series:
Won't Fix
Bug description:
Our md-tag-create-multiple (/v2/metadefs/namespaces/{namespace_name}/tags)
[1] API overwrites existing tags for specified namespace rather than creating
new one in addition to the existing tags.
Where as if you try to create different tags using md-tag-create
(/v2/metadefs/namespaces/{namespace_name}/tags/{tag_name}) it is working as
expected, means adding new tag in addition to existing ones.
Steps to reproduce:
1. source using admin credentials
$ source devstack/openrc admin admin
2. Create new public namespace
$ glance md-namespace-create TagsBugNamespace --visibility public
++--+
| Property | Value|
++--+
| created_at | 2021-08-06T17:43:03Z |
| namespace | TagsBugNamespace |
| owner | a14a058e2d1540c3a0dc7c397c55174e |
| protected | False|
| schema | /v2/schemas/metadefs/namespace |
| updated_at | 2021-08-06T17:43:03Z |
| visibility | public |
++--+
3. Create single tag using md-tag-create command
$ glance md-tag-create TagsBugNamespace --name tag1
++--+
| Property | Value|
++--+
| created_at | 2021-08-06T17:57:37Z |
| name | tag1 |
| updated_at | 2021-08-06T17:57:37Z |
++--+
4. Create another tag
$ glance md-tag-create TagsBugNamespace --name tag2
++--+
| Property | Value|
++--+
| created_at | 2021-08-06T17:57:37Z |
| name | tag2 |
| updated_at | 2021-08-06T17:57:37Z |
++--+
5. Verify that we have two tags in the list
$ glance md-tag-list TagsBugNamespace
+--+
| name |
+--+
| tag2 |
| tag1 |
+--+
6. Add more tags using md-tag-crate-multiple command
$ glance md-tag-create-multiple TagsBugNamespace --names
TestTag1141=TestTag2411 --delim =
+-+
| name|
+-+
| TestTag1141 |
| TestTag2411 |
+-+
7. Now run tags list command again
$ glance md-tag-list TagsBugNamespace
+-+
| name|
+-+
| TestTag2411 |
| TestTag1141 |
+-+
Expected result:
These new tags should have been added to existing tags.
Actual result:
Existing tags gets deleted and only newly added tags using
md-tag-crate-multiple command remains.
* This is further to show that adding new tag using md-tag-create command now
will add a new tag and does not overwrite existing ones.
$ glance md-tag-create TagsBugNamespace --name tag3
++--+
| Property | Value|
++--+
| created_at | 2021-08-06T18:12:14Z |
| name | tag3 |
| updated_at | 2021-08-06T18:12:14Z |
++--+
* Verify that we have not overwritten existing tags now;
$ glance md-tag-list TagsBugNamespace
+-+
| name|
+-+
| tag3|
| TestTag2411 |
| TestTag1141 |
+-+
[1] https://docs.openstack.org/api-ref/image/v2/metadefs-
index.html?expanded=create-tag-definition-detail,create-tags-
detail,get-tag-definition-detail,delete-all-tag-definitions-
detail#create-tags
To manage notifications about this bug go to:
https://bugs.launchpad.net/glance/+bug/1939169/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1928922] Re: evacuation tests in nova-live-migration post hook fails with VirtualInterfaceCreateException due to vif-plugged event received before nova starts waiting for it.
*** This bug is a duplicate of bug 1901707 *** https://bugs.launchpad.net/bugs/1901707 ** This bug has been marked a duplicate of bug 1901707 race condition on port binding vs instance being resumed for live-migrations -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/1928922 Title: evacuation tests in nova-live-migration post hook fails with VirtualInterfaceCreateException due to vif-plugged event received before nova starts waiting for it. Status in OpenStack Compute (nova): Triaged Bug description: Example POST_FAILURE run https://zuul.opendev.org/t/openstack/build/4d0d5072c1e0479db616a211e0afda42/logs https://zuul.opendev.org/t/openstack/build/4d0d5072c1e0479db616a211e0afda42/log/controller/logs/screen- n-cpu.txt#13322 May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova- compute[94366]: ERROR nova.compute.manager [None req-778f8336-8e29-4282-a112-91a876303fe3 demo admin] [instance: 021396f6-40ff-434c-acbe-7092a6b1bcd9] Setting instance vm_state to ERROR: nova.exception.VirtualInterfaceCreateException: Virtual Interface creation failed May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova- compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-434c-acbe-7092a6b1bcd9] Traceback (most recent call last): May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova- compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-434c-acbe-7092a6b1bcd9] File "/opt/stack/nova/nova/virt/libvirt/driver.py", line 7188, in _create_guest_with_network May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova- compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-434c-acbe-7092a6b1bcd9] guest = self._create_guest( May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova- compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-434c-acbe-7092a6b1bcd9] File "/usr/lib/python3.8/contextlib.py", line 120, in __exit__ May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova- compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-434c-acbe-7092a6b1bcd9] next(self.gen) May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova- compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-434c-acbe-7092a6b1bcd9] File "/opt/stack/nova/nova/compute/manager.py", line 479, in wait_for_instance_event May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova- compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-434c-acbe-7092a6b1bcd9] actual_event = event.wait() May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova- compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-434c-acbe-7092a6b1bcd9] File "/usr/local/lib/python3.8/dist-packages/eventlet/event.py", line 125, in wait May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova- compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-434c-acbe-7092a6b1bcd9] result = hub.switch() May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova- compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-434c-acbe-7092a6b1bcd9] File "/usr/local/lib/python3.8/dist-packages/eventlet/hubs/hub.py", line 313, in switch May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova- compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-434c-acbe-7092a6b1bcd9] return self.greenlet.switch() May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova- compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-434c-acbe-7092a6b1bcd9] eventlet.timeout.Timeout: 300 seconds May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova- compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-434c-acbe-7092a6b1bcd9] May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova- compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-434c-acbe-7092a6b1bcd9] During handling of the above exception, another exception occurred: May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova- compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-434c-acbe-7092a6b1bcd9] May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova- compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-434c-acbe-7092a6b1bcd9] Traceback (most recent call last): May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova- compute[94366]: ERROR nova.compute.manager [instance: 021396f6-40ff-434c-acbe-7092a6b1bcd9] File "/opt/stack/nova/nova/compute/manager.py", line 10090, in _error_out_instance_on_exception May 14 16:30:35.248552 ubuntu-focal-ovh-bhs1-0024684290 nova- compute[94366]: ERROR nova.compute.manager [instance:
[Yahoo-eng-team] [Bug 1940791] Re: sr0 not available causes cloud-init.target not run on focal cloud image
** Changed in: cloud-images
Status: Expired => Incomplete
** Changed in: cloud-init
Status: Expired => Incomplete
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1940791
Title:
sr0 not available causes cloud-init.target not run on focal cloud
image
Status in cloud-images:
Incomplete
Status in cloud-init:
Incomplete
Bug description:
Focal image cloud-init generator reports:
'cloud-init is enabled but no datasource found, disabling'
looks to be related to ds-identify not finding the cdrom drive (and
caching it) on first run. Not sure why /dev/sr0 would not be available
early enough.
cat /run/cloud-init/ds-identify.log
...
ISO9660_DEVS=
...
No ds found [mode=search, notfound=disabled]. Disabled cloud-init [1]
[up 1.20s] returning 1
root@ubuntu:~# /usr/lib/cloud-init/ds-identify --force
[up 200.71s] ds-identify --force
...
ISO9660_DEVS=/dev/sr0=cidata
...
Found single datasource: NoCloud
[up 200.79s] returning 0
Booting https://cloud-images.ubuntu.com/focal/current/focal-server-
cloudimg-amd64-disk-kvm.img as of Aug 22, 2021 in KVM (created with
virt-install and libvirt) along with cloud-config ISO
$ cat /tmp/cloud
#cloud-config
hostname: proxy1
$ cloud-localds /tmp/test.iso /tmp/cloud
cloud-init.target never reached and network doesn't come up (default
behavior for cloud-init is eth0 DHCP). If I manually start `systemctl
start cloud-init.target` then I get what I expected, but by then it is
"too late" and I also have to kick systemd-networkd.
cloud-init starts up as expected with the same environment when using
Bionic (https://cloud-images.ubuntu.com/bionic/current/bionic-server-
cloudimg-amd64.img)
The focal image never touches cloud-init.target. Note that there is no
reverse dependency in focal.
root@ubuntu:~# systemctl list-dependencies --reverse cloud-init.target
cloud-init.target
Both images have default target of "graphical.target"
There is mention of a "generator" and "detection" in the cloud-init
docs. https://cloudinit.readthedocs.io/en/latest/topics/boot.html
The generator appears to be what is adding the "wants" of cloud-init.target
to multi-user.target
from /lib/systemd/system-generators/cloud-init-generator:
local target_name="multi-user.target" gen_d="$early_d"
local link_path="$gen_d/${target_name}.wants/${CLOUD_TARGET_NAME}"
Bionic:
root@proxy1:~# systemctl get-default
graphical.target
root@proxy1:~#
UNIT LOAD ACTIVE SUBDESCRIPTION
basic.target loaded active active Basic System
cloud-config.targetloaded active active Cloud-config availability
cloud-init.target loaded active active Cloud-init target
...
root@proxy1:~# systemctl list-dependencies --reverse cloud-init.target
cloud-init.target
● └─multi-user.target
● └─graphical.target
root@proxy1:/etc/systemd/system# cat /run/cloud-init/cloud-init-generator.log
/lib/systemd/system-generators/cloud-init-generator
normal=/run/systemd/generator early=/run/systemd/generator.early
late=/run/systemd/generator.late
kernel command line (/proc/cmdline):
BOOT_IMAGE=/boot/vmlinuz-4.15.0-154-generic root=LABEL=cloudimg-rootfs ro
console=tty1 console=ttyS0
kernel_cmdline found unset
etc_file found unset
default found enabled
checking for datasource
ds-identify rc=0
ds-identify _RET=found
enabled via
/run/systemd/generator.early/multi-user.target.wants/cloud-init.target ->
/lib/systemd/system/cloud-init.target
Focal:
root@ubuntu:~# systemctl get-default
graphical.target
root@ubuntu:~# systemctl list-units --type=target --all
UNITLOAD ACTIVE SUB
>
basic.targetloaded active
activ>
blockdev@dev-disk-by\x2dlabel-cloudimg\x2drootfs.target loaded inactive
dead >
blockdev@dev-disk-by\x2dlabel-UEFI.target loaded inactive
dead >
blockdev@dev-loop0.target loaded inactive
dead >
blockdev@dev-loop1.target loaded inactive
dead >
blockdev@dev-loop2.target loaded inactive
dead >
blockdev@dev-vda15.target loaded inactive
dead >
cloud-config.target loaded inactive
dead >
cloud-init.target loaded inactive
dead >
root@ubuntu:~# systemctl list-unit-files
...
cloud-config.service enabled enabled
cloud-final.serviceenabled enabled
cloud-init-local.service enabled enabled
cloud-init.service enabled enabled
...
root@ubuntu:~# systemctl list-dependencies --reverse
[Yahoo-eng-team] [Bug 1962167] [NEW] [ntp] "test_network_attached_with_two_routers" randomly failing
Public bug reported: Failures detected in neutron-lib CI. Example: https://c0a6da4decb9452ac635-13a95957352e8408b84a588b720357c0.ssl.cf2.rackcdn.com/828738/6/check/neutron- tempest-plugin-api/90fd383/testr_results.html Error: https://paste.opendev.org/show/bH5jPQGIgO5TF70wDXMv/ ** Affects: neutron Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1962167 Title: [ntp] "test_network_attached_with_two_routers" randomly failing Status in neutron: New Bug description: Failures detected in neutron-lib CI. Example: https://c0a6da4decb9452ac635-13a95957352e8408b84a588b720357c0.ssl.cf2.rackcdn.com/828738/6/check/neutron- tempest-plugin-api/90fd383/testr_results.html Error: https://paste.opendev.org/show/bH5jPQGIgO5TF70wDXMv/ To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1962167/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1930402] Re: SSH timeouts happens very often in the ovn based CI jobs
Closing it as neutron-ovn-tempest-slow is moved to periodic in master branch and is no longer seeing ssh failures, if it happen again this can be reopened or new issue can be created. ** Changed in: neutron Status: Confirmed => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1930402 Title: SSH timeouts happens very often in the ovn based CI jobs Status in neutron: Fix Released Bug description: I saw those errors mostly in the neutron-ovn-tempest-slow but probably it happens also in other jobs. Example of failures: https://storage.gra.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_0de/791365/7/check/neutron-ovn-tempest-slow/0de1c30/testr_results.html https://2f7fb53980d59c550f7f-e09de525732b656a1c483807eeb06fc8.ssl.cf2.rackcdn.com/793369/3/check/neutron-ovn-tempest-slow/8a116c7/testr_results.html https://f86d217b949ada82d82c-f669355b5e0e599ce4f84e6e473a124c.ssl.cf2.rackcdn.com/791365/6/check/neutron-ovn-tempest-slow/5dc8d92/testr_results.html In all those cases, common thing is that VMs seems to get IP address from dhcp properly, cloud-init seems to be working fine but SSH to the FIP is not possible. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1930402/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1851500] Re: "test_show_port_chain" conflicts with Flow Classifier
** Changed in: neutron
Status: Confirmed => Fix Released
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to neutron.
https://bugs.launchpad.net/bugs/1851500
Title:
"test_show_port_chain" conflicts with Flow Classifier
Status in networking-sfc:
New
Status in neutron:
Fix Released
Bug description:
Test case:
neutron_tempest_plugin.sfc.tests.api.test_sfc_extensions.SfcExtensionTestJSON.test_show_port_chain
Error:
```
Details: {'type': 'PortChainFlowClassifierInConflict', 'message': 'Flow
Classifier a3d6ec45-5ea8-4e3a-a602-d24baa3ca360 conflicts with Flow Classifier
a978b8ee-651b-4a7f-9475-58b224486827 in port chain
b9fb54ee-1868-46ba-9abd-b09e69360a2d.', 'detail': ''}
```
Log:
https://storage.bhs1.cloud.ovh.net/v1/AUTH_dcaab5e32b234d56b626f72581e3644c/zuul_opendev_logs_6e4/692822/4/check/neutron-tempest-plugin-sfc/6e436c3/testr_results.html.gz
To manage notifications about this bug go to:
https://bugs.launchpad.net/networking-sfc/+bug/1851500/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1961620] Re: cloud-init can add users in wrong filesystem (race with `mount /home`)
Thanks for looking into this. I thought about pivoting into /target too, that should save us from worrying about mounts at all, but requires changes in how subiquity operates, and in general moves a bit away from the idea that installs done from ISO should be treatable like cloud instances, which is what allows us to use cloud-init on bare metal after all. It a good guiding principle, as it helps convergence between bare metal server systems and cloud instances. On "not making cloud-init wait forever": I see your point, however in the case of subiquity we're speaking of a freshly installed system, which can't be in production. Between blocking boot and booting but misconfiguring the system, I'm not sure the latter is better. ** Also affects: subiquity Importance: Undecided Status: New -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to cloud-init. https://bugs.launchpad.net/bugs/1961620 Title: cloud-init can add users in wrong filesystem (race with `mount /home`) Status in cloud-init: Triaged Status in subiquity: New Bug description: When cloud-init is used to configure a new Ubuntu Server system installed from the ISO images, and /home is configured as a separate partition, there is a (slow) race between the user creation and /home being mounted. This can lead to the user $HOME being created in the wrong filesystem. Steps to reproduce: 1. Prepare to install focal-live-server-amd64.iso in a VM. In my case I used one of the 20.04.4 dailies. 2. Proceed with all-defaults but for storage. Configure the storage so / is in a dedicated partition, while /home in a an *encrypted* LVM volume. (The only purpose of encryption is to add delay in the /home mount, see the next point.) 3. Finish the install and reboot. At the dm-crypt password prompt stop and wait a few minutes. At some point cloud-init will proceed creating the configured username, but /home is not mounted yet! The user's $HOME is now in the same filesystem as /. 4. Enter the dm-crypt password. This will cause /home to be mounted from the encrypted volume, and this will shadow the actual $HOME. 5. Login with the configured credentials and verify that $HOME is inaccessible. To manage notifications about this bug go to: https://bugs.launchpad.net/cloud-init/+bug/1961620/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1962153] [NEW] Transition to SQLAlchemy 2.0
Public bug reported: The ``Session.autocommit`` parameter is deprecated and will be removed in SQLAlchemy version 2.0. The Session now features ``autobegin`` behavior such that the Session.begin() method may be called if a transaction has not yet been started yet. See the section session_explicit_begin for background. In Neutron/neutron-lib, the property ``Session.is_active`` [1] is used to determine if a session has an active transaction. With "autocommit=False", the transaction is never closed once created. That means the property will return "True" even when the transaction has flushed all pending commands. In order to mimic the previous behaviour, we need to find an alternative method to decide if the session has any pending command to be flushed. Related topic: https://review.opendev.org/q/topic:sqlalchemy-20 [1]https://github.com/sqlalchemy/sqlalchemy/blob/878c37614efd311794aa50467dbb9e3fe972fdff/lib/sqlalchemy/orm/session.py#L3809-L3837 ** Affects: neutron Importance: Undecided Assignee: Rodolfo Alonso (rodolfo-alonso-hernandez) Status: In Progress ** Description changed: The ``Session.autocommit`` parameter is deprecated and will be removed in SQLAlchemy version 2.0. The Session now features ``autobegin`` behavior such that the Session.begin() method may be called if a transaction has not yet been started yet. See the section session_explicit_begin for background. In Neutron/neutron-lib, the property ``Session.is_active`` [1] is used to determine if a session has an active transaction. With "autocommit=False", the transaction is never closed once created. That means the property will return "True" even when the transaction has flushed all pending commands. In order to mimic the previous behaviour, we need to find an alternative method to decide if the session has any pending command to be flushed. + Related topic: https://review.opendev.org/q/topic:sqlalchemy-20 [1]https://github.com/sqlalchemy/sqlalchemy/blob/878c37614efd311794aa50467dbb9e3fe972fdff/lib/sqlalchemy/orm/session.py#L3809-L3837 ** Changed in: neutron Assignee: (unassigned) => Rodolfo Alonso (rodolfo-alonso-hernandez) -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1962153 Title: Transition to SQLAlchemy 2.0 Status in neutron: In Progress Bug description: The ``Session.autocommit`` parameter is deprecated and will be removed in SQLAlchemy version 2.0. The Session now features ``autobegin`` behavior such that the Session.begin() method may be called if a transaction has not yet been started yet. See the section session_explicit_begin for background. In Neutron/neutron-lib, the property ``Session.is_active`` [1] is used to determine if a session has an active transaction. With "autocommit=False", the transaction is never closed once created. That means the property will return "True" even when the transaction has flushed all pending commands. In order to mimic the previous behaviour, we need to find an alternative method to decide if the session has any pending command to be flushed. Related topic: https://review.opendev.org/q/topic:sqlalchemy-20 [1]https://github.com/sqlalchemy/sqlalchemy/blob/878c37614efd311794aa50467dbb9e3fe972fdff/lib/sqlalchemy/orm/session.py#L3809-L3837 To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1962153/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
[Yahoo-eng-team] [Bug 1962150] [NEW] status symlink non-atomicity traceback with status --wait
Public bug reported:
MAAS run system-tests on a regular basis using LXD containers to set
things up.
A recent run failed waiting for the LXD container to finish booting,
with a traceback from cloud-init.
After launching the container, the script runs `timeout 2000 cloud-init
status --wait --long` to ensure that the commands we pass through as
user-data are complete.
Here's a redacted snippet from the logs
2022-02-23 17:21:00 INFO : Waiting for boot to finish...
2022-02-23 17:21:00 INFO timeout 2000 cloud-init status --wait --long
2022-02-23 17:21:04 INFO ..
2022-02-23 17:21:04 INFO Traceback (most recent call last):
2022-02-23 17:21:04 INFO File "/usr/bin/cloud-init", line 11, in
2022-02-23 17:21:04 INFO load_entry_point('cloud-init==21.4',
'console_scripts', 'cloud-init')()
2022-02-23 17:21:04 INFO File
"/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 927, in main
2022-02-23 17:21:04 INFO retval = util.log_time(
2022-02-23 17:21:04 INFO File
"/usr/lib/python3/dist-packages/cloudinit/util.py", line 2472, in log_time
2022-02-23 17:21:04 INFO ret = func(*args, **kwargs)
2022-02-23 17:21:04 INFO File
"/usr/lib/python3/dist-packages/cloudinit/cmd/status.py", line 60, in
handle_status_args
2022-02-23 17:21:04 INFO status, status_detail, time =
_get_status_details(init.paths)
2022-02-23 17:21:04 INFO File
"/usr/lib/python3/dist-packages/cloudinit/cmd/status.py", line 123, in
_get_status_details
2022-02-23 17:21:04 INFO status_v1 =
load_json(load_file(status_file)).get('v1', {})
2022-02-23 17:21:04 INFO File
"/usr/lib/python3/dist-packages/cloudinit/util.py", line 1361, in load_file
2022-02-23 17:21:04 INFO with open(fname, 'rb') as ifh:
2022-02-23 17:21:04 INFO FileNotFoundError: [Errno 2] No such file or
directory: '/run/cloud-init/status.json'
You can see from the ... that there are a few successful attempts to
wait, but then it fails.
** Affects: cloud-init
Importance: Undecided
Status: New
--
You received this bug notification because you are a member of Yahoo!
Engineering Team, which is subscribed to cloud-init.
https://bugs.launchpad.net/bugs/1962150
Title:
status symlink non-atomicity traceback with status --wait
Status in cloud-init:
New
Bug description:
MAAS run system-tests on a regular basis using LXD containers to set
things up.
A recent run failed waiting for the LXD container to finish booting,
with a traceback from cloud-init.
After launching the container, the script runs `timeout 2000 cloud-
init status --wait --long` to ensure that the commands we pass through
as user-data are complete.
Here's a redacted snippet from the logs
2022-02-23 17:21:00 INFO : Waiting for boot to finish...
2022-02-23 17:21:00 INFO timeout 2000 cloud-init status --wait --long
2022-02-23 17:21:04 INFO ..
2022-02-23 17:21:04 INFO Traceback (most recent call last):
2022-02-23 17:21:04 INFO File "/usr/bin/cloud-init", line 11, in
2022-02-23 17:21:04 INFO load_entry_point('cloud-init==21.4',
'console_scripts', 'cloud-init')()
2022-02-23 17:21:04 INFO File
"/usr/lib/python3/dist-packages/cloudinit/cmd/main.py", line 927, in main
2022-02-23 17:21:04 INFO retval = util.log_time(
2022-02-23 17:21:04 INFO File
"/usr/lib/python3/dist-packages/cloudinit/util.py", line 2472, in log_time
2022-02-23 17:21:04 INFO ret = func(*args, **kwargs)
2022-02-23 17:21:04 INFO File
"/usr/lib/python3/dist-packages/cloudinit/cmd/status.py", line 60, in
handle_status_args
2022-02-23 17:21:04 INFO status, status_detail, time =
_get_status_details(init.paths)
2022-02-23 17:21:04 INFO File
"/usr/lib/python3/dist-packages/cloudinit/cmd/status.py", line 123, in
_get_status_details
2022-02-23 17:21:04 INFO status_v1 =
load_json(load_file(status_file)).get('v1', {})
2022-02-23 17:21:04 INFO File
"/usr/lib/python3/dist-packages/cloudinit/util.py", line 1361, in load_file
2022-02-23 17:21:04 INFO with open(fname, 'rb') as ifh:
2022-02-23 17:21:04 INFO FileNotFoundError: [Errno 2] No such file or
directory: '/run/cloud-init/status.json'
You can see from the ... that there are a few successful attempts to
wait, but then it fails.
To manage notifications about this bug go to:
https://bugs.launchpad.net/cloud-init/+bug/1962150/+subscriptions
--
Mailing list: https://launchpad.net/~yahoo-eng-team
Post to : yahoo-eng-team@lists.launchpad.net
Unsubscribe : https://launchpad.net/~yahoo-eng-team
More help : https://help.launchpad.net/ListHelp
