** Description changed: + [Impact] + + When the ovn-controller daemon elects a new leader is expected that + clients reconnect to that new instance, for the case of Xena the + reconnect attempt will also call register_metadata_agent()[0] and this + method enforces that OVS system-id is formatted as UUID which is not + true for Charmed OpenStack deployed with OVN, this produces that the + neutron-ovn-metadata-agent daemon stays running but disconnected and new + launched VMs won't have access to the metadata service. + + [0] + https://github.com/openstack/neutron/blob/stable/xena/neutron/agent/ovn/metadata/agent.py#L157 + + + [Test Plan] + + 1. Deploy an OpenStack cloud using OVN + + ``` + git clone https://git.launchpad.net/stsstack-bundles + cd stsstack-bundles/openstack + ``` + + Focal Xena: + ./generate-bundle.sh --series focal --release xena --ovn --name focal-xena --run + + Impish: + ./generate-bundle.sh --series impish --ovn --name focal-xena --run + + 2. Configure the cloud creating networks, subnets, etc. + + ``` + source ~/novarc + ./configure + ``` + + 3. Launch an instance + + ``` + source ./novarc + ./tools/instance_launch 1 focal + ``` + + 4. Check the net namespace was correctly provisioned + + ``` + juju ssh nova-compute/0 sudo ip netns + ``` + + Example output: + + $ juju ssh nova-compute/0 sudo ip netns | grep ovnmeta + ovnmeta-0211506b-233e-4773-a034-3950dfefe23d (id: 0) + + 5. Delete the instance: `openstack server delete focal-150930` + + 6. Check the netns was removed. + + $ juju ssh nova-compute/0 sudo ip netns | grep ovnmeta + Connection to 10.5.2.148 closed. + + 7. Restart ovn controller leader unit to force a new leader. + + juju ssh $(juju status ovn-central | grep leader | tail -n 1 | awk + '{print $1}' | tr -d '*') sudo reboot + + 8. Wait a few minutes and then launch a new instance + ``` + source ./novarc + ./tools/instance_launch 1 focal + ``` + + 9. Wait a few minutes (~5m) and check cloud-init's output and the + ovnmeta netns + + ``` + openstack console log show <UUID> + juju ssh nova-compute/0 sudo ip netns | grep ovnmeta + ``` + + Expected result: + * The launched instance is able to read its configuration from the metadata service and not timing out. + * The ovnmeta-<NET_UUID> namespace gets created. + + Actual result: + + * The instance launched can't be accessed via ssh, because cloud-init timed out trying to access the metadata service. + * The ovnmeta-<NET_UUID> namespace is missing from the nova-compute unit. + + + [Where problems could occur] + + * This patch changes the way the UUID used to identify the neutron-ovn- + metadata-agent service is generated, hence issues would manifest as the + daemon not starting (check `systemctl status neutron-ovn-metadata- + agent`) or starting but not being able to connect and provision the + datapath needed when launching new instances in the faulty compute unit + and those instances would have cloud-init timing out. + + [Other Info] + + + [Original Description] + When creating VMs, they will randomly not get access to metadata service. Openstack focal/Xena, with stock OVN 21.09.0-0ubuntu1~cloud0. For testing, I created 32 instances (at once), and 19 have access to metadata service and the other 13 do not. The proportion will vary depending on the iteration and tend to be about 50%. Because of that, I cannot enter those machines via SSH (I can see in the console logs they are not able to get anything from the agent). If I create all of them using "ConfigDrive" option then all of them get SSH keys. When entering them and trying to 'curl' the metadata ip address, I get the correct response on some and timeout on others. I don't see any correlation between the failures and specific compute hosts. I don't see any suspecting messages in {nova,ovn,neutron,openvswitch} logs for the hypervisor that have a problematic vm or for the dedicated gateway. Note: this cloud has 2 extra nodes running ovn-dedicated-chassis and those two are the only nodes that have a way out to provider-networks. Network tests, except for the metadata problem, seem to be ok, including routers and security groups. This has been very consistent between batches of vm deploys and even across redeploys of the cloud.
-- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1951841 Title: [SRU] ovn metadata agent randomly timing out To manage notifications about this bug go to: https://bugs.launchpad.net/charm-ovn-chassis/+bug/1951841/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
