Reviewed: https://review.opendev.org/c/openstack/neutron/+/843253 Committed: https://opendev.org/openstack/neutron/commit/e09b128f416a809cd7734aba8ab52220ea01b2e2 Submitter: "Zuul (22348)" Branch: master
commit e09b128f416a809cd7734aba8ab52220ea01b2e2 Author: Henning Eggers <henn...@eggers.name> Date: Wed May 25 11:17:43 2022 +0200 Defer flow deletion in openvswitch firewall Reduces the deletion time of conjunction flows on hypervisors where virtual machines reside which are part of a security group that has remote security groups as target which contain thousands of ports. Without deferred deletion the agent will call ovs-ofctl several hundred times in succession, during this time the agent will block any new vm creation or neutron port modifications on this hypervisor. This patch has been tested using a single network with a single vm with a security group that points to a remote security group with 2000 ports. During testing without the patch, the iteration time for deletion was at around 500 seconds. After adding the patch to the l2 agent on the test environment the same deletion time went down to 4 seconds. Closes-Bug: #1975674 Change-Id: I46b1fe94b2e358f7f4b2cd4943a74ebaf84f51b8 ** Changed in: neutron Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to neutron. https://bugs.launchpad.net/bugs/1975674 Title: Neutron agent blocks during VM deletion when a remote security group is involved Status in neutron: Fix Released Bug description: When deleting a VM that has a security group referring to a remote security group, the neutron agent will block for as long as it takes to remove the respective flows. This happens when the remote security group contains many (thousands) ports referring to other VMs. Steps to reproduce: - Create a VM with security group A - Add a rule to security group A allowing access from a remote security group B - Add a large number or ports to security group B (e.g. 2000) - The respective ovs flows will be added - Delete the VM - The ovs flows will be removed Expected: - VM and flow to be deleted within seconds - No impact to other VMs on the same hypervisor Actual: - Flow deletion takes a long time, sometimes up to 10 minutes - While flows are being deleted, no VMs can be created on the same hypervisor The reason for this behavior is that under the hood the agent calls ovs-ofctl (via execve()) once for each port in the remote security group. These calls quickly add up to minutes if there are many ports. The proposed solution would be to use deferred execution for the flow deletion. In that case it becomes a bulk operation and around 400 flows are deleted in one call. In addition it runs in the background and does not block the agent for other operations. To manage notifications about this bug go to: https://bugs.launchpad.net/neutron/+bug/1975674/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : yahoo-eng-team@lists.launchpad.net Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp