** Description changed: + [Impact] + + * We run neutron-ovs-cleanup in startup if neutron installed. If nova-compute + does not wait for completion it will try to use veth/bridge devices that may + may be in the process of bring deleted. + + [Test Case] + + * Create neutron (ovs) network and boot an instance with this network + as --nic + + * Check that creation was successful and network is functional. Also make a + note corresponding veth and bridge devices (ip a). + + * Reboot system, check that expected veth and bridge devices are still + there and that nova-compute is happy e.g. try sshing to your instance. Also + check /var/log/upstart/nova-compute.log to see if service waited for + ovs-cleanup to finish. + + [Regression Potential] + + * None + + + ---- ---- ---- ---- + There is a race when both neutron-ovs-cleanup and nova-compute trying to do operations on the qvb*** and qvo*** devices. Below is a scenario I recently met, 1. nova-compute was started and creating the veth_pair for VM instances running on the host - https://github.com/openstack/nova/blob/stable/icehouse/nova/network/linux_net.py#L1298 2. neutron-ovs-cleanup was kicked off and deleted all the ports. 3. when nova-compute tried to set the MTU at https://github.com/openstack/nova/blob/stable/icehouse/nova/network/linux_net.py#L1280 , Stderr: u'Cannot find device "qvo***"\n' was reported. Because the device that was just created was deleted again by neutron-ovs-cleanup. As they both operate on the same resources, there needs a way to synchronize the operations the two processes do on those resources.
** Description changed: [Impact] - * We run neutron-ovs-cleanup in startup if neutron installed. If nova-compute - does not wait for completion it will try to use veth/bridge devices that may - may be in the process of bring deleted. + * We run neutron-ovs-cleanup in startup if neutron installed. If + nova-compute does not wait for completion it will try to use + veth/bridge devices that may be in the process of bring deleted. [Test Case] - * Create neutron (ovs) network and boot an instance with this network - as --nic + * Create neutron (ovs) network and boot an instance with this network + as --nic - * Check that creation was successful and network is functional. Also make a - note corresponding veth and bridge devices (ip a). + * Check that creation was successful and network is functional. Also make + a note corresponding veth and bridge devices (ip a). - * Reboot system, check that expected veth and bridge devices are still - there and that nova-compute is happy e.g. try sshing to your instance. Also - check /var/log/upstart/nova-compute.log to see if service waited for - ovs-cleanup to finish. + * Reboot system, check that expected veth and bridge devices are still + there and that nova-compute is happy e.g. try sshing to your instance. + Also check /var/log/upstart/nova-compute.log to see if service waited + for ovs-cleanup to finish. [Regression Potential] - * None - + * None ---- ---- ---- ---- There is a race when both neutron-ovs-cleanup and nova-compute trying to do operations on the qvb*** and qvo*** devices. Below is a scenario I recently met, 1. nova-compute was started and creating the veth_pair for VM instances running on the host - https://github.com/openstack/nova/blob/stable/icehouse/nova/network/linux_net.py#L1298 2. neutron-ovs-cleanup was kicked off and deleted all the ports. 3. when nova-compute tried to set the MTU at https://github.com/openstack/nova/blob/stable/icehouse/nova/network/linux_net.py#L1280 , Stderr: u'Cannot find device "qvo***"\n' was reported. Because the device that was just created was deleted again by neutron-ovs-cleanup. As they both operate on the same resources, there needs a way to synchronize the operations the two processes do on those resources. -- You received this bug notification because you are a member of Ubuntu Server Team, which is subscribed to nova in Ubuntu. https://bugs.launchpad.net/bugs/1420572 Title: race between neutron-ovs-cleanup and nova-compute To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/nova/+bug/1420572/+subscriptions -- Ubuntu-server-bugs mailing list Ubuntu-server-bugs@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-server-bugs
