Reviewed: https://review.opendev.org/c/openstack/nova/+/954613 Committed: https://opendev.org/openstack/nova/commit/f37cdf0c4182103ad81dbf39188ff39955da3850 Submitter: "Zuul (22348)" Branch: master
commit f37cdf0c4182103ad81dbf39188ff39955da3850 Author: Balazs Gibizer <[email protected]> Date: Tue Jul 8 16:55:55 2025 +0200 [PCI tracker]Remove non configured devs when freed The PCI tracker handles the case when a device spec is removed from the configuration while a device is still being allocated. It keeps the device until the VM is deleted to avoid inconsistencies. However the full removal of such a device needs not just the VM deletion, but also a nova-compute restart. The device tracker just frees the device during VM deletion but does not removed them until the next nova-compute startup. This allows the device to be re-allocated by another VM even though the device is not allowed by a device_spec. This change adds yet another in memory dict to the pci tracker to track these devices that are only kept until they are freed. Then during free() this list is consulted and if the device is in the list then the device is marked for removal as well. This kills two birds with one stone: * We prevent the re-allocation of the device as the state of the device will be set to REMOVED not AVAILABLE during VM deletion. * As PCI in Placement relies on the state of the device to decide what to track in placement, this change makes sure that a device that needs to be removed, is now removed from placement too. Note that we have another bug that prevents this removal for now. But at least the reproducers of that bug now starts to behave the same regardless of how many device belongs to the same RP in placement. Related-Bug: #2115905 Change-Id: I63c8fb2669a3c6b3adb77d210c0f9b39d3657c80 Signed-off-by: Balazs Gibizer <[email protected]> ** Changed in: nova Status: In Progress => Fix Released -- You received this bug notification because you are a member of Yahoo! Engineering Team, which is subscribed to OpenStack Compute (nova). https://bugs.launchpad.net/bugs/2115905 Title: Cannot delete a VM that uses a PCI device where the matching device_spec is removed if PCI in Placement is enabled Status in OpenStack Compute (nova): Fix Released Bug description: If a device_spec is removed while the device matching it is in use nova-compute will raise a warning at startup. Our doc says: https://docs.openstack.org/nova/latest/admin/pci-passthrough.html#pci- tracking-in-placement Reconfiguring the PCI devices on the hypervisor or changing the pci.device_spec configuration option and restarting the nova-compute service is supported in the following cases: * new devices are added * devices without allocation are removed Removing a device that has allocations is not supported. If a device having any allocation is removed then the nova-compute service will keep the device and the allocation exists in the nova DB and in placement and logs a warning. If a device with any allocation is reconfigured in a way that an allocated PF is removed and VFs from the same PF is configured (or vice versa) then nova-compute will refuse to start as it would create a situation where both the PF and its VFs are made available for consumption. The actual warning says: Unable to remove device with status 'allocated' and ownership 818f2460-61ff-449e-b3c4-9e3626e01645 because of PCI device 1:0000:07:10.2 is allocated instead of ['available', 'unavailable', 'unclaimable']. Check your [pci]device_spec configuration to make sure this allocated device is whitelisted. If you have removed the device from the whitelist intentionally or the device is no longer available on the host you will need to delete the server or migrate it to another host to silence this warning.: nova.exception.PciDeviceInvalidStatus: PCI device 1:0000:07:10.2 is allocated instead of ['available', 'unavailable', 'unclaimable'] But the suggestion of delete the server (and probably the other to migrate it) is wrong. Trying to delete the server in this state causing the VM to go to ERROR state and cannot be deleted. The only way to delete this VM then is to manually delete the placement allocation of the VM first, then delete the VM. This is pretty dangerous, so it should not be suggested. The clean way to avoid this is not to remove the dev_spec whil the device is in use. Or if it is removed then put it back, delete the VM and then remove the dev_spec. However if this problem is triggered not by a manual reconfiguration of the dev_spec but a device disappearing from the hypervisor, putting it back to delete the VM is not an option. See the full reproduction steps and stack traces are in: https://paste.opendev.org/show/bn0uXg4JqOLPUKg4AKd5/ To manage notifications about this bug go to: https://bugs.launchpad.net/nova/+bug/2115905/+subscriptions -- Mailing list: https://launchpad.net/~yahoo-eng-team Post to : [email protected] Unsubscribe : https://launchpad.net/~yahoo-eng-team More help : https://help.launchpad.net/ListHelp
