On Mon, Jan 12, 2026 at 14:28:02 -0000, [email protected] wrote: > Hi all, > > I'm part of Cyberus Technology and currently working on the Cloud > Hypervisor driver backend of libvirt. We are currently implementing a > feature to fix device bus addresses in the domain definitions for PCI > devices, guaranteeing that devices remain available at the same > address even after live migration and restarts, and across libvirt > versions. In doing so, some questions came up about the intended > semantics of the `--persistent` and `--live` flags in virsh commands > and the synchronization between runtime and persistent configuration. > So we wanted to ask you these questions.
Please note that there's quite a difference between migration and all the other APIs, mostly stemming from historical differences when the code was added. > > Our first question revolves around `virsh mirgate --persistent`. The > documentation says `leaves the domain persistent on the destination > host physical machine`, but what domain definition is used from the > source side to migrate the VM? Is it that the live definition is sent, > then migrated, and persisted on the receiver side? Or is the source's > persistent XML sent to the receiver, and the receiver’s live > definition created from the persistent definition received? This one is fun. I had my fair share of messing up things here. The extra fun in this one is that the definition, when transferred to the destination can (or sometimes needs to) be modified to work on the destination host (e.g. adapt paths to disk images if the shared filesystem is mounted in a different destination). There are also multiple mechanisms to achieve the modification. Either the user can provide a new XML or there's the possibility to invoke a hook script which can do the modification. The behaviour depends also e.g. on whether the source VM even has a persistent definition (you can start a VM without a persistent definition using virDomainCreateXML()). If there's no defnition, the running one is copied. If there is one that one is used. Then the user can also specify 'persist_xml' which is a modified persistent XML and that one is used if provided. You can read about my mis-fortune with trying to make sense of the above mess in 6f237f4642764b8389ca36ce5771cee4e6cd2666. I also added notes to documentation that users need to pass the persist XML if they want correct behaviour (055379df2352556a0ab76dd73131acd23ed37b64). > The second question is about synchronization between live and > persistent configuration. To achieve the goal of keeping the same > device at the same PCI bus address across restarts and live > migrations, to our understanding, we have to ensure that one device is > written to both domain definitions with the same PCI bus address. Is > this right, or do we miss something? First thing to remember is that 'def' and 'newDef' can be same (similar as running def has some additional runtime data) but also *completely different*. This is possible since you can either use the APIs to modify only one of the definitions or even use the define API to define something completely new. This means that the logic in the compound APIs (the ones that can modify both live and persistent) has to be able to deal with the difference. By dealing with it I don't necessarily want to suggest that it needs to succeed. Rather that it needs to not e.g. crash in such case, but if the definition simply doesn't support what the user wants there's nothign to do other than fail. > The next questions are about attaching/detaching devices (live). With > respect to our intention to implement fixed PCI bus addresses for > devices, what would be the right semantics for the following cases? > Are they even valid? The source code of the qemu driver leaves those (sorry I had to break your formatting, your mail client decided to put everything in a massive paragraph with no line breaks so I had to break it up) > questions open: > 1) We attach a device `A` with `--persistent`and > remove it with `--live`. If we now attach `B`, either via > `--persistent` and `--live`, we expect that the PCI bus address > formerly assigned to `A` is not reused for `B`. Is this correct? In the qemu driver we parse the definition twice if both states need to be modified. Depending on what the user requested (e.g. if they filled in the address or not) a PCI address might be generated. Each address is assigned in context of the VM definition it will be applied to. This means that the addresses might differ, if e.g. the live definition has one taken. You obviously *can* apply some extra logic to try to make them same, but in the end there might be a situation when that won't be possible. > 2)> If we attach a device `A` with `--live` and a second device `B` with > `--persistent`, then `A` will not be present in the persistent domain > XML. Still, we expect that `B` is added to both live and persistent > definitions, using the same PCI bus address. By doing so, we would No if you use only --persistent you *must not* touch the live config. This is just a cold-plug request. > expect the attachment routines for the persistent definition not to > assign the address handed out by the attachment routines of the live > configuration for device `A`. In other words, even if `A` is not > present in the persistent configuration, we would expect the address > of `A` not be reused when attaching `B` to the persistent definition. > Are those semantics correct? No, at least the qemu driver doesn't do that. If the definitions differ you might have no choice. > The last question is about target names for network devices > (`ifname`). libvirt does generate names for such devices in certain > situations. Whenever a device definition is parsed that contains an > `ifname` that matches libvirt's generation rules, e.g. vnet*, it is > blanked out (see `virDomainNetDefParseXML`, in domain_conf.c). When IIUC 'ifname' is a runtime setting so there's no need to deal with it in the next-start definition. Any of the runtime setting are allocated when the object is starting to be used, so at hotplug or startup time thus the cold definitions don't have them. What is important though is e.g. interface 'mac' address since that is guest visible and also an identifier. It's also auto-generated when the user-supplied def doesn't have it. The qemu driver if you attach the device with --live and --persistent as a transaction it ensures that the generated MAC is the same. Obviously if the user does two separate calls with --live and --persistent respectively the interface will get different MAC generated. There's not much you can do about that. > writing code that tries to identify network devices by their > `ifname`s, we found that this would be impossible because devices with > an autogenerated name will have a blank `ifname` in the definition. > What is the expected semantics around fixed PCI bus addresses for such > devices? Would it be right to assign a custom `ifname` to them in this > case? > > Thanks in advance! Pascal >
