Hi, all
I've been having an annoying issue with live VM + volume migration. When
I try to use |virDomainMigrate2| to migrate a VM and some of its volumes
(but not all) I get the following error:
|unable to execute QEMU command 'block-export-add': Block node is read-only |
Some additional context:
* I originally found this issue using the java bindings, but I've been
able to reproduce it using virsh;
* I'm using Libvirt 8.0.0 and Qemu 6.2.0;
* This only happens when the VM has volumes on NFS storage;
* The issue does not happen when migrating all the volumes that are on
NFS storage. Note that, if the VM has volumes that are on some other
type of storage, such as ISCSI, those do not need to be migrated.
However, migrating any volume will raise the error if the volumes on
NFS are not migrated as well;
* I've tried a number of valid combinations along the
|VIR_MIGRATE_LIVE| flag, using the flags: |VIR_MIGRATE_PAUSED|,
|VIR_MIGRATE_NON_SHARED_DISK|, |VIR_MIGRATE_NON_SHARED_INC|,
|VIR_MIGRATE_UNSAFE|, |VIR_MIGRATE_COMPRESSED|.
The easiest way I've found to reproduce this error was using the
following steps:
1. Create a VM using virt-manager, it must have at least two volumes,
at least one of them must be on NFS storage. Here's the XML of one
of the VMs I've tested with:
|<domain type='kvm' id='4'> <name>ubuntu22.04</name>
<uuid>9965e409-c89d-408c-a9f7-488184566d56</uuid> <metadata>
<libosinfo:libosinfo
xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0";>
<libosinfo:os id="http://ubuntu.com/ubuntu/22.04"/>
</libosinfo:libosinfo> </metadata> <memory unit='KiB'>2097152</memory>
<currentMemory unit='KiB'>2097152</currentMemory> <vcpu
placement='static'>1</vcpu> <resource> <partition>/machine</partition>
</resource> <os> <type arch='x86_64' machine='pc-q35-6.2'>hvm</type>
<boot dev='hd'/> </os> <features> <acpi/> <apic/> <vmport state='off'/>
</features> <cpu mode='host-passthrough' check='none' migratable='on'/>
<clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer
name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/>
</clock> <on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <pm>
<suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm>
<devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk
type='file' device='disk'> <driver name='qemu' type='qcow2'/> <source
file='/mnt/nfs-storage/template.qcow2' index='1'/> <backingStore/>
<target dev='vda' bus='virtio'/> <alias name='virtio-disk0'/> <address
type='pci' domain='0x0000' bus='0x04' slot='0x00' function='0x0'/>
</disk> <disk type='file' device='disk'> <driver name='qemu'
type='qcow2'/> <source file='/mnt/nfs-storage/vol.qcow2' index='2'/>
<backingStore/> <target dev='vdb' bus='virtio'/> <alias
name='virtio-disk1'/> <address type='pci' domain='0x0000' bus='0x08'
slot='0x00' function='0x0'/> </disk> <controller type='usb' index='0'
model='qemu-xhci' ports='15'> <alias name='usb'/> <address type='pci'
domain='0x0000' bus='0x02' slot='0x00' function='0x0'/> </controller>
<controller type='pci' index='0' model='pcie-root'> <alias
name='pcie.0'/> </controller> <controller type='pci' index='1'
model='pcie-root-port'> <model name='pcie-root-port'/> <target
chassis='1' port='0x10'/> <alias name='pci.1'/> <address type='pci'
domain='0x0000' bus='0x00' slot='0x02' function='0x0'
multifunction='on'/> </controller> <controller type='pci' index='2'
model='pcie-root-port'> <model name='pcie-root-port'/> <target
chassis='2' port='0x11'/> <alias name='pci.2'/> <address type='pci'
domain='0x0000' bus='0x00' slot='0x02' function='0x1'/> </controller>
<controller type='pci' index='3' model='pcie-root-port'> <model
name='pcie-root-port'/> <target chassis='3' port='0x12'/> <alias
name='pci.3'/> <address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x2'/> </controller> <controller type='pci'
index='4' model='pcie-root-port'> <model name='pcie-root-port'/> <target
chassis='4' port='0x13'/> <alias name='pci.4'/> <address type='pci'
domain='0x0000' bus='0x00' slot='0x02' function='0x3'/> </controller>
<controller type='pci' index='5' model='pcie-root-port'> <model
name='pcie-root-port'/> <target chassis='5' port='0x14'/> <alias
name='pci.5'/> <address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x4'/> </controller> <controller type='pci'
index='6' model='pcie-root-port'> <model name='pcie-root-port'/> <target
chassis='6' port='0x15'/> <alias name='pci.6'/> <address type='pci'
domain='0x0000' bus='0x00' slot='0x02' function='0x5'/> </controller>
<controller type='pci' index='7' model='pcie-root-port'> <model
name='pcie-root-port'/> <target chassis='7' port='0x16'/> <alias
name='pci.7'/> <address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x6'/> </controller> <controller type='pci'
index='8' model='pcie-root-port'> <model name='pcie-root-port'/> <target
chassis='8' port='0x17'/> <alias name='pci.8'/> <address type='pci'
domain='0x0000' bus='0x00' slot='0x02' function='0x7'/> </controller>
<controller type='pci' index='9' model='pcie-root-port'> <model
name='pcie-root-port'/> <target chassis='9' port='0x18'/> <alias
name='pci.9'/> <address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x0' multifunction='on'/> </controller>
<controller type='pci' index='10' model='pcie-root-port'> <model
name='pcie-root-port'/> <target chassis='10' port='0x19'/> <alias
name='pci.10'/> <address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x1'/> </controller> <controller type='pci'
index='11' model='pcie-root-port'> <model name='pcie-root-port'/>
<target chassis='11' port='0x1a'/> <alias name='pci.11'/> <address
type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x2'/>
</controller> <controller type='pci' index='12' model='pcie-root-port'>
<model name='pcie-root-port'/> <target chassis='12' port='0x1b'/> <alias
name='pci.12'/> <address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x3'/> </controller> <controller type='pci'
index='13' model='pcie-root-port'> <model name='pcie-root-port'/>
<target chassis='13' port='0x1c'/> <alias name='pci.13'/> <address
type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x4'/>
</controller> <controller type='pci' index='14' model='pcie-root-port'>
<model name='pcie-root-port'/> <target chassis='14' port='0x1d'/> <alias
name='pci.14'/> <address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x5'/> </controller> <controller type='sata'
index='0'> <alias name='ide'/> <address type='pci' domain='0x0000'
bus='0x00' slot='0x1f' function='0x2'/> </controller> <controller
type='virtio-serial' index='0'> <alias name='virtio-serial0'/> <address
type='pci' domain='0x0000' bus='0x03' slot='0x00' function='0x0'/>
</controller> <controller type='scsi' index='0' model='virtio-scsi'>
<alias name='scsi0'/> <address type='pci' domain='0x0000' bus='0x07'
slot='0x00' function='0x0'/> </controller> <interface type='bridge'>
<mac address='52:54:00:c4:a7:6d'/> <source bridge='br-test'/> <target
dev='vnet6'/> <model type='virtio'/> <alias name='net0'/> <address
type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</interface> <serial type='pty'> <source path='/dev/pts/3'/> <target
type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias
name='serial0'/> </serial> <console type='pty' tty='/dev/pts/3'> <source
path='/dev/pts/3'/> <target type='serial' port='0'/> <alias
name='serial0'/> </console> <channel type='unix'> <source mode='bind'
path='/var/lib/libvirt/qemu/channel/target/domain-4-ubuntu22.04/org.qemu.guest_agent.0'/>
<target type='virtio' name='org.qemu.guest_agent.0'
state='disconnected'/> <alias name='channel0'/> <address
type='virtio-serial' controller='0' bus='0' port='1'/> </channel>
<channel type='spicevmc'> <target type='virtio'
name='com.redhat.spice.0' state='connected'/> <alias name='channel1'/>
<address type='virtio-serial' controller='0' bus='0' port='2'/>
</channel> <input type='tablet' bus='usb'> <alias name='input0'/>
<address type='usb' bus='0' port='1'/> </input> <input type='mouse'
bus='ps2'> <alias name='input1'/> </input> <input type='keyboard'
bus='ps2'> <alias name='input2'/> </input> <graphics type='spice'
port='5901' autoport='yes' listen='127.0.0.1'> <listen type='address'
address='127.0.0.1'/> </graphics> <sound model='ich9'> <alias
name='sound0'/> <address type='pci' domain='0x0000' bus='0x00'
slot='0x1b' function='0x0'/> </sound> <audio id='1' type='spice'/>
<video> <model type='qxl' ram='65536' vram='65536' vgamem='16384'
heads='1' primary='yes'/> <alias name='video0'/> <address type='pci'
domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </video>
<redirdev bus='usb' type='spicevmc'> <alias name='redir0'/> <address
type='usb' bus='0' port='2'/> </redirdev> <redirdev bus='usb'
type='spicevmc'> <alias name='redir1'/> <address type='usb' bus='0'
port='3'/> </redirdev> <memballoon model='virtio'> <alias
name='balloon0'/> <address type='pci' domain='0x0000' bus='0x05'
slot='0x00' function='0x0'/> </memballoon> <rng model='virtio'> <backend
model='random'>/dev/urandom</backend> <alias name='rng0'/> <address
type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
</rng> </devices> <seclabel type='dynamic' model='dac' relabel='yes'>
<label>+0:+0</label> <imagelabel>+0:+0</imagelabel> </seclabel> </domain> |
2. Use |virsh migrate --domain <name> --copy-storage-all --live
--desturi <other_host_uri> --xml ./newxml| to migrate the VM and one
of its volumes, while leaving a volume that is on NFS without
migration. I've tested migrating the volume to another NFS storage,
as well as migrating to a ISCSI storage that is mounted on both
hosts, the result is the same. Here's the content of the |newxml|
for this example:
|<domain type='kvm' id='4'> <name>ubuntu22.04</name>
<uuid>9965e409-c89d-408c-a9f7-488184566d56</uuid> <metadata>
<libosinfo:libosinfo
xmlns:libosinfo="http://libosinfo.org/xmlns/libvirt/domain/1.0";>
<libosinfo:os id="http://ubuntu.com/ubuntu/22.04"/>
</libosinfo:libosinfo> </metadata> <memory unit='KiB'>2097152</memory>
<currentMemory unit='KiB'>2097152</currentMemory> <vcpu
placement='static'>1</vcpu> <resource> <partition>/machine</partition>
</resource> <os> <type arch='x86_64' machine='pc-q35-6.2'>hvm</type>
<boot dev='hd'/> </os> <features> <acpi/> <apic/> <vmport state='off'/>
</features> <cpu mode='host-passthrough' check='none' migratable='on'/>
<clock offset='utc'> <timer name='rtc' tickpolicy='catchup'/> <timer
name='pit' tickpolicy='delay'/> <timer name='hpet' present='no'/>
</clock> <on_poweroff>destroy</on_poweroff>
<on_reboot>restart</on_reboot> <on_crash>destroy</on_crash> <pm>
<suspend-to-mem enabled='no'/> <suspend-to-disk enabled='no'/> </pm>
<devices> <emulator>/usr/bin/qemu-system-x86_64</emulator> <disk
type='file' device='disk'> <driver name='qemu' type='qcow2'
cache='none'/> <source file='/mnt/nfs-storage/template1.qcow2'
index='1'/> <backingStore/> <target dev='vda' bus='virtio'/> <alias
name='virtio-disk0'/> <address type='pci' domain='0x0000' bus='0x04'
slot='0x00' function='0x0'/> </disk> <disk type='file' device='disk'>
<driver name='qemu' type='qcow2' cache='none'/> <source
file='/mnt/other-storage/vol.qcow2' index='2'/> <backingStore/> <target
dev='vdb' bus='virtio'/> <alias name='virtio-disk1'/> <address
type='pci' domain='0x0000' bus='0x08' slot='0x00' function='0x0'/>
</disk> <controller type='usb' index='0' model='qemu-xhci' ports='15'>
<alias name='usb'/> <address type='pci' domain='0x0000' bus='0x02'
slot='0x00' function='0x0'/> </controller> <controller type='pci'
index='0' model='pcie-root'> <alias name='pcie.0'/> </controller>
<controller type='pci' index='1' model='pcie-root-port'> <model
name='pcie-root-port'/> <target chassis='1' port='0x10'/> <alias
name='pci.1'/> <address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x0' multifunction='on'/> </controller>
<controller type='pci' index='2' model='pcie-root-port'> <model
name='pcie-root-port'/> <target chassis='2' port='0x11'/> <alias
name='pci.2'/> <address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x1'/> </controller> <controller type='pci'
index='3' model='pcie-root-port'> <model name='pcie-root-port'/> <target
chassis='3' port='0x12'/> <alias name='pci.3'/> <address type='pci'
domain='0x0000' bus='0x00' slot='0x02' function='0x2'/> </controller>
<controller type='pci' index='4' model='pcie-root-port'> <model
name='pcie-root-port'/> <target chassis='4' port='0x13'/> <alias
name='pci.4'/> <address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x3'/> </controller> <controller type='pci'
index='5' model='pcie-root-port'> <model name='pcie-root-port'/> <target
chassis='5' port='0x14'/> <alias name='pci.5'/> <address type='pci'
domain='0x0000' bus='0x00' slot='0x02' function='0x4'/> </controller>
<controller type='pci' index='6' model='pcie-root-port'> <model
name='pcie-root-port'/> <target chassis='6' port='0x15'/> <alias
name='pci.6'/> <address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x5'/> </controller> <controller type='pci'
index='7' model='pcie-root-port'> <model name='pcie-root-port'/> <target
chassis='7' port='0x16'/> <alias name='pci.7'/> <address type='pci'
domain='0x0000' bus='0x00' slot='0x02' function='0x6'/> </controller>
<controller type='pci' index='8' model='pcie-root-port'> <model
name='pcie-root-port'/> <target chassis='8' port='0x17'/> <alias
name='pci.8'/> <address type='pci' domain='0x0000' bus='0x00'
slot='0x02' function='0x7'/> </controller> <controller type='pci'
index='9' model='pcie-root-port'> <model name='pcie-root-port'/> <target
chassis='9' port='0x18'/> <alias name='pci.9'/> <address type='pci'
domain='0x0000' bus='0x00' slot='0x03' function='0x0'
multifunction='on'/> </controller> <controller type='pci' index='10'
model='pcie-root-port'> <model name='pcie-root-port'/> <target
chassis='10' port='0x19'/> <alias name='pci.10'/> <address type='pci'
domain='0x0000' bus='0x00' slot='0x03' function='0x1'/> </controller>
<controller type='pci' index='11' model='pcie-root-port'> <model
name='pcie-root-port'/> <target chassis='11' port='0x1a'/> <alias
name='pci.11'/> <address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x2'/> </controller> <controller type='pci'
index='12' model='pcie-root-port'> <model name='pcie-root-port'/>
<target chassis='12' port='0x1b'/> <alias name='pci.12'/> <address
type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x3'/>
</controller> <controller type='pci' index='13' model='pcie-root-port'>
<model name='pcie-root-port'/> <target chassis='13' port='0x1c'/> <alias
name='pci.13'/> <address type='pci' domain='0x0000' bus='0x00'
slot='0x03' function='0x4'/> </controller> <controller type='pci'
index='14' model='pcie-root-port'> <model name='pcie-root-port'/>
<target chassis='14' port='0x1d'/> <alias name='pci.14'/> <address
type='pci' domain='0x0000' bus='0x00' slot='0x03' function='0x5'/>
</controller> <controller type='sata' index='0'> <alias name='ide'/>
<address type='pci' domain='0x0000' bus='0x00' slot='0x1f'
function='0x2'/> </controller> <controller type='virtio-serial'
index='0'> <alias name='virtio-serial0'/> <address type='pci'
domain='0x0000' bus='0x03' slot='0x00' function='0x0'/> </controller>
<controller type='scsi' index='0' model='virtio-scsi'> <alias
name='scsi0'/> <address type='pci' domain='0x0000' bus='0x07'
slot='0x00' function='0x0'/> </controller> <interface type='bridge'>
<mac address='52:54:00:c4:a7:6d'/> <source bridge='br-test'/> <target
dev='vnet6'/> <model type='virtio'/> <alias name='net0'/> <address
type='pci' domain='0x0000' bus='0x01' slot='0x00' function='0x0'/>
</interface> <serial type='pty'> <source path='/dev/pts/3'/> <target
type='isa-serial' port='0'> <model name='isa-serial'/> </target> <alias
name='serial0'/> </serial> <console type='pty' tty='/dev/pts/3'> <source
path='/dev/pts/3'/> <target type='serial' port='0'/> <alias
name='serial0'/> </console> <channel type='unix'> <source mode='bind'
path='/var/lib/libvirt/qemu/channel/target/domain-4-ubuntu22.04/org.qemu.guest_agent.0'/>
<target type='virtio' name='org.qemu.guest_agent.0'
state='disconnected'/> <alias name='channel0'/> <address
type='virtio-serial' controller='0' bus='0' port='1'/> </channel>
<channel type='spicevmc'> <target type='virtio'
name='com.redhat.spice.0' state='connected'/> <alias name='channel1'/>
<address type='virtio-serial' controller='0' bus='0' port='2'/>
</channel> <input type='tablet' bus='usb'> <alias name='input0'/>
<address type='usb' bus='0' port='1'/> </input> <input type='mouse'
bus='ps2'> <alias name='input1'/> </input> <input type='keyboard'
bus='ps2'> <alias name='input2'/> </input> <graphics type='spice'
port='5901' autoport='yes' listen='127.0.0.1'> <listen type='address'
address='127.0.0.1'/> </graphics> <sound model='ich9'> <alias
name='sound0'/> <address type='pci' domain='0x0000' bus='0x00'
slot='0x1b' function='0x0'/> </sound> <audio id='1' type='spice'/>
<video> <model type='qxl' ram='65536' vram='65536' vgamem='16384'
heads='1' primary='yes'/> <alias name='video0'/> <address type='pci'
domain='0x0000' bus='0x00' slot='0x01' function='0x0'/> </video>
<redirdev bus='usb' type='spicevmc'> <alias name='redir0'/> <address
type='usb' bus='0' port='2'/> </redirdev> <redirdev bus='usb'
type='spicevmc'> <alias name='redir1'/> <address type='usb' bus='0'
port='3'/> </redirdev> <memballoon model='virtio'> <alias
name='balloon0'/> <address type='pci' domain='0x0000' bus='0x05'
slot='0x00' function='0x0'/> </memballoon> <rng model='virtio'> <backend
model='random'>/dev/urandom</backend> <alias name='rng0'/> <address
type='pci' domain='0x0000' bus='0x06' slot='0x00' function='0x0'/>
</rng> </devices> <seclabel type='dynamic' model='dac' relabel='yes'>
<label>+0:+0</label> <imagelabel>+0:+0</imagelabel> </seclabel> </domain> |
Right after the command is sent, the |unable to execute QEMU command
'block-export-add': Block node is read-only| happens. If I do the same
steps, but migrate all volumes that are on NFS (even if they stay on the
same storage, only changing names), the error does not happen.
Here are some other examples just to clarify when the error happens,
examples with green headers do not present the error, while examples
with red headers do:
Volume Origin Destination
ROOT iSCSI1 iSCSI2
DATA1 iSCSI1 ---
DATA2 iSCSI1 ---
Volume Origin Destination
ROOT iSCSI1 ---
DATA1 iSCSI1 iSCSI2
DATA2 iSCSI2 iSCSI1
Volume Origin Destination
ROOT NFS1 NFS2
DATA1 NFS2 NFS1
DATA2 iSCSI1 ---
Volume Origin Destination
ROOT iSCSI1 NFS2
DATA1 NFS2 ---
DATA2 NFS2 ---
Volume Origin Destination
ROOT iSCSI1 iSCSI1
DATA1 NFS1 ---
DATA2 NFS1 ---
Volume Origin Destination
ROOT iSCSI1 ---
DATA1 NFS2 NFS2
DATA2 NFS2 ---
Has anyone experienced this before? If so, is there another workaround?
As migrating all the volumes that are on NFS storage might be costly in
terms of time and storage/network resources.
Best regards,
João Jandre.
_______________________________________________
Users mailing list -- users@lists.libvirt.org
To unsubscribe send an email to users-le...@lists.libvirt.org
