GitHub user mwaag closed a discussion: VMWare to KVM Migration fails - "nbdcopy command failed"
Hi, iam testing the vmware-kvm-feature in our testing-environment right now. We are running vmware-only environments with advanced networking features in both, production and test. So i had to build a kvm-zone with advanced networking first. For setting up the migration-scenario i tried to stick to Shapeblues Guide https://www.shapeblue.com/cloudstack-vmware-to-kvm-migration-tool/ as strict as possible with one significant difference. We use Debian 12 as KVM-OS. Knowing, this isn't officially supported i got the feedback at a user group meeting, that it should technically work as it is very similar to ubuntu. We now have a Cloudstack 4.19.2.0-Installation with zwo Zones: 1x VMware (2 Hosts) + 1x KVM (1 Host) Everything generally seems to work and we have all functionalities we are used to. But migration of a vmware-vm to kvm unfortunately does not finish successfully atm. I can monitor the export in vcenter and after it i can follow virt-v2v-conversion via cloudstack-agent-log. But in this process the failure happens. I suspect an virt-v2v-error 'nbdcopy command failed' to be responsible for the failing migration. See logs and Screenshots attached. Any suggestions on this? Versions Cloudstack-Management-Server: 4.19.2.0 Libvirt: 9.0.0 ESX / Vsphere: 7.0.3 Cloudstack-Agent-Log: `2025-05-09 13:20:35,141 INFO [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (Script-4:null) (logid:) (virt-v2v ovf source: /mnt/712cdae3-3474-3a73-985a-5165c940db32/d40281df-aa8e-4169-959d-513f56782381/i-4-57-VM/ progress) virt-v2v: The QEMU Guest Agent will be installed for this guest at first boot. 2025-05-09 13:21:07,563 DEBUG [kvm.resource.LibvirtConnection] (Thread-1:null) (logid:) Looking for libvirtd connection at: qemu:///system 2025-05-09 13:21:07,565 DEBUG [kvm.resource.KVMHAMonitor] (Thread-1:null) (logid:) Found NFS storage pool [e336c1bb-917c-30ca-82df-7b118b5fda18] in libvirt, continuing. 2025-05-09 13:21:07,565 DEBUG [agent.properties.AgentPropertiesFileHandler] (Thread-1:null) (logid:) Property [kvm.scripts.dir] has empty or null value. Using default value [scripts/vm/hypervisor/kvm]. 2025-05-09 13:21:07,565 DEBUG [utils.script.Script] (Thread-1:null) (logid:) Looking for kvmheartbeat.sh in the classpath 2025-05-09 13:21:07,565 DEBUG [utils.script.Script] (Thread-1:null) (logid:) System resource: null 2025-05-09 13:21:07,566 DEBUG [utils.script.Script] (Thread-1:null) (logid:) Classpath resource: null 2025-05-09 13:21:07,566 DEBUG [utils.script.Script] (Thread-1:null) (logid:) Looking for kvmheartbeat.sh 2025-05-09 13:21:07,566 DEBUG [utils.script.Script] (Thread-1:null) (logid:) Current binaries reside at /usr/share/cloudstack-agent/lib 2025-05-09 13:21:07,566 DEBUG [utils.script.Script] (Thread-1:null) (logid:) Looking for kvmheartbeat.sh in /usr/share/cloudstack-agent/lib/scripts/vm/hypervisor/kvm/kvmheartbeat.sh 2025-05-09 13:21:07,566 DEBUG [utils.script.Script] (Thread-1:null) (logid:) Looking for kvmheartbeat.sh in /usr/share/cloudstack-agent/lib/scripts/vm/hypervisor/kvmheartbeat.sh 2025-05-09 13:21:07,566 DEBUG [utils.script.Script] (Thread-1:null) (logid:) Looking for kvmheartbeat.sh in /usr/share/cloudstack-agent/lib/scripts/vm/kvmheartbeat.sh 2025-05-09 13:21:07,566 DEBUG [utils.script.Script] (Thread-1:null) (logid:) Looking for kvmheartbeat.sh in /usr/share/cloudstack-agent/lib/scripts/kvmheartbeat.sh 2025-05-09 13:21:07,566 DEBUG [utils.script.Script] (Thread-1:null) (logid:) Looking for kvmheartbeat.sh in /usr/share/cloudstack-agent/lib/kvmheartbeat.sh 2025-05-09 13:21:07,566 DEBUG [utils.script.Script] (Thread-1:null) (logid:) Looking for kvmheartbeat.sh in /usr/share/cloudstack-agent/kvmheartbeat.sh 2025-05-09 13:21:07,566 DEBUG [utils.script.Script] (Thread-1:null) (logid:) Looking for kvmheartbeat.sh in /usr/share/kvmheartbeat.sh 2025-05-09 13:21:07,566 DEBUG [utils.script.Script] (Thread-1:null) (logid:) Looking for kvmheartbeat.sh in /usr/kvmheartbeat.sh 2025-05-09 13:21:07,566 DEBUG [utils.script.Script] (Thread-1:null) (logid:) Looking for kvmheartbeat.sh in /kvmheartbeat.sh 2025-05-09 13:21:07,566 DEBUG [utils.script.Script] (Thread-1:null) (logid:) Searching in environment.properties 2025-05-09 13:21:07,566 DEBUG [utils.script.Script] (Thread-1:null) (logid:) environment.properties says scripts should be in /usr/share/cloudstack-common 2025-05-09 13:21:07,567 DEBUG [utils.script.Script] (Thread-1:null) (logid:) Looking for kvmheartbeat.sh in /usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh 2025-05-09 13:21:07,567 DEBUG [kvm.storage.LibvirtStoragePool] (Thread-1:null) (logid:) Executing command [/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i 192.168.2.4 -p /bi_nfs_testenv_cloudstack0_pri_kvm -m /mnt/e336c1bb-917c-30ca-82df-7b118b5fda18 -h 192.168.4.106 ]. 2025-05-09 13:21:07,584 DEBUG [kvm.storage.LibvirtStoragePool] (Thread-1:null) (logid:) Successfully executed process [167793] for command [/usr/share/cloudstack-common/scripts/vm/hypervisor/kvm/kvmheartbeat.sh -i 192.168.2.4 -p /bi_nfs_testenv_cloudstack0_pri_kvm -m /mnt/e336c1bb-917c-30ca-82df-7b118b5fda18 -h 192.168.4.106 ]. 2025-05-09 13:21:15,682 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) (logid:) Executing command [/usr/share/cloudstack-common/scripts/vm/network/security_group.py get_rule_logs_for_vms ]. 2025-05-09 13:21:15,793 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) (logid:) Successfully executed process [167800] for command [/usr/share/cloudstack-common/scripts/vm/network/security_group.py get_rule_logs_for_vms ]. 2025-05-09 13:21:15,793 DEBUG [agent.properties.AgentPropertiesFileHandler] (UgentTask-5:null) (logid:) Property [hypervisor.uri] has empty or null value. Using default value [null]. 2025-05-09 13:21:15,793 DEBUG [kvm.resource.LibvirtConnection] (UgentTask-5:null) (logid:) Looking for libvirtd connection at: qemu:///system 2025-05-09 13:21:15,798 DEBUG [kvm.resource.LibvirtComputingResource] (UgentTask-5:null) (logid:) Host health check script path is not specified 2025-05-09 13:21:15,799 DEBUG [cloud.agent.Agent] (UgentTask-5:null) (logid:) Sending ping: Seq 18-1581: { Cmd , MgmtId: -1, via: 18, Ver: v1, Flags: 11, [{"com.cloud.agent.api.PingRoutingWithNwGroupsCommand":{"newGroupStates":{},"_hostVmStateReport":{"r-146-VM":{"state":"PowerOn","host":"b-kvm-test-001.test.c.fraunhofer.de"},"s-144-VM":{"state":"PowerOn","host":"b-kvm-test-001.test.c.fraunhofer.de"},"i-4-134-VM":{"state":"PowerOn","host":"b-kvm-test-001.test.c.fraunhofer.de"},"v-145-VM":{"state":"PowerOn","host":"b-kvm-test-001.test.c.fraunhofer.de"}},"_gatewayAccessible":"true","_vnetAccessible":"true","hostType":"Routing","hostId":"18","outOfBand":"false","wait":"0","bypassHostMaintenance":"false"}}] } 2025-05-09 13:21:15,873 DEBUG [cloud.agent.Agent] (Agent-Handler-3:null) (logid:d26bf530) Received response: Seq 18-1581: { Ans: , MgmtId: 345051427970, via: 18, Ver: v1, Flags: 100010, [{"com.cloud.agent.api.PingAnswer":{"_command":{"hostType":"Routing","hostId":"18","outOfBand":"false","wait":"0","bypassHostMaintenance":"false"},"sendStartup":"false","result":"true","wait":"0","bypassHostMaintenance":"false"}}] } 2025-05-09 13:21:19,318 INFO [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (Script-4:null) (logid:) (virt-v2v ovf source: /mnt/712cdae3-3474-3a73-985a-5165c940db32/d40281df-aa8e-4169-959d-513f56782381/i-4-57-VM/ progress) virt-v2v: This guest has virtio drivers installed. 2025-05-09 13:21:19,532 INFO [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (Script-4:null) (logid:) (virt-v2v ovf source: /mnt/712cdae3-3474-3a73-985a-5165c940db32/d40281df-aa8e-4169-959d-513f56782381/i-4-57-VM/ progress) [ 94.6] Mapping filesystem data to avoid copying unused and blank areas 2025-05-09 13:21:23,247 INFO [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (Script-4:null) (logid:) (virt-v2v ovf source: /mnt/712cdae3-3474-3a73-985a-5165c940db32/d40281df-aa8e-4169-959d-513f56782381/i-4-57-VM/ progress) [ 98.3] Closing the overlay 2025-05-09 13:21:23,534 INFO [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (Script-4:null) (logid:) (virt-v2v ovf source: /mnt/712cdae3-3474-3a73-985a-5165c940db32/d40281df-aa8e-4169-959d-513f56782381/i-4-57-VM/ progress) [ 98.6] Assigning disks to buses 2025-05-09 13:21:23,534 INFO [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (Script-4:null) (logid:) (virt-v2v ovf source: /mnt/712cdae3-3474-3a73-985a-5165c940db32/d40281df-aa8e-4169-959d-513f56782381/i-4-57-VM/ progress) [ 98.6] Checking if the guest needs BIOS or UEFI to boot 2025-05-09 13:21:23,534 INFO [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (Script-4:null) (logid:) (virt-v2v ovf source: /mnt/712cdae3-3474-3a73-985a-5165c940db32/d40281df-aa8e-4169-959d-513f56782381/i-4-57-VM/ progress) [ 98.6] Setting up the destination: -o disk -os /mnt/712cdae3-3474-3a73-985a-5165c940db32 2025-05-09 13:21:24,627 INFO [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (Script-4:null) (logid:) (virt-v2v ovf source: /mnt/712cdae3-3474-3a73-985a-5165c940db32/d40281df-aa8e-4169-959d-513f56782381/i-4-57-VM/ progress) [ 99.7] Copying disk 1/1 2025-05-09 13:21:24,628 INFO [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (Script-4:null) (logid:) (virt-v2v ovf source: /mnt/712cdae3-3474-3a73-985a-5165c940db32/d40281df-aa8e-4169-959d-513f56782381/i-4-57-VM/ progress) virt-v2v: error: nbdcopy command failed, see earlier error messages 2025-05-09 13:21:24,628 INFO [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (Script-4:null) (logid:) (virt-v2v ovf source: /mnt/712cdae3-3474-3a73-985a-5165c940db32/d40281df-aa8e-4169-959d-513f56782381/i-4-57-VM/ progress) 2025-05-09 13:21:24,628 INFO [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (Script-4:null) (logid:) (virt-v2v ovf source: /mnt/712cdae3-3474-3a73-985a-5165c940db32/d40281df-aa8e-4169-959d-513f56782381/i-4-57-VM/ progress) If reporting bugs, run virt-v2v with debugging enabled and include the complete output: 2025-05-09 13:21:24,628 INFO [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (Script-4:null) (logid:) (virt-v2v ovf source: /mnt/712cdae3-3474-3a73-985a-5165c940db32/d40281df-aa8e-4169-959d-513f56782381/i-4-57-VM/ progress) 2025-05-09 13:21:24,628 INFO [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (Script-4:null) (logid:) (virt-v2v ovf source: /mnt/712cdae3-3474-3a73-985a-5165c940db32/d40281df-aa8e-4169-959d-513f56782381/i-4-57-VM/ progress) virt-v2v -v -x [...] 2025-05-09 13:21:24,644 WARN [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (agentRequest-Handler-1:null) (logid:11512efa) Execution of process [167638] for command [virt-v2v --root first -i ova /mnt/712cdae3-3474-3a73-985a-5165c940db32/d40281df-aa8e-4169-959d-513f56782381/i-4-57-VM/ -o local -os /mnt/712cdae3-3474-3a73-985a-5165c940db32 -of qcow2 -on 20406168-eef9-416d-baa6-2cb029febf31 ] failed. 2025-05-09 13:21:24,644 DEBUG [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (agentRequest-Handler-1:null) (logid:11512efa) Exit value of process [167638] for command [virt-v2v --root first -i ova /mnt/712cdae3-3474-3a73-985a-5165c940db32/d40281df-aa8e-4169-959d-513f56782381/i-4-57-VM/ -o local -os /mnt/712cdae3-3474-3a73-985a-5165c940db32 -of qcow2 -on 20406168-eef9-416d-baa6-2cb029febf31 ] is [1]. 2025-05-09 13:21:24,690 WARN [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (agentRequest-Handler-1:null) (logid:11512efa) Exception [Stream closed] occurred when attempting to run command [virt-v2v --root first -i ova /mnt/712cdae3-3474-3a73-985a-5165c940db32/d40281df-aa8e-4169-959d-513f56782381/i-4-57-VM/ -o local -os /mnt/712cdae3-3474-3a73-985a-5165c940db32 -of qcow2 -on 20406168-eef9-416d-baa6-2cb029febf31 ]. java.io.IOException: Stream closed at java.base/java.io.BufferedInputStream.getBufIfOpen(BufferedInputStream.java:168) at java.base/java.io.BufferedInputStream.read(BufferedInputStream.java:334) at java.base/sun.nio.cs.StreamDecoder.readBytes(StreamDecoder.java:287) at java.base/sun.nio.cs.StreamDecoder.implRead(StreamDecoder.java:330) at java.base/sun.nio.cs.StreamDecoder.read(StreamDecoder.java:190) at java.base/java.io.InputStreamReader.read(InputStreamReader.java:177) at java.base/java.io.BufferedReader.fill(BufferedReader.java:162) at java.base/java.io.BufferedReader.readLine(BufferedReader.java:329) at java.base/java.io.BufferedReader.readLine(BufferedReader.java:396) at com.cloud.utils.script.OutputInterpreter.processError(OutputInterpreter.java:38) at com.cloud.utils.script.Script.execute(Script.java:313) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtConvertInstanceCommandWrapper.performInstanceConversion(LibvirtConvertInstanceCommandWrapper.java:239) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtConvertInstanceCommandWrapper.execute(LibvirtConvertInstanceCommandWrapper.java:120) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtConvertInstanceCommandWrapper.execute(LibvirtConvertInstanceCommandWrapper.java:47) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1930) at com.cloud.agent.Agent.processRequest(Agent.java:683) at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1106) at com.cloud.utils.nio.Task.call(Task.java:83) at com.cloud.utils.nio.Task.call(Task.java:29) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635) at java.base/java.lang.Thread.run(Thread.java:840) 2025-05-09 13:21:24,690 ERROR [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (agentRequest-Handler-1:null) (logid:11512efa) The virt-v2v conversion for the OVF d40281df-aa8e-4169-959d-513f56782381 failed. Please check the agent logs for the virt-v2v output. Please try on a different kvm host which has a different virt-v2v version. 2025-05-09 13:21:24,690 DEBUG [resource.wrapper.LibvirtConvertInstanceCommandWrapper] (agentRequest-Handler-1:null) (logid:11512efa) Cleaning up exported OVA at dir /mnt/712cdae3-3474-3a73-985a-5165c940db32/d40281df-aa8e-4169-959d-513f56782381 2025-05-09 13:21:24,752 DEBUG [cloud.utils.FileUtil] (agentRequest-Handler-1:null) (logid:11512efa) Deleted path: /mnt/712cdae3-3474-3a73-985a-5165c940db32/d40281df-aa8e-4169-959d-513f56782381 2025-05-09 13:21:24,753 DEBUG [cloud.agent.Agent] (agentRequest-Handler-1:null) (logid:11512efa) Seq 18-2943383831463329824: { Ans: , MgmtId: 345051427970, via: 18, Ver: v1, Flags: 10, [{"com.cloud.agent.api.Answer":{"result":"false","details":"The virt-v2v conversion for the OVF d40281df-aa8e-4169-959d-513f56782381 failed. Please check the agent logs for the virt-v2v output. Please try on a different kvm host which has a different virt-v2v version.","wait":"0","bypassHostMaintenance":"false"}}] }` Screenshots of Migration-Process (redacted):    GitHub link: https://github.com/apache/cloudstack/discussions/10841 ---- This is an automatically sent email for users@cloudstack.apache.org. To unsubscribe, please send an email to: users-unsubscr...@cloudstack.apache.org
