> On 2015-05-08 13:58, Daan Hoogland wrote: > > Moving this to dev, Mike. Probably more understanding of the matter available > there. > > I don't understand the periode between 2015-05-08 09:58:13,766 until > 2015-05-08 09:58:14,491 > > All of your log from the source host is between only two of the log entries > from the destination. > > Can you maybe share the management log from the same attempt? > >>
Below you'll find a fresh set of logs for me trying the migration. Source / destination agent and management server logs are all included. ==== = Agent Running VM ==== 2015-05-08 15:06:03,870 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) Request:Seq 18-630503947831870702: { Cmd , MgmtId: 114374076129940, via: 18, Ver: v1, Flags: 100011, [{"com.cloud.agent.api.MigrateCommand":{"vmName":"r-93-VM","destIp":"172.16.16.30","hostGuid":"b86fc0dc-1017-3776-a9a3-2775684a0c84-LibvirtComputingResource","isWindows":false,"vmTO":{"id":93,"name":"r-93-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian GNU/Linux 7(64-bit)","platformEmulator":"Other","bootArgs":"","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"2b74e551cba07566","params":{},"uuid":"5269e919-564a-4986-92c1-07464e99e3ce","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"c1b45654-5e74-420e-bf91-5beb745546f8","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"d5174632-721e-39ed-b93b-462539f63c77","id":1,"poolType":"NetworkFilesystem","host":"172.16.16.25","path":"/srv/cloudstack/primary/0001","port":2049,"url":"NetworkFilesystem://172.16.16.25/srv/cloudstack/primary/0001/?ROLE=Primary&STOREUUID=d5174632-721e-39ed-b93b-462539f63c77"}},"name":"ROOT-93","size":2621440000,"path":"c1b45654-5e74-420e-bf91-5beb745546f8","volumeId":132,"vmName":"r-93-VM","accountId":2,"format":"QCOW2","id":132,"deviceId":0,"cacheMode":"NONE","hypervisorType":"KVM"}},"diskSeq":0,"path":"c1b45654-5e74-420e-bf91-5beb745546f8","ty pe":"ROOT","_details":{"managed":"false","storagePort":"2049","storageHost":"172.16.16.25","volumeSize":"2621440000"}}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":false,"nicUuid":"3ae999b5-9d59-431a-a41b-7dbeb90c5509","uuid":"a011df28-a322-4d94-bf31-1f4fbb7a869b","ip":"172.16.18.1","netmask":"255.255.255.0","mac":"02:00:6b:5b:00:03","dns1":"172.16.17.2","dns2":"","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://1221","isolationUri":"vlan://1221","isSecurityGroupEnabled":false,"name":"cloudbr1"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"nicUuid":"cd17fc0c-164e-4fe8-acc0-36a13f37f5aa","uuid":"0e669946-5f4d-4c51-9b2e-564edc1dc06a","ip":"169.254.2.186","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:ba","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false},{"deviceId":2,"networkRateMbps":200,"defaultNic":true,"nicUuid":"9bc37269-68d5-40b8-ac08-4b60ccab5d0f","uuid":"80c92696-16cb-427e-855a-a2ac01c34ebc","i p":"172.16.17.202","netmask":"255.255.255.0","gateway":"172.16.17.2","mac":"06:13:ca:00:00:36","dns1":"172.16.17.2","dns2":"","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://untagged","isolationUri":"vlan://untagged","isSecurityGroupEnabled":false,"name":"cloudbr0"}]},"executeInSequence":false,"wait":0}}] } 2015-05-08 15:06:03,870 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) Processing command: com.cloud.agent.api.MigrateCommand 2015-05-08 15:06:03,893 INFO [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) Live migration of instance r-93-VM initiated 2015-05-08 15:06:04,194 INFO [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) Migration thread for r-93-VM is done 2015-05-08 15:06:04,194 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-5:null) Failed to execute while migrating domain: org.libvirt.LibvirtException: Cannot get interface MTU on 'breth0-1221': No such device 2015-05-08 15:06:04,198 DEBUG [cloud.agent.Agent] (agentRequest-Handler-5:null) Seq 18-630503947831870702: { Ans: , MgmtId: 114374076129940, via: 18, Ver: v1, Flags: 10, [{"com.cloud.agent.api.MigrateAnswer":{"result":false,"details":"org.libvirt.LibvirtException: Cannot get interface MTU on 'breth0-1221': No such device","wait":0}}] } === = Destination Agent === 2015-05-08 15:06:03,751 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) Request:Seq 1-5821465468329789166: { Cmd , [17/1826] 14374076129940, via: 1, Ver: v1, Flags: 100111, [{"com.cloud.agent.api.PrepareForMigrationCommand":{"vm":{"id":93,"name":"r-93-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian GNU/Lin ux 7(64-bit)","platformEmulator":"Other","bootArgs":"","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"2b74e551cba07566","params":{},"uuid":"5269e919-564a-4986-92c1-07464e99e3ce","disks":[{"data":{"org.apache .cloudstack.storage.to.VolumeObjectTO":{"uuid":"c1b45654-5e74-420e-bf91-5beb745546f8","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"d5174632-721e-39ed-b93b-462539f63c77","id":1,"poolType":"NetworkFilesystem","host":" 172.16.16.25","path":"/srv/cloudstack/primary/0001","port":2049,"url":"NetworkFilesystem://172.16.16.25/srv/cloudstack/primary/0001/?ROLE=Primary&STOREUUID=d5174632-721e-39ed-b93b-462539f63c77"}},"name":"ROOT-93","size":2621440000,"path":"c1b45654-5e74-420e-bf91-5be b745546f8","volumeId":132,"vmName":"r-93-VM","accountId":2,"format":"QCOW2","id":132,"deviceId":0,"cacheMode":"NONE","hypervisorType":"KVM"}},"diskSeq":0,"path":"c1b45654-5e74-420e-bf91-5beb745546f8","type":"ROOT","_details":{"managed":"false","storagePort":"2049"," storageHost":"172.16.16.25","volumeSize":"2621440000"}}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":false,"nicUuid":"3a e999b5-9d59-431a-a41b-7dbeb90c5509","uuid":"a011df28-a322-4d94-bf31-1f4fbb7a869b","ip":"172.16.18.1","netmask":"255.255.255.0","mac": "02:00:6b:5b:00:03","dns1":"172.16.17.2","dns2":"","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://1221","isolationUri":"vlan://1221","isSecurityGroupEnabled":false,"name":"cloudbr1"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"nicUuid":"cd17 fc0c-164e-4fe8-acc0-36a13f37f5aa","uuid":"0e669946-5f4d-4c51-9b2e-564edc1dc06a","ip":"169.254.2.186","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:ba","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false},{"deviceId":2, "networkRateMbps":200,"defaultNic":true,"nicUuid":"9bc37269-68d5-40b8-ac08-4b60ccab5d0f","uuid":"80c92696-16cb-427e-855a-a2ac01c34ebc ","ip":"172.16.17.202","netmask":"255.255.255.0","gateway":"172.16.17.2","mac":"06:13:ca:00:00:36","dns1":"172.16.17.2","dns2":"","br oadcastType":"Vlan","type":"Public","broadcastUri":"vlan://untagged","isolationUri":"vlan://untagged","isSecurityGroupEnabled":false, "name":"cloudbr0"}]},"wait":0}}] } 2015-05-08 15:06:03,752 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) Processing command: com.cloud.agent.api.PrepareForMig rationCommand 2015-05-08 15:06:03,752 DEBUG [kvm.resource.LibvirtComputingResource] (agentRequest-Handler-3:null) Preparing host for migrating com. cloud.agent.api.to.VirtualMachineTO@19b45f09 2015-05-08 15:06:03,754 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-3:null) can't find connection: KVM, for vm: r-93 -VM, continue 2015-05-08 15:06:03,755 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-3:null) can't find connection: LXC, for vm: r-93 -VM, continue 2015-05-08 15:06:03,755 DEBUG [kvm.resource.LibvirtConnection] (agentRequest-Handler-3:null) can't find which hypervisor the vm used , then use the default hypervisor 2015-05-08 15:06:03,756 DEBUG [kvm.resource.BridgeVifDriver] (agentRequest-Handler-3:null) nic=[Nic:Guest-172.16.18.1-vlan://1221] 2015-05-08 15:06:03,756 DEBUG [kvm.resource.BridgeVifDriver] (agentRequest-Handler-3:null) creating a vNet dev and bridge for guest t raffic per traffic label cloudbr1 2015-05-08 15:06:03,756 DEBUG [kvm.resource.BridgeVifDriver] (agentRequest-Handler-3:null) Executing: /usr/share/cloudstack-common/sc ripts/vm/network/vnet/modifyvlan.sh -v 1221 -p bond0 -b brbond0-1221 -o add 2015-05-08 15:06:03,782 DEBUG [kvm.resource.BridgeVifDriver] (agentRequest-Handler-3:null) Execution is successful. 2015-05-08 15:06:03,782 DEBUG [kvm.resource.BridgeVifDriver] (agentRequest-Handler-3:null) Set name-type for VLAN subsystem. Should b e visible in /proc/net/vlan/config 2015-05-08 15:06:03,783 DEBUG [kvm.resource.BridgeVifDriver] (agentRequest-Handler-3:null) nic=[Nic:Control-169.254.2.186-null] 2015-05-08 15:06:03,783 DEBUG [utils.script.Script] (agentRequest-Handler-3:null) Executing: /bin/bash -c ip route | grep 169.254.0.0 /16 2015-05-08 15:06:03,790 DEBUG [utils.script.Script] (agentRequest-Handler-3:null) Execution is successful. 2015-05-08 15:06:03,791 DEBUG [kvm.resource.BridgeVifDriver] (agentRequest-Handler-3:null) nic=[Nic:Public-172.16.17.202-vlan://untag ged] 2015-05-08 15:06:03,873 DEBUG [cloud.agent.Agent] (agentRequest-Handler-3:null) Seq 1-5821465468329789166: { Ans: , MgmtId: 11437407 6129940, via: 1, Ver: v1, Flags: 110, [{"com.cloud.agent.api.PrepareForMigrationAnswer":{"result":true,"wait":0}}] } === = Management Server === 2015-05-08 15:06:03,267 DEBUG [c.c.a.ApiServlet] (http-6443-exec-16:ctx-8e20b368) ===START=== 172.16.8.151 -- GET command[195/1797] stemVm&hostid=fdd1b20a-d1b3-4e3b-b1d2-0a1127b729aa&virtualmachineid=5269e919-564a-4986-92c1-07464e99e3ce&response=json&sessionkey=ZT0 %2FJYmjsGusXAdfGH%2BGqRN8D4A%3D&_=1431111965567 2015-05-08 15:06:03,285 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (http-6443-exec-16:ctx-8e20b368 ctx-801c1285) submit async job-4142, details: AsyncJobVO {id:4142, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.a dmin.systemvm.MigrateSystemVMCmd, cmdInfo: {"response":"json","sessionkey":"ZT0/JYmjsGusXAdfGH+GqRN8D4Au003d","ctxDetails":"{"com.c loud.vm.VirtualMachine":"5269e919-564a-4986-92c1-07464e99e3ce","com.cloud.host.Host":"fdd1b20a-d1b3-4e3b-b1d2-0a1127b729aa"}", "virtualmachineid":"5269e919-564a-4986-92c1-07464e99e3ce","cmdEventType":"VM.MIGRATE","hostid":"fdd1b20a-d1b3-4e3b-b1d2-0a1127b729aa" ,"ctxUserId":"2","httpmethod":"GET","_":"1431111965567","ctxAccountId":"2","ctxStartEventId":"1952"}, cmdVersion: 0, status: IN_PROGR ESS, processStatus: 0, resultCode: 0, result: null, initMsid: 114374076129940, completeMsid: null, lastUpdated: null, lastPolled: nul l, created: null} 2015-05-08 15:06:03,285 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-20:ctx-d79c631c job-4142) Add job-4142 into job monitor ing 2015-05-08 15:06:03,285 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-20:ctx-d79c631c job-4142) Executing AsyncJobVO {id: 4142, userId: 2, accountId: 2, instanceType: None, instanceId: null, cmd: org.apache.cloudstack.api.command.admin.systemvm.MigrateSys temVMCmd, cmdInfo: {"response":"json","sessionkey":"ZT0/JYmjsGusXAdfGH+GqRN8D4Au003d","ctxDetails":"{"com.cloud.vm.VirtualMachine" :"5269e919-564a-4986-92c1-07464e99e3ce","com.cloud.host.Host":"fdd1b20a-d1b3-4e3b-b1d2-0a1127b729aa"}","virtualmachineid":"5269 e919-564a-4986-92c1-07464e99e3ce","cmdEventType":"VM.MIGRATE","hostid":"fdd1b20a-d1b3-4e3b-b1d2-0a1127b729aa","ctxUserId":"2","httpme thod":"GET","_":"1431111965567","ctxAccountId":"2","ctxStartEventId":"1952"}, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, r esultCode: 0, result: null, initMsid: 114374076129940, completeMsid: null, lastUpdated: null, lastPolled: null, created: null} 2015-05-08 15:06:03,286 DEBUG [c.c.a.ApiServlet] (http-6443-exec-16:ctx-8e20b368 ctx-801c1285) ===END=== 172.16.8.151 -- GET comman d=migrateSystemVm&hostid=fdd1b20a-d1b3-4e3b-b1d2-0a1127b729aa&virtualmachineid=5269e919-564a-4986-92c1-07464e99e3ce&response=json&ses sionkey=ZT0%2FJYmjsGusXAdfGH%2BGqRN8D4A%3D&_=1431111965567 2015-05-08 15:06:03,312 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-20:ctx-d79c631c job-4142 ctx-ca445082) Sync job-414 3 execution on object VmWorkJobQueue.93 2015-05-08 15:06:03,314 WARN [c.c.u.d.Merovingian2] (API-Job-Executor-20:ctx-d79c631c job-4142 ctx-ca445082) Was unable to find lock for the key vm_instance93 and thread id 2016601761 2015-05-08 15:06:03,695 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-adbfde0d) Execute sync-queue item: SyncQ ueueItemVO {id:106, queueId: 120, contentType: AsyncJob, contentId: 4143, lastProcessMsid: null, lastprocessNumber: null, lastProcess Time: null, created: Fri May 08 15:06:03 EDT 2015} 2015-05-08 15:06:03,696 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (AsyncJobMgr-Heartbeat-1:ctx-adbfde0d) Schedule queued job-4143 2015-05-08 15:06:03,700 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143) Add job-4143 into j ob monitoring 2015-05-08 15:06:03,700 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143) Executing Async JobVO {id:4143, userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkMigrate, cmdInfo: rO0ABXNyABpj b20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZRdxQXtPtzYqAgAGSgAJc3JjSG9zdElkTAAJY2x1c3RlcklkdAAQTGphdmEvbGFuZy9Mb25nO0wABmhvc3RJZHEAfgABTAAFcG9kS WRxAH4AAUwAB3N0b3JhZ2V0AA9MamF2YS91dGlsL01hcDtMAAZ6b25lSWRxAH4AAXhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2 VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAXXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGw AAAAAAAAAEnNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAAAXEAfgAJcQB-AAlwcQB- AAk, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 114374076129940, completeMsid: null , lastUpdated: null, lastPolled: null, created: Fri May 08 15:06:03 EDT 2015} 2015-05-08 15:06:03,701 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143) Run VM work job: com. cloud.vm.VmWorkMigrate for VM 93, job origin: 4142 2015-05-08 15:06:03,702 DEBUG [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Execut e VM work job: com.cloud.vm.VmWorkMigrate{"zoneId":1,"podId":1,"clusterId":1,"hostId":1,"srcHostId":18,"userId":2,"accountId":2,"vmId ":93,"handlerName":"VirtualMachineManagerImpl"} 2015-05-08 15:06:03,706 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Mi grating VM[DomainRouter|r-93-VM] to Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))] : Dest[Zone(1)-Pod (1)-Cluster(1)-Host(1)-Storage()] 2015-05-08 15:06:03,712 DEBUG [c.c.n.NetworkModelImpl] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Service Sec urityGroup is not supported in the network id=216 2015-05-08 15:06:03,723 DEBUG [c.c.n.NetworkModelImpl] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Service Sec urityGroup is not supported in the network id=216 2015-05-08 15:06:03,731 DEBUG [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Prepa ring 1 volumes for VM[DomainRouter|r-93-VM] 2015-05-08 15:06:03,745 DEBUG [c.c.a.t.Request] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Seq 1-582146546832 9789166: Sending { Cmd , MgmtId: 114374076129940, via: 1(rubella.nusku.biz), Ver: v1, Flags: 100111, [{"com.cloud.agent.api.PrepareF orMigrationCommand":{"vm":{"id":93,"name":"r-93-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":268435456," maxRam":268435456,"arch":"x86_64","os":"Debian GNU/Linux 7(64-bit)","platformEmulator":"Other","bootArgs":"","rebootOnCrash":false,"e nableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"2b74e551cba07566","params":{},"uuid":"5269e919-564a -4986-92c1-07464e99e3ce","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"c1b45654-5e74-420e-bf91-5beb745 546f8","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"d5174632-721e-39ed-b93b-462539 f63c77","id":1,"poolType":"NetworkFilesystem","host":"172.16.16.25","path":"/srv/cloudstack/primary/0001","port":2049,"url":"NetworkF ilesystem://172.16.16.25/srv/cloudstack/primary/0001/?ROLE=Primary&STOREUUID=d5174632-721e-39ed-b93b-462539f63c77"}},"name":"ROOT-93" ,"size":2621440000,"path":"c1b45654-5e74-420e-bf91-5beb745546f8","volumeId":132,"vmName":"r-93-VM","accountId":2,"format":"QCOW2","id ":132,"deviceId":0,"cacheMode":"NONE","hypervisorType":"KVM"}},"diskSeq":0,"path":"c1b45654-5e74-420e-bf91-5beb745546f8","type":"ROOT ","_details":{"managed":"false","storagePort":"2049","storageHost":"172.16.16.25","volumeSize":"2621440000"}}],"nics":[{"deviceId":0, "networkRateMbps":200,"defaultNic":false,"nicUuid":"3ae999b5-9d59-431a-a41b-7dbeb90c5509","uuid":"a011df28-a322-4d94-bf31-1f4fbb7a869 b","ip":"172.16.18.1","netmask":"255.255.255.0","mac":"02:00:6b:5b:00:03","dns1":"172.16.17.2","dns2":"","broadcastType":"Vlan","type ":"Guest","broadcastUri":"vlan://1221","isolationUri":"vlan://1221","isSecurityGroupEnabled":false,"name":"cloudbr1"},{"deviceId":1," networkRateMbps":-1,"defaultNic":false,"nicUuid":"cd17fc0c-164e-4fe8-acc0-36a13f37f5aa","uuid":"0e669946-5f4d-4c51-9b2e-564edc1dc06a" ,"ip":"169.254.2.186","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:ba","broadcastType":"LinkLocal","type":"C ontrol","isSecurityGroupEnabled":false},{"deviceId":2,"networkRateMbps":200,"defaultNic":true,"nicUuid":"9bc37269-68d5-40b8-ac08-4b60 ccab5d0f","uuid":"80c92696-16cb-427e-855a-a2ac01c34ebc","ip":"172.16.17.202","netmask":"255.255.255.0","gateway":"172.16.17.2","mac": "06:13:ca:00:00:36","dns1":"172.16.17.2","dns2":"","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://untagged","isolation Uri":"vlan://untagged","isSecurityGroupEnabled":false,"name":"cloudbr0"}]},"wait":0}}] } 2015-05-08 15:06:03,874 DEBUG [c.c.a.t.Request] (AgentManager-Handler-3:null) Seq 1-5821465468329789166: Processing: { Ans: , MgmtId : 114374076129940, via: 1, Ver: v1, Flags: 110, [{"com.cloud.agent.api.PrepareForMigrationAnswer":{"result":true,"wait":0}}] } 2015-05-08 15:06:03,874 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-3:null) Seq 1-5821465468329789166: No more commands found 2015-05-08 15:06:03,874 DEBUG [c.c.a.t.Request] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Seq 1-582146546832 9789166: Received: { Ans: , MgmtId: 114374076129940, via: 1, Ver: v1, Flags: 110, { PrepareForMigrationAnswer } } 2015-05-08 15:06:03,882 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) VM state transitted from :Running to Migrating with event: MigrationRequestedvm's original host id: 18 new host id: 1 host id before state tr ansition: 18 2015-05-08 15:06:03,886 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Hosts's actual total CPU: 24480 and CPU after applying overprovisioning: 24480 2015-05-08 15:06:03,886 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) We are a llocating VM, increasing the used capacity of this host:1 2015-05-08 15:06:03,886 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Current Used CPU: 10500 , Free CPU:13980 ,Requested CPU: 500 2015-05-08 15:06:03,887 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Current Used RAM: 11005853696 , Free RAM:10032242688 ,Requested RAM: 268435456 2015-05-08 15:06:03,887 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) CPU STAT S after allocation: for host: 1, old used: 10500, old reserved: 0, actual total: 24480, total with overprovisioning: 24480; new used: 11000, reserved:0; requested cpu:500,alloc_from_last:false 2015-05-08 15:06:03,887 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) RAM STAT S after allocation: for host: 1, old used: 11005853696, old reserved: 0, total: 21038096384; new used: 11274289152, reserved: 0; requ ested mem: 268435456,alloc_from_last:false 2015-05-08 15:06:03,893 DEBUG [c.c.a.t.Request] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Seq 18-63050394783 1870702: Sending { Cmd , MgmtId: 114374076129940, via: 18(leprosy.nusku.biz), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.Migrate Command":{"vmName":"r-93-VM","destIp":"172.16.16.30","hostGuid":"b86fc0dc-1017-3776-a9a3-2775684a0c84-LibvirtComputingResource","isWi ndows":false,"vmTO":{"id":93,"name":"r-93-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":268435456,"maxRam ":268435456,"arch":"x86_64","os":"Debian GNU/Linux 7(64-bit)","platformEmulator":"Other","bootArgs":"","rebootOnCrash":false,"enableH A":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"2b74e551cba07566","params":{},"uuid":"5269e919-564a-4986- 92c1-07464e99e3ce","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"c1b45654-5e74-420e-bf91-5beb745546f8" ,"volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"d5174632-721e-39ed-b93b-462539f63c77 ","id":1,"poolType":"NetworkFilesystem","host":"172.16.16.25","path":"/srv/cloudstack/primary/0001","port":2049,"url":"NetworkFilesys tem://172.16.16.25/srv/cloudstack/primary/0001/?ROLE=Primary&STOREUUID=d5174632-721e-39ed-b93b-462539f63c77"}},"name":"ROOT-93","size ":2621440000,"path":"c1b45654-5e74-420e-bf91-5beb745546f8","volumeId":132,"vmName":"r-93-VM","accountId":2,"format":"QCOW2","id":132, "deviceId":0,"cacheMode":"NONE","hypervisorType":"KVM"}},"diskSeq":0,"path":"c1b45654-5e74-420e-bf91-5beb745546f8","type":"ROOT","_de tails":{"managed":"false","storagePort":"2049","storageHost":"172.16.16.25","volumeSize":"2621440000"}}],"nics":[{"deviceId":0,"netwo rkRateMbps":200,"defaultNic":false,"nicUuid":"3ae999b5-9d59-431a-a41b-7dbeb90c5509","uuid":"a011df28-a322-4d94-bf31-1f4fbb7a869b","ip ":"172.16.18.1","netmask":"255.255.255.0","mac":"02:00:6b:5b:00:03","dns1":"172.16.17.2","dns2":"","broadcastType":"Vlan","type":"Gue st","broadcastUri":"vlan://1221","isolationUri":"vlan://1221","isSecurityGroupEnabled":false,"name":"cloudbr1"},{"deviceId":1,"networ kRateMbps":-1,"defaultNic":false,"nicUuid":"cd17fc0c-164e-4fe8-acc0-36a13f37f5aa","uuid":"0e669946-5f4d-4c51-9b2e-564edc1dc06a","ip": "169.254.2.186","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:ba","broadcastType":"LinkLocal","type":"Control ","isSecurityGroupEnabled":false},{"deviceId":2,"networkRateMbps":200,"defaultNic":true,"nicUuid":"9bc37269-68d5-40b8-ac08-4b60ccab5d 0f","uuid":"80c92696-16cb-427e-855a-a2ac01c34ebc","ip":"172.16.17.202","netmask":"255.255.255.0","gateway":"172.16.17.2","mac":"06:13 :ca:00:00:36","dns1":"172.16.17.2","dns2":"","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://untagged","isolationUri":" vlan://untagged","isSecurityGroupEnabled":false,"name":"cloudbr0"}]},"executeInSequence":false,"wait":0}}] } 2015-05-08 15:06:04,239 DEBUG [c.c.a.t.Request] (AgentManager-Handler-11:null) Seq 18-630503947831870702: Processing: { Ans: , MgmtI d: 114374076129940, via: 18, Ver: v1, Flags: 10, [{"com.cloud.agent.api.MigrateAnswer":{"result":false,"details":"org.libvirt.Libvirt Exception: Cannot get interface MTU on 'breth0-1221': No such device","wait":0}}] } 2015-05-08 15:06:04,239 DEBUG [c.c.a.t.Request] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Seq 18-63050394783 1870702: Received: { Ans: , MgmtId: 114374076129940, via: 18, Ver: v1, Flags: 10, { MigrateAnswer } } 2015-05-08 15:06:04,239 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Mi gration was unsuccessful. Cleaning up: VM[DomainRouter|r-93-VM] 2015-05-08 15:06:04,244 WARN [o.a.c.alerts] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) alertType:: 15 // da taCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Unable to migrate vm r-93-VM from host leprosy.nusku.biz in zone Nusku N etworks - MI0001 and pod POD0001 2015-05-08 15:06:04,244 DEBUG [c.c.a.AlertManagerImpl] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Have alread y sent: 1 emails for alert type '15' -- skipping send email 2015-05-08 15:06:04,245 DEBUG [c.c.a.m.AgentAttache] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Seq 1-5821465 468329789167: Routed from 114374076129940 2015-05-08 15:06:04,245 DEBUG [c.c.a.t.Request] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Seq 1-582146546832 9789167: Sending { Cmd , MgmtId: 114374076129940, via: 1(rubella.nusku.biz), Ver: v1, Flags: 100011, [{"com.cloud.agent.api.StopComm and":{"isProxy":false,"executeInSequence":false,"checkBeforeCleanup":false,"vmName":"r-93-VM","wait":0}}] } 2015-05-08 15:06:04,251 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) VM state transitted from :Migrating to Running with event: OperationFailedvm's original host id: 18 new host id: 18 host id before state tran sition: 1 2015-05-08 15:06:04,255 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Hosts's actual total CPU: 24480 and CPU after applying overprovisioning: 24480 2015-05-08 15:06:04,255 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Hosts's actual total RAM: 21038096384 and RAM after applying overprovisioning: 21038096384 2015-05-08 15:06:04,255 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) release cpu from host: 1, old used: 11000,reserved: 0, actual total: 24480, total with overprovisioning: 24480; new used: 10500,reserved:0; m ovedfromreserved: false,moveToReserveredfalse 2015-05-08 15:06:04,255 DEBUG [c.c.c.CapacityManagerImpl] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) release mem from host: 1, old used: 11274289152,reserved: 0, total: 21038096384; new used: 11005853696,reserved:0; movedfromreserved: false,m oveToReserveredfalse 2015-05-08 15:06:04,262 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Invoca tion exception, caused by: com.cloud.utils.exception.CloudRuntimeException: Unable to migrate due to org.libvirt.LibvirtException: Ca nnot get interface MTU on 'breth0-1221': No such device 2015-05-08 15:06:04,263 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Rethro w exception com.cloud.utils.exception.CloudRuntimeException: Unable to migrate due to org.libvirt.LibvirtException: Cannot get interf ace MTU on 'breth0-1221': No such device 2015-05-08 15:06:04,263 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143) Done with run of VM w ork job: com.cloud.vm.VmWorkMigrate for VM 93, job origin: 4142 2015-05-08 15:06:04,263 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143) Unable to complete As yncJobVO {id:4143, userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkMigrate, cmdInfo: rO0ABXNyA Bpjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZRdxQXtPtzYqAgAGSgAJc3JjSG9zdElkTAAJY2x1c3RlcklkdAAQTGphdmEvbGFuZy9Mb25nO0wABmhvc3RJZHEAfgABTAAFcG 9kSWRxAH4AAUwAB3N0b3JhZ2V0AA9MamF2YS91dGlsL01hcDtMAAZ6b25lSWRxAH4AAXhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ 1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAXXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlcklt cGwAAAAAAAAAEnNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAAAXEAfgAJcQB-AAlwc QB-AAk, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 114374076129940, completeMsid: n ull, lastUpdated: null, lastPolled: null, created: Fri May 08 15:06:03 EDT 2015}, job origin:4142 com.cloud.utils.exception.CloudRuntimeException: Unable to migrate due to org.libvirt.LibvirtException: Cannot get interface MTU on ' breth0-1221': No such device at com.cloud.vm.VirtualMachineManagerImpl.migrate(VirtualMachineManagerImpl.java:1922) at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrate(VirtualMachineManagerImpl.java:1822) at com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrate(VirtualMachineManagerImpl.java:5223) at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:606) at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) at com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5346) at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:502) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:459) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2015-05-08 15:06:04,265 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143) Complete async job-4143, jobStatus: FAILED, resultCode: 0, result: rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFuZy5Fe GNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARMAAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2 V0ABJMamF2YS9sYW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS9 1dGlsL0xpc3Q7eHBxAH4AB3QAjEpvYiBmYWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBVbmFibGUgdG8gbWlncmF0ZSBkdWUgdG8gb3JnLmxpYnZpcnQuTGlidmlydEV4Y2VwdGlv bjogQ2Fubm90IGdldCBpbnRlcmZhY2UgTVRVIG9uICdicmV0aDAtMTIyMSc6IE5vIHN1Y2ggZGV2aWNldXIAHltMamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50OwJGKjw8_ SI5AgAAeHAAAAANc3IAG2phdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudGEJxZomNt2FAgAESQAKbGluZU51bWJlckwADmRlY2xhcmluZ0NsYXNzcQB-AARMAAhmaWxlTmFtZX EAfgAETAAKbWV0aG9kTmFtZXEAfgAEeHAAAABydAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2J zcQB-AAsAAAH2dAA_b3JnLmFwYWNoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwuQXN5bmNKb2JNYW5hZ2VySW1wbCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5q YXZhdAAMcnVuSW5Db250ZXh0c3EAfgALAAAAMXQAPm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZSQxdAAbTWFuY WdlZENvbnRleHRSdW5uYWJsZS5qYXZhdAADcnVuc3EAfgALAAAAOHQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZE NvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHNxAH4ACwAAAGd0AEBvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGw uRGVmYXVsdE1hbmFnZWRDb250ZXh0cQB-ABp0AA9jYWxsV2l0aENvbnRleHRzcQB-AAsAAAA1cQB-AB1xAH4AGnQADnJ1bldpdGhDb250ZXh0c3EAfgALAAAALnQAPG9yZy5h cGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAWcQB-ABdzcQB-AAsAAAHLcQB-ABFxAH4AEnEAfgAXc3EAfgALAAAB1 3QALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAbc3EAfgALAAABBnQAH2phdmEudXRpbC5jb25jdX JyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4AF3NxAH4ACwAABHl0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWR Qb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnNxAH4ACwAAAmd0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-ACxxAH4AF3Nx AH4ACwAAAul0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4AF3NyACZqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlTGlzdPwPJTG17I4QAgABT AAEbGlzdHEAfgAGeHIALGphdmEudXRpbC5Db2xsZWN0aW9ucyRVbm1vZGlmaWFibGVDb2xsZWN0aW9uGUIAgMte9x4CAAFMAAFjdAAWTGphdmEvdXRpbC9Db2xsZWN0aW9uO3 hwc3IAE2phdmEudXRpbC5BcnJheUxpc3R4gdIdmcdhnQMAAUkABHNpemV4cAAAAAB3BAAAAAB4cQB-ADh4 2015-05-08 15:06:04,272 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143) Done executing com.cloud.vm.VmWorkMigrate for job-4143 2015-05-08 15:06:04,282 DEBUG [o.a.c.f.j.i.SyncQueueManagerImpl] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143) Sync queue (12 0) is currently empty 2015-05-08 15:06:04,283 INFO [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143) Remove job-4143 fro m job monitoring 2015-05-08 15:06:04,286 ERROR [c.c.a.ApiAsyncJobDispatcher] (API-Job-Executor-20:ctx-d79c631c job-4142) Unexpected exception while ex ecuting org.apache.cloudstack.api.command.admin.systemvm.MigrateSystemVMCmd java.lang.RuntimeException: Job failed due to exception Unable to migrate due to org.libvirt.LibvirtException: Cannot get interface M TU on 'breth0-1221': No such device at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:502) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:459) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask.run(FutureTask.java:262) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) at java.lang.Thread.run(Thread.java:745) 2015-05-08 15:06:04,287 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-20:ctx-d79c631c job-4142) Complete async job-4142, jobStatus: FAILED, resultCode: 530, result: org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Job failed due to exception Unable to migrate due to org.libvirt.LibvirtException: Cannot get interface MTU on u0027breth0-1221u0027: No such device"} 2015-05-08 15:06:04,293 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl] (API-Job-Executor-20:ctx-d79c631c job-4142) Done executing org.apache.cloudstack.api.command.admin.systemvm.MigrateSystemVMCmd for job-4142 2015-05-08 15:06:04,296 INFO [o.a.c.f.j.i.AsyncJobMonitor] (API-Job-Executor-20:ctx-d79c631c job-4142) Remove job-4142 from job monitoring 2015-05-08 15:06:04,381 DEBUG [c.c.a.t.Request] (AgentManager-Handler-14:null) Seq 1-5821465468329789167: Processing: { Ans: , MgmtId: 114374076129940, via: 1, Ver: v1, Flags: 10, [{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] } 2015-05-08 15:06:04,382 DEBUG [c.c.a.m.AgentAttache] (AgentManager-Handler-14:null) Seq 1-5821465468329789167: Unable to find listener. 2015-05-08 15:06:06,296 DEBUG [c.c.a.ApiServlet] (http-6443-exec-13:ctx-c3e12033) ===START=== 172.16.8.151 -- GET command=queryAsyncJobResult&jobId=4770b267-27ac-4fe5-9915-05923e9c8eaf&response=json&sessionkey=ZT0%2FJYmjsGusXAdfGH%2BGqRN8D4A%3D&_=1431111968594 2015-05-08 15:06:06,310 DEBUG [c.c.a.ApiServlet] (http-6443-exec-13:ctx-c3e12033 ctx-ba0d1ec2) ===END=== 172.16.8.151 -- GET command=queryAsyncJobResult&jobId=4770b267-27ac-4fe5-9915-05923e9c8eaf&response=json&sessionkey=ZT0%2FJYmjsGusXAdfGH%2BGqRN8D4A%3D&_=1431111968594