Can you provide some data from the /var/log/libvirt/libvirtd log, and also your
network configuration for both hosts? Also the contents of
/etc/cloudstack/agent.properties would be useful.
- Si
________________________________________
From: Mike C <mcrosson_cloudst...@nusku.net>
Sent: Friday, May 08, 2015 2:14 PM
To: Daan Hoogland
Cc: dev
Subject: Re: libvirt exception
> On 2015-05-08 13:58, Daan Hoogland wrote:
>
> Moving this to dev, Mike. Probably more understanding of the matter available
> there.
>
> I don't understand the periode between 2015-05-08 09:58:13,766 until
> 2015-05-08 09:58:14,491
>
> All of your log from the source host is between only two of the log entries
> from the destination.
>
> Can you maybe share the management log from the same attempt?
>
>>
Below you'll find a fresh set of logs for me trying the migration.
Source / destination agent and management server logs are all included.
====
= Agent Running VM
====
2015-05-08 15:06:03,870 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) Request:Seq 18-630503947831870702: { Cmd ,
MgmtId: 114374076129940, via: 18, Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.MigrateCommand":{"vmName":"r-93-VM","destIp":"172.16.16.30","hostGuid":"b86fc0dc-1017-3776-a9a3-2775684a0c84-LibvirtComputingResource","isWindows":false,"vmTO":{"id":93,"name":"r-93-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian
GNU/Linux
7(64-bit)","platformEmulator":"Other","bootArgs":"","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"2b74e551cba07566","params":{},"uuid":"5269e919-564a-4986-92c1-07464e99e3ce","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"c1b45654-5e74-420e-bf91-5beb745546f8","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"d5174632-721e-39ed-b93b-462539f63c77","id":1,"poolType":"NetworkFilesystem","host":"172.16.16.25","path":"/srv/cloudstack/primary/0001","port":2049,"url":"NetworkFilesystem://172.16.16.25/srv/cloudstack/primary/0001/?ROLE=Primary&STOREUUID=d5174632-721e-39ed-b93b-462539f63c77"}},"name":"ROOT-93","size":2621440000,"path":"c1b45654-5e74-420e-bf91-5beb745546f8","volumeId":132,"vmName":"r-93-VM","accountId":2,"format":"QCOW2","id":132,"deviceId":0,"cacheMode":"NONE","hypervisorType":"KVM"}},"diskSeq":0,"path":"c1b45654-5e74-420e-bf91-5beb745546f8","ty
pe":"ROOT","_details":{"managed":"false","storagePort":"2049","storageHost":"172.16.16.25","volumeSize":"2621440000"}}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":false,"nicUuid":"3ae999b5-9d59-431a-a41b-7dbeb90c5509","uuid":"a011df28-a322-4d94-bf31-1f4fbb7a869b","ip":"172.16.18.1","netmask":"255.255.255.0","mac":"02:00:6b:5b:00:03","dns1":"172.16.17.2","dns2":"","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://1221","isolationUri":"vlan://1221","isSecurityGroupEnabled":false,"name":"cloudbr1"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"nicUuid":"cd17fc0c-164e-4fe8-acc0-36a13f37f5aa","uuid":"0e669946-5f4d-4c51-9b2e-564edc1dc06a","ip":"169.254.2.186","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:ba","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false},{"deviceId":2,"networkRateMbps":200,"defaultNic":true,"nicUuid":"9bc37269-68d5-40b8-ac08-4b60ccab5d0f","uuid":"80c92696-16cb-427e-855a-a2ac01c34ebc","i
p":"172.16.17.202","netmask":"255.255.255.0","gateway":"172.16.17.2","mac":"06:13:ca:00:00:36","dns1":"172.16.17.2","dns2":"","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://untagged","isolationUri":"vlan://untagged","isSecurityGroupEnabled":false,"name":"cloudbr0"}]},"executeInSequence":false,"wait":0}}]
}
2015-05-08 15:06:03,870 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) Processing command:
com.cloud.agent.api.MigrateCommand
2015-05-08 15:06:03,893 INFO [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) Live migration of instance r-93-VM
initiated
2015-05-08 15:06:04,194 INFO [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) Migration thread for r-93-VM is done
2015-05-08 15:06:04,194 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-5:null) Failed to execute while migrating domain:
org.libvirt.LibvirtException: Cannot get interface MTU on 'breth0-1221':
No such device
2015-05-08 15:06:04,198 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-5:null) Seq 18-630503947831870702: { Ans: ,
MgmtId: 114374076129940, via: 18, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.MigrateAnswer":{"result":false,"details":"org.libvirt.LibvirtException:
Cannot get interface MTU on 'breth0-1221': No such device","wait":0}}] }
===
= Destination Agent
===
2015-05-08 15:06:03,751 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:null) Request:Seq 1-5821465468329789166: { Cmd ,
[17/1826]
14374076129940, via: 1, Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.PrepareForMigrationCommand":{"vm":{"id":93,"name":"r-93-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":268435456,"maxRam":268435456,"arch":"x86_64","os":"Debian
GNU/Lin
ux
7(64-bit)","platformEmulator":"Other","bootArgs":"","rebootOnCrash":false,"enableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"2b74e551cba07566","params":{},"uuid":"5269e919-564a-4986-92c1-07464e99e3ce","disks":[{"data":{"org.apache
.cloudstack.storage.to.VolumeObjectTO":{"uuid":"c1b45654-5e74-420e-bf91-5beb745546f8","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"d5174632-721e-39ed-b93b-462539f63c77","id":1,"poolType":"NetworkFilesystem","host":"
172.16.16.25","path":"/srv/cloudstack/primary/0001","port":2049,"url":"NetworkFilesystem://172.16.16.25/srv/cloudstack/primary/0001/?ROLE=Primary&STOREUUID=d5174632-721e-39ed-b93b-462539f63c77"}},"name":"ROOT-93","size":2621440000,"path":"c1b45654-5e74-420e-bf91-5be
b745546f8","volumeId":132,"vmName":"r-93-VM","accountId":2,"format":"QCOW2","id":132,"deviceId":0,"cacheMode":"NONE","hypervisorType":"KVM"}},"diskSeq":0,"path":"c1b45654-5e74-420e-bf91-5beb745546f8","type":"ROOT","_details":{"managed":"false","storagePort":"2049","
storageHost":"172.16.16.25","volumeSize":"2621440000"}}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":false,"nicUuid":"3a
e999b5-9d59-431a-a41b-7dbeb90c5509","uuid":"a011df28-a322-4d94-bf31-1f4fbb7a869b","ip":"172.16.18.1","netmask":"255.255.255.0","mac":
"02:00:6b:5b:00:03","dns1":"172.16.17.2","dns2":"","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://1221","isolationUri":"vlan://1221","isSecurityGroupEnabled":false,"name":"cloudbr1"},{"deviceId":1,"networkRateMbps":-1,"defaultNic":false,"nicUuid":"cd17
fc0c-164e-4fe8-acc0-36a13f37f5aa","uuid":"0e669946-5f4d-4c51-9b2e-564edc1dc06a","ip":"169.254.2.186","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:ba","broadcastType":"LinkLocal","type":"Control","isSecurityGroupEnabled":false},{"deviceId":2,
"networkRateMbps":200,"defaultNic":true,"nicUuid":"9bc37269-68d5-40b8-ac08-4b60ccab5d0f","uuid":"80c92696-16cb-427e-855a-a2ac01c34ebc
","ip":"172.16.17.202","netmask":"255.255.255.0","gateway":"172.16.17.2","mac":"06:13:ca:00:00:36","dns1":"172.16.17.2","dns2":"","br
oadcastType":"Vlan","type":"Public","broadcastUri":"vlan://untagged","isolationUri":"vlan://untagged","isSecurityGroupEnabled":false,
"name":"cloudbr0"}]},"wait":0}}] }
2015-05-08 15:06:03,752 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:null) Processing command:
com.cloud.agent.api.PrepareForMig
rationCommand
2015-05-08 15:06:03,752 DEBUG [kvm.resource.LibvirtComputingResource]
(agentRequest-Handler-3:null) Preparing host for migrating com.
cloud.agent.api.to.VirtualMachineTO@19b45f09
2015-05-08 15:06:03,754 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:null) can't find connection: KVM, for vm: r-93
-VM, continue
2015-05-08 15:06:03,755 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:null) can't find connection: LXC, for vm: r-93
-VM, continue
2015-05-08 15:06:03,755 DEBUG [kvm.resource.LibvirtConnection]
(agentRequest-Handler-3:null) can't find which hypervisor the vm used
, then use the default hypervisor
2015-05-08 15:06:03,756 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-3:null) nic=[Nic:Guest-172.16.18.1-vlan://1221]
2015-05-08 15:06:03,756 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-3:null) creating a vNet dev and bridge for guest t
raffic per traffic label cloudbr1
2015-05-08 15:06:03,756 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-3:null) Executing: /usr/share/cloudstack-common/sc
ripts/vm/network/vnet/modifyvlan.sh -v 1221 -p bond0 -b brbond0-1221 -o
add
2015-05-08 15:06:03,782 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-3:null) Execution is successful.
2015-05-08 15:06:03,782 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-3:null) Set name-type for VLAN subsystem. Should b
e visible in /proc/net/vlan/config
2015-05-08 15:06:03,783 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-3:null) nic=[Nic:Control-169.254.2.186-null]
2015-05-08 15:06:03,783 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) Executing: /bin/bash -c ip route | grep
169.254.0.0
/16
2015-05-08 15:06:03,790 DEBUG [utils.script.Script]
(agentRequest-Handler-3:null) Execution is successful.
2015-05-08 15:06:03,791 DEBUG [kvm.resource.BridgeVifDriver]
(agentRequest-Handler-3:null) nic=[Nic:Public-172.16.17.202-vlan://untag
ged]
2015-05-08 15:06:03,873 DEBUG [cloud.agent.Agent]
(agentRequest-Handler-3:null) Seq 1-5821465468329789166: { Ans: ,
MgmtId: 11437407
6129940, via: 1, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.PrepareForMigrationAnswer":{"result":true,"wait":0}}]
}
===
= Management Server
===
2015-05-08 15:06:03,267 DEBUG [c.c.a.ApiServlet]
(http-6443-exec-16:ctx-8e20b368) ===START=== 172.16.8.151 -- GET
command[195/1797]
stemVm&hostid=fdd1b20a-d1b3-4e3b-b1d2-0a1127b729aa&virtualmachineid=5269e919-564a-4986-92c1-07464e99e3ce&response=json&sessionkey=ZT0
%2FJYmjsGusXAdfGH%2BGqRN8D4A%3D&_=1431111965567
2015-05-08 15:06:03,285 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(http-6443-exec-16:ctx-8e20b368 ctx-801c1285) submit async job-4142,
details: AsyncJobVO {id:4142, userId: 2, accountId: 2, instanceType:
None, instanceId: null, cmd: org.apache.cloudstack.api.command.a
dmin.systemvm.MigrateSystemVMCmd, cmdInfo:
{"response":"json","sessionkey":"ZT0/JYmjsGusXAdfGH+GqRN8D4Au003d","ctxDetails":"{"com.c
loud.vm.VirtualMachine":"5269e919-564a-4986-92c1-07464e99e3ce","com.cloud.host.Host":"fdd1b20a-d1b3-4e3b-b1d2-0a1127b729aa"}",
"virtualmachineid":"5269e919-564a-4986-92c1-07464e99e3ce","cmdEventType":"VM.MIGRATE","hostid":"fdd1b20a-d1b3-4e3b-b1d2-0a1127b729aa"
,"ctxUserId":"2","httpmethod":"GET","_":"1431111965567","ctxAccountId":"2","ctxStartEventId":"1952"},
cmdVersion: 0, status: IN_PROGR
ESS, processStatus: 0, resultCode: 0, result: null, initMsid:
114374076129940, completeMsid: null, lastUpdated: null, lastPolled: nul
l, created: null}
2015-05-08 15:06:03,285 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-20:ctx-d79c631c job-4142) Add job-4142 into job
monitor
ing
2015-05-08 15:06:03,285 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-20:ctx-d79c631c job-4142) Executing AsyncJobVO {id:
4142, userId: 2, accountId: 2, instanceType: None, instanceId: null,
cmd: org.apache.cloudstack.api.command.admin.systemvm.MigrateSys
temVMCmd, cmdInfo:
{"response":"json","sessionkey":"ZT0/JYmjsGusXAdfGH+GqRN8D4Au003d","ctxDetails":"{"com.cloud.vm.VirtualMachine"
:"5269e919-564a-4986-92c1-07464e99e3ce","com.cloud.host.Host":"fdd1b20a-d1b3-4e3b-b1d2-0a1127b729aa"}","virtualmachineid":"5269
e919-564a-4986-92c1-07464e99e3ce","cmdEventType":"VM.MIGRATE","hostid":"fdd1b20a-d1b3-4e3b-b1d2-0a1127b729aa","ctxUserId":"2","httpme
thod":"GET","_":"1431111965567","ctxAccountId":"2","ctxStartEventId":"1952"},
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, r
esultCode: 0, result: null, initMsid: 114374076129940, completeMsid:
null, lastUpdated: null, lastPolled: null, created: null}
2015-05-08 15:06:03,286 DEBUG [c.c.a.ApiServlet]
(http-6443-exec-16:ctx-8e20b368 ctx-801c1285) ===END=== 172.16.8.151 --
GET comman
d=migrateSystemVm&hostid=fdd1b20a-d1b3-4e3b-b1d2-0a1127b729aa&virtualmachineid=5269e919-564a-4986-92c1-07464e99e3ce&response=json&ses
sionkey=ZT0%2FJYmjsGusXAdfGH%2BGqRN8D4A%3D&_=1431111965567
2015-05-08 15:06:03,312 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-20:ctx-d79c631c job-4142 ctx-ca445082) Sync job-414
3 execution on object VmWorkJobQueue.93
2015-05-08 15:06:03,314 WARN [c.c.u.d.Merovingian2]
(API-Job-Executor-20:ctx-d79c631c job-4142 ctx-ca445082) Was unable to
find lock
for the key vm_instance93 and thread id 2016601761
2015-05-08 15:06:03,695 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-adbfde0d) Execute sync-queue item: SyncQ
ueueItemVO {id:106, queueId: 120, contentType: AsyncJob, contentId:
4143, lastProcessMsid: null, lastprocessNumber: null, lastProcess
Time: null, created: Fri May 08 15:06:03 EDT 2015}
2015-05-08 15:06:03,696 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(AsyncJobMgr-Heartbeat-1:ctx-adbfde0d) Schedule queued job-4143
2015-05-08 15:06:03,700 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143) Add job-4143 into
j
ob monitoring
2015-05-08 15:06:03,700 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143) Executing Async
JobVO {id:4143, userId: 2, accountId: 2, instanceType: null, instanceId:
null, cmd: com.cloud.vm.VmWorkMigrate, cmdInfo: rO0ABXNyABpj
b20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZRdxQXtPtzYqAgAGSgAJc3JjSG9zdElkTAAJY2x1c3RlcklkdAAQTGphdmEvbGFuZy9Mb25nO0wABmhvc3RJZHEAfgABTAAFcG9kS
WRxAH4AAUwAB3N0b3JhZ2V0AA9MamF2YS91dGlsL01hcDtMAAZ6b25lSWRxAH4AAXhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2
VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAXXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGw
AAAAAAAAAEnNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAAAXEAfgAJcQB-AAlwcQB-
AAk, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode:
0, result: null, initMsid: 114374076129940, completeMsid: null
, lastUpdated: null, lastPolled: null, created: Fri May 08 15:06:03 EDT
2015}
2015-05-08 15:06:03,701 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143) Run VM work job:
com.
cloud.vm.VmWorkMigrate for VM 93, job origin: 4142
2015-05-08 15:06:03,702 DEBUG [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc)
Execut
e VM work job:
com.cloud.vm.VmWorkMigrate{"zoneId":1,"podId":1,"clusterId":1,"hostId":1,"srcHostId":18,"userId":2,"accountId":2,"vmId
":93,"handlerName":"VirtualMachineManagerImpl"}
2015-05-08 15:06:03,706 INFO [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Mi
grating VM[DomainRouter|r-93-VM] to
Dest[Zone(Id)-Pod(Id)-Cluster(Id)-Host(Id)-Storage(Volume(Id|Type-->Pool(Id))]
: Dest[Zone(1)-Pod
(1)-Cluster(1)-Host(1)-Storage()]
2015-05-08 15:06:03,712 DEBUG [c.c.n.NetworkModelImpl]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc)
Service Sec
urityGroup is not supported in the network id=216
2015-05-08 15:06:03,723 DEBUG [c.c.n.NetworkModelImpl]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc)
Service Sec
urityGroup is not supported in the network id=216
2015-05-08 15:06:03,731 DEBUG [o.a.c.e.o.VolumeOrchestrator]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Prepa
ring 1 volumes for VM[DomainRouter|r-93-VM]
2015-05-08 15:06:03,745 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Seq
1-582146546832
9789166: Sending { Cmd , MgmtId: 114374076129940, via:
1(rubella.nusku.biz), Ver: v1, Flags: 100111,
[{"com.cloud.agent.api.PrepareF
orMigrationCommand":{"vm":{"id":93,"name":"r-93-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":268435456,"
maxRam":268435456,"arch":"x86_64","os":"Debian GNU/Linux
7(64-bit)","platformEmulator":"Other","bootArgs":"","rebootOnCrash":false,"e
nableHA":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"2b74e551cba07566","params":{},"uuid":"5269e919-564a
-4986-92c1-07464e99e3ce","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"c1b45654-5e74-420e-bf91-5beb745
546f8","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"d5174632-721e-39ed-b93b-462539
f63c77","id":1,"poolType":"NetworkFilesystem","host":"172.16.16.25","path":"/srv/cloudstack/primary/0001","port":2049,"url":"NetworkF
ilesystem://172.16.16.25/srv/cloudstack/primary/0001/?ROLE=Primary&STOREUUID=d5174632-721e-39ed-b93b-462539f63c77"}},"name":"ROOT-93"
,"size":2621440000,"path":"c1b45654-5e74-420e-bf91-5beb745546f8","volumeId":132,"vmName":"r-93-VM","accountId":2,"format":"QCOW2","id
":132,"deviceId":0,"cacheMode":"NONE","hypervisorType":"KVM"}},"diskSeq":0,"path":"c1b45654-5e74-420e-bf91-5beb745546f8","type":"ROOT
","_details":{"managed":"false","storagePort":"2049","storageHost":"172.16.16.25","volumeSize":"2621440000"}}],"nics":[{"deviceId":0,
"networkRateMbps":200,"defaultNic":false,"nicUuid":"3ae999b5-9d59-431a-a41b-7dbeb90c5509","uuid":"a011df28-a322-4d94-bf31-1f4fbb7a869
b","ip":"172.16.18.1","netmask":"255.255.255.0","mac":"02:00:6b:5b:00:03","dns1":"172.16.17.2","dns2":"","broadcastType":"Vlan","type
":"Guest","broadcastUri":"vlan://1221","isolationUri":"vlan://1221","isSecurityGroupEnabled":false,"name":"cloudbr1"},{"deviceId":1,"
networkRateMbps":-1,"defaultNic":false,"nicUuid":"cd17fc0c-164e-4fe8-acc0-36a13f37f5aa","uuid":"0e669946-5f4d-4c51-9b2e-564edc1dc06a"
,"ip":"169.254.2.186","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:ba","broadcastType":"LinkLocal","type":"C
ontrol","isSecurityGroupEnabled":false},{"deviceId":2,"networkRateMbps":200,"defaultNic":true,"nicUuid":"9bc37269-68d5-40b8-ac08-4b60
ccab5d0f","uuid":"80c92696-16cb-427e-855a-a2ac01c34ebc","ip":"172.16.17.202","netmask":"255.255.255.0","gateway":"172.16.17.2","mac":
"06:13:ca:00:00:36","dns1":"172.16.17.2","dns2":"","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://untagged","isolation
Uri":"vlan://untagged","isSecurityGroupEnabled":false,"name":"cloudbr0"}]},"wait":0}}]
}
2015-05-08 15:06:03,874 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-3:null) Seq 1-5821465468329789166: Processing: {
Ans: , MgmtId
: 114374076129940, via: 1, Ver: v1, Flags: 110,
[{"com.cloud.agent.api.PrepareForMigrationAnswer":{"result":true,"wait":0}}]
}
2015-05-08 15:06:03,874 DEBUG [c.c.a.m.AgentAttache]
(AgentManager-Handler-3:null) Seq 1-5821465468329789166: No more
commands found
2015-05-08 15:06:03,874 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Seq
1-582146546832
9789166: Received: { Ans: , MgmtId: 114374076129940, via: 1, Ver: v1,
Flags: 110, { PrepareForMigrationAnswer } }
2015-05-08 15:06:03,882 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) VM
state
transitted from :Running to Migrating with event:
MigrationRequestedvm's original host id: 18 new host id: 1 host id
before state tr
ansition: 18
2015-05-08 15:06:03,886 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc)
Hosts's
actual total CPU: 24480 and CPU after applying overprovisioning: 24480
2015-05-08 15:06:03,886 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) We
are a
llocating VM, increasing the used capacity of this host:1
2015-05-08 15:06:03,886 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc)
Current
Used CPU: 10500 , Free CPU:13980 ,Requested CPU: 500
2015-05-08 15:06:03,887 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc)
Current
Used RAM: 11005853696 , Free RAM:10032242688 ,Requested RAM: 268435456
2015-05-08 15:06:03,887 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) CPU
STAT
S after allocation: for host: 1, old used: 10500, old reserved: 0,
actual total: 24480, total with overprovisioning: 24480; new used:
11000, reserved:0; requested cpu:500,alloc_from_last:false
2015-05-08 15:06:03,887 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) RAM
STAT
S after allocation: for host: 1, old used: 11005853696, old reserved: 0,
total: 21038096384; new used: 11274289152, reserved: 0; requ
ested mem: 268435456,alloc_from_last:false
2015-05-08 15:06:03,893 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Seq
18-63050394783
1870702: Sending { Cmd , MgmtId: 114374076129940, via:
18(leprosy.nusku.biz), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.Migrate
Command":{"vmName":"r-93-VM","destIp":"172.16.16.30","hostGuid":"b86fc0dc-1017-3776-a9a3-2775684a0c84-LibvirtComputingResource","isWi
ndows":false,"vmTO":{"id":93,"name":"r-93-VM","type":"DomainRouter","cpus":1,"minSpeed":500,"maxSpeed":500,"minRam":268435456,"maxRam
":268435456,"arch":"x86_64","os":"Debian GNU/Linux
7(64-bit)","platformEmulator":"Other","bootArgs":"","rebootOnCrash":false,"enableH
A":true,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"2b74e551cba07566","params":{},"uuid":"5269e919-564a-4986-
92c1-07464e99e3ce","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"c1b45654-5e74-420e-bf91-5beb745546f8"
,"volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"d5174632-721e-39ed-b93b-462539f63c77
","id":1,"poolType":"NetworkFilesystem","host":"172.16.16.25","path":"/srv/cloudstack/primary/0001","port":2049,"url":"NetworkFilesys
tem://172.16.16.25/srv/cloudstack/primary/0001/?ROLE=Primary&STOREUUID=d5174632-721e-39ed-b93b-462539f63c77"}},"name":"ROOT-93","size
":2621440000,"path":"c1b45654-5e74-420e-bf91-5beb745546f8","volumeId":132,"vmName":"r-93-VM","accountId":2,"format":"QCOW2","id":132,
"deviceId":0,"cacheMode":"NONE","hypervisorType":"KVM"}},"diskSeq":0,"path":"c1b45654-5e74-420e-bf91-5beb745546f8","type":"ROOT","_de
tails":{"managed":"false","storagePort":"2049","storageHost":"172.16.16.25","volumeSize":"2621440000"}}],"nics":[{"deviceId":0,"netwo
rkRateMbps":200,"defaultNic":false,"nicUuid":"3ae999b5-9d59-431a-a41b-7dbeb90c5509","uuid":"a011df28-a322-4d94-bf31-1f4fbb7a869b","ip
":"172.16.18.1","netmask":"255.255.255.0","mac":"02:00:6b:5b:00:03","dns1":"172.16.17.2","dns2":"","broadcastType":"Vlan","type":"Gue
st","broadcastUri":"vlan://1221","isolationUri":"vlan://1221","isSecurityGroupEnabled":false,"name":"cloudbr1"},{"deviceId":1,"networ
kRateMbps":-1,"defaultNic":false,"nicUuid":"cd17fc0c-164e-4fe8-acc0-36a13f37f5aa","uuid":"0e669946-5f4d-4c51-9b2e-564edc1dc06a","ip":
"169.254.2.186","netmask":"255.255.0.0","gateway":"169.254.0.1","mac":"0e:00:a9:fe:02:ba","broadcastType":"LinkLocal","type":"Control
","isSecurityGroupEnabled":false},{"deviceId":2,"networkRateMbps":200,"defaultNic":true,"nicUuid":"9bc37269-68d5-40b8-ac08-4b60ccab5d
0f","uuid":"80c92696-16cb-427e-855a-a2ac01c34ebc","ip":"172.16.17.202","netmask":"255.255.255.0","gateway":"172.16.17.2","mac":"06:13
:ca:00:00:36","dns1":"172.16.17.2","dns2":"","broadcastType":"Vlan","type":"Public","broadcastUri":"vlan://untagged","isolationUri":"
vlan://untagged","isSecurityGroupEnabled":false,"name":"cloudbr0"}]},"executeInSequence":false,"wait":0}}]
}
2015-05-08 15:06:04,239 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-11:null) Seq 18-630503947831870702: Processing: {
Ans: , MgmtI
d: 114374076129940, via: 18, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.MigrateAnswer":{"result":false,"details":"org.libvirt.Libvirt
Exception: Cannot get interface MTU on 'breth0-1221': No such
device","wait":0}}] }
2015-05-08 15:06:04,239 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Seq
18-63050394783
1870702: Received: { Ans: , MgmtId: 114374076129940, via: 18, Ver: v1,
Flags: 10, { MigrateAnswer } }
2015-05-08 15:06:04,239 INFO [c.c.v.VirtualMachineManagerImpl]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Mi
gration was unsuccessful. Cleaning up: VM[DomainRouter|r-93-VM]
2015-05-08 15:06:04,244 WARN [o.a.c.alerts]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc)
alertType:: 15 // da
taCenterId:: 1 // podId:: 1 // clusterId:: null // message:: Unable to
migrate vm r-93-VM from host leprosy.nusku.biz in zone Nusku N
etworks - MI0001 and pod POD0001
2015-05-08 15:06:04,244 DEBUG [c.c.a.AlertManagerImpl]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Have
alread
y sent: 1 emails for alert type '15' -- skipping send email
2015-05-08 15:06:04,245 DEBUG [c.c.a.m.AgentAttache]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Seq
1-5821465
468329789167: Routed from 114374076129940
2015-05-08 15:06:04,245 DEBUG [c.c.a.t.Request]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) Seq
1-582146546832
9789167: Sending { Cmd , MgmtId: 114374076129940, via:
1(rubella.nusku.biz), Ver: v1, Flags: 100011,
[{"com.cloud.agent.api.StopComm
and":{"isProxy":false,"executeInSequence":false,"checkBeforeCleanup":false,"vmName":"r-93-VM","wait":0}}]
}
2015-05-08 15:06:04,251 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc) VM
state
transitted from :Migrating to Running with event: OperationFailedvm's
original host id: 18 new host id: 18 host id before state tran
sition: 1
2015-05-08 15:06:04,255 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc)
Hosts's
actual total CPU: 24480 and CPU after applying overprovisioning: 24480
2015-05-08 15:06:04,255 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc)
Hosts's
actual total RAM: 21038096384 and RAM after applying overprovisioning:
21038096384
2015-05-08 15:06:04,255 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc)
release
cpu from host: 1, old used: 11000,reserved: 0, actual total: 24480,
total with overprovisioning: 24480; new used: 10500,reserved:0; m
ovedfromreserved: false,moveToReserveredfalse
2015-05-08 15:06:04,255 DEBUG [c.c.c.CapacityManagerImpl]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc)
release
mem from host: 1, old used: 11274289152,reserved: 0, total: 21038096384;
new used: 11005853696,reserved:0; movedfromreserved: false,m
oveToReserveredfalse
2015-05-08 15:06:04,262 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc)
Invoca
tion exception, caused by:
com.cloud.utils.exception.CloudRuntimeException: Unable to migrate due
to org.libvirt.LibvirtException: Ca
nnot get interface MTU on 'breth0-1221': No such device
2015-05-08 15:06:04,263 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143 ctx-044b39fc)
Rethro
w exception com.cloud.utils.exception.CloudRuntimeException: Unable to
migrate due to org.libvirt.LibvirtException: Cannot get interf
ace MTU on 'breth0-1221': No such device
2015-05-08 15:06:04,263 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143) Done with run of
VM w
ork job: com.cloud.vm.VmWorkMigrate for VM 93, job origin: 4142
2015-05-08 15:06:04,263 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143) Unable to complete
As
yncJobVO {id:4143, userId: 2, accountId: 2, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkMigrate, cmdInfo: rO0ABXNyA
Bpjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZRdxQXtPtzYqAgAGSgAJc3JjSG9zdElkTAAJY2x1c3RlcklkdAAQTGphdmEvbGFuZy9Mb25nO0wABmhvc3RJZHEAfgABTAAFcG
9kSWRxAH4AAUwAB3N0b3JhZ2V0AA9MamF2YS91dGlsL01hcDtMAAZ6b25lSWRxAH4AAXhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ
1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAXXQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlcklt
cGwAAAAAAAAAEnNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAAAXEAfgAJcQB-AAlwc
QB-AAk, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0,
resultCode: 0, result: null, initMsid: 114374076129940, completeMsid: n
ull, lastUpdated: null, lastPolled: null, created: Fri May 08 15:06:03
EDT 2015}, job origin:4142
com.cloud.utils.exception.CloudRuntimeException: Unable to migrate due
to org.libvirt.LibvirtException: Cannot get interface MTU on '
breth0-1221': No such device
at
com.cloud.vm.VirtualMachineManagerImpl.migrate(VirtualMachineManagerImpl.java:1922)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrate(VirtualMachineManagerImpl.java:1822)
at
com.cloud.vm.VirtualMachineManagerImpl.orchestrateMigrate(VirtualMachineManagerImpl.java:5223)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5346)
at
com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:502)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:459)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2015-05-08 15:06:04,265 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143) Complete async
job-4143, jobStatus: FAILED, resultCode: 0, result:
rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFuZy5Fe
GNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARMAAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2
V0ABJMamF2YS9sYW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS9
1dGlsL0xpc3Q7eHBxAH4AB3QAjEpvYiBmYWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBVbmFibGUgdG8gbWlncmF0ZSBkdWUgdG8gb3JnLmxpYnZpcnQuTGlidmlydEV4Y2VwdGlv
bjogQ2Fubm90IGdldCBpbnRlcmZhY2UgTVRVIG9uICdicmV0aDAtMTIyMSc6IE5vIHN1Y2ggZGV2aWNldXIAHltMamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW50OwJGKjw8_
SI5AgAAeHAAAAANc3IAG2phdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudGEJxZomNt2FAgAESQAKbGluZU51bWJlckwADmRlY2xhcmluZ0NsYXNzcQB-AARMAAhmaWxlTmFtZX
EAfgAETAAKbWV0aG9kTmFtZXEAfgAEeHAAAABydAAgY29tLmNsb3VkLnZtLlZtV29ya0pvYkRpc3BhdGNoZXJ0ABhWbVdvcmtKb2JEaXNwYXRjaGVyLmphdmF0AAZydW5Kb2J
zcQB-AAsAAAH2dAA_b3JnLmFwYWNoZS5jbG91ZHN0YWNrLmZyYW1ld29yay5qb2JzLmltcGwuQXN5bmNKb2JNYW5hZ2VySW1wbCQ1dAAYQXN5bmNKb2JNYW5hZ2VySW1wbC5q
YXZhdAAMcnVuSW5Db250ZXh0c3EAfgALAAAAMXQAPm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZSQxdAAbTWFuY
WdlZENvbnRleHRSdW5uYWJsZS5qYXZhdAADcnVuc3EAfgALAAAAOHQAQm9yZy5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdWx0TWFuYWdlZE
NvbnRleHQkMXQAGkRlZmF1bHRNYW5hZ2VkQ29udGV4dC5qYXZhdAAEY2FsbHNxAH4ACwAAAGd0AEBvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGw
uRGVmYXVsdE1hbmFnZWRDb250ZXh0cQB-ABp0AA9jYWxsV2l0aENvbnRleHRzcQB-AAsAAAA1cQB-AB1xAH4AGnQADnJ1bldpdGhDb250ZXh0c3EAfgALAAAALnQAPG9yZy5h
cGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuTWFuYWdlZENvbnRleHRSdW5uYWJsZXEAfgAWcQB-ABdzcQB-AAsAAAHLcQB-ABFxAH4AEnEAfgAXc3EAfgALAAAB1
3QALmphdmEudXRpbC5jb25jdXJyZW50LkV4ZWN1dG9ycyRSdW5uYWJsZUFkYXB0ZXJ0AA5FeGVjdXRvcnMuamF2YXEAfgAbc3EAfgALAAABBnQAH2phdmEudXRpbC5jb25jdX
JyZW50LkZ1dHVyZVRhc2t0AA9GdXR1cmVUYXNrLmphdmFxAH4AF3NxAH4ACwAABHl0ACdqYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3J0ABdUaHJlYWR
Qb29sRXhlY3V0b3IuamF2YXQACXJ1bldvcmtlcnNxAH4ACwAAAmd0AC5qYXZhLnV0aWwuY29uY3VycmVudC5UaHJlYWRQb29sRXhlY3V0b3IkV29ya2VycQB-ACxxAH4AF3Nx
AH4ACwAAAul0ABBqYXZhLmxhbmcuVGhyZWFkdAALVGhyZWFkLmphdmFxAH4AF3NyACZqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlTGlzdPwPJTG17I4QAgABT
AAEbGlzdHEAfgAGeHIALGphdmEudXRpbC5Db2xsZWN0aW9ucyRVbm1vZGlmaWFibGVDb2xsZWN0aW9uGUIAgMte9x4CAAFMAAFjdAAWTGphdmEvdXRpbC9Db2xsZWN0aW9uO3
hwc3IAE2phdmEudXRpbC5BcnJheUxpc3R4gdIdmcdhnQMAAUkABHNpemV4cAAAAAB3BAAAAAB4cQB-ADh4
2015-05-08 15:06:04,272 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143) Done executing
com.cloud.vm.VmWorkMigrate for job-4143
2015-05-08 15:06:04,282 DEBUG [o.a.c.f.j.i.SyncQueueManagerImpl]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143) Sync queue (12
0) is currently empty
2015-05-08 15:06:04,283 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(Work-Job-Executor-23:ctx-68095d22 job-4142/job-4143) Remove job-4143
fro
m job monitoring
2015-05-08 15:06:04,286 ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-20:ctx-d79c631c job-4142) Unexpected exception while
ex
ecuting
org.apache.cloudstack.api.command.admin.systemvm.MigrateSystemVMCmd
java.lang.RuntimeException: Job failed due to exception Unable to
migrate due to org.libvirt.LibvirtException: Cannot get interface M
TU on 'breth0-1221': No such device
at
com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:502)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:459)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask.run(FutureTask.java:262)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
2015-05-08 15:06:04,287 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-20:ctx-d79c631c job-4142) Complete async job-4142,
jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Job
failed due to exception Unable to migrate due to
org.libvirt.LibvirtException: Cannot get interface MTU on
u0027breth0-1221u0027: No such device"}
2015-05-08 15:06:04,293 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-20:ctx-d79c631c job-4142) Done executing
org.apache.cloudstack.api.command.admin.systemvm.MigrateSystemVMCmd for
job-4142
2015-05-08 15:06:04,296 INFO [o.a.c.f.j.i.AsyncJobMonitor]
(API-Job-Executor-20:ctx-d79c631c job-4142) Remove job-4142 from job
monitoring
2015-05-08 15:06:04,381 DEBUG [c.c.a.t.Request]
(AgentManager-Handler-14:null) Seq 1-5821465468329789167: Processing: {
Ans: , MgmtId: 114374076129940, via: 1, Ver: v1, Flags: 10,
[{"com.cloud.agent.api.StopAnswer":{"result":true,"wait":0}}] }
2015-05-08 15:06:04,382 DEBUG [c.c.a.m.AgentAttache]
(AgentManager-Handler-14:null) Seq 1-5821465468329789167: Unable to find
listener.
2015-05-08 15:06:06,296 DEBUG [c.c.a.ApiServlet]
(http-6443-exec-13:ctx-c3e12033) ===START=== 172.16.8.151 -- GET
command=queryAsyncJobResult&jobId=4770b267-27ac-4fe5-9915-05923e9c8eaf&response=json&sessionkey=ZT0%2FJYmjsGusXAdfGH%2BGqRN8D4A%3D&_=1431111968594
2015-05-08 15:06:06,310 DEBUG [c.c.a.ApiServlet]
(http-6443-exec-13:ctx-c3e12033 ctx-ba0d1ec2) ===END=== 172.16.8.151 --
GET
command=queryAsyncJobResult&jobId=4770b267-27ac-4fe5-9915-05923e9c8eaf&response=json&sessionkey=ZT0%2FJYmjsGusXAdfGH%2BGqRN8D4A%3D&_=1431111968594
