[ https://issues.apache.org/jira/browse/CLOUDSTACK-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sateesh Chodapuneedi resolved CLOUDSTACK-2962. ---------------------------------------------- Resolution: Fixed > [SM][VMWare]: migrate a virtualMachine with volumes is failing > -------------------------------------------------------------- > > Key: CLOUDSTACK-2962 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-2962 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server > Affects Versions: 4.2.0 > Reporter: Srikanteswararao Talluri > Assignee: Sateesh Chodapuneedi > Priority: Blocker > Fix For: 4.2.0 > > Attachments: Archive.zip > > > Have a vmware deployment with two clusters in a zone and zonewide primary > storage > 1. Now try to migrate a virtual machine with volumes. > ===START=== 10.252.192.7 -- GET > command=findHostsForMigration&VirtualMachineId=88d92066-ca9c-4ecc-b204-7c07edc117a2&response=json&sessionkey=5ez1jBXiPUJzcam0%2Ft9UtTy9L%2BY%3D&_=1371038241533 > 2013-06-12 22:52:48,366 DEBUG [cloud.storage.StorageManagerImpl] > (catalina-exec-8:null) Checking pool 2 for storage, totalSize: 5902284800000, > usedBytes: 3289819828224, usedPct: 0.5573807330042766, disable threshold: 0.85 > 2013-06-12 22:52:48,447 DEBUG [cloud.server.StatsCollector] > (StatsCollector-1:null) VmStatsCollector is running... > 2013-06-12 22:52:48,464 DEBUG [cloud.storage.StorageManagerImpl] > (catalina-exec-8:null) Checking pool: 2 for volume allocation > [Vol[14|vm=8|ROOT]], maxSize : 11804569600000, totalAllocatedSize : > 2147483648, askingSize : 0, allocated disable threshold: 0.85 > 2013-06-12 22:52:48,484 DEBUG [allocator.impl.FirstFitAllocator] > (catalina-exec-8:null) FirstFitAllocator has 1 hosts to check for allocation: > [Host[-1-Routing]] > 2013-06-12 22:52:48,487 DEBUG [agent.manager.DirectAgentAttache] > (DirectAgent-439:null) Seq 1-715325919: Executing request > 2013-06-12 22:52:48,507 DEBUG [allocator.impl.FirstFitAllocator] > (catalina-exec-8:null) Found 1 hosts for allocation after prioritization: > [Host[-1-Routing]] > 2013-06-12 22:52:48,507 DEBUG [allocator.impl.FirstFitAllocator] > (catalina-exec-8:null) Looking for speed=100Mhz, Ram=512 > 2013-06-12 22:52:48,518 DEBUG [cloud.capacity.CapacityManagerImpl] > (catalina-exec-8:null) Checking if host: 1 has enough capacity for requested > CPU: 100 and requested RAM: 536870912 , cpuOverprovisioningFactor: 1.0 > 2013-06-12 22:52:48,523 DEBUG [cloud.capacity.CapacityManagerImpl] > (catalina-exec-8:null) Hosts's actual total CPU: 12368 and CPU after applying > overprovisioning: 12368 > 2013-06-12 22:52:48,523 DEBUG [cloud.capacity.CapacityManagerImpl] > (catalina-exec-8:null) considerReservedCapacity isfalse , not considering > reserved capacity for calculating free capacity > 2013-06-12 22:52:48,523 DEBUG [cloud.capacity.CapacityManagerImpl] > (catalina-exec-8:null) Free CPU: 10768 , Requested CPU: 100 > 2013-06-12 22:52:48,524 DEBUG [cloud.capacity.CapacityManagerImpl] > (catalina-exec-8:null) Free RAM: 6565937152 , Requested RAM: 536870912 > 2013-06-12 22:52:48,524 DEBUG [cloud.capacity.CapacityManagerImpl] > (catalina-exec-8:null) Host has enough CPU and RAM available > 2013-06-12 22:52:48,524 DEBUG [cloud.capacity.CapacityManagerImpl] > (catalina-exec-8:null) STATS: Can alloc CPU from host: 1, used: 1600, > reserved: 0, actual total: 12368, total with overprovisioning: 12368; > requested cpu:100,alloc_from_last_host?:false ,considerReservedCapacity?: > false > 2013-06-12 22:52:48,524 DEBUG [cloud.capacity.CapacityManagerImpl] > (catalina-exec-8:null) STATS: Can alloc MEM from host: 1, used: 2013265920, > reserved: 0, total: 8579203072; requested mem: > 536870912,alloc_from_last_host?:false ,considerReservedCapacity?: false > 2013-06-12 22:52:48,524 DEBUG [allocator.impl.FirstFitAllocator] > (catalina-exec-8:null) Found a suitable host, adding to list: 1 > 2013-06-12 22:52:48,524 DEBUG [allocator.impl.FirstFitAllocator] > (catalina-exec-8:null) Host Allocator returning 1 suitable hosts > 2013-06-12 22:52:48,524 DEBUG [cloud.server.ManagementServerImpl] > (catalina-exec-8:null) Hosts having capacity and suitable for migration: > [Host[-1-Routing]] > 2013-06-12 22:52:48,561 DEBUG [cloud.api.ApiServlet] (catalina-exec-8:null) > ===END=== 10.252.192.7 -- GET > command=findHostsForMigration&VirtualMachineId=88d92066-ca9c-4ecc-b204-7c07edc117a2&response=json&sessionkey=5ez1jBXiPUJzcam0%2Ft9UtTy9L%2BY%3D&_=1371038241533 > 2013-06-12 22:52:48,598 DEBUG [vmware.mo.HostMO] > (DirectAgent-439:10.147.40.24) find VM i-2-9-VM on host > 2013-06-12 22:52:48,599 DEBUG [vmware.mo.HostMO] > (DirectAgent-439:10.147.40.24) load VM cache on host > 2013-06-12 22:52:48,608 DEBUG [agent.manager.DirectAgentAttache] > (DirectAgent-439:null) Seq 1-715325919: Response Received: > 2013-06-12 22:52:48,609 DEBUG [agent.transport.Request] > (StatsCollector-1:null) Seq 1-715325919: Received: { Ans: , MgmtId: > 7566222426160, via: 1, Ver: v1, Flags: 10, { GetVmStatsAnswer } } > 2013-06-12 22:52:48,637 DEBUG [agent.manager.DirectAgentAttache] > (DirectAgent-5:null) Seq 6-835518759: Executing request > 2013-06-12 22:52:48,716 DEBUG [vmware.mo.HostMO] (DirectAgent-5:10.147.40.30) > find VM i-2-7-VM on host > 2013-06-12 22:52:48,717 DEBUG [vmware.mo.HostMO] (DirectAgent-5:10.147.40.30) > load VM cache on host > 2013-06-12 22:52:48,733 DEBUG [vmware.mo.HostMO] (DirectAgent-5:10.147.40.30) > find VM i-2-8-VM on host > 2013-06-12 22:52:48,733 DEBUG [vmware.mo.HostMO] (DirectAgent-5:10.147.40.30) > VM i-2-8-VM found in host cache > 2013-06-12 22:52:48,738 DEBUG [agent.manager.DirectAgentAttache] > (DirectAgent-5:null) Seq 6-835518759: Response Received: > 2013-06-12 22:52:48,738 DEBUG [agent.transport.Request] > (StatsCollector-1:null) Seq 6-835518759: Received: { Ans: , MgmtId: > 7566222426160, via: 6, Ver: v1, Flags: 10, { GetVmStatsAnswer } } > 2013-06-12 22:52:48,926 DEBUG [cloud.api.ApiServlet] (catalina-exec-1:null) > ===START=== 10.252.192.7 -- GET > command=queryAsyncJobResult&jobId=8c775230-f312-428d-a77e-248d9f0d107c&response=json&sessionkey=5ez1jBXiPUJzcam0%2Ft9UtTy9L%2BY%3D&_=1371038242151 > 2013-06-12 22:52:48,947 DEBUG [cloud.async.AsyncJobManagerImpl] > (catalina-exec-1:null) Async job-35 completed > 2013-06-12 22:52:48,956 DEBUG [cloud.api.ApiServlet] (catalina-exec-1:null) > ===END=== 10.252.192.7 -- GET > command=queryAsyncJobResult&jobId=8c775230-f312-428d-a77e-248d9f0d107c&response=json&sessionkey=5ez1jBXiPUJzcam0%2Ft9UtTy9L%2BY%3D&_=1371038242151 > 2013-06-12 22:52:49,842 DEBUG [cloud.api.ApiServlet] (catalina-exec-15:null) > ===START=== 10.252.192.7 -- GET > command=migrateVirtualMachineWithVolume&hostid=2e452c2f-1898-4784-b971-937e357fc781&virtualmachineid=88d92066-ca9c-4ecc-b204-7c07edc117a2&response=json&sessionkey=5ez1jBXiPUJzcam0%2Ft9UtTy9L%2BY%3D&_=1371038243067 > 2013-06-12 22:52:49,892 DEBUG [cloud.async.AsyncJobManagerImpl] > (catalina-exec-15:null) submit async job-51, details: AsyncJobVO {id:51, > userId: 2, accountId: 2, sessionKey: null, instanceType: None, instanceId: > null, cmd: > org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd, > cmdOriginator: null, cmdInfo: > {"response":"json","sessionkey":"5ez1jBXiPUJzcam0/t9UtTy9L+Y\u003d","virtualmachineid":"88d92066-ca9c-4ecc-b204-7c07edc117a2","hostid":"2e452c2f-1898-4784-b971-937e357fc781","ctxUserId":"2","httpmethod":"GET","_":"1371038243067","ctxAccountId":"2","ctxStartEventId":"180"}, > cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0, > processStatus: 0, resultCode: 0, result: null, initMsid: 7566222426160, > completeMsid: null, lastUpdated: null, lastPolled: null, created: null} > 2013-06-12 22:52:49,896 DEBUG [cloud.api.ApiServlet] (catalina-exec-15:null) > ===END=== 10.252.192.7 -- GET > command=migrateVirtualMachineWithVolume&hostid=2e452c2f-1898-4784-b971-937e357fc781&virtualmachineid=88d92066-ca9c-4ecc-b204-7c07edc117a2&response=json&sessionkey=5ez1jBXiPUJzcam0%2Ft9UtTy9L%2BY%3D&_=1371038243067 > 2013-06-12 22:52:49,901 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-39:job-51) Executing > org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd > for job-51 > 2013-06-12 22:52:50,002 DEBUG [cloud.storage.StorageManagerImpl] > (Job-Executor-39:job-51) Checking pool 2 for storage, totalSize: > 5902284800000, usedBytes: 3289819828224, usedPct: 0.5573807330042766, disable > threshold: 0.85 > 2013-06-12 22:52:50,037 DEBUG [cloud.storage.StorageManagerImpl] > (Job-Executor-39:job-51) Checking pool: 2 for volume allocation > [Vol[14|vm=8|ROOT]], maxSize : 11804569600000, totalAllocatedSize : > 2147483648, askingSize : 0, allocated disable threshold: 0.85 > 2013-06-12 22:52:50,061 DEBUG [cloud.network.NetworkModelImpl] > (Job-Executor-39:job-51) Service SecurityGroup is not supported in the > network id=204 > 2013-06-12 22:52:50,069 DEBUG [cloud.storage.VolumeManagerImpl] > (Job-Executor-39:job-51) Preparing 1 volumes for VM[User|prim2] > 2013-06-12 22:52:50,091 DEBUG [hypervisor.guru.VMwareGuru] > (Job-Executor-39:job-51) Nested virtualization requested, adding flag to vm > configuration > 2013-06-12 22:52:50,121 DEBUG [cloud.capacity.CapacityManagerImpl] > (Job-Executor-39:job-51) VM state transitted from :Running to Migrating with > event: MigrationRequestedvm's original host id: 6 new host id: 1 host id > before state transition: 6 > 2013-06-12 22:52:50,135 DEBUG [cloud.capacity.CapacityManagerImpl] > (Job-Executor-39:job-51) Hosts's actual total CPU: 12368 and CPU after > applying overprovisioning: 12368 > 2013-06-12 22:52:50,136 DEBUG [cloud.capacity.CapacityManagerImpl] > (Job-Executor-39:job-51) We are allocating VM, increasing the used capacity > of this host:1 > 2013-06-12 22:52:50,136 DEBUG [cloud.capacity.CapacityManagerImpl] > (Job-Executor-39:job-51) Current Used CPU: 1600 , Free CPU:10768 ,Requested > CPU: 100 > 2013-06-12 22:52:50,136 DEBUG [cloud.capacity.CapacityManagerImpl] > (Job-Executor-39:job-51) Current Used RAM: 2013265920 , Free RAM:6565937152 > ,Requested RAM: 536870912 > 2013-06-12 22:52:50,136 DEBUG [cloud.capacity.CapacityManagerImpl] > (Job-Executor-39:job-51) CPU STATS after allocation: for host: 1, old used: > 1600, old reserved: 0, actual total: 12368, total with overprovisioning: > 12368; new used:1700, reserved:0; requested cpu:100,alloc_from_last:false > 2013-06-12 22:52:50,136 DEBUG [cloud.capacity.CapacityManagerImpl] > (Job-Executor-39:job-51) RAM STATS after allocation: for host: 1, old used: > 2013265920, old reserved: 0, total: 8579203072; new used: 2550136832, > reserved: 0; requested mem: 536870912,alloc_from_last:false > 2013-06-12 22:52:50,193 DEBUG [agent.transport.Request] > (Job-Executor-39:job-51) Seq 6-835518760: Sending { Cmd , MgmtId: > 7566222426160, via: 6, Ver: v1, Flags: 100111, > [{"MigrateWithStorageCommand":{"vm":{"id":8,"name":"i-2-8-VM","bootloader":"HVM","type":"User","cpus":1,"minSpeed":100,"maxSpeed":100,"minRam":536870912,"maxRam":536870912,"arch":"x86_64","os":"CentOS > 5.3 > (64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"409dc14c63c9fb0f","params":{"rootDiskController":"ide","nicAdapter":"E1000","nestedVirtualizationFlag":"false"},"uuid":"88d92066-ca9c-4ecc-b204-7c07edc117a2","disks":[{"id":14,"name":"ROOT-8","mountPoint":"/export/home/talluri/mysetup_prim2","path":"ROOT-8-14","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"ece502dc-5d06-3043-afec-38e3d54fe995","deviceId":0}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"90463686-3e63-4a3c-99f7-5ab36bd8a1a8","ip":"10.1.1.120","netmask":"255.255.255.0","gateway":"10.1.1.1","mac":"02:00:70:f1:00:05","dns1":"10.103.128.16","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://944","isolationUri":"vlan://944","isSecurityGroupEnabled":false}]},"volumeToFiler":{"Vol[14|ROOT|ROOT-8-14|2147483648]":{"id":2,"uuid":"ece502dc-5d06-3043-afec-38e3d54fe995","host":"10.147.28.7","path":"/export/home/talluri/mysetup_prim2","port":2049,"type":"NetworkFilesystem"}},"tgtHost":"HostSystem:host-7108@10.147.60.15","wait":0}}] > } > 2013-06-12 22:52:50,195 DEBUG [agent.transport.Request] > (Job-Executor-39:job-51) Seq 6-835518760: Executing: { Cmd , MgmtId: > 7566222426160, via: 6, Ver: v1, Flags: 100111, > [{"MigrateWithStorageCommand":{"vm":{"id":8,"name":"i-2-8-VM","bootloader":"HVM","type":"User","cpus":1,"minSpeed":100,"maxSpeed":100,"minRam":536870912,"maxRam":536870912,"arch":"x86_64","os":"CentOS > 5.3 > (64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"409dc14c63c9fb0f","params":{"rootDiskController":"ide","nicAdapter":"E1000","nestedVirtualizationFlag":"false"},"uuid":"88d92066-ca9c-4ecc-b204-7c07edc117a2","disks":[{"id":14,"name":"ROOT-8","mountPoint":"/export/home/talluri/mysetup_prim2","path":"ROOT-8-14","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"ece502dc-5d06-3043-afec-38e3d54fe995","deviceId":0}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"90463686-3e63-4a3c-99f7-5ab36bd8a1a8","ip":"10.1.1.120","netmask":"255.255.255.0","gateway":"10.1.1.1","mac":"02:00:70:f1:00:05","dns1":"10.103.128.16","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://944","isolationUri":"vlan://944","isSecurityGroupEnabled":false}]},"volumeToFiler":{"Vol[14|ROOT|ROOT-8-14|2147483648]":{"id":2,"uuid":"ece502dc-5d06-3043-afec-38e3d54fe995","host":"10.147.28.7","path":"/export/home/talluri/mysetup_prim2","port":2049,"type":"NetworkFilesystem"}},"tgtHost":"HostSystem:host-7108@10.147.60.15","wait":0}}] > } > 2013-06-12 22:52:50,212 DEBUG [agent.manager.DirectAgentAttache] > (DirectAgent-327:null) Seq 6-835518760: Executing request > 2013-06-12 22:52:50,224 INFO [vmware.resource.VmwareResource] > (DirectAgent-327:10.147.40.30) Executing resource MigrateWithStorageCommand: > {"vm":{"id":8,"name":"i-2-8-VM","bootloader":"HVM","type":"User","cpus":1,"minSpeed":100,"maxSpeed":100,"minRam":536870912,"maxRam":536870912,"arch":"x86_64","os":"CentOS > 5.3 > (64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"409dc14c63c9fb0f","params":{"rootDiskController":"ide","nicAdapter":"E1000","nestedVirtualizationFlag":"false"},"uuid":"88d92066-ca9c-4ecc-b204-7c07edc117a2","disks":[{"id":14,"name":"ROOT-8","mountPoint":"/export/home/talluri/mysetup_prim2","path":"ROOT-8-14","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"ece502dc-5d06-3043-afec-38e3d54fe995","deviceId":0}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"90463686-3e63-4a3c-99f7-5ab36bd8a1a8","ip":"10.1.1.120","netmask":"255.255.255.0","gateway":"10.1.1.1","mac":"02:00:70:f1:00:05","dns1":"10.103.128.16","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://944","isolationUri":"vlan://944","isSecurityGroupEnabled":false}]},"volumeToFiler":{"Vol[14|ROOT|ROOT-8-14|2147483648]":{"id":2,"uuid":"ece502dc-5d06-3043-afec-38e3d54fe995","host":"10.147.28.7","path":"/export/home/talluri/mysetup_prim2","port":2049,"type":"NetworkFilesystem"}},"tgtHost":"HostSystem:host-7108@10.147.60.15","wait":0} > 2013-06-12 22:52:50,237 WARN [vmware.resource.VmwareResource] > (DirectAgent-327:10.147.40.30) MigrationCommand failed due to Exception: > com.cloud.utils.exception.CloudRuntimeException > Message: Source host & target host are in different datacentesr > com.cloud.utils.exception.CloudRuntimeException: Source host & target host > are in different datacentesr > at > com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3494) > at > com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:418) > at > com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > 2013-06-12 22:52:50,241 DEBUG [agent.manager.DirectAgentAttache] > (DirectAgent-327:null) Seq 6-835518760: Response Received: > 2013-06-12 22:52:50,241 DEBUG [agent.transport.Request] > (DirectAgent-327:null) Seq 6-835518760: Processing: { Ans: , MgmtId: > 7566222426160, via: 6, Ver: v1, Flags: 110, > [{"MigrateWithStorageAnswer":{"result":false,"details":"Exception: > com.cloud.utils.exception.CloudRuntimeException\nMessage: Source host & > target host are in different datacentesr\nStack: > com.cloud.utils.exception.CloudRuntimeException: Source host & target host > are in different datacentesr\n\tat > com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3494)\n\tat > > com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:418)\n\tat > > com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)\n\tat > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat > java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)\n\tat > java.util.concurrent.FutureTask.run(FutureTask.java:166)\n\tat > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)\n\tat > > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)\n\tat > > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)\n\tat > > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)\n\tat > java.lang.Thread.run(Thread.java:679)\n","wait":0}}] } > 2013-06-12 22:52:50,242 DEBUG [agent.transport.Request] > (Job-Executor-39:job-51) Seq 6-835518760: Received: { Ans: , MgmtId: > 7566222426160, via: 6, Ver: v1, Flags: 110, { MigrateWithStorageAnswer } } > 2013-06-12 22:52:50,242 ERROR [storage.motion.VmwareStorageMotionStrategy] > (Job-Executor-39:job-51) Migration with storage of vm VM[User|prim2] failed. > Details: Exception: com.cloud.utils.exception.CloudRuntimeException > Message: Source host & target host are in different datacentesr > Stack: com.cloud.utils.exception.CloudRuntimeException: Source host & target > host are in different datacentesr > at > com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3494) > at > com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:418) > at > com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > 2013-06-12 22:52:50,243 ERROR [storage.motion.VmwareStorageMotionStrategy] > (Job-Executor-39:job-51) copy failed > com.cloud.utils.exception.CloudRuntimeException: Error while migrating the vm > VM[User|prim2] to host Host[-1-Routing]. Exception: > com.cloud.utils.exception.CloudRuntimeException > Message: Source host & target host are in different datacentesr > Stack: com.cloud.utils.exception.CloudRuntimeException: Source host & target > host are in different datacentesr > at > com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3494) > at > com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:418) > at > com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > at > org.apache.cloudstack.storage.motion.VmwareStorageMotionStrategy.migrateVmWithVolumesAcrossCluster(VmwareStorageMotionStrategy.java:147) > at > org.apache.cloudstack.storage.motion.VmwareStorageMotionStrategy.copyAsync(VmwareStorageMotionStrategy.java:106) > at > org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:71) > at > org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:693) > at > com.cloud.storage.VolumeManagerImpl.migrateVolumes(VolumeManagerImpl.java:2171) > at > com.cloud.vm.VirtualMachineManagerImpl.migrateWithStorage(VirtualMachineManagerImpl.java:1676) > at > com.cloud.vm.UserVmManagerImpl.migrateVirtualMachineWithVolume(UserVmManagerImpl.java:4056) > at > com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at > org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd.execute(MigrateVirtualMachineWithVolumeCmd.java:137) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155) > at > com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > 2013-06-12 22:52:50,246 DEBUG [agent.manager.AgentAttache] > (DirectAgent-327:null) Seq 6-835518760: No more commands found > 2013-06-12 22:52:50,259 DEBUG [cloud.storage.VolumeManagerImpl] > (Job-Executor-39:job-51) Failed to migrated vm VM[User|prim2] along with its > volumes. com.cloud.utils.exception.CloudRuntimeException: Error while > migrating the vm VM[User|prim2] to host Host[-1-Routing]. Exception: > com.cloud.utils.exception.CloudRuntimeException > Message: Source host & target host are in different datacentesr > Stack: com.cloud.utils.exception.CloudRuntimeException: Source host & target > host are in different datacentesr > at > com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3494) > at > com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:418) > at > com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > 2013-06-12 22:52:50,259 INFO [cloud.vm.VirtualMachineManagerImpl] > (Job-Executor-39:job-51) Migration was unsuccessful. Cleaning up: > VM[User|prim2] > 2013-06-12 22:52:50,260 WARN [apache.cloudstack.alerts] > (Job-Executor-39:job-51) alertType:: 17 // dataCenterId:: 1 // podId:: 1 // > clusterId:: null // message:: Unable to migrate vm i-2-8-VM from host > 10.147.40.30 in zone zwpszone and pod zwpszone > 2013-06-12 22:52:50,264 DEBUG [cloud.alert.AlertManagerImpl] > (Job-Executor-39:job-51) Have already sent: 1 emails for alert type '17' -- > skipping send email > 2013-06-12 22:52:50,267 DEBUG [agent.manager.AgentAttache] > (Job-Executor-39:job-51) Seq 1-715325920: Routed from 7566222426160 > 2013-06-12 22:52:50,267 DEBUG [agent.transport.Request] > (Job-Executor-39:job-51) Seq 1-715325920: Sending { Cmd , MgmtId: > 7566222426160, via: 1, Ver: v1, Flags: 100111, > [{"StopCommand":{"isProxy":false,"vmName":"i-2-8-VM","wait":0}}] } > 2013-06-12 22:52:50,268 DEBUG [agent.transport.Request] > (Job-Executor-39:job-51) Seq 1-715325920: Executing: { Cmd , MgmtId: > 7566222426160, via: 1, Ver: v1, Flags: 100111, > [{"StopCommand":{"isProxy":false,"vmName":"i-2-8-VM","wait":0}}] } > 2013-06-12 22:52:50,269 DEBUG [agent.manager.DirectAgentAttache] > (DirectAgent-440:null) Seq 1-715325920: Executing request > 2013-06-12 22:52:50,270 INFO [vmware.resource.VmwareResource] > (DirectAgent-440:10.147.40.24) Executing resource StopCommand: > {"isProxy":false,"vmName":"i-2-8-VM","wait":0} > 2013-06-12 22:52:50,273 DEBUG [vmware.mo.HostMO] > (DirectAgent-440:10.147.40.24) find VM i-2-8-VM on host > 2013-06-12 22:52:50,273 DEBUG [vmware.mo.HostMO] > (DirectAgent-440:10.147.40.24) load VM cache on host > 2013-06-12 22:52:50,291 INFO [vmware.resource.VmwareResource] > (DirectAgent-440:10.147.40.24) VM i-2-8-VM is no longer in vSphere > 2013-06-12 22:52:50,292 DEBUG [agent.manager.DirectAgentAttache] > (DirectAgent-440:null) Seq 1-715325920: Response Received: > 2013-06-12 22:52:50,293 DEBUG [agent.transport.Request] > (DirectAgent-440:null) Seq 1-715325920: Processing: { Ans: , MgmtId: > 7566222426160, via: 1, Ver: v1, Flags: 110, > [{"StopAnswer":{"vncPort":0,"result":true,"details":"VM i-2-8-VM is no longer > in vSphere","wait":0}}] } > 2013-06-12 22:52:50,294 DEBUG [agent.manager.AgentAttache] > (DirectAgent-440:null) Seq 1-715325920: Unable to find listener. > 2013-06-12 22:52:50,294 DEBUG [agent.manager.AgentAttache] > (DirectAgent-440:null) Seq 1-715325920: No more commands found > 2013-06-12 22:52:50,297 DEBUG [cloud.capacity.CapacityManagerImpl] > (Job-Executor-39:job-51) VM state transitted from :Migrating to Running with > event: OperationFailedvm's original host id: 6 new host id: 6 host id before > state transition: 1 > 2013-06-12 22:52:50,310 DEBUG [cloud.capacity.CapacityManagerImpl] > (Job-Executor-39:job-51) Hosts's actual total CPU: 12368 and CPU after > applying overprovisioning: 12368 > 2013-06-12 22:52:50,310 DEBUG [cloud.capacity.CapacityManagerImpl] > (Job-Executor-39:job-51) Hosts's actual total RAM: 8579203072 and RAM after > applying overprovisioning: 8579203072 > 2013-06-12 22:52:50,310 DEBUG [cloud.capacity.CapacityManagerImpl] > (Job-Executor-39:job-51) release cpu from host: 1, old used: 1700,reserved: > 0, actual total: 12368, total with overprovisioning: 12368; new used: > 1600,reserved:0; movedfromreserved: false,moveToReserveredfalse > 2013-06-12 22:52:50,311 DEBUG [cloud.capacity.CapacityManagerImpl] > (Job-Executor-39:job-51) release mem from host: 1, old used: > 2550136832,reserved: 0, total: 8579203072; new used: 2013265920,reserved:0; > movedfromreserved: false,moveToReserveredfalse > 2013-06-12 22:52:50,336 ERROR [cloud.async.AsyncJobManagerImpl] > (Job-Executor-39:job-51) Unexpected exception while executing > org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd > com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm > VM[User|prim2] along with its volumes. > com.cloud.utils.exception.CloudRuntimeException: Error while migrating the vm > VM[User|prim2] to host Host[-1-Routing]. Exception: > com.cloud.utils.exception.CloudRuntimeException > Message: Source host & target host are in different datacentesr > Stack: com.cloud.utils.exception.CloudRuntimeException: Source host & target > host are in different datacentesr > at > com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3494) > at > com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:418) > at > com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > at > com.cloud.storage.VolumeManagerImpl.migrateVolumes(VolumeManagerImpl.java:2176) > at > com.cloud.vm.VirtualMachineManagerImpl.migrateWithStorage(VirtualMachineManagerImpl.java:1676) > at > com.cloud.vm.UserVmManagerImpl.migrateVirtualMachineWithVolume(UserVmManagerImpl.java:4056) > at > com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125) > at > org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd.execute(MigrateVirtualMachineWithVolumeCmd.java:137) > at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155) > at > com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > 2013-06-12 22:52:50,338 DEBUG [cloud.async.AsyncJobManagerImpl] > (Job-Executor-39:job-51) Complete async job-51, jobStatus: 2, resultCode: > 530, result: Error Code: 530 Error text: Failed to migrated vm VM[User|prim2] > along with its volumes. com.cloud.utils.exception.CloudRuntimeException: > Error while migrating the vm VM[User|prim2] to host Host[-1-Routing]. > Exception: com.cloud.utils.exception.CloudRuntimeException > Message: Source host & target host are in different datacentesr > Stack: com.cloud.utils.exception.CloudRuntimeException: Source host & target > host are in different datacentesr > at > com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3494) > at > com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:418) > at > com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > 2013-06-12 22:52:51,583 DEBUG [agent.manager.AgentManagerImpl] > (AgentManager-Handler-15:null) SeqA 3-882: Processing Seq 3-882: { Cmd , > MgmtId: -1, via: 3, Ver: v1, Flags: 11, [{"Con -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira