Srikanteswararao Talluri created CLOUDSTACK-2962:
----------------------------------------------------
Summary: [SM][VMWare]: migrate a virtualMachine with volumes is
failing
Key: CLOUDSTACK-2962
URL: https://issues.apache.org/jira/browse/CLOUDSTACK-2962
Project: CloudStack
Issue Type: Bug
Security Level: Public (Anyone can view this level - this is the default.)
Components: Management Server
Affects Versions: 4.2.0
Reporter: Srikanteswararao Talluri
Priority: Blocker
Fix For: 4.2.0
Have a vmware deployment with two clusters in a zone and zonewide primary
storage
1. Now try to migrate a virtual machine with volumes.
===START=== 10.252.192.7 -- GET
command=findHostsForMigration&VirtualMachineId=88d92066-ca9c-4ecc-b204-7c07edc117a2&response=json&sessionkey=5ez1jBXiPUJzcam0%2Ft9UtTy9L%2BY%3D&_=1371038241533
2013-06-12 22:52:48,366 DEBUG [cloud.storage.StorageManagerImpl]
(catalina-exec-8:null) Checking pool 2 for storage, totalSize: 5902284800000,
usedBytes: 3289819828224, usedPct: 0.5573807330042766, disable threshold: 0.85
2013-06-12 22:52:48,447 DEBUG [cloud.server.StatsCollector]
(StatsCollector-1:null) VmStatsCollector is running...
2013-06-12 22:52:48,464 DEBUG [cloud.storage.StorageManagerImpl]
(catalina-exec-8:null) Checking pool: 2 for volume allocation
[Vol[14|vm=8|ROOT]], maxSize : 11804569600000, totalAllocatedSize : 2147483648,
askingSize : 0, allocated disable threshold: 0.85
2013-06-12 22:52:48,484 DEBUG [allocator.impl.FirstFitAllocator]
(catalina-exec-8:null) FirstFitAllocator has 1 hosts to check for allocation:
[Host[-1-Routing]]
2013-06-12 22:52:48,487 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-439:null) Seq 1-715325919: Executing request
2013-06-12 22:52:48,507 DEBUG [allocator.impl.FirstFitAllocator]
(catalina-exec-8:null) Found 1 hosts for allocation after prioritization:
[Host[-1-Routing]]
2013-06-12 22:52:48,507 DEBUG [allocator.impl.FirstFitAllocator]
(catalina-exec-8:null) Looking for speed=100Mhz, Ram=512
2013-06-12 22:52:48,518 DEBUG [cloud.capacity.CapacityManagerImpl]
(catalina-exec-8:null) Checking if host: 1 has enough capacity for requested
CPU: 100 and requested RAM: 536870912 , cpuOverprovisioningFactor: 1.0
2013-06-12 22:52:48,523 DEBUG [cloud.capacity.CapacityManagerImpl]
(catalina-exec-8:null) Hosts's actual total CPU: 12368 and CPU after applying
overprovisioning: 12368
2013-06-12 22:52:48,523 DEBUG [cloud.capacity.CapacityManagerImpl]
(catalina-exec-8:null) considerReservedCapacity isfalse , not considering
reserved capacity for calculating free capacity
2013-06-12 22:52:48,523 DEBUG [cloud.capacity.CapacityManagerImpl]
(catalina-exec-8:null) Free CPU: 10768 , Requested CPU: 100
2013-06-12 22:52:48,524 DEBUG [cloud.capacity.CapacityManagerImpl]
(catalina-exec-8:null) Free RAM: 6565937152 , Requested RAM: 536870912
2013-06-12 22:52:48,524 DEBUG [cloud.capacity.CapacityManagerImpl]
(catalina-exec-8:null) Host has enough CPU and RAM available
2013-06-12 22:52:48,524 DEBUG [cloud.capacity.CapacityManagerImpl]
(catalina-exec-8:null) STATS: Can alloc CPU from host: 1, used: 1600, reserved:
0, actual total: 12368, total with overprovisioning: 12368; requested
cpu:100,alloc_from_last_host?:false ,considerReservedCapacity?: false
2013-06-12 22:52:48,524 DEBUG [cloud.capacity.CapacityManagerImpl]
(catalina-exec-8:null) STATS: Can alloc MEM from host: 1, used: 2013265920,
reserved: 0, total: 8579203072; requested mem:
536870912,alloc_from_last_host?:false ,considerReservedCapacity?: false
2013-06-12 22:52:48,524 DEBUG [allocator.impl.FirstFitAllocator]
(catalina-exec-8:null) Found a suitable host, adding to list: 1
2013-06-12 22:52:48,524 DEBUG [allocator.impl.FirstFitAllocator]
(catalina-exec-8:null) Host Allocator returning 1 suitable hosts
2013-06-12 22:52:48,524 DEBUG [cloud.server.ManagementServerImpl]
(catalina-exec-8:null) Hosts having capacity and suitable for migration:
[Host[-1-Routing]]
2013-06-12 22:52:48,561 DEBUG [cloud.api.ApiServlet] (catalina-exec-8:null)
===END=== 10.252.192.7 -- GET
command=findHostsForMigration&VirtualMachineId=88d92066-ca9c-4ecc-b204-7c07edc117a2&response=json&sessionkey=5ez1jBXiPUJzcam0%2Ft9UtTy9L%2BY%3D&_=1371038241533
2013-06-12 22:52:48,598 DEBUG [vmware.mo.HostMO] (DirectAgent-439:10.147.40.24)
find VM i-2-9-VM on host
2013-06-12 22:52:48,599 DEBUG [vmware.mo.HostMO] (DirectAgent-439:10.147.40.24)
load VM cache on host
2013-06-12 22:52:48,608 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-439:null) Seq 1-715325919: Response Received:
2013-06-12 22:52:48,609 DEBUG [agent.transport.Request] (StatsCollector-1:null)
Seq 1-715325919: Received: { Ans: , MgmtId: 7566222426160, via: 1, Ver: v1,
Flags: 10, { GetVmStatsAnswer } }
2013-06-12 22:52:48,637 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-5:null) Seq 6-835518759: Executing request
2013-06-12 22:52:48,716 DEBUG [vmware.mo.HostMO] (DirectAgent-5:10.147.40.30)
find VM i-2-7-VM on host
2013-06-12 22:52:48,717 DEBUG [vmware.mo.HostMO] (DirectAgent-5:10.147.40.30)
load VM cache on host
2013-06-12 22:52:48,733 DEBUG [vmware.mo.HostMO] (DirectAgent-5:10.147.40.30)
find VM i-2-8-VM on host
2013-06-12 22:52:48,733 DEBUG [vmware.mo.HostMO] (DirectAgent-5:10.147.40.30)
VM i-2-8-VM found in host cache
2013-06-12 22:52:48,738 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-5:null) Seq 6-835518759: Response Received:
2013-06-12 22:52:48,738 DEBUG [agent.transport.Request] (StatsCollector-1:null)
Seq 6-835518759: Received: { Ans: , MgmtId: 7566222426160, via: 6, Ver: v1,
Flags: 10, { GetVmStatsAnswer } }
2013-06-12 22:52:48,926 DEBUG [cloud.api.ApiServlet] (catalina-exec-1:null)
===START=== 10.252.192.7 -- GET
command=queryAsyncJobResult&jobId=8c775230-f312-428d-a77e-248d9f0d107c&response=json&sessionkey=5ez1jBXiPUJzcam0%2Ft9UtTy9L%2BY%3D&_=1371038242151
2013-06-12 22:52:48,947 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-1:null) Async job-35 completed
2013-06-12 22:52:48,956 DEBUG [cloud.api.ApiServlet] (catalina-exec-1:null)
===END=== 10.252.192.7 -- GET
command=queryAsyncJobResult&jobId=8c775230-f312-428d-a77e-248d9f0d107c&response=json&sessionkey=5ez1jBXiPUJzcam0%2Ft9UtTy9L%2BY%3D&_=1371038242151
2013-06-12 22:52:49,842 DEBUG [cloud.api.ApiServlet] (catalina-exec-15:null)
===START=== 10.252.192.7 -- GET
command=migrateVirtualMachineWithVolume&hostid=2e452c2f-1898-4784-b971-937e357fc781&virtualmachineid=88d92066-ca9c-4ecc-b204-7c07edc117a2&response=json&sessionkey=5ez1jBXiPUJzcam0%2Ft9UtTy9L%2BY%3D&_=1371038243067
2013-06-12 22:52:49,892 DEBUG [cloud.async.AsyncJobManagerImpl]
(catalina-exec-15:null) submit async job-51, details: AsyncJobVO {id:51,
userId: 2, accountId: 2, sessionKey: null, instanceType: None, instanceId:
null, cmd:
org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd,
cmdOriginator: null, cmdInfo:
{"response":"json","sessionkey":"5ez1jBXiPUJzcam0/t9UtTy9L+Y\u003d","virtualmachineid":"88d92066-ca9c-4ecc-b204-7c07edc117a2","hostid":"2e452c2f-1898-4784-b971-937e357fc781","ctxUserId":"2","httpmethod":"GET","_":"1371038243067","ctxAccountId":"2","ctxStartEventId":"180"},
cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
processStatus: 0, resultCode: 0, result: null, initMsid: 7566222426160,
completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
2013-06-12 22:52:49,896 DEBUG [cloud.api.ApiServlet] (catalina-exec-15:null)
===END=== 10.252.192.7 -- GET
command=migrateVirtualMachineWithVolume&hostid=2e452c2f-1898-4784-b971-937e357fc781&virtualmachineid=88d92066-ca9c-4ecc-b204-7c07edc117a2&response=json&sessionkey=5ez1jBXiPUJzcam0%2Ft9UtTy9L%2BY%3D&_=1371038243067
2013-06-12 22:52:49,901 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-39:job-51) Executing
org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd
for job-51
2013-06-12 22:52:50,002 DEBUG [cloud.storage.StorageManagerImpl]
(Job-Executor-39:job-51) Checking pool 2 for storage, totalSize: 5902284800000,
usedBytes: 3289819828224, usedPct: 0.5573807330042766, disable threshold: 0.85
2013-06-12 22:52:50,037 DEBUG [cloud.storage.StorageManagerImpl]
(Job-Executor-39:job-51) Checking pool: 2 for volume allocation
[Vol[14|vm=8|ROOT]], maxSize : 11804569600000, totalAllocatedSize : 2147483648,
askingSize : 0, allocated disable threshold: 0.85
2013-06-12 22:52:50,061 DEBUG [cloud.network.NetworkModelImpl]
(Job-Executor-39:job-51) Service SecurityGroup is not supported in the network
id=204
2013-06-12 22:52:50,069 DEBUG [cloud.storage.VolumeManagerImpl]
(Job-Executor-39:job-51) Preparing 1 volumes for VM[User|prim2]
2013-06-12 22:52:50,091 DEBUG [hypervisor.guru.VMwareGuru]
(Job-Executor-39:job-51) Nested virtualization requested, adding flag to vm
configuration
2013-06-12 22:52:50,121 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-39:job-51) VM state transitted from :Running to Migrating with
event: MigrationRequestedvm's original host id: 6 new host id: 1 host id before
state transition: 6
2013-06-12 22:52:50,135 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-39:job-51) Hosts's actual total CPU: 12368 and CPU after applying
overprovisioning: 12368
2013-06-12 22:52:50,136 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-39:job-51) We are allocating VM, increasing the used capacity of
this host:1
2013-06-12 22:52:50,136 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-39:job-51) Current Used CPU: 1600 , Free CPU:10768 ,Requested
CPU: 100
2013-06-12 22:52:50,136 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-39:job-51) Current Used RAM: 2013265920 , Free RAM:6565937152
,Requested RAM: 536870912
2013-06-12 22:52:50,136 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-39:job-51) CPU STATS after allocation: for host: 1, old used:
1600, old reserved: 0, actual total: 12368, total with overprovisioning: 12368;
new used:1700, reserved:0; requested cpu:100,alloc_from_last:false
2013-06-12 22:52:50,136 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-39:job-51) RAM STATS after allocation: for host: 1, old used:
2013265920, old reserved: 0, total: 8579203072; new used: 2550136832, reserved:
0; requested mem: 536870912,alloc_from_last:false
2013-06-12 22:52:50,193 DEBUG [agent.transport.Request]
(Job-Executor-39:job-51) Seq 6-835518760: Sending { Cmd , MgmtId:
7566222426160, via: 6, Ver: v1, Flags: 100111,
[{"MigrateWithStorageCommand":{"vm":{"id":8,"name":"i-2-8-VM","bootloader":"HVM","type":"User","cpus":1,"minSpeed":100,"maxSpeed":100,"minRam":536870912,"maxRam":536870912,"arch":"x86_64","os":"CentOS
5.3
(64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"409dc14c63c9fb0f","params":{"rootDiskController":"ide","nicAdapter":"E1000","nestedVirtualizationFlag":"false"},"uuid":"88d92066-ca9c-4ecc-b204-7c07edc117a2","disks":[{"id":14,"name":"ROOT-8","mountPoint":"/export/home/talluri/mysetup_prim2","path":"ROOT-8-14","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"ece502dc-5d06-3043-afec-38e3d54fe995","deviceId":0}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"90463686-3e63-4a3c-99f7-5ab36bd8a1a8","ip":"10.1.1.120","netmask":"255.255.255.0","gateway":"10.1.1.1","mac":"02:00:70:f1:00:05","dns1":"10.103.128.16","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://944","isolationUri":"vlan://944","isSecurityGroupEnabled":false}]},"volumeToFiler":{"Vol[14|ROOT|ROOT-8-14|2147483648]":{"id":2,"uuid":"ece502dc-5d06-3043-afec-38e3d54fe995","host":"10.147.28.7","path":"/export/home/talluri/mysetup_prim2","port":2049,"type":"NetworkFilesystem"}},"tgtHost":"HostSystem:host-7108@10.147.60.15","wait":0}}]
}
2013-06-12 22:52:50,195 DEBUG [agent.transport.Request]
(Job-Executor-39:job-51) Seq 6-835518760: Executing: { Cmd , MgmtId:
7566222426160, via: 6, Ver: v1, Flags: 100111,
[{"MigrateWithStorageCommand":{"vm":{"id":8,"name":"i-2-8-VM","bootloader":"HVM","type":"User","cpus":1,"minSpeed":100,"maxSpeed":100,"minRam":536870912,"maxRam":536870912,"arch":"x86_64","os":"CentOS
5.3
(64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"409dc14c63c9fb0f","params":{"rootDiskController":"ide","nicAdapter":"E1000","nestedVirtualizationFlag":"false"},"uuid":"88d92066-ca9c-4ecc-b204-7c07edc117a2","disks":[{"id":14,"name":"ROOT-8","mountPoint":"/export/home/talluri/mysetup_prim2","path":"ROOT-8-14","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"ece502dc-5d06-3043-afec-38e3d54fe995","deviceId":0}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"90463686-3e63-4a3c-99f7-5ab36bd8a1a8","ip":"10.1.1.120","netmask":"255.255.255.0","gateway":"10.1.1.1","mac":"02:00:70:f1:00:05","dns1":"10.103.128.16","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://944","isolationUri":"vlan://944","isSecurityGroupEnabled":false}]},"volumeToFiler":{"Vol[14|ROOT|ROOT-8-14|2147483648]":{"id":2,"uuid":"ece502dc-5d06-3043-afec-38e3d54fe995","host":"10.147.28.7","path":"/export/home/talluri/mysetup_prim2","port":2049,"type":"NetworkFilesystem"}},"tgtHost":"HostSystem:host-7108@10.147.60.15","wait":0}}]
}
2013-06-12 22:52:50,212 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-327:null) Seq 6-835518760: Executing request
2013-06-12 22:52:50,224 INFO [vmware.resource.VmwareResource]
(DirectAgent-327:10.147.40.30) Executing resource MigrateWithStorageCommand:
{"vm":{"id":8,"name":"i-2-8-VM","bootloader":"HVM","type":"User","cpus":1,"minSpeed":100,"maxSpeed":100,"minRam":536870912,"maxRam":536870912,"arch":"x86_64","os":"CentOS
5.3
(64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"409dc14c63c9fb0f","params":{"rootDiskController":"ide","nicAdapter":"E1000","nestedVirtualizationFlag":"false"},"uuid":"88d92066-ca9c-4ecc-b204-7c07edc117a2","disks":[{"id":14,"name":"ROOT-8","mountPoint":"/export/home/talluri/mysetup_prim2","path":"ROOT-8-14","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"ece502dc-5d06-3043-afec-38e3d54fe995","deviceId":0}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"90463686-3e63-4a3c-99f7-5ab36bd8a1a8","ip":"10.1.1.120","netmask":"255.255.255.0","gateway":"10.1.1.1","mac":"02:00:70:f1:00:05","dns1":"10.103.128.16","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://944","isolationUri":"vlan://944","isSecurityGroupEnabled":false}]},"volumeToFiler":{"Vol[14|ROOT|ROOT-8-14|2147483648]":{"id":2,"uuid":"ece502dc-5d06-3043-afec-38e3d54fe995","host":"10.147.28.7","path":"/export/home/talluri/mysetup_prim2","port":2049,"type":"NetworkFilesystem"}},"tgtHost":"HostSystem:host-7108@10.147.60.15","wait":0}
2013-06-12 22:52:50,237 WARN [vmware.resource.VmwareResource]
(DirectAgent-327:10.147.40.30) MigrationCommand failed due to Exception:
com.cloud.utils.exception.CloudRuntimeException
Message: Source host & target host are in different datacentesr
com.cloud.utils.exception.CloudRuntimeException: Source host & target host are
in different datacentesr
at
com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3494)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:418)
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-06-12 22:52:50,241 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-327:null) Seq 6-835518760: Response Received:
2013-06-12 22:52:50,241 DEBUG [agent.transport.Request] (DirectAgent-327:null)
Seq 6-835518760: Processing: { Ans: , MgmtId: 7566222426160, via: 6, Ver: v1,
Flags: 110, [{"MigrateWithStorageAnswer":{"result":false,"details":"Exception:
com.cloud.utils.exception.CloudRuntimeException\nMessage: Source host & target
host are in different datacentesr\nStack:
com.cloud.utils.exception.CloudRuntimeException: Source host & target host are
in different datacentesr\n\tat
com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3494)\n\tat
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:418)\n\tat
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)\n\tat
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat
java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)\n\tat
java.util.concurrent.FutureTask.run(FutureTask.java:166)\n\tat
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)\n\tat
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)\n\tat
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)\n\tat
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)\n\tat
java.lang.Thread.run(Thread.java:679)\n","wait":0}}] }
2013-06-12 22:52:50,242 DEBUG [agent.transport.Request]
(Job-Executor-39:job-51) Seq 6-835518760: Received: { Ans: , MgmtId:
7566222426160, via: 6, Ver: v1, Flags: 110, { MigrateWithStorageAnswer } }
2013-06-12 22:52:50,242 ERROR [storage.motion.VmwareStorageMotionStrategy]
(Job-Executor-39:job-51) Migration with storage of vm VM[User|prim2] failed.
Details: Exception: com.cloud.utils.exception.CloudRuntimeException
Message: Source host & target host are in different datacentesr
Stack: com.cloud.utils.exception.CloudRuntimeException: Source host & target
host are in different datacentesr
at
com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3494)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:418)
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-06-12 22:52:50,243 ERROR [storage.motion.VmwareStorageMotionStrategy]
(Job-Executor-39:job-51) copy failed
com.cloud.utils.exception.CloudRuntimeException: Error while migrating the vm
VM[User|prim2] to host Host[-1-Routing]. Exception:
com.cloud.utils.exception.CloudRuntimeException
Message: Source host & target host are in different datacentesr
Stack: com.cloud.utils.exception.CloudRuntimeException: Source host & target
host are in different datacentesr
at
com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3494)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:418)
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
at
org.apache.cloudstack.storage.motion.VmwareStorageMotionStrategy.migrateVmWithVolumesAcrossCluster(VmwareStorageMotionStrategy.java:147)
at
org.apache.cloudstack.storage.motion.VmwareStorageMotionStrategy.copyAsync(VmwareStorageMotionStrategy.java:106)
at
org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:71)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:693)
at
com.cloud.storage.VolumeManagerImpl.migrateVolumes(VolumeManagerImpl.java:2171)
at
com.cloud.vm.VirtualMachineManagerImpl.migrateWithStorage(VirtualMachineManagerImpl.java:1676)
at
com.cloud.vm.UserVmManagerImpl.migrateVirtualMachineWithVolume(UserVmManagerImpl.java:4056)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd.execute(MigrateVirtualMachineWithVolumeCmd.java:137)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-06-12 22:52:50,246 DEBUG [agent.manager.AgentAttache]
(DirectAgent-327:null) Seq 6-835518760: No more commands found
2013-06-12 22:52:50,259 DEBUG [cloud.storage.VolumeManagerImpl]
(Job-Executor-39:job-51) Failed to migrated vm VM[User|prim2] along with its
volumes. com.cloud.utils.exception.CloudRuntimeException: Error while migrating
the vm VM[User|prim2] to host Host[-1-Routing]. Exception:
com.cloud.utils.exception.CloudRuntimeException
Message: Source host & target host are in different datacentesr
Stack: com.cloud.utils.exception.CloudRuntimeException: Source host & target
host are in different datacentesr
at
com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3494)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:418)
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-06-12 22:52:50,259 INFO [cloud.vm.VirtualMachineManagerImpl]
(Job-Executor-39:job-51) Migration was unsuccessful. Cleaning up:
VM[User|prim2]
2013-06-12 22:52:50,260 WARN [apache.cloudstack.alerts]
(Job-Executor-39:job-51) alertType:: 17 // dataCenterId:: 1 // podId:: 1 //
clusterId:: null // message:: Unable to migrate vm i-2-8-VM from host
10.147.40.30 in zone zwpszone and pod zwpszone
2013-06-12 22:52:50,264 DEBUG [cloud.alert.AlertManagerImpl]
(Job-Executor-39:job-51) Have already sent: 1 emails for alert type '17' --
skipping send email
2013-06-12 22:52:50,267 DEBUG [agent.manager.AgentAttache]
(Job-Executor-39:job-51) Seq 1-715325920: Routed from 7566222426160
2013-06-12 22:52:50,267 DEBUG [agent.transport.Request]
(Job-Executor-39:job-51) Seq 1-715325920: Sending { Cmd , MgmtId:
7566222426160, via: 1, Ver: v1, Flags: 100111,
[{"StopCommand":{"isProxy":false,"vmName":"i-2-8-VM","wait":0}}] }
2013-06-12 22:52:50,268 DEBUG [agent.transport.Request]
(Job-Executor-39:job-51) Seq 1-715325920: Executing: { Cmd , MgmtId:
7566222426160, via: 1, Ver: v1, Flags: 100111,
[{"StopCommand":{"isProxy":false,"vmName":"i-2-8-VM","wait":0}}] }
2013-06-12 22:52:50,269 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-440:null) Seq 1-715325920: Executing request
2013-06-12 22:52:50,270 INFO [vmware.resource.VmwareResource]
(DirectAgent-440:10.147.40.24) Executing resource StopCommand:
{"isProxy":false,"vmName":"i-2-8-VM","wait":0}
2013-06-12 22:52:50,273 DEBUG [vmware.mo.HostMO] (DirectAgent-440:10.147.40.24)
find VM i-2-8-VM on host
2013-06-12 22:52:50,273 DEBUG [vmware.mo.HostMO] (DirectAgent-440:10.147.40.24)
load VM cache on host
2013-06-12 22:52:50,291 INFO [vmware.resource.VmwareResource]
(DirectAgent-440:10.147.40.24) VM i-2-8-VM is no longer in vSphere
2013-06-12 22:52:50,292 DEBUG [agent.manager.DirectAgentAttache]
(DirectAgent-440:null) Seq 1-715325920: Response Received:
2013-06-12 22:52:50,293 DEBUG [agent.transport.Request] (DirectAgent-440:null)
Seq 1-715325920: Processing: { Ans: , MgmtId: 7566222426160, via: 1, Ver: v1,
Flags: 110, [{"StopAnswer":{"vncPort":0,"result":true,"details":"VM i-2-8-VM is
no longer in vSphere","wait":0}}] }
2013-06-12 22:52:50,294 DEBUG [agent.manager.AgentAttache]
(DirectAgent-440:null) Seq 1-715325920: Unable to find listener.
2013-06-12 22:52:50,294 DEBUG [agent.manager.AgentAttache]
(DirectAgent-440:null) Seq 1-715325920: No more commands found
2013-06-12 22:52:50,297 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-39:job-51) VM state transitted from :Migrating to Running with
event: OperationFailedvm's original host id: 6 new host id: 6 host id before
state transition: 1
2013-06-12 22:52:50,310 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-39:job-51) Hosts's actual total CPU: 12368 and CPU after applying
overprovisioning: 12368
2013-06-12 22:52:50,310 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-39:job-51) Hosts's actual total RAM: 8579203072 and RAM after
applying overprovisioning: 8579203072
2013-06-12 22:52:50,310 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-39:job-51) release cpu from host: 1, old used: 1700,reserved: 0,
actual total: 12368, total with overprovisioning: 12368; new used:
1600,reserved:0; movedfromreserved: false,moveToReserveredfalse
2013-06-12 22:52:50,311 DEBUG [cloud.capacity.CapacityManagerImpl]
(Job-Executor-39:job-51) release mem from host: 1, old used:
2550136832,reserved: 0, total: 8579203072; new used: 2013265920,reserved:0;
movedfromreserved: false,moveToReserveredfalse
2013-06-12 22:52:50,336 ERROR [cloud.async.AsyncJobManagerImpl]
(Job-Executor-39:job-51) Unexpected exception while executing
org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd
com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
VM[User|prim2] along with its volumes.
com.cloud.utils.exception.CloudRuntimeException: Error while migrating the vm
VM[User|prim2] to host Host[-1-Routing]. Exception:
com.cloud.utils.exception.CloudRuntimeException
Message: Source host & target host are in different datacentesr
Stack: com.cloud.utils.exception.CloudRuntimeException: Source host & target
host are in different datacentesr
at
com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3494)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:418)
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
at
com.cloud.storage.VolumeManagerImpl.migrateVolumes(VolumeManagerImpl.java:2176)
at
com.cloud.vm.VirtualMachineManagerImpl.migrateWithStorage(VirtualMachineManagerImpl.java:1676)
at
com.cloud.vm.UserVmManagerImpl.migrateVirtualMachineWithVolume(UserVmManagerImpl.java:4056)
at
com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
at
org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd.execute(MigrateVirtualMachineWithVolumeCmd.java:137)
at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155)
at
com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-06-12 22:52:50,338 DEBUG [cloud.async.AsyncJobManagerImpl]
(Job-Executor-39:job-51) Complete async job-51, jobStatus: 2, resultCode: 530,
result: Error Code: 530 Error text: Failed to migrated vm VM[User|prim2] along
with its volumes. com.cloud.utils.exception.CloudRuntimeException: Error while
migrating the vm VM[User|prim2] to host Host[-1-Routing]. Exception:
com.cloud.utils.exception.CloudRuntimeException
Message: Source host & target host are in different datacentesr
Stack: com.cloud.utils.exception.CloudRuntimeException: Source host & target
host are in different datacentesr
at
com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3494)
at
com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:418)
at
com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
at java.lang.Thread.run(Thread.java:679)
2013-06-12 22:52:51,583 DEBUG [agent.manager.AgentManagerImpl]
(AgentManager-Handler-15:null) SeqA 3-882: Processing Seq 3-882: { Cmd ,
MgmtId: -1, via: 3, Ver: v1, Flags: 11, [{"Con
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
