[
https://issues.apache.org/jira/browse/CLOUDSTACK-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13681181#comment-13681181
]
Sateesh Chodapuneedi commented on CLOUDSTACK-2962:
--------------------------------------------------
Per logs, it seems source and target hosts belongs to different datacenters.
Please confirm.
VMware storage live migration is expected to fail if source and target hosts
are in different datacenters.
BTW, wondering how could clusters from 2 datacenters allowed in single
cloudstack zone? May be setup issue or a bug in zone to dc mapping feature,
please raise a bug in such a case.
Is this upgraded setup or fresh install?
Please provide commit id & full management server logs.
> [SM][VMWare]: migrate a virtualMachine with volumes is failing
> --------------------------------------------------------------
>
> Key: CLOUDSTACK-2962
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-2962
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: 4.2.0
> Reporter: Srikanteswararao Talluri
> Assignee: Sateesh Chodapuneedi
> Priority: Blocker
> Fix For: 4.2.0
>
>
> Have a vmware deployment with two clusters in a zone and zonewide primary
> storage
> 1. Now try to migrate a virtual machine with volumes.
> ===START=== 10.252.192.7 -- GET
> command=findHostsForMigration&VirtualMachineId=88d92066-ca9c-4ecc-b204-7c07edc117a2&response=json&sessionkey=5ez1jBXiPUJzcam0%2Ft9UtTy9L%2BY%3D&_=1371038241533
> 2013-06-12 22:52:48,366 DEBUG [cloud.storage.StorageManagerImpl]
> (catalina-exec-8:null) Checking pool 2 for storage, totalSize: 5902284800000,
> usedBytes: 3289819828224, usedPct: 0.5573807330042766, disable threshold: 0.85
> 2013-06-12 22:52:48,447 DEBUG [cloud.server.StatsCollector]
> (StatsCollector-1:null) VmStatsCollector is running...
> 2013-06-12 22:52:48,464 DEBUG [cloud.storage.StorageManagerImpl]
> (catalina-exec-8:null) Checking pool: 2 for volume allocation
> [Vol[14|vm=8|ROOT]], maxSize : 11804569600000, totalAllocatedSize :
> 2147483648, askingSize : 0, allocated disable threshold: 0.85
> 2013-06-12 22:52:48,484 DEBUG [allocator.impl.FirstFitAllocator]
> (catalina-exec-8:null) FirstFitAllocator has 1 hosts to check for allocation:
> [Host[-1-Routing]]
> 2013-06-12 22:52:48,487 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-439:null) Seq 1-715325919: Executing request
> 2013-06-12 22:52:48,507 DEBUG [allocator.impl.FirstFitAllocator]
> (catalina-exec-8:null) Found 1 hosts for allocation after prioritization:
> [Host[-1-Routing]]
> 2013-06-12 22:52:48,507 DEBUG [allocator.impl.FirstFitAllocator]
> (catalina-exec-8:null) Looking for speed=100Mhz, Ram=512
> 2013-06-12 22:52:48,518 DEBUG [cloud.capacity.CapacityManagerImpl]
> (catalina-exec-8:null) Checking if host: 1 has enough capacity for requested
> CPU: 100 and requested RAM: 536870912 , cpuOverprovisioningFactor: 1.0
> 2013-06-12 22:52:48,523 DEBUG [cloud.capacity.CapacityManagerImpl]
> (catalina-exec-8:null) Hosts's actual total CPU: 12368 and CPU after applying
> overprovisioning: 12368
> 2013-06-12 22:52:48,523 DEBUG [cloud.capacity.CapacityManagerImpl]
> (catalina-exec-8:null) considerReservedCapacity isfalse , not considering
> reserved capacity for calculating free capacity
> 2013-06-12 22:52:48,523 DEBUG [cloud.capacity.CapacityManagerImpl]
> (catalina-exec-8:null) Free CPU: 10768 , Requested CPU: 100
> 2013-06-12 22:52:48,524 DEBUG [cloud.capacity.CapacityManagerImpl]
> (catalina-exec-8:null) Free RAM: 6565937152 , Requested RAM: 536870912
> 2013-06-12 22:52:48,524 DEBUG [cloud.capacity.CapacityManagerImpl]
> (catalina-exec-8:null) Host has enough CPU and RAM available
> 2013-06-12 22:52:48,524 DEBUG [cloud.capacity.CapacityManagerImpl]
> (catalina-exec-8:null) STATS: Can alloc CPU from host: 1, used: 1600,
> reserved: 0, actual total: 12368, total with overprovisioning: 12368;
> requested cpu:100,alloc_from_last_host?:false ,considerReservedCapacity?:
> false
> 2013-06-12 22:52:48,524 DEBUG [cloud.capacity.CapacityManagerImpl]
> (catalina-exec-8:null) STATS: Can alloc MEM from host: 1, used: 2013265920,
> reserved: 0, total: 8579203072; requested mem:
> 536870912,alloc_from_last_host?:false ,considerReservedCapacity?: false
> 2013-06-12 22:52:48,524 DEBUG [allocator.impl.FirstFitAllocator]
> (catalina-exec-8:null) Found a suitable host, adding to list: 1
> 2013-06-12 22:52:48,524 DEBUG [allocator.impl.FirstFitAllocator]
> (catalina-exec-8:null) Host Allocator returning 1 suitable hosts
> 2013-06-12 22:52:48,524 DEBUG [cloud.server.ManagementServerImpl]
> (catalina-exec-8:null) Hosts having capacity and suitable for migration:
> [Host[-1-Routing]]
> 2013-06-12 22:52:48,561 DEBUG [cloud.api.ApiServlet] (catalina-exec-8:null)
> ===END=== 10.252.192.7 -- GET
> command=findHostsForMigration&VirtualMachineId=88d92066-ca9c-4ecc-b204-7c07edc117a2&response=json&sessionkey=5ez1jBXiPUJzcam0%2Ft9UtTy9L%2BY%3D&_=1371038241533
> 2013-06-12 22:52:48,598 DEBUG [vmware.mo.HostMO]
> (DirectAgent-439:10.147.40.24) find VM i-2-9-VM on host
> 2013-06-12 22:52:48,599 DEBUG [vmware.mo.HostMO]
> (DirectAgent-439:10.147.40.24) load VM cache on host
> 2013-06-12 22:52:48,608 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-439:null) Seq 1-715325919: Response Received:
> 2013-06-12 22:52:48,609 DEBUG [agent.transport.Request]
> (StatsCollector-1:null) Seq 1-715325919: Received: { Ans: , MgmtId:
> 7566222426160, via: 1, Ver: v1, Flags: 10, { GetVmStatsAnswer } }
> 2013-06-12 22:52:48,637 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-5:null) Seq 6-835518759: Executing request
> 2013-06-12 22:52:48,716 DEBUG [vmware.mo.HostMO] (DirectAgent-5:10.147.40.30)
> find VM i-2-7-VM on host
> 2013-06-12 22:52:48,717 DEBUG [vmware.mo.HostMO] (DirectAgent-5:10.147.40.30)
> load VM cache on host
> 2013-06-12 22:52:48,733 DEBUG [vmware.mo.HostMO] (DirectAgent-5:10.147.40.30)
> find VM i-2-8-VM on host
> 2013-06-12 22:52:48,733 DEBUG [vmware.mo.HostMO] (DirectAgent-5:10.147.40.30)
> VM i-2-8-VM found in host cache
> 2013-06-12 22:52:48,738 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-5:null) Seq 6-835518759: Response Received:
> 2013-06-12 22:52:48,738 DEBUG [agent.transport.Request]
> (StatsCollector-1:null) Seq 6-835518759: Received: { Ans: , MgmtId:
> 7566222426160, via: 6, Ver: v1, Flags: 10, { GetVmStatsAnswer } }
> 2013-06-12 22:52:48,926 DEBUG [cloud.api.ApiServlet] (catalina-exec-1:null)
> ===START=== 10.252.192.7 -- GET
> command=queryAsyncJobResult&jobId=8c775230-f312-428d-a77e-248d9f0d107c&response=json&sessionkey=5ez1jBXiPUJzcam0%2Ft9UtTy9L%2BY%3D&_=1371038242151
> 2013-06-12 22:52:48,947 DEBUG [cloud.async.AsyncJobManagerImpl]
> (catalina-exec-1:null) Async job-35 completed
> 2013-06-12 22:52:48,956 DEBUG [cloud.api.ApiServlet] (catalina-exec-1:null)
> ===END=== 10.252.192.7 -- GET
> command=queryAsyncJobResult&jobId=8c775230-f312-428d-a77e-248d9f0d107c&response=json&sessionkey=5ez1jBXiPUJzcam0%2Ft9UtTy9L%2BY%3D&_=1371038242151
> 2013-06-12 22:52:49,842 DEBUG [cloud.api.ApiServlet] (catalina-exec-15:null)
> ===START=== 10.252.192.7 -- GET
> command=migrateVirtualMachineWithVolume&hostid=2e452c2f-1898-4784-b971-937e357fc781&virtualmachineid=88d92066-ca9c-4ecc-b204-7c07edc117a2&response=json&sessionkey=5ez1jBXiPUJzcam0%2Ft9UtTy9L%2BY%3D&_=1371038243067
> 2013-06-12 22:52:49,892 DEBUG [cloud.async.AsyncJobManagerImpl]
> (catalina-exec-15:null) submit async job-51, details: AsyncJobVO {id:51,
> userId: 2, accountId: 2, sessionKey: null, instanceType: None, instanceId:
> null, cmd:
> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd,
> cmdOriginator: null, cmdInfo:
> {"response":"json","sessionkey":"5ez1jBXiPUJzcam0/t9UtTy9L+Y\u003d","virtualmachineid":"88d92066-ca9c-4ecc-b204-7c07edc117a2","hostid":"2e452c2f-1898-4784-b971-937e357fc781","ctxUserId":"2","httpmethod":"GET","_":"1371038243067","ctxAccountId":"2","ctxStartEventId":"180"},
> cmdVersion: 0, callbackType: 0, callbackAddress: null, status: 0,
> processStatus: 0, resultCode: 0, result: null, initMsid: 7566222426160,
> completeMsid: null, lastUpdated: null, lastPolled: null, created: null}
> 2013-06-12 22:52:49,896 DEBUG [cloud.api.ApiServlet] (catalina-exec-15:null)
> ===END=== 10.252.192.7 -- GET
> command=migrateVirtualMachineWithVolume&hostid=2e452c2f-1898-4784-b971-937e357fc781&virtualmachineid=88d92066-ca9c-4ecc-b204-7c07edc117a2&response=json&sessionkey=5ez1jBXiPUJzcam0%2Ft9UtTy9L%2BY%3D&_=1371038243067
> 2013-06-12 22:52:49,901 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-39:job-51) Executing
> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd
> for job-51
> 2013-06-12 22:52:50,002 DEBUG [cloud.storage.StorageManagerImpl]
> (Job-Executor-39:job-51) Checking pool 2 for storage, totalSize:
> 5902284800000, usedBytes: 3289819828224, usedPct: 0.5573807330042766, disable
> threshold: 0.85
> 2013-06-12 22:52:50,037 DEBUG [cloud.storage.StorageManagerImpl]
> (Job-Executor-39:job-51) Checking pool: 2 for volume allocation
> [Vol[14|vm=8|ROOT]], maxSize : 11804569600000, totalAllocatedSize :
> 2147483648, askingSize : 0, allocated disable threshold: 0.85
> 2013-06-12 22:52:50,061 DEBUG [cloud.network.NetworkModelImpl]
> (Job-Executor-39:job-51) Service SecurityGroup is not supported in the
> network id=204
> 2013-06-12 22:52:50,069 DEBUG [cloud.storage.VolumeManagerImpl]
> (Job-Executor-39:job-51) Preparing 1 volumes for VM[User|prim2]
> 2013-06-12 22:52:50,091 DEBUG [hypervisor.guru.VMwareGuru]
> (Job-Executor-39:job-51) Nested virtualization requested, adding flag to vm
> configuration
> 2013-06-12 22:52:50,121 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-39:job-51) VM state transitted from :Running to Migrating with
> event: MigrationRequestedvm's original host id: 6 new host id: 1 host id
> before state transition: 6
> 2013-06-12 22:52:50,135 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-39:job-51) Hosts's actual total CPU: 12368 and CPU after
> applying overprovisioning: 12368
> 2013-06-12 22:52:50,136 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-39:job-51) We are allocating VM, increasing the used capacity
> of this host:1
> 2013-06-12 22:52:50,136 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-39:job-51) Current Used CPU: 1600 , Free CPU:10768 ,Requested
> CPU: 100
> 2013-06-12 22:52:50,136 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-39:job-51) Current Used RAM: 2013265920 , Free RAM:6565937152
> ,Requested RAM: 536870912
> 2013-06-12 22:52:50,136 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-39:job-51) CPU STATS after allocation: for host: 1, old used:
> 1600, old reserved: 0, actual total: 12368, total with overprovisioning:
> 12368; new used:1700, reserved:0; requested cpu:100,alloc_from_last:false
> 2013-06-12 22:52:50,136 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-39:job-51) RAM STATS after allocation: for host: 1, old used:
> 2013265920, old reserved: 0, total: 8579203072; new used: 2550136832,
> reserved: 0; requested mem: 536870912,alloc_from_last:false
> 2013-06-12 22:52:50,193 DEBUG [agent.transport.Request]
> (Job-Executor-39:job-51) Seq 6-835518760: Sending { Cmd , MgmtId:
> 7566222426160, via: 6, Ver: v1, Flags: 100111,
> [{"MigrateWithStorageCommand":{"vm":{"id":8,"name":"i-2-8-VM","bootloader":"HVM","type":"User","cpus":1,"minSpeed":100,"maxSpeed":100,"minRam":536870912,"maxRam":536870912,"arch":"x86_64","os":"CentOS
> 5.3
> (64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"409dc14c63c9fb0f","params":{"rootDiskController":"ide","nicAdapter":"E1000","nestedVirtualizationFlag":"false"},"uuid":"88d92066-ca9c-4ecc-b204-7c07edc117a2","disks":[{"id":14,"name":"ROOT-8","mountPoint":"/export/home/talluri/mysetup_prim2","path":"ROOT-8-14","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"ece502dc-5d06-3043-afec-38e3d54fe995","deviceId":0}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"90463686-3e63-4a3c-99f7-5ab36bd8a1a8","ip":"10.1.1.120","netmask":"255.255.255.0","gateway":"10.1.1.1","mac":"02:00:70:f1:00:05","dns1":"10.103.128.16","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://944","isolationUri":"vlan://944","isSecurityGroupEnabled":false}]},"volumeToFiler":{"Vol[14|ROOT|ROOT-8-14|2147483648]":{"id":2,"uuid":"ece502dc-5d06-3043-afec-38e3d54fe995","host":"10.147.28.7","path":"/export/home/talluri/mysetup_prim2","port":2049,"type":"NetworkFilesystem"}},"tgtHost":"HostSystem:host-7108@10.147.60.15","wait":0}}]
> }
> 2013-06-12 22:52:50,195 DEBUG [agent.transport.Request]
> (Job-Executor-39:job-51) Seq 6-835518760: Executing: { Cmd , MgmtId:
> 7566222426160, via: 6, Ver: v1, Flags: 100111,
> [{"MigrateWithStorageCommand":{"vm":{"id":8,"name":"i-2-8-VM","bootloader":"HVM","type":"User","cpus":1,"minSpeed":100,"maxSpeed":100,"minRam":536870912,"maxRam":536870912,"arch":"x86_64","os":"CentOS
> 5.3
> (64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"409dc14c63c9fb0f","params":{"rootDiskController":"ide","nicAdapter":"E1000","nestedVirtualizationFlag":"false"},"uuid":"88d92066-ca9c-4ecc-b204-7c07edc117a2","disks":[{"id":14,"name":"ROOT-8","mountPoint":"/export/home/talluri/mysetup_prim2","path":"ROOT-8-14","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"ece502dc-5d06-3043-afec-38e3d54fe995","deviceId":0}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"90463686-3e63-4a3c-99f7-5ab36bd8a1a8","ip":"10.1.1.120","netmask":"255.255.255.0","gateway":"10.1.1.1","mac":"02:00:70:f1:00:05","dns1":"10.103.128.16","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://944","isolationUri":"vlan://944","isSecurityGroupEnabled":false}]},"volumeToFiler":{"Vol[14|ROOT|ROOT-8-14|2147483648]":{"id":2,"uuid":"ece502dc-5d06-3043-afec-38e3d54fe995","host":"10.147.28.7","path":"/export/home/talluri/mysetup_prim2","port":2049,"type":"NetworkFilesystem"}},"tgtHost":"HostSystem:host-7108@10.147.60.15","wait":0}}]
> }
> 2013-06-12 22:52:50,212 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-327:null) Seq 6-835518760: Executing request
> 2013-06-12 22:52:50,224 INFO [vmware.resource.VmwareResource]
> (DirectAgent-327:10.147.40.30) Executing resource MigrateWithStorageCommand:
> {"vm":{"id":8,"name":"i-2-8-VM","bootloader":"HVM","type":"User","cpus":1,"minSpeed":100,"maxSpeed":100,"minRam":536870912,"maxRam":536870912,"arch":"x86_64","os":"CentOS
> 5.3
> (64-bit)","bootArgs":"","rebootOnCrash":false,"enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":false,"vncPassword":"409dc14c63c9fb0f","params":{"rootDiskController":"ide","nicAdapter":"E1000","nestedVirtualizationFlag":"false"},"uuid":"88d92066-ca9c-4ecc-b204-7c07edc117a2","disks":[{"id":14,"name":"ROOT-8","mountPoint":"/export/home/talluri/mysetup_prim2","path":"ROOT-8-14","size":2147483648,"type":"ROOT","storagePoolType":"NetworkFilesystem","storagePoolUuid":"ece502dc-5d06-3043-afec-38e3d54fe995","deviceId":0}],"nics":[{"deviceId":0,"networkRateMbps":200,"defaultNic":true,"uuid":"90463686-3e63-4a3c-99f7-5ab36bd8a1a8","ip":"10.1.1.120","netmask":"255.255.255.0","gateway":"10.1.1.1","mac":"02:00:70:f1:00:05","dns1":"10.103.128.16","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://944","isolationUri":"vlan://944","isSecurityGroupEnabled":false}]},"volumeToFiler":{"Vol[14|ROOT|ROOT-8-14|2147483648]":{"id":2,"uuid":"ece502dc-5d06-3043-afec-38e3d54fe995","host":"10.147.28.7","path":"/export/home/talluri/mysetup_prim2","port":2049,"type":"NetworkFilesystem"}},"tgtHost":"HostSystem:host-7108@10.147.60.15","wait":0}
> 2013-06-12 22:52:50,237 WARN [vmware.resource.VmwareResource]
> (DirectAgent-327:10.147.40.30) MigrationCommand failed due to Exception:
> com.cloud.utils.exception.CloudRuntimeException
> Message: Source host & target host are in different datacentesr
> com.cloud.utils.exception.CloudRuntimeException: Source host & target host
> are in different datacentesr
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3494)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:418)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-06-12 22:52:50,241 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-327:null) Seq 6-835518760: Response Received:
> 2013-06-12 22:52:50,241 DEBUG [agent.transport.Request]
> (DirectAgent-327:null) Seq 6-835518760: Processing: { Ans: , MgmtId:
> 7566222426160, via: 6, Ver: v1, Flags: 110,
> [{"MigrateWithStorageAnswer":{"result":false,"details":"Exception:
> com.cloud.utils.exception.CloudRuntimeException\nMessage: Source host &
> target host are in different datacentesr\nStack:
> com.cloud.utils.exception.CloudRuntimeException: Source host & target host
> are in different datacentesr\n\tat
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3494)\n\tat
>
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:418)\n\tat
>
> com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)\n\tat
>
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)\n\tat
> java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)\n\tat
> java.util.concurrent.FutureTask.run(FutureTask.java:166)\n\tat
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)\n\tat
>
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)\n\tat
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)\n\tat
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)\n\tat
> java.lang.Thread.run(Thread.java:679)\n","wait":0}}] }
> 2013-06-12 22:52:50,242 DEBUG [agent.transport.Request]
> (Job-Executor-39:job-51) Seq 6-835518760: Received: { Ans: , MgmtId:
> 7566222426160, via: 6, Ver: v1, Flags: 110, { MigrateWithStorageAnswer } }
> 2013-06-12 22:52:50,242 ERROR [storage.motion.VmwareStorageMotionStrategy]
> (Job-Executor-39:job-51) Migration with storage of vm VM[User|prim2] failed.
> Details: Exception: com.cloud.utils.exception.CloudRuntimeException
> Message: Source host & target host are in different datacentesr
> Stack: com.cloud.utils.exception.CloudRuntimeException: Source host & target
> host are in different datacentesr
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3494)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:418)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-06-12 22:52:50,243 ERROR [storage.motion.VmwareStorageMotionStrategy]
> (Job-Executor-39:job-51) copy failed
> com.cloud.utils.exception.CloudRuntimeException: Error while migrating the vm
> VM[User|prim2] to host Host[-1-Routing]. Exception:
> com.cloud.utils.exception.CloudRuntimeException
> Message: Source host & target host are in different datacentesr
> Stack: com.cloud.utils.exception.CloudRuntimeException: Source host & target
> host are in different datacentesr
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3494)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:418)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> at
> org.apache.cloudstack.storage.motion.VmwareStorageMotionStrategy.migrateVmWithVolumesAcrossCluster(VmwareStorageMotionStrategy.java:147)
> at
> org.apache.cloudstack.storage.motion.VmwareStorageMotionStrategy.copyAsync(VmwareStorageMotionStrategy.java:106)
> at
> org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:71)
> at
> org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolumes(VolumeServiceImpl.java:693)
> at
> com.cloud.storage.VolumeManagerImpl.migrateVolumes(VolumeManagerImpl.java:2171)
> at
> com.cloud.vm.VirtualMachineManagerImpl.migrateWithStorage(VirtualMachineManagerImpl.java:1676)
> at
> com.cloud.vm.UserVmManagerImpl.migrateVirtualMachineWithVolume(UserVmManagerImpl.java:4056)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd.execute(MigrateVirtualMachineWithVolumeCmd.java:137)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-06-12 22:52:50,246 DEBUG [agent.manager.AgentAttache]
> (DirectAgent-327:null) Seq 6-835518760: No more commands found
> 2013-06-12 22:52:50,259 DEBUG [cloud.storage.VolumeManagerImpl]
> (Job-Executor-39:job-51) Failed to migrated vm VM[User|prim2] along with its
> volumes. com.cloud.utils.exception.CloudRuntimeException: Error while
> migrating the vm VM[User|prim2] to host Host[-1-Routing]. Exception:
> com.cloud.utils.exception.CloudRuntimeException
> Message: Source host & target host are in different datacentesr
> Stack: com.cloud.utils.exception.CloudRuntimeException: Source host & target
> host are in different datacentesr
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3494)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:418)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-06-12 22:52:50,259 INFO [cloud.vm.VirtualMachineManagerImpl]
> (Job-Executor-39:job-51) Migration was unsuccessful. Cleaning up:
> VM[User|prim2]
> 2013-06-12 22:52:50,260 WARN [apache.cloudstack.alerts]
> (Job-Executor-39:job-51) alertType:: 17 // dataCenterId:: 1 // podId:: 1 //
> clusterId:: null // message:: Unable to migrate vm i-2-8-VM from host
> 10.147.40.30 in zone zwpszone and pod zwpszone
> 2013-06-12 22:52:50,264 DEBUG [cloud.alert.AlertManagerImpl]
> (Job-Executor-39:job-51) Have already sent: 1 emails for alert type '17' --
> skipping send email
> 2013-06-12 22:52:50,267 DEBUG [agent.manager.AgentAttache]
> (Job-Executor-39:job-51) Seq 1-715325920: Routed from 7566222426160
> 2013-06-12 22:52:50,267 DEBUG [agent.transport.Request]
> (Job-Executor-39:job-51) Seq 1-715325920: Sending { Cmd , MgmtId:
> 7566222426160, via: 1, Ver: v1, Flags: 100111,
> [{"StopCommand":{"isProxy":false,"vmName":"i-2-8-VM","wait":0}}] }
> 2013-06-12 22:52:50,268 DEBUG [agent.transport.Request]
> (Job-Executor-39:job-51) Seq 1-715325920: Executing: { Cmd , MgmtId:
> 7566222426160, via: 1, Ver: v1, Flags: 100111,
> [{"StopCommand":{"isProxy":false,"vmName":"i-2-8-VM","wait":0}}] }
> 2013-06-12 22:52:50,269 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-440:null) Seq 1-715325920: Executing request
> 2013-06-12 22:52:50,270 INFO [vmware.resource.VmwareResource]
> (DirectAgent-440:10.147.40.24) Executing resource StopCommand:
> {"isProxy":false,"vmName":"i-2-8-VM","wait":0}
> 2013-06-12 22:52:50,273 DEBUG [vmware.mo.HostMO]
> (DirectAgent-440:10.147.40.24) find VM i-2-8-VM on host
> 2013-06-12 22:52:50,273 DEBUG [vmware.mo.HostMO]
> (DirectAgent-440:10.147.40.24) load VM cache on host
> 2013-06-12 22:52:50,291 INFO [vmware.resource.VmwareResource]
> (DirectAgent-440:10.147.40.24) VM i-2-8-VM is no longer in vSphere
> 2013-06-12 22:52:50,292 DEBUG [agent.manager.DirectAgentAttache]
> (DirectAgent-440:null) Seq 1-715325920: Response Received:
> 2013-06-12 22:52:50,293 DEBUG [agent.transport.Request]
> (DirectAgent-440:null) Seq 1-715325920: Processing: { Ans: , MgmtId:
> 7566222426160, via: 1, Ver: v1, Flags: 110,
> [{"StopAnswer":{"vncPort":0,"result":true,"details":"VM i-2-8-VM is no longer
> in vSphere","wait":0}}] }
> 2013-06-12 22:52:50,294 DEBUG [agent.manager.AgentAttache]
> (DirectAgent-440:null) Seq 1-715325920: Unable to find listener.
> 2013-06-12 22:52:50,294 DEBUG [agent.manager.AgentAttache]
> (DirectAgent-440:null) Seq 1-715325920: No more commands found
> 2013-06-12 22:52:50,297 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-39:job-51) VM state transitted from :Migrating to Running with
> event: OperationFailedvm's original host id: 6 new host id: 6 host id before
> state transition: 1
> 2013-06-12 22:52:50,310 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-39:job-51) Hosts's actual total CPU: 12368 and CPU after
> applying overprovisioning: 12368
> 2013-06-12 22:52:50,310 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-39:job-51) Hosts's actual total RAM: 8579203072 and RAM after
> applying overprovisioning: 8579203072
> 2013-06-12 22:52:50,310 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-39:job-51) release cpu from host: 1, old used: 1700,reserved:
> 0, actual total: 12368, total with overprovisioning: 12368; new used:
> 1600,reserved:0; movedfromreserved: false,moveToReserveredfalse
> 2013-06-12 22:52:50,311 DEBUG [cloud.capacity.CapacityManagerImpl]
> (Job-Executor-39:job-51) release mem from host: 1, old used:
> 2550136832,reserved: 0, total: 8579203072; new used: 2013265920,reserved:0;
> movedfromreserved: false,moveToReserveredfalse
> 2013-06-12 22:52:50,336 ERROR [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-39:job-51) Unexpected exception while executing
> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd
> com.cloud.utils.exception.CloudRuntimeException: Failed to migrated vm
> VM[User|prim2] along with its volumes.
> com.cloud.utils.exception.CloudRuntimeException: Error while migrating the vm
> VM[User|prim2] to host Host[-1-Routing]. Exception:
> com.cloud.utils.exception.CloudRuntimeException
> Message: Source host & target host are in different datacentesr
> Stack: com.cloud.utils.exception.CloudRuntimeException: Source host & target
> host are in different datacentesr
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3494)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:418)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> at
> com.cloud.storage.VolumeManagerImpl.migrateVolumes(VolumeManagerImpl.java:2176)
> at
> com.cloud.vm.VirtualMachineManagerImpl.migrateWithStorage(VirtualMachineManagerImpl.java:1676)
> at
> com.cloud.vm.UserVmManagerImpl.migrateVirtualMachineWithVolume(UserVmManagerImpl.java:4056)
> at
> com.cloud.utils.component.ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept(ComponentInstantiationPostProcessor.java:125)
> at
> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd.execute(MigrateVirtualMachineWithVolumeCmd.java:137)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:155)
> at
> com.cloud.async.AsyncJobManagerImpl$1.run(AsyncJobManagerImpl.java:437)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-06-12 22:52:50,338 DEBUG [cloud.async.AsyncJobManagerImpl]
> (Job-Executor-39:job-51) Complete async job-51, jobStatus: 2, resultCode:
> 530, result: Error Code: 530 Error text: Failed to migrated vm VM[User|prim2]
> along with its volumes. com.cloud.utils.exception.CloudRuntimeException:
> Error while migrating the vm VM[User|prim2] to host Host[-1-Routing].
> Exception: com.cloud.utils.exception.CloudRuntimeException
> Message: Source host & target host are in different datacentesr
> Stack: com.cloud.utils.exception.CloudRuntimeException: Source host & target
> host are in different datacentesr
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3494)
> at
> com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:418)
> at
> com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-06-12 22:52:51,583 DEBUG [agent.manager.AgentManagerImpl]
> (AgentManager-Handler-15:null) SeqA 3-882: Processing Seq 3-882: { Cmd ,
> MgmtId: -1, via: 3, Ver: v1, Flags: 11, [{"Con
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
