I do not have access to ACS code right now, but I suggest starting
debugging here:
> 2017-01-24 18:05:28,427 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-156:ctx-14b86f5e job-60216/job-191380 ctx-175d37df) VM
> state transitted from :Running to Stopping with event: StopRequestedvm's
> original host id: 224 new host id: 15 host id before state transition: 15
>
I would try to understand first why ACS requested the hypervisor to stop
the VM. I mean the following, check the source code for conditions that
would make ACS request the shutdown of VMs.
BTW: are you migrating within a cluster or across clusters?
I will be able to analyze this further only after work hours, so if you
find anything, keep me posted.
On Wed, Jan 25, 2017 at 11:45 AM, Francois Scheurer <
francois.scheu...@everyware.ch> wrote:
> Hello Rafael
>
>
> I think the important log lines are these:
>
> ewcstack-man02-prod: 2017-01-24 18:05:28,407 INFO
> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-156:ctx-14b86f5e
> job-60216/job-191380 ctx-175d37df) Migration cancelled because state has
> changed: VM[User|i-638-1736-VM]
> ewcstack-man02-prod: 2017-01-24 18:05:28,407 DEBUG
> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-156:ctx-14b86f5e
> job-60216/job-191380 ctx-175d37df) Unable to migrate VM due to: Migration
> cancelled because state has changed: VM[User|i-638-1736-VM]
> ewcstack-man02-prod: 2017-01-24 18:05:28,427 DEBUG
> [c.c.c.CapacityManagerImpl] (Work-Job-Executor-156:ctx-14b86f5e
> job-60216/job-191380 ctx-175d37df) VM state transitted from :Running to
> Stopping with event: StopRequestedvm's original host id: 224 new host id:
> 15 host id before state transition: 15
>
> The really disturbing thing is that CS stop again the VM if we start it
> manually, apparently because the migratingjob is still running.
>
> Note the the VM HA Flag is enabled.
>
>
>
> Thank you.
>
>
> Best Regards.
>
> Francois
>
>
>
>
>
>
> more details:
>
>
>
> ewcstack-man02-prod: 2017-01-24 18:05:28,148 DEBUG
> [o.a.c.e.o.VolumeOrchestrator] (Work-Job-Executor-156:ctx-14b86f5e
> job-60216/job-191380 ctx-175d37df) Preparing 3 volumes for
> VM[User|i-638-1736-VM]
> ewcstack-man02-prod: 2017-01-24 18:05:28,155 DEBUG
> [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-122:ctx-a8bacc64
> job-14953/job-191351) Done with run of VM work job:
> com.cloud.vm.VmWorkMigrateAway for VM 1411, job origin: 14953
> ewcstack-man02-prod: 2017-01-24 18:05:28,155 DEBUG
> [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-122:ctx-a8bacc64
> job-14953/job-191351) Done executing com.cloud.vm.VmWorkMigrateAway for
> job-191351
> ewcstack-man02-prod: 2017-01-24 18:05:28,160 INFO
> [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-122:ctx-a8bacc64
> job-14953/job-191351) Remove job-191351 from job monitoring
> ewcstack-man02-prod: 2017-01-24 18:05:28,168 INFO
> [c.c.h.HighAvailabilityManagerImpl] (HA-Worker-2:ctx-f3bcf43c work-1989)
> Completed HAWork[1989-Migration-1411-Running-Migrating]
> ewcstack-man02-prod: 2017-01-24 18:05:28,246 DEBUG
> [c.c.a.m.ClusteredAgentAttache] (Work-Job-Executor-156:ctx-14b86f5e
> job-60216/job-191380 ctx-175d37df) Seq 19-8526158519542516375: Forwarding
> Seq 19-8526158519542516375: { Cmd , MgmtId: 345049103441, via:
> 19(ewcstack-vh010-prod), Ver: v1, Flags: 100111, [{
> "com.cloud.agent.api.PrepareForMigrationCommand":{"vm":{"id"
> :1736,"name":"i-638-1736-VM","bootloader":"PyGrub","type":"
> User","cpus":4,"minSpeed":2100,"maxSpeed":2100,"minRam":
> 17179869184,"maxRam":17179869184,"arch":"x86_64","os":"Other
> (64-bit)","platformEmulator":"Other install media","bootArgs":"","enable
> HA":true,"limitCpuUse":true,"enableDynamicallyScaleVm":true,
> "vncPassword":"pC1WUC7h1DBH9J36q3H9pg==","params":{"memoryOv
> ercommitRatio":"1.0","platform":"viridian:true;acpi:1;apic:
> true;pae:true;nx:true;timeoffset:-3601","keyboard":"us","
> Message.ReservedCapacityFreed.Flag":"false","cpuOvercommitRatio":"4.0","
> hypervisortoolsversion":"xenserver61"},"uuid":"5bff01de-
> c033-4925-9e47-30dd14539272","disks":[{"data":{"org.apache.
> cloudstack.storage.to.VolumeObjectTO":{"uuid":"1ab514b2-f2de-4d85-8e0d-
> 9768184a1349","volumeType":"ROOT","dataStore":{"org.
> apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"b8751b89-00f
> 6-5079-ed7a-c073579c7563","id":64,"poolType":"PreSetup","hos
> t":"localhost","path":"/b8751b89-00f6-5079-ed7a-c073579c7563
> ","port":0,"url":"PreSetup://localhost/b8751b89-00f6-5079-
> ed7a-c073579c7563/?ROLE=Primary&STOREUUID=b8751b89-
> 00f6-5079-ed7a-c073579c7563"}},"name":"ROOT-1736","size":
> 53687091200,"path":"d3
> f87e3c-1efe-42ad-aa93-609d5b980a34","volumeId":8655,"vmName"
> :"i-638-1736-VM","accountId":638,"format":"VHD","
> provisioningType":"THIN","id":8655,"deviceId":0,"hypervisorT
> ype":"XenServer"}},"diskSeq":0,"path":"d3f87e3c-1efe-42ad-
> aa93-609d5b980a34","type":"ROOT","_details":{"managed":"
> false","storagePort":"0",
> "storageHost":"localhost","volumeSize":"53687091200"}},{"dat
> a":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid
> ":"733df9dc-5e24-4ab9-b22b-bce78322428e","volumeType":"
> DATADISK","dataStore":{"org.apache.cloudstack.storage.to.P
> rimaryDataStoreTO":{"uuid":"b8751b89-00f6-5079-ed7a-c073579c
> 7563","id":64,"
> poolType":"PreSetup","host":"localhost","path":"/b8751b89-00
> f6-5079-ed7a-c073579c7563","port":0,"url":"PreSetup://localh
> ost/b8751b89-00f6-5079-ed7a-c073579c7563/?ROLE=Primary&
> STOREUUID=b8751b89-00f6-5079-ed7a-c073579c7563"}},"name":"
> WELLA-DB1-NCSA_D","size":268435456000,"path":"265f168a-
> bb92-4591-a32b-ee3e2
> 2c8cb68","volumeId":8656,"vmName":"i-638-1736-VM","accountId
> ":638,"format":"VHD","provisioningType":"THIN","id":
> 8656,"deviceId":1,"hypervisorType":"XenServer"}},"diskSeq":
> 1,"path":"265f168a-bb92-4591-a32b-ee3e22c8cb68","type":"
> DATADISK","_details":{"managed":"false","storagePort":"0","
> storageHost":"localhos
> t","volumeSize":"268435456000"}},{"data":{"org.apache.clouds
> tack.storage.to.VolumeObjectTO":{"uuid":"bda6f335-b8a3-479c-
> 8c0d-41b344d2a50d","volumeType":"DATADISK","dataStore":{"org.apache.
> cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"09df3ca6-
> 0a20-e4f0-3d79-12a6ed67a13d","id":89,"poolType":"PreSetup","
> host":"localhost","path":"/09df3ca6-0a20-e4f0-3d79-12a6ed67a
> 13d","port":0,"url":"PreSetup://localhost/09df3ca6-0a20-
> e4f0-3d79-12a6ed67a13d/?ROLE=Primary&STOREUUID=09df3ca6-
> 0a20-e4f0-3d79-12a6ed67a13d"}},"name":"WELLA-DB1-NCSA_E","
> size":429496729600,"path":"612cb586-a70c-4896-b120-
> 062ee7a67894","volumeId":86
> 57,"vmName":"i-638-1736-VM","accountId":638,"format":"VHD","
> provisioningType":"THIN","id":8657,"deviceId":2,"hypervisorT
> ype":"XenServer"}},"diskSeq":2,"path":"612cb586-a70c-4896-
> b120-062ee7a67894","type":"DATADISK","_details":{"managed
> ":"false","storagePort":"0","storageHost":"localhost","volumeSize":"42949
> 6729600"}},{"data":{"org.apache.cloudstack.storage.to.Templa
> teObjectTO":{"uuid":"0e5bf4ff-caca-4b08-abed-7bbcdf3c42ef","
> id":201,"format":"ISO","accountId":1,"hvm":true,"
> displayText":"xen-pv-drv-iso","name":"xs-tools.iso","guestOsType":"CentOS
> 4.5 (32-bit)","hypervisorType":"XenServer"}},"diskSeq":3,"type":"
> ISO"}],"nics":[{"deviceId":0,"networkRateMbps":1000,"default
> Nic":true,"pxeDisable":false,"nicUuid":"3523c3d3-9f92-4076-
> ad99-f8864d0f0edd","uuid":"e6fbb257-16e1-415f-88fb-
> 9a79b263b3a8","ip":"192.168.1.42","netmask":"255.255.255.0",
> "gateway":"192.168.1.1","mac":"02:00:0c:ae:00:0f","dns1":"8.
> 8.8.8","dns2":"8.8
> .4.4","broadcastType":"Vlan","type":"Guest","broadcastUri":"
> vlan://2285","isolationUri":"vlan://2285","isSecurityGroupEn
> abled":false,"name":"bond0"}],"vcpuMaxLimit":16},"wait":0}}] } to
> 345049098498
> ewcstack-man02-prod: 2017-01-24 18:05:28,399 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-10:null) Seq 19-8526158519542516375: Processing: {
> Ans: , MgmtId: 345049103441, via: 19, Ver: v1, Flags: 110,
> [{"com.cloud.agent.api.PrepareForMigrationAnswer":{"result":true,"wait":0}}]
> }
> ewcstack-man02-prod: 2017-01-24 18:05:28,399 DEBUG [c.c.a.m.AgentAttache]
> (AgentManager-Handler-10:null) Seq 19-8526158519542516375: No more commands
> found
> ewcstack-man02-prod: 2017-01-24 18:05:28,399 DEBUG [c.c.a.t.Request]
> (Work-Job-Executor-156:ctx-14b86f5e job-60216/job-191380 ctx-175d37df)
> Seq 19-8526158519542516375: Received: { Ans: , MgmtId: 345049103441, via:
> 19, Ver: v1, Flags: 110, { PrepareForMigrationAnswer } }
>
> ewcstack-man02-prod: 2017-01-24 18:05:28,407 INFO
> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-156:ctx-14b86f5e
> job-60216/job-191380 ctx-175d37df) Migration cancelled because state has
> changed: VM[User|i-638-1736-VM]
> ewcstack-man02-prod: 2017-01-24 18:05:28,407 DEBUG
> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-156:ctx-14b86f5e
> job-60216/job-191380 ctx-175d37df) Unable to migrate VM due to: Migration
> cancelled because state has changed: VM[User|i-638-1736-VM]
> ewcstack-man02-prod: 2017-01-24 18:05:28,427 DEBUG
> [c.c.c.CapacityManagerImpl] (Work-Job-Executor-156:ctx-14b86f5e
> job-60216/job-191380 ctx-175d37df) VM state transitted from :Running to
> Stopping with event: StopRequestedvm's original host id: 224 new host id:
> 15 host id before state transition: 15
>
> ewcstack-man02-prod: 2017-01-24 18:05:28,438 DEBUG
> [c.c.a.m.ClusteredAgentAttache] (Work-Job-Executor-156:ctx-14b86f5e
> job-60216/job-191380 ctx-175d37df) Seq 15-6889944479923437643: Forwarding
> Seq 15-6889944479923437643: { Cmd , MgmtId: 345049103441, via:
> 15(ewcstack-vh011-prod), Ver: v1, Flags: 100011,
> [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"
> executeInSequence":false,"checkBeforeCleanup":false,"vmName":"i-638-1736-VM","wait":0}}]
> } to 345049098498
> ewcstack-man02-prod: 2017-01-24 18:05:30,335 DEBUG
> [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-1:null) SeqA 254-800498:
> Processing Seq 254-800498: { Cmd , MgmtId: -1, via: 254, Ver: v1, Flags:
> 11, [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_
> proxyVmId":1530,"_loadInfo":"{\n \"connections\": []\n}","wait":0}}] }
> ewcstack-man02-prod: 2017-01-24 18:05:30,340 DEBUG
> [c.c.a.m.AgentManagerImpl] (AgentManager-Handler-1:null) SeqA 254-800498:
> Sending Seq 254-800498: { Ans: , MgmtId: 345049103441, via: 254, Ver: v1,
> Flags: 100010,
> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}]
> }
> ewcstack-man02-prod: 2017-01-24 18:05:32,447 DEBUG [c.c.a.t.Request]
> (AgentManager-Handler-11:null) Seq 15-6889944479923437632: Processing: {
> Ans: , MgmtId: 345049103441, via: 15, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.StopAnswer":{"platform":"viridian:
> true;acpi:1;apic:true;pae:true;nx:true;timeoffset:3601","result":true,"details":"Stop
> VM i-638-1735-VM Succeed","wait":0}}] }
> ewcstack-man02-prod: 2017-01-24 18:05:32,447 DEBUG [c.c.a.t.Request]
> (Work-Job-Executor-115:ctx-0d5db528 job-13733/job-191379 ctx-bbabfeae)
> Seq 15-6889944479923437632: Received: { Ans: , MgmtId: 345049103441, via:
> 15, Ver: v1, Flags: 10, { StopAnswer } }
> ewcstack-man02-prod: 2017-01-24 18:05:32,470 DEBUG
> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-115:ctx-0d5db528
> job-13733/job-191379 ctx-bbabfeae) VM[User|i-638-1735-VM] is stopped on the
> host. Proceeding to release resource held.
> ewcstack-man02-prod: 2017-01-24 18:05:32,475 DEBUG
> [c.c.n.NetworkModelImpl] (Work-Job-Executor-115:ctx-0d5db528
> job-13733/job-191379 ctx-bbabfeae) Service SecurityGroup is not supported
> in the network id=519
> ewcstack-man02-prod: 2017-01-24 18:05:32,478 DEBUG
> [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-115:ctx-0d5db528
> job-13733/job-191379 ctx-bbabfeae) Changing active number of nics for
> network id=519 on -1
> ewcstack-man02-prod: 2017-01-24 18:05:32,488 DEBUG
> [o.a.c.e.o.NetworkOrchestrator] (Work-Job-Executor-115:ctx-0d5db528
> job-13733/job-191379 ctx-bbabfeae) Asking VpcVirtualRouter to release
> NicProfile[17403-1735-3364083b-dcc0-4eb9-a7f9-f3e85bf8c7d3-
> 192.168.1.11-null
> ewcstack-man02-prod: 2017-01-24 18:05:32,488 DEBUG
> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-115:ctx-0d5db528
> job-13733/job-191379 ctx-bbabfeae) Successfully released network resources
> for the vm VM[User|i-638-1735-VM]
> ewcstack-man02-prod: 2017-01-24 18:05:32,488 DEBUG
> [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-115:ctx-0d5db528
> job-13733/job-191379 ctx-bbabfeae) Successfully released storage resources
> for the vm VM[User|i-638-1735-VM]
> ewcstack-man02-prod: 2017-01-24 18:05:32,496 DEBUG
> [c.c.c.CapacityManagerImpl] (Work-Job-Executor-115:ctx-0d5db528
> job-13733/job-191379 ctx-bbabfeae) VM state transitted from :Stopping to
> Stopped with event: OperationSucceededvm's original host id: 15 new host
> id: null host id before state transition: 15
> ewcstack-man02-prod: 2017-01-24 18:05:32,505 DEBUG
> [c.c.c.CapacityManagerImpl] (Work-Job-Executor-115:ctx-0d5db528
> job-13733/job-191379 ctx-bbabfeae) Hosts's actual total CPU: 82976 and CPU
> after applying overprovisioning: 331904
> ewcstack-man02-prod: 2017-01-24 18:05:32,505 DEBUG
> [c.c.c.CapacityManagerImpl] (Work-Job-Executor-115:ctx-0d5db528
> job-13733/job-191379 ctx-bbabfeae) Hosts's actual total RAM: 261961927808
> and RAM after applying overprovisioning: 261961924608
> ewcstack-man02-prod: 2017-01-24 18:05:32,505 DEBUG
> [c.c.c.CapacityManagerImpl] (Work-Job-Executor-115:ctx-0d5db528
> job-13733/job-191379 ctx-bbabfeae) release cpu from host: 15, old used:
> 71300,reserved: 2100, actual total: 82976, total with overprovisioning:
> 331904; new used: 67100,reserved:6300; movedfromreserved:
> false,moveToReserveredtrue
> ewcstack-man02-prod: 2017-01-24 18:05:32,505 DEBUG
> [c.c.c.CapacityManagerImpl] (Work-Job-Executor-115:ctx-0d5db528
> job-13733/job-191379 ctx-bbabfeae) release mem from host: 15, old used:
> 98784247808,reserved: 1073741824, total: 261961924608; new used:
> 94489280512,reserved:5368709120 <(53)%206870-9120>; movedfromreserved:
> false,moveToReserveredtrue
> ewcstack-man02-prod: 2017-01-24 18:05:32,538 DEBUG
> [c.c.n.NetworkModelImpl] (Work-Job-Executor-115:ctx-0d5db528
> job-13733/job-191379 ctx-bbabfeae) Service SecurityGroup is not supported
> in the network id=519
> ewcstack-man02-prod: 2017-01-24 18:05:32,557 DEBUG
> [c.c.n.NetworkModelImpl] (Work-Job-Executor-115:ctx-0d5db528
> job-13733/job-191379 ctx-bbabfeae) Service SecurityGroup is not supported
> in the network id=519
>
> ewcstack-man02-prod: 2017-01-24 18:05:32,588 ERROR
> [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-115:ctx-0d5db528
> job-13733/job-191379 ctx-bbabfeae) Invocation exception, caused by:
> com.cloud.utils.exception.CloudRuntimeException: Unable to migrate
> VM[User|i-638-1735-VM]
> ewcstack-man02-prod: 2017-01-24 18:05:32,589 INFO
> [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-115:ctx-0d5db528
> job-13733/job-191379 ctx-bbabfeae) Rethrow exception
> com.cloud.utils.exception.CloudRuntimeException: Unable to migrate
> VM[User|i-638-1735-VM]
> ewcstack-man02-prod: 2017-01-24 18:05:32,589 DEBUG
> [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-115:ctx-0d5db528
> job-13733/job-191379) Done with run of VM work job:
> com.cloud.vm.VmWorkMigrateAway for VM 1735, job origin: 13733
> ewcstack-man02-prod: 2017-01-24 18:05:32,589 ERROR
> [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-115:ctx-0d5db528
> job-13733/job-191379) Unable to complete AsyncJobVO {id:191379, userId: 1,
> accountId: 1, instanceType: null, instanceId: null, cmd:
> com.cloud.vm.VmWorkMigrateAway, cmdInfo: rO0ABXNyAB5jb20uY2xvdWQudm0uVm
> 1Xb3JrTWlncmF0ZUF3YXmt4MX4jtcEmwIAAUoACXNyY0hvc3RJZHhyABNjb2
> 0uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2
> VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZz
> t4cAAAAAAAAAABAAAAAAAAAAEAAAAAAAAGx3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAA4A,
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 345049098498, completeMsid: null, lastUpdated:
> null, lastPolled: null, created: Tue Jan 24 18:05:00 CET 2017}, job
> origin:13733
> ewcstack-man02-prod: com.cloud.utils.exception.CloudRuntimeException:
> Unable to migrate VM[User|i-638-1735-VM]
> ewcstack-man02-prod: at com.cloud.vm.VirtualMachineMan
> agerImpl.orchestrateMigrateAway(VirtualMachineManagerImpl.java:2405)
> ewcstack-man02-prod: at com.cloud.vm.VirtualMachineMan
> agerImpl.orchestrateMigrateAway(VirtualMachineManagerImpl.java:4517)
> ewcstack-man02-prod: at
> sun.reflect.GeneratedMethodAccessor402.invoke(Unknown
> Source)
> ewcstack-man02-prod: at sun.reflect.DelegatingMethodAc
> cessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> ewcstack-man02-prod: at java.lang.reflect.Method.invok
> e(Method.java:606)
> ewcstack-man02-prod: at com.cloud.vm.VmWorkJobHandlerP
> roxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> ewcstack-man02-prod: at com.cloud.vm.VirtualMachineMan
> agerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4636)
> ewcstack-man02-prod: at com.cloud.vm.VmWorkJobDispatch
> er.runJob(VmWorkJobDispatcher.java:103)
> ewcstack-man02-prod: at org.apache.cloudstack.framewor
> k.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManag
> erImpl.java:537)
> ewcstack-man02-prod: at org.apache.cloudstack.managed.
> context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> ewcstack-man02-prod: at org.apache.cloudstack.managed.
> context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> ewcstack-man02-prod: at org.apache.cloudstack.managed.
> context.impl.DefaultManagedContext.callWithContext(DefaultMa
> nagedContext.java:103)
> ewcstack-man02-prod: at org.apache.cloudstack.managed.
> context.impl.DefaultManagedContext.runWithContext(DefaultMan
> agedContext.java:53)
> ewcstack-man02-prod: at org.apache.cloudstack.managed.
> context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> ewcstack-man02-prod: at org.apache.cloudstack.framewor
> k.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:494)
> ewcstack-man02-prod: at java.util.concurrent.Executors
> $RunnableAdapter.call(Executors.java:471)
> ewcstack-man02-prod: at java.util.concurrent.FutureTas
> k.run(FutureTask.java:262)
> ewcstack-man02-prod: at java.util.concurrent.ThreadPoo
> lExecutor.runWorker(ThreadPoolExecutor.java:1145)
> ewcstack-man02-prod: at java.util.concurrent.ThreadPoo
> lExecutor$Worker.run(ThreadPoolExecutor.java:615)
> ewcstack-man02-prod: at java.lang.Thread.run(Thread.java:745)
> ewcstack-man02-prod: 2017-01-24 18:05:32,590 DEBUG
> [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-115:ctx-0d5db528
> job-13733/job-191379) Complete async job-191379, jobStatus: FAILED,
> resultCode: 0, result: rO0ABXNyAC9jb20uY2xvdWQudXRpbH
> MuZXhjZXB0aW9uLkNsb3VkUnVudGltZUV4Y2VwdGlvbgAAAABWTU9yAwABSQ
> ALY3NFcnJvckNvZGV4cgAaamF2YS5sYW5nLlJ1bnRpbWVFeGNlcHRpb26eXw
> ZHCjSD5QIAAHhyABNqYXZhLmxhbmcuRXhjZXB0aW9u0P0fPho7HMQCAAB4cg
> ATamF2YS5sYW5nLlRocm93YWJsZdXGNSc5d7jLAwAETAAFY2F1c2V0ABVMam
> F2YS9sYW5nL1Rocm93YWJsZTtMAA1kZXRhaWxNZXNzYWdldAASTGphdmEvbG
> FuZy9TdHJpbmc7WwAKc3RhY2tUcmFjZXQAHltMamF2YS9sYW5nL1N0YWNrVH
> JhY2VFbGVtZW50O0wAFHN1cHByZXNzZWRFeGNlcHRpb25zdAAQTGphdmEvdX
> RpbC9MaXN0O3hwcQB-AAh0AChVbmFibGUgdG8gbWlncmF0ZSBWTVtVc2VyfG
> ktNjM4LTE3MzUtVk1ddXIAHltMamF2YS5sYW5nLlN0YWNrVHJhY2VFbGVtZW
> 50OwJGKjw8_SI5AgAAeHAAAAAUc3IAG2phdmEubGFuZy5TdGFja1RyYWNlRW
> xlbWVudGEJxZomNt2FAgAESQAKbGluZU51bWJlckwADmRlY2xhcmluZ0NsYX
> NzcQB-AAVMAAhmaWxlTmFtZXEAfgAFTAAKbWV0aG9kTmFtZXEAfgAFeHAAAA
> lldAAmY29tLmNsb3VkLnZtLlZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGx0AB
> 5WaXJ0dWFsTWFjaGluZU1hbmFnZXJJbXBsLmphdmF0ABZvcmNoZXN0cmF0ZU
> 1pZ3JhdGVBd2F5c3EAfgAMAAARpXEAfgAOcQB-AA9xAH4AEHNxAH4ADP____
> 90ACZzdW4ucmVmbGVjdC5HZW5lcmF0ZWRNZXRob2RBY2Nlc3NvcjQwMnB0AA
> ZpbnZva2VzcQB-AAwAAAArdAAoc3VuLnJlZmxlY3QuRGVsZWdhdGluZ01ldG
> hvZEFjY2Vzc29ySW1wbHQAIURlbGVnYXRpbmdNZXRob2RBY2Nlc3NvckltcG
> wuamF2YXEAfgAUc3EAfgAMAAACXnQAGGphdmEubGFuZy5yZWZsZWN0Lk1ldG
> hvZHQAC01ldGhvZC5qYXZhcQB-ABRzcQB-AAwAAABrdAAiY29tLmNsb3
> VkLnZtLlZtV29ya0pvYkhhbmRsZXJQcm94eXQAGlZtV29ya0pvYkhhbmRsZX
> JQcm94eS5qYXZhdAAPaGFuZGxlVm1Xb3JrSm9ic3EAfgAMAAASHHEAfgAOcQ
> B-AA9xAH4AHnNxAH4ADAAAAGd0ACBjb20uY2xvdWQudm0uVm1Xb3JrSm9iRG
> lzcGF0Y2hlcnQAGFZtV29ya0pvYkRpc3BhdGNoZXIuamF2YXQABnJ1bkpvYn
> NxAH4ADAAAAhl0AD9vcmcuYXBhY2hlLmNsb3Vkc3RhY2suZnJhbWV3b3JrLm
> pvYnMuaW1wbC5Bc3luY0pvYk1hbmFnZXJJbXBsJDV0ABhBc3luY0pvYk1hbm
> FnZXJJbXBsLmphdmF0AAxydW5JbkNvbnRleHRzcQB-AAwAAAAxdAA-b3JnLm
> FwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ29udG
> V4dFJ1bm5hYmxlJDF0ABtNYW5hZ2VkQ29udGV4dFJ1bm5hYmxlLmphdmF0AA
> NydW5zcQB-AAwAAAA4dABCb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZW
> QuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dCQxdAAaRGVmYX
> VsdE1hbmFnZWRDb250ZXh0LmphdmF0AARjYWxsc3EAfgAMAAAAZ3QAQG9yZy
> 5hcGFjaGUuY2xvdWRzdGFjay5tYW5hZ2VkLmNvbnRleHQuaW1wbC5EZWZhdW
> x0TWFuYWdlZENvbnRleHRxAH4ALnQAD2NhbGxXaXRoQ29udGV4dHNxAH4ADA
> AAADVxAH4AMXEAfgAudAAOcnVuV2l0aENvbnRleHRzcQB-AAwAAAAudAA8b3
> JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5NYW5hZ2VkQ2
> 9udGV4dFJ1bm5hYmxlcQB-ACpxAH4AK3NxAH4ADAAAAe5xAH4AJXEAfgAmcQ
> B-ACtzcQB-AAwAAAHXdAAuamF2YS51dGlsLmNvbmN1cnJlbnQuRXhlY3V0b3
> JzJFJ1bm5hYmxlQWRhcHRlcnQADkV4ZWN1dG9ycy5qYXZhcQB-AC9zcQB-AA
> wAAAEGdAAfamF2YS51dGlsLmNvbmN1cnJlbnQuRnV0dXJlVGFza3QAD0Z1dH
> VyZVRhc2suamF2YXEAfgArc3EAfgAMAAAEeXQAJ2phdmEudXRpbC5jb25jdX
> JyZW50LlRocmVhZFBvb2xFeGVjdXRvcnQAF1RocmVhZFBvb2xFeGVjdXRvci
> 5qYXZhdAAJcnVuV29ya2Vyc3EAfgAMAAACZ3QALmphdmEudXRpbC5jb25jdX
> JyZW50LlRocmVhZFBvb2xFeGVjdXRvciRXb3JrZXJxAH4AQHEAfgArc3EAfg
> AMAAAC6XQAEGphdmEubGFuZy5UaHJlYWR0AAtUaHJlYWQuamF2YXEAfgArc3
> IAJmphdmEudXRpbC5Db2xsZWN0aW9ucyRVbm1vZGlmaWFibGVMaXN0_A8lMb
> XsjhACAAFMAARsaXN0cQB-AAd4cgAsamF2YS51dGlsLkNvbGxlY3Rpb25zJF
> VubW9kaWZpYWJsZUNvbGxlY3Rpb24ZQgCAy173HgIAAUwAAWN0ABZMamF2YS
> 91dGlsL0NvbGxlY3Rpb247eHBzcgATamF2YS51dGlsLkFycmF5TGlzdHiB0h
> 2Zx2GdAwABSQAEc2l6ZXhwAAAAAHcEAAAAAHhxAH4ATHgAABCadwgAAAAAAAAAAHg
> ewcstack-man02-prod: 2017-01-24 18:05:32,592 DEBUG
> [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-115:ctx-0d5db528
> job-13733/job-191379) Publish async job-191379 complete on message bus
> ewcstack-man02-prod: 2017-01-24 18:05:32,592 DEBUG
> [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-115:ctx-0d5db528
> job-13733/job-191379) Wake up jobs related to job-191379
> ewcstack-man02-prod: 2017-01-24 18:05:32,592 DEBUG
> [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-115:ctx-0d5db528
> job-13733/job-191379) Update db status for job-191379
> ewcstack-man02-prod: 2017-01-24 18:05:32,594 DEBUG
> [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-115:ctx-0d5db528
> job-13733/job-191379) Wake up jobs joined with job-191379 and disjoin all
> subjobs created from job- 191379
> ewcstack-man02-prod: 2017-01-24 18:05:32,604 DEBUG
> [o.a.c.f.j.i.AsyncJobManagerImpl] (Work-Job-Executor-115:ctx-0d5db528
> job-13733/job-191379) Done executing com.cloud.vm.VmWorkMigrateAway for
> job-191379
> ewcstack-man02-prod: 2017-01-24 18:05:32,608 INFO
> [o.a.c.f.j.i.AsyncJobMonitor] (Work-Job-Executor-115:ctx-0d5db528
> job-13733/job-191379) Remove job-191379 from job monitoring
>
>
>
> On 25.01.2017 15:53, Rafael Weingärtner wrote:
>
>> Did you check the ACS log files looking for some exceptions when the
>> migrations for this specific VM started?
>>
>> On Wed, Jan 25, 2017 at 9:49 AM, Francois Scheurer <
>> francois.scheu...@everyware.ch> wrote:
>>
>> Dear CS contributors
>>>
>>>
>>> We use CS 4.5.1 on a 3 Clusters with XenServer 6.5. and shared primary
>>> storage.
>>>
>>> When we put a host in maintenance, cs evacuates all VM's on other hosts.
>>>
>>> But it happens regularly that some VM's are stopped instead of live
>>> migrated.
>>>
>>> If we restart that stopped VM's, CS will then stop them again later.
>>>
>>> We need to start them several times until they stays up.
>>>
>>> Is it a known issue? Is it fixed on CS 4.9.2 ?
>>>
>>>
>>> Migrating manually all VM's is working but impracticable because too much
>>> time consuming.
>>>
>>>
>>> Many thanks in advance for your help.
>>>
>>>
>>>
>>> Best Regards
>>> Francois
>>>
>>>
>>>
>>>
>>>
>>>
>>> --
>>>
>>>
>>> EveryWare AG
>>> François Scheurer
>>> Senior Systems Engineer
>>> Zurlindenstrasse 52a
>>> CH-8003 Zürich
>>>
>>> tel: +41 44 466 60 00
>>> fax: +41 44 466 60 10
>>> mail: francois.scheu...@everyware.ch
>>> web: http://www.everyware.ch
>>>
>>>
>>
>>
> --
>
>
> EveryWare AG
> François Scheurer
> Senior Systems Engineer
> Zurlindenstrasse 52a
> CH-8003 Zürich
>
> tel: +41 44 466 60 00
> fax: +41 44 466 60 10
> mail: francois.scheu...@everyware.ch
> web: http://www.everyware.ch
>
>
--
Rafael Weingärtner
