[ https://issues.apache.org/jira/browse/CLOUDSTACK-5665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
angeline shen updated CLOUDSTACK-5665: -------------------------------------- Attachment: management-server.log.gz > XEN patch/hotfix certification - XS 6.0.2 XS602E030 patch installation fail > --------------------------------------------------------------------------- > > Key: CLOUDSTACK-5665 > URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5665 > Project: CloudStack > Issue Type: Bug > Security Level: Public(Anyone can view this level - this is the > default.) > Components: Management Server > Affects Versions: 4.2.1 > Environment: MS CloudPlatform-4.2.1-2-rhel6.3.tar.gz > host 10.223.51.3 XS 6.0.2 base + HF XS602E001 to XS602E029 + CSPs > 10.223.51.4 XS 6.0.2 base + HF XS602E001 to XS602E029 + CSPs > Reporter: angeline shen > Priority: Critical > Fix For: 4.2.1 > > Attachments: management-server.log.gz > > > http://wiki-ccp.citrix.com/pages/viewpage.action?pageId=11829631 : > XEN hotfix installation procedure > 1. Hosts master slave installed with > XS 6.0.2 base + Hotfix XS602E001 to XS602E029 + CSPs > 2. Basic zone with cluster of above master & slave hosts > security group default with NO security rules > Deploy two VMs > 3. MS: master host put to maintenance mode - VMs migrate to slave host > 4. MS: unmanage cluster > 5. master host: install hotfix XS602E030 + CSP , reboot master > 6. MS: master host cancel maintenance mode > 7. MS: manage cluster > 8. MS: slave host put to maintenance mode, VMs FAIL migrate to master > Unable to complete hotfix installation > 2013-12-26 16:51:25,307 INFO [xen.resource.CitrixResourceBase] > (DirectAgent-56:null) VM does not exist on > XenServerc6c75483-b35b-4a7c-9e2d-2274e34dd437 > 2013-12-26 16:51:25,307 DEBUG [agent.manager.DirectAgentAttache] > (DirectAgent-56:null) Seq 1-265617550: Response Received: > 2013-12-26 16:51:25,307 DEBUG [agent.transport.Request] (DirectAgent-56:null) > Seq 1-265617550: Processing: { Ans: , MgmtId: 7692017993539, via: 1, Ver: > v1, Flags: 110, > [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM does not > exist","wait":0}}] } > 2013-12-26 16:51:25,307 DEBUG [agent.manager.AgentAttache] > (DirectAgent-56:null) Seq 1-265617550: Unable to find listener. > 2013-12-26 16:51:25,307 DEBUG [agent.manager.AgentAttache] > (DirectAgent-56:null) Seq 1-265617550: No more commands found > 2013-12-26 16:51:25,313 DEBUG [vm.dao.VMInstanceDaoImpl] > (HA-Worker-3:work-19) Unable to update VM[DomainRouter|r-4-VM]: DB > Data={Host=4; State=Running; updated=13; time=Thu Dec 26 16:51:25 PST 2013} > New Data: {Host=4; State=Stopping; updated=12; time=Thu Dec 26 16:51:25 PST > 2013} Stale Data: {Host=4; State=Running; updated=11; time=Thu Dec 26 > 16:51:22 PST 2013} > 2013-12-26 16:51:25,313 DEBUG [cloud.vm.VirtualMachineManagerImpl] > (HA-Worker-3:work-19) Unable to stop VM due to VM is being operated on. > 2013-12-26 16:51:25,313 WARN [cloud.ha.HighAvailabilityManagerImpl] > (HA-Worker-3:work-19) Unable to migrate vm from 4 > 2013-12-26 16:51:25,314 DEBUG [cloud.resource.ResourceManagerImpl] > (HA-Worker-3:work-19) No next resource state for host 4 while current state > is ErrorInMaintenance with event UnableToMigrate > com.cloud.utils.fsm.NoTransitionException: No next resource state found for > current state =ErrorInMaintenance event =UnableToMigrate > at > com.cloud.resource.ResourceManagerImpl.resourceStateTransitTo(ResourceManagerImpl.java:1178) > at > com.cloud.resource.ResourceManagerImpl.maintenanceFailed(ResourceManagerImpl.java:2313) > at > com.cloud.ha.HighAvailabilityManagerImpl.migrate(HighAvailabilityManagerImpl.java:602) > at > com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.run(HighAvailabilityManagerImpl.java:858) > 2013-12-26 16:51:25,314 INFO [cloud.ha.HighAvailabilityManagerImpl] > (HA-Worker-3:work-19) Completed HAWork[19-Migration-4-Running-Migrating] > 2013-12-26 16:51:25,324 WARN [xen.resource.CitrixResourceBase] > (DirectAgent-57:null) Task failed! Task record: uuid: > 38ce6cd4-9ab2-7e46-4d10-dafe25a83084 > nameLabel: Async.VM.pool_migrate > nameDescription: > allowedOperations: [] > currentOperations: {} > created: Thu Dec 26 16:55:03 PST 2013 > finished: Thu Dec 26 16:55:03 PST 2013 > status: failure > residentOn: com.xensource.xenapi.Host@40561b9c > progress: 1.0 > type: <none/> > result: > errorInfo: [VM_REQUIRES_SR, > OpaqueRef:6e9c09e8-9ed5-1e21-2048-d848a084a15a, > OpaqueRef:eec84d31-9691-1ba4-f7bb-39db42ac8628] > otherConfig: {} > subtaskOf: com.xensource.xenapi.Task@aaf13f6f > subtasks: [] > 2013-12-26 16:51:25,332 WARN [xen.resource.CitrixResourceBase] > (DirectAgent-57:null) Unable to migrate VM(s-1-VM) from > host(d044a044-1a30-4c01-ae83-11d2631a4ffe) due to Task failed! Task record: > uuid: 38ce6cd4-9ab2-7e46-4d10-dafe25a83084 > nameLabel: Async.VM.pool_migrate > nameDescription: > allowedOperations: [] > currentOperations: {} > created: Thu Dec 26 16:55:03 PST 2013 > finished: Thu Dec 26 16:55:03 PST 2013 > status: failure > residentOn: com.xensource.xenapi.Host@40561b9c > progress: 1.0 > type: <none/> > result: > errorInfo: [VM_REQUIRES_SR, > OpaqueRef:6e9c09e8-9ed5-1e21-2048-d848a084a15a, > OpaqueRef:eec84d31-9691-1ba4-f7bb-39db42ac8628] > otherConfig: {} > subtaskOf: com.xensource.xenapi.Task@aaf13f6f > subtasks: [] > Task failed! Task record: uuid: > 38ce6cd4-9ab2-7e46-4d10-dafe25a83084 > nameLabel: Async.VM.pool_migrate > nameDescription: > : > residentOn: com.xensource.xenapi.Host@40561b9c > progress: 1.0 > type: <none/> > result: > errorInfo: [VM_REQUIRES_SR, > OpaqueRef:6e9c09e8-9ed5-1e21-2048-d848a084a15a, > OpaqueRef:eec84d31-9691-1ba4-f7bb-39db42ac8628] > otherConfig: {} > subtaskOf: com.xensource.xenapi.Task@aaf13f6f > subtasks: [] > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.migrateVM(CitrixResourceBase.java:3789) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.execute(CitrixResourceBase.java:3346) > at > com.cloud.hypervisor.xen.resource.CitrixResourceBase.executeRequest(CitrixResourceBase.java:511) > at > com.cloud.hypervisor.xen.resource.XenServer56Resource.executeRequest(XenServer56Resource.java:59) > at > com.cloud.agent.manager.DirectAgentAttache$Task.run(DirectAgentAttache.java:186) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) > at java.util.concurrent.FutureTask.run(FutureTask.java:166) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$101(ScheduledThreadPoolExecutor.java:165) > at > java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:266) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) > at java.lang.Thread.run(Thread.java:679) > 2013-12-26 16:51:25,342 DEBUG [xen.resource.CitrixResourceBase] > (DirectAgent-57:null) 6. The VM s-1-VM is in Migrating state > 2013-12-26 16:51:25,342 DEBUG [agent.manager.DirectAgentAttache] > (DirectAgent-57:null) Seq 4-961806364: Response Received: > 2013-12-26 16:51:25,342 DEBUG [agent.transport.Request] (DirectAgent-57:null) > Seq 4-961806364: Processing: { Ans: , MgmtId: 7692017993539, via: 4, Ver: > v1, Flags: 110, > [{"com.cloud.agent.api.MigrateAnswer":{"result":false,"details":"Catch > Exception com.cloud.utils.exception.CloudRuntimeException: Migration failed > due to com.cloud.utils.exception.CloudRuntimeException: Unable to migrate > VM(s-1-VM) from host(d044a044-1a30-4c01-ae83-11d2631a4ffe) due to Task > failed! Task record: uuid: > 38ce6cd4-9ab2-7e46-4d10-dafe25a83084\n nameLabel: > Async.VM.pool_migrate\n nameDescription: \n allowedOperations: []\n > currentOperations: {}\n created: Thu Dec 26 16:55:03 PST 2013\n > finished: Thu Dec 26 16:55:03 PST 2013\n status: > failure\n residentOn: com.xensource.xenapi.Host@40561b9c\n > progress: 1.0\n type: <none/>\n result: \n > errorInfo: [VM_REQUIRES_SR, > OpaqueRef:6e9c09e8-9ed5-1e21-2048-d848a084a15a, > OpaqueRef:eec84d31-9691-1ba4-f7bb-39db42ac8628]\n otherConfig: {}\n > subtaskOf: com.xensource.xenapi.Task@aaf13f6f\n subtasks: > []\n","wait":0}}] } > 2013-12-26 16:51:25,342 DEBUG [agent.transport.Request] (HA-Worker-0:work-16) > Seq 4-961806364: Received: { Ans: , MgmtId: 7692017993539, via: 4, Ver: v1, > Flags: 110, { MigrateAnswer } } > 2013-12-26 16:51:25,342 ERROR [cloud.vm.VirtualMachineManagerImpl] > (HA-Worker-0:work-16) Unable to migrate due to Catch Exception > com.cloud.utils.exception.CloudRuntimeException: Migration failed due to > com.cloud.utils.exception.CloudRuntimeException: Unable to migrate VM(s-1-VM) > from host(d044a044-1a30-4c01-ae83-11d2631a4ffe) due to Task failed! Task > record: uuid: 38ce6cd4-9ab2-7e46-4d10-dafe25a83084 > nameLabel: Async.VM.pool_migrate > nameDescription: > allowedOperations: [] > currentOperations: {} > created: Thu Dec 26 16:55:03 PST 2013 > finished: Thu Dec 26 16:55:03 PST 2013 > status: failure > residentOn: com.xensource.xenapi.Host@40561b9c > progress: 1.0 > type: <none/> > result: > errorInfo: [VM_REQUIRES_SR, > OpaqueRef:6e9c09e8-9ed5-1e21-2048-d848a084a15a, > OpaqueRef:eec84d31-9691-1ba4-f7bb-39db42ac8628] > otherConfig: {} > subtaskOf: com.xensource.xenapi.Task@aaf13f6f > subtasks: [] > 2013-12-26 16:51:25,342 INFO [cloud.vm.VirtualMachineManagerImpl] > (HA-Worker-0:work-16) Migration was unsuccessful. Cleaning up: > VM[SecondaryStorageVm|s-1-VM] > 2013-12-26 16:51:25,346 DEBUG [agent.manager.AgentAttache] > (DirectAgent-57:null) Seq 4-961806364: No more commands found > 2013-12-26 16:51:25,355 WARN [apache.cloudstack.alerts] > (HA-Worker-0:work-16) alertType:: 17 // dataCenterId:: 1 // podId:: 1 // > clusterId:: null // message:: Unable to migrate vm s-1-VM from host > Rack2Host19.lab.vmops.com in zone z1 and pod z1p1 > 2013-12-26 16:51:25,356 DEBUG [cloud.alert.AlertManagerImpl] > (HA-Worker-0:work-16) Have already sent: 1 emails for alert type '17' -- > skipping send email > 2013-12-26 16:51:25,357 DEBUG [agent.manager.AgentAttache] > (HA-Worker-0:work-16) Seq 1-265617551: Routed from 7692017993539 > 2013-12-26 16:51:25,358 DEBUG [agent.transport.Request] (HA-Worker-0:work-16) > Seq 1-265617551: Sending { Cmd , MgmtId: 7692017993539, via: 1, Ver: v1, > Flags: 100111, > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":true,"vmName":"s-1-VM","wait":0}}] > } > 2013-12-26 16:51:25,358 DEBUG [agent.transport.Request] (HA-Worker-0:work-16) > Seq 1-265617551: Executing: { Cmd , MgmtId: 7692017993539, via: 1, Ver: v1, > Flags: 100111, > [{"com.cloud.agent.api.StopCommand":{"isProxy":false,"executeInSequence":true,"vmName":"s-1-VM","wait":0}}] > } > 2013-12-26 16:51:25,358 DEBUG [agent.manager.DirectAgentAttache] > (DirectAgent-62:null) Seq 1-265617551: Executing request > 2013-12-26 16:51:25,368 DEBUG [cloud.capacity.CapacityManagerImpl] > (HA-Worker-0:work-16) VM state transitted from :Migrating to Running with > event: OperationFailedvm's original host id: 4 new host id: 4 host id before > state transition: 1 > 2013-12-26 16:51:25,375 DEBUG [cloud.capacity.CapacityManagerImpl] > (HA-Worker-0:work-16) Hosts's actual total CPU: 9044 and CPU after applying > overprovisioning: 9044 > 2013-12-26 16:51:25,375 DEBUG [cloud.capacity.CapacityManagerImpl] > (HA-Worker-0:work-16) Hosts's actual total RAM: 15988871808 and RAM after > applying overprovisioning: 15988872192 > 2013-12-26 16:51:25,375 DEBUG [cloud.capacity.CapacityManagerImpl] > (HA-Worker-0:work-16) release cpu from host: 1, old used: 500,reserved: 0, > actual total: 9044, total with overprovisioning: 9044; new used: > 0,reserved:0; movedfromreserved: false,moveToReserveredfalse > 2013-12-26 16:51:25,376 DEBUG [cloud.capacity.CapacityManagerImpl] > (HA-Worker-0:work-16) release mem from host: 1, old used: 268435456,reserved: > 0, total: 15988872192; new used: 0,reserved:0; movedfromreserved: > false,moveToReserveredfalse > 2013-12-26 16:51:25,385 INFO [xen.resource.CitrixResourceBase] > (DirectAgent-62:null) VM does not exist on > XenServerc6c75483-b35b-4a7c-9e2d-2274e34dd437 > 2013-12-26 16:51:25,385 DEBUG [agent.manager.DirectAgentAttache] > (DirectAgent-62:null) Seq 1-265617551: Response Received: > 2013-12-26 16:51:25,386 DEBUG [agent.transport.Request] (DirectAgent-62:null) > Seq 1-265617551: Processing: { Ans: , MgmtId: 7692017993539, via: 1, Ver: > v1, Flags: 110, > [{"com.cloud.agent.api.StopAnswer":{"result":true,"details":"VM does not > exist","wait":0}}] } > 2013-12-26 16:51:25,386 DEBUG [agent.manager.AgentAttache] > (DirectAgent-62:null) Seq 1-265617551: Unable to find listener. > 2013-12-26 16:51:25,386 DEBUG [agent.manager.AgentAttache] > (DirectAgent-62:null) Seq 1-265617551: No more commands found > 2013-12-26 16:51:25,390 DEBUG [vm.dao.VMInstanceDaoImpl] > (HA-Worker-0:work-16) Unable to update VM[SecondaryStorageVm|s-1-VM]: DB > Data={Host=4; State=Running; updated=13; time=Thu Dec 26 16:51:25 PST 2013} > New Data: {Host=4; State=Stopping; updated=12; time=Thu Dec 26 16:51:25 PST > 2013} Stale Data: {Host=4; State=Running; updated=11; time=Thu Dec 26 > 16:51:23 PST 2013} > 2013-12-26 16:51:25,390 DEBUG [cloud.vm.VirtualMachineManagerImpl] > (HA-Worker-0:work-16) Unable to stop VM due to VM is being operated on. > 2013-12-26 16:51:25,390 WARN [cloud.ha.HighAvailabilityManagerImpl] > (HA-Worker-0:work-16) Unable to migrate vm from 4 > 2013-12-26 16:51:25,391 DEBUG [cloud.resource.ResourceManagerImpl] > (HA-Worker-0:work-16) No next resource state for host 4 while current state > is ErrorInMaintenance with event UnableToMigrate > com.cloud.utils.fsm.NoTransitionException: No next resource state found for > current state =ErrorInMaintenance event =UnableToMigrate > at > com.cloud.resource.ResourceManagerImpl.resourceStateTransitTo(ResourceManagerImpl.java:1178) > at > com.cloud.resource.ResourceManagerImpl.maintenanceFailed(ResourceManagerImpl.java:2313) > at > com.cloud.ha.HighAvailabilityManagerImpl.migrate(HighAvailabilityManagerImpl.java:602) > at > com.cloud.ha.HighAvailabilityManagerImpl$WorkerThread.run(HighAvailabilityManagerImpl.java:858) > 2013-12-26 16:51:25,392 INFO [cloud.ha.HighAvailabilityManagerImpl] > (HA-Worker-0:work-16) Completed HAWork[16-Migration-1-Running-Migrating] > 2013-12-26 16:51:26,339 DEBUG [agent.manager.AgentManagerImpl] > (AgentManager-Handler-10:null) SeqA 2-416: Processing Seq 2-416: { Cmd , > MgmtId: -1, via: 2, Ver: v1, Flags: 11, > [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":2,"_loadInfo":"{\n > \"connections\": []\n}","wait":0}}] } > 2013-12-26 16:51:26,346 DEBUG [agent.manager.AgentManagerImpl] > (AgentManager-Handler-10:null) SeqA 2-416: Sending Seq 2-416: { Ans: , > MgmtId: 7692017993539, via: 2, Ver: v1, Flags: 100010, [{"com.cloud.agent.ap -- This message was sent by Atlassian JIRA (v6.1.5#6160)