[
https://issues.apache.org/jira/browse/CLOUDSTACK-8412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Likitha Shetty resolved CLOUDSTACK-8412.
----------------------------------------
Resolution: Fixed
+Root Cause Analysis+
When a request is made to migrate a VM along with its storage, CS spawns an
internal VM work job (VmWorkMigrateWithStorage). If this job is being processed
by a MS that is not the owner of the host that the VM belongs to, then the
request will be forwarded to the MS that owns the VM's host. In case of any
forwarded request, CCP attempts to de-serialize the JSON request. But in case
of 'MigrateWithStorageCommand' this de-serialization fails
This issue will be intermittently observed in a clustered Management Server
(MS) because the issue will only occur if the VM Migrate job is not being
processed by the MS that owns the host that the VM being migrated belongs to.
+Steps to reproduce+
1. Setup a clustered management server environment.
2. Have at least 2 hosts and 2 primary storages in one cluster.
3. Deploy at least a couple of VMs.
4. Call 'migrateVirtualMachineWithVolume' API to migrate the VMs between hosts
and storage pools.
5. Retry till the error is seen in the management server that owns the source
host.
> VM migration with storage fails in a clustered management server setup
> ----------------------------------------------------------------------
>
> Key: CLOUDSTACK-8412
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-8412
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Reporter: Likitha Shetty
> Assignee: Likitha Shetty
> Priority: Critical
> Fix For: 4.6.0
>
>
> When a VM migration with storage request is forwarded by CS from one MS to
> another, de-serialization of the JSON request will fail in second management
> server with the below error -
> {noformat}
> 2015-04-18 10:43:33,232 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-52:ctx-d5bbb06c job-120140) (logid:88d08ca3) Add job-120140
> into job monitoring
> 2015-04-18 10:43:33,245 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-52:ctx-d5bbb06c job-120140) (logid:58b8c3e1) Executing
> AsyncJobVO {id:120140, userId: 42, accountId: 2, instanceType: None,
> instanceId: null, cmd:
> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd,
> cmdInfo:
> {"virtualmachineid":"cf879643-3143-4822-a727-0223783d0553","cmdEventType":"VM.MIGRATE","ctxUserId":"42","httpmethod":"GET","migrateto[0].pool":"f04a44e1-d75e-3829-813d-8bb6f356f3c2","migrateto[0].volume":"933adc86-44aa-4afd-b704-d5572f513a13","apiKey":"LrQgAYP2C8u47ZnFahkISur_4eKlWLTZK7fnkOJw5AuMtgz3s0KFKP1d6ow_mx5ecbI3zS8w18fG-ljIZ2m5dA","response":"json","ctxDetails":"{\"com.cloud.vm.VirtualMachine\":\"cf879643-3143-4822-a727-0223783d0553\",\"com.cloud.host.Host\":\"9a74034e-2dbd-4b4f-84e3-9a7d828fdcb4\"}","hostid":"9a74034e-2dbd-4b4f-84e3-9a7d828fdcb4","ctxAccountId":"2","ctxStartEventId":"64200","signature":"gmEZXBo5X0Lwcc4fQ9MAgvsAoW8\u003d"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
> null, initMsid: 345049612948, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: null}
> 2015-04-18 10:43:33,245 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (catalina-exec-17:ctx-53d5ddc1 ctx-7d9c3e95 ctx-ba112db2) (logid:69812f7a)
> submit async job-120140, details: AsyncJobVO {id:120140, userId: 42,
> accountId: 2, instanceType: None, instanceId: null, cmd:
> org.apache.cloudstack.api.command.admin.vm.MigrateVirtualMachineWithVolumeCmd,
> cmdInfo:
> {"virtualmachineid":"cf879643-3143-4822-a727-0223783d0553","cmdEventType":"VM.MIGRATE","ctxUserId":"42","httpmethod":"GET","migrateto[0].pool":"f04a44e1-d75e-3829-813d-8bb6f356f3c2","migrateto[0].volume":"933adc86-44aa-4afd-b704-d5572f513a13","apiKey":"LrQgAYP2C8u47ZnFahkISur_4eKlWLTZK7fnkOJw5AuMtgz3s0KFKP1d6ow_mx5ecbI3zS8w18fG-ljIZ2m5dA","response":"json","ctxDetails":"{\"com.cloud.vm.VirtualMachine\":\"cf879643-3143-4822-a727-0223783d0553\",\"com.cloud.host.Host\":\"9a74034e-2dbd-4b4f-84e3-9a7d828fdcb4\"}","hostid":"9a74034e-2dbd-4b4f-84e3-9a7d828fdcb4","ctxAccountId":"2","ctxStartEventId":"64200","signature":"gmEZXBo5X0Lwcc4fQ9MAgvsAoW8\u003d"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
> null, initMsid: 345049612948, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: null}
> 2015-04-18 10:43:33,301 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-52:ctx-d5bbb06c job-120140 ctx-36e40c6e) (logid:58b8c3e1)
> Sync job-120141 execution on object VmWorkJobQueue.2082
> 2015-04-18 10:43:33,985 ERROR [c.c.a.t.Request] (AgentManager-Handler-1:null)
> (logid:) Caught problem with
> [{"com.cloud.agent.api.MigrateWithStorageCommand":{"vm":{"id":2082,"name":"i-3-2082-VM","bootloader":"HVM","type":"User","cpus":2,"minSpeed":1860,"maxSpeed":1860,"minRam":4294967296,"maxRam":4294967296,"hostName":"PKNSRV-DC01","arch":"x86_64","os":"Windows
> Server 2012 R2
> (64-bit)","platformEmulator":"windows8Server64Guest","bootArgs":"","enableHA":false,"limitCpuUse":false,"enableDynamicallyScaleVm":true,"vncPassword":"f1ce1313c3a2ce1c","params":{"memoryOvercommitRatio":"1","Message.ReservedCapacityFreed.Flag":"true","dataDiskController":"osdefault","cpuOvercommitRatio":"4.0","nicAdapter":"Vmxnet3","vmware.reserve.cpu":"false","nestedVirtualizationFlag":"false","rootDiskController":"osdefault","vmware.reserve.mem":"false"},"uuid":"cf879643-3143-4822-a727-0223783d0553","disks":[{"data":{"org.apache.cloudstack.storage.to.VolumeObjectTO":{"uuid":"933adc86-44aa-4afd-b704-d5572f513a13","volumeType":"ROOT","dataStore":{"org.apache.cloudstack.storage.to.PrimaryDataStoreTO":{"uuid":"f14aba0e-cd98-3397-81da-a8e00f23f1eb","id":216,"poolType":"VMFS","host":"VMFS
> datastore:
> /REALCLOUD/RCBLNSAN03-VMCLP-Vol02","path":"/REALCLOUD/RCBLNSAN03-VMCLP-Vol02","port":0,"url":"VMFS://VMFS
> datastore:
> /REALCLOUD/RCBLNSAN03-VMCLP-Vol02/REALCLOUD/RCBLNSAN03-VMCLP-Vol02/?ROLE\u003dPrimary\u0026STOREUUID\u003df14aba0e-cd98-3397-81da-a8e00f23f1eb"}},"name":"ROOT-2082","size":42949672960,"path":"ROOT-2082-000001","volumeId":2790,"vmName":"i-3-2082-VM","accountId":3,"chainInfo":"{\"diskDeviceBusName\":\"scsi0:0\",\"diskChain\":[\"[RCBLNSAN03-VMCLP-Vol02]
>
> i-3-2082-VM/ROOT-2082-000001.vmdk\"]}","format":"OVA","provisioningType":"THIN","id":2790,"deviceId":0,"cacheMode":"NONE","hypervisorType":"VMware"}},"diskSeq":0,"path":"ROOT-2082-000001","type":"ROOT","_details":{"managed":"false","storagePort":"0","storageHost":"VMFS
> datastore:
> /REALCLOUD/RCBLNSAN03-VMCLP-Vol02","volumeSize":"42949672960"}},{"data":{"org.apache.cloudstack.storage.to.TemplateObjectTO":{"id":0,"format":"ISO","accountId":0,"hvm":false}},"diskSeq":3,"type":"ISO"}],"nics":[{"deviceId":0,"networkRateMbps":1000,"defaultNic":true,"pxeDisable":false,"nicUuid":"e6cab6cc-4378-4797-9dbf-e901d7d6c806","uuid":"aa7ff2c7-738e-47dc-bdaf-8e081c63053d","ip":"10.10.1.8","netmask":"255.255.248.0","gateway":"10.10.4.4","mac":"06:92:2a:00:05:dc","dns1":"8.8.8.8","dns2":"8.8.4.4","broadcastType":"Vlan","type":"Guest","broadcastUri":"vlan://2","isolationUri":"vlan://2","isSecurityGroupEnabled":false,"name":"vSwitch2"}]},"volumeToFiler":{"Vol[2790|ROOT|ROOT-2082-000001|42949672960]":{"id":244,"uuid":"f04a44e1-d75e-3829-813d-8bb6f356f3c2","host":"VMFS
> datastore:
> /REALCLOUD/RCBLNEMC01-VMCLP-Vol05","path":"/REALCLOUD/RCBLNEMC01-VMCLP-Vol05","port":0,"type":"VMFS"}},"tgtHost":"HostSystem:host-25811@10.90.2.11","contextMap":{"job":"job-120140/job-120141","logid":"58b8c3e1"},"wait":0}}]
> com.google.gson.JsonParseException: Expecting object found:
> "Vol[2790|ROOT|ROOT-2082-000001|42949672960]"
> at
> com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:100)
> at
> com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
> at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
> at
> com.google.gson.JsonDeserializationContextDefault.fromJsonPrimitive(JsonDeserializationContextDefault.java:85)
> at
> com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:56)
> at com.google.gson.MapTypeAdapter.deserialize(MapTypeAdapter.java:67)
> at com.google.gson.MapTypeAdapter.deserialize(MapTypeAdapter.java:33)
> at
> com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
> at
> com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
> at
> com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:117)
> at
> com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
> at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
> at
> com.google.gson.JsonDeserializationContextDefault.fromJsonObject(JsonDeserializationContextDefault.java:76)
> at
> com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:54)
> at com.google.gson.Gson.fromJson(Gson.java:551)
> at com.google.gson.Gson.fromJson(Gson.java:521)
> at
> com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:80)
> at
> com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:40)
> at
> com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
> at
> com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
> at
> com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:117)
> at
> com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
> at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
> at
> com.google.gson.JsonDeserializationContextDefault.fromJsonObject(JsonDeserializationContextDefault.java:76)
> at
> com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:54)
> at com.google.gson.Gson.fromJson(Gson.java:551)
> at com.google.gson.Gson.fromJson(Gson.java:521)
> at
> com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:80)
> at
> com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:40)
> at
> com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
> at
> com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
> at
> com.google.gson.JsonDeserializationVisitor.visitUsingCustomHandler(JsonDeserializationVisitor.java:80)
> at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:101)
> at
> com.google.gson.JsonDeserializationContextDefault.fromJsonArray(JsonDeserializationContextDefault.java:67)
> at
> com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:52)
> at com.google.gson.Gson.fromJson(Gson.java:551)
> at com.google.gson.Gson.fromJson(Gson.java:498)
> at com.cloud.agent.transport.Request.getCommands(Request.java:249)
> at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:347)
> at
> com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:656)
> at com.cloud.utils.nio.Task.run(Task.java:84)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> 2015-04-18 10:43:33,986 WARN [c.c.u.n.Task] (AgentManager-Handler-1:null)
> (logid:) Caught the following exception but pushing on
> com.google.gson.JsonParseException: Expecting object found:
> "Vol[2790|ROOT|ROOT-2082-000001|42949672960]"
> at
> com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:100)
> at
> com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
> at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
> at
> com.google.gson.JsonDeserializationContextDefault.fromJsonPrimitive(JsonDeserializationContextDefault.java:85)
> at
> com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:56)
> at com.google.gson.MapTypeAdapter.deserialize(MapTypeAdapter.java:67)
> at com.google.gson.MapTypeAdapter.deserialize(MapTypeAdapter.java:33)
> at
> com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
> at
> com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
> at
> com.google.gson.JsonObjectDeserializationVisitor.visitFieldUsingCustomHandler(JsonObjectDeserializationVisitor.java:117)
> at
> com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
> at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
> at
> com.google.gson.JsonDeserializationContextDefault.fromJsonObject(JsonDeserializationContextDefault.java:76)
> at
> com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:54)
> at com.google.gson.Gson.fromJson(Gson.java:551)
> at com.google.gson.Gson.fromJson(Gson.java:521)
> at
> com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:80)
> at
> com.cloud.agent.transport.ArrayTypeAdaptor.deserialize(ArrayTypeAdaptor.java:40)
> at
> com.google.gson.JsonDeserializerExceptionWrapper.deserialize(JsonDeserializerExceptionWrapper.java:51)
> at
> com.google.gson.JsonDeserializationVisitor.invokeCustomDeserializer(JsonDeserializationVisitor.java:92)
> at
> com.google.gson.JsonDeserializationVisitor.visitUsingCustomHandler(JsonDeserializationVisitor.java:80)
> at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:101)
> at
> com.google.gson.JsonDeserializationContextDefault.fromJsonArray(JsonDeserializationContextDefault.java:67)
> at
> com.google.gson.JsonDeserializationContextDefault.deserialize(JsonDeserializationContextDefault.java:52)
> at com.google.gson.Gson.fromJson(Gson.java:551)
> at com.google.gson.Gson.fromJson(Gson.java:498)
> at com.cloud.agent.transport.Request.getCommands(Request.java:249)
> at com.cloud.agent.manager.AgentAttache.send(AgentAttache.java:347)
> at
> com.cloud.agent.manager.ClusteredAgentManagerImpl$ClusteredAgentHandler.doTask(ClusteredAgentManagerImpl.java:656)
> at com.cloud.utils.nio.Task.run(Task.java:84)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:722)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
