[
https://issues.apache.org/jira/browse/CLOUDSTACK-5676?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ram Ganesh updated CLOUDSTACK-5676:
-----------------------------------
Assignee: Kelven Yang (was: Devdeep Singh)
> Live migration of VM is failing with a NPE
> ------------------------------------------
>
> Key: CLOUDSTACK-5676
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-5676
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: 4.3.0
> Environment: hyperv , 4.3
> Reporter: Abhinav Roy
> Assignee: Kelven Yang
> Priority: Blocker
> Labels: vmsync
> Fix For: 4.3.0
>
>
> Steps to Reproduce:
> ================
> 1.Bring up CS in advanced zone with 2 hosts in a hyper-v cluster using CIFS
> for both primary and secondary storage
> 2.Deploy one or two guest vms using default cent os template.
> 3.Migrate one of the VMs from Host1 to Host2
> Expected Behaviour:
> =============
> Live VM migration should be successful
> Observed Behaviour:
> ===========
> Live VM migration fails with :
> 013-12-30 16:41:29,564 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Job-Executor-1:ctx-2ba86ed3 ctx-ca0be041) Sync job-95 execution on object
> VmWorkJobQueue.16
> 2013-12-30 16:41:31,261 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-c85fb697) Execute sync-queue item:
> SyncQueueItemVO {id:30, queueId: 27, contentType: AsyncJob, contentId: 95,
> lastProcessMsid: null, lastprocessNumber: null, lastProcessTime: null,
> created: Mon Dec 30 16:41:29 IST 2013}
> 2013-12-30 16:41:31,263 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-c85fb697) Schedule queued job-95
> 2013-12-30 16:41:31,277 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (Job-Executor-2:ctx-ba975c56) Add job-95 into job monitoring
> 2013-12-30 16:41:31,278 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Job-Executor-2:ctx-ba975c56) Executing AsyncJobVO {id:95, userId: 2,
> accountId: 2, instanceType: null, instanceId: null, cmd:
> com.cloud.vm.VmWorkMigrate, cmdInfo:
> rO0ABXNyABpjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZRdxQXtPtzYqAgAGSgAJc3JjSG9zdElkTAAJY2x1c3RlcklkdAAQTGphdmEvbGFuZy9Mb25nO0wABmhvc3RJZHEAfgABTAAFcG9kSWRxAH4AAUwAB3N0b3JhZ2V0AA9MamF2YS91dGlsL01hcDtMAAZ6b25lSWRxAH4AAXhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAEHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAXNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAAAXNxAH4ABwAAAAAAAAAEcQB-AAlwcQB-AAk,
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
> null, initMsid: 280320865129348, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: Mon Dec 30 16:41:29 IST 2013}
> 2013-12-30 16:41:31,279 DEBUG [c.c.v.VmWorkJobDispatcher]
> (Job-Executor-2:ctx-ba975c56) Run VM work job: com.cloud.vm.VmWorkMigrate
> 2013-12-30 16:41:31,287 ERROR [c.c.v.VmWorkJobDispatcher]
> (Job-Executor-2:ctx-ba975c56 ctx-ca0be041) Unable to complete AsyncJobVO
> {id:95, userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd:
> com.cloud.vm.VmWorkMigrate, cmdInfo:
> rO0ABXNyABpjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZRdxQXtPtzYqAgAGSgAJc3JjSG9zdElkTAAJY2x1c3RlcklkdAAQTGphdmEvbGFuZy9Mb25nO0wABmhvc3RJZHEAfgABTAAFcG9kSWRxAH4AAUwAB3N0b3JhZ2V0AA9MamF2YS91dGlsL01hcDtMAAZ6b25lSWRxAH4AAXhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAEHQAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwAAAAAAAAAAXNyAA5qYXZhLmxhbmcuTG9uZzuL5JDMjyPfAgABSgAFdmFsdWV4cgAQamF2YS5sYW5nLk51bWJlcoaslR0LlOCLAgAAeHAAAAAAAAAAAXNxAH4ABwAAAAAAAAAEcQB-AAlwcQB-AAk,
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
> null, initMsid: 280320865129348, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: Mon Dec 30 16:41:29 IST 2013}
> java.lang.NullPointerException
> at
> com.cloud.vm.VmWorkMigrate.getDeployDestination(VmWorkMigrate.java:60)
> at
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:4743)
> at
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:99)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:522)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-12-30 16:41:31,291 ERROR [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Job-Executor-2:ctx-ba975c56) Unexpected exception
> java.lang.NullPointerException
> at
> com.google.gson.JsonPrimitive.isPrimitiveOrString(JsonPrimitive.java:316)
> at com.google.gson.JsonPrimitive.setValue(JsonPrimitive.java:97)
> at com.google.gson.JsonPrimitive.<init>(JsonPrimitive.java:67)
> at
> org.apache.cloudstack.framework.jobs.impl.JobSerializerHelper$ThrowableTypeAdapter.serialize(JobSerializerHelper.java:195)
> at
> org.apache.cloudstack.framework.jobs.impl.JobSerializerHelper$ThrowableTypeAdapter.serialize(JobSerializerHelper.java:158)
> at
> com.google.gson.JsonSerializationVisitor.findAndInvokeCustomSerializer(JsonSerializationVisitor.java:184)
> at
> com.google.gson.JsonSerializationVisitor.visitFieldUsingCustomHandler(JsonSerializationVisitor.java:204)
> at
> com.google.gson.ReflectingFieldNavigator.visitFieldsReflectively(ReflectingFieldNavigator.java:63)
> at com.google.gson.ObjectNavigator.accept(ObjectNavigator.java:120)
> at
> com.google.gson.JsonSerializationContextDefault.serialize(JsonSerializationContextDefault.java:62)
> at
> com.google.gson.JsonSerializationContextDefault.serialize(JsonSerializationContextDefault.java:53)
> at com.google.gson.Gson.toJsonTree(Gson.java:220)
> at com.google.gson.Gson.toJson(Gson.java:260)
> at com.google.gson.Gson.toJson(Gson.java:240)
> at
> org.apache.cloudstack.framework.jobs.impl.JobSerializerHelper.toSerializedString(JobSerializerHelper.java:65)
> at
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:105)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:522)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
> at java.util.concurrent.FutureTask.run(FutureTask.java:166)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603)
> at java.lang.Thread.run(Thread.java:679)
> 2013-12-30 16:41:31,296 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Job-Executor-2:ctx-ba975c56) Complete async job-95, jobStatus: FAILED,
> resultCode: 530, result: null
> 2013-12-30 16:41:31,313 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (Job-Executor-3:ctx-71346dc1) Add job-94 into job monitoring
> 2013-12-30 16:41:31,313 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Job-Executor-3:ctx-71346dc1) Executing AsyncJobVO {id:94, userId: 2,
> accountId: 2, instanceType: None, instanceId: null, cmd:
> org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd, cmdInfo:
> {"response":"json","sessionkey":"lIed4xLaMsDL4yCyAJYyUK0pAJk\u003d","virtualmachineid":"462e6bfc-7c79-4290-9967-8b897ada09e1","cmdEventType":"VM.MIGRATE","hostid":"655f6fb2-82b7-4388-9462-3c15f824e5f9","ctxUserId":"2","httpmethod":"GET","_":"1388401853884","ctxAccountId":"2","ctxStartEventId":"216"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
> null, initMsid: 280320865129348, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: Mon Dec 30 16:41:29 IST 2013}
> 2013-12-30 16:41:31,328 WARN [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Job-Executor-3:ctx-71346dc1) job-94 is scheduled for wakeup run, but there
> is no joining info anymore
> 2013-12-30 16:41:31,333 ERROR [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Job-Executor-3:ctx-71346dc1) Unable to find a wakeup dispatcher from the
> joined job: AsyncJobVO {id:94, userId: 2, accountId: 2, instanceType: None,
> instanceId: null, cmd:
> org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd, cmdInfo:
> {"response":"json","sessionkey":"lIed4xLaMsDL4yCyAJYyUK0pAJk\u003d","virtualmachineid":"462e6bfc-7c79-4290-9967-8b897ada09e1","cmdEventType":"VM.MIGRATE","hostid":"655f6fb2-82b7-4388-9462-3c15f824e5f9","ctxUserId":"2","httpmethod":"GET","_":"1388401853884","ctxAccountId":"2","ctxStartEventId":"216"},
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
> null, initMsid: 280320865129348, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: Mon Dec 30 16:41:29 IST 2013}
> 2013-12-30 16:41:31,334 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Job-Executor-3:ctx-71346dc1) Done executing
> org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd for job-94
> 2013-12-30 16:41:31,336 DEBUG [o.a.c.f.j.i.SyncQueueManagerImpl]
> (Job-Executor-2:ctx-ba975c56) Sync queue (27) is currently empty
> 2013-12-30 16:41:31,337 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (Job-Executor-2:ctx-ba975c56) Remove job-95 from job monitoring
> 2013-12-30 16:41:31,347 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (Job-Executor-3:ctx-71346dc1) Remove job-94 from job monitoring
> 2013-12-30 16:41:31,364 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Job-Executor-1:ctx-2ba86ed3 ctx-ca0be041) Complete async job-94, jobStatus:
> SUCCEEDED, resultCode: 0, result:
> org.apache.cloudstack.api.response.UserVmResponse/virtualmachine/{"id":"462e6bfc-7c79-4290-9967-8b897ada09e1","name":"av8","displayname":"av8","account":"admin","domainid":"3478c004-7129-11e3-89a3-fef34996d384","domain":"ROOT","created":"2013-12-30T16:31:47+0530","state":"Running","haenable":false,"zoneid":"aaa51135-ad9e-4053-9023-17b7923b8694","zonename":"BLR","hostid":"cd56ae63-24c8-4e55-be45-c133f9ecb50f","hostname":"10.102.192.14","templateid":"3483f6ae-7129-11e3-89a3-fef34996d384","templatename":"CentOS
> 6.4(64-bit) GUI (Hyperv)","templatedisplaytext":"CentOS 6.4 (64-bit) GUI
> (Hyperv)","passwordenabled":false,"serviceofferingid":"869e9873-ab4c-48c1-a57b-58b73f2939ec","serviceofferingname":"Small
>
> Instance","cpunumber":1,"cpuspeed":500,"memory":512,"cpuused":"0%","networkkbsread":1,"networkkbswrite":1,"guestosid":"34a20e96-7129-11e3-89a3-fef34996d384","rootdeviceid":0,"rootdevicetype":"ROOT","securitygroup":[],"nic":[{"id":"e897a7eb-4bea-4335-8925-a240c95df9da","networkid":"80e01c55-75b3-4e7e-9668-fe46ecae4ea8","networkname":"an1","netmask":"255.255.255.0","gateway":"10.1.1.1","ipaddress":"10.1.1.7","isolationuri":"vlan://680","broadcasturi":"vlan://680","traffictype":"Guest","type":"Isolated","isdefault":true,"macaddress":"02:00:08:a2:00:09"}],"hypervisor":"Hyperv","instancename":"i-2-16-VM","tags":[],"affinitygroup":[],"displayvm":true,"isdynamicallyscalable":false}
> 2013-12-30 16:41:31,372 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Job-Executor-1:ctx-2ba86ed3) Done executing
> org.apache.cloudstack.api.command.admin.vm.MigrateVMCmd for job-94
> 2013-12-30 16:41:31,382 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (Job-Executor-1:ctx-2ba86ed3) Remove job-94 from job monitoring
> 2013-12-30 16:41:32,507 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-16:ctx-5f391aea) ===START=== 10.144.7.20 -- GET
> command=queryAsyncJobResult&jobId=f94905d3-a6e7-4119-aa2a-a58c673979c4&response=json&sessionkey=lIed4xLaMsDL4yCyAJYyUK0pAJk%3D&_=1388401856795
> Workaround :
> =========================
> Add a new entry in cloud.configurations table with
> vm.job.enabled set to "false"
> restart management server. After this Live VM migration works fine.
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
