Likitha Shetty created CLOUDSTACK-8118: ------------------------------------------
Summary: Root volume migration fails with 'No such disk device' in case of vCenter 5.5 setup Key: CLOUDSTACK-8118 URL: https://issues.apache.org/jira/browse/CLOUDSTACK-8118 Project: CloudStack Issue Type: Bug Security Level: Public (Anyone can view this level - this is the default.) Affects Versions: 4.5.0 Reporter: Likitha Shetty Assignee: Likitha Shetty Priority: Critical Fix For: Future +Steps to reproduce+ 1. Setup CS with vCenter 5.5. 2. Deploy a VM in cluster. Say ROOT volume is in primary storage P1 3. Add another primary storage to the same cluster (P2). 4. Migrate ROOT volume to primary storage P2. Migration fails with - {noformat} 2014-12-24 15:13:29,299 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-325:ctx-0dbf332a) Seq 5-6870804181507113730: Executing request 2014-11-24 15:13:29,299 INFO [c.c.h.v.r.VmwareResource] (DirectAgent-325:ctx-0dbf332a 10.147.40.14, job-760/job-761, cmd: MigrateVolumeCommand) Executing resource Mi grateVolumeCommand: {"volumeId":106,"volumePath":"ROOT-40-000001","pool": {"id":8,"uuid":"251a42e1-7316-30a8-ad11-bf03421f39f2","host":"10.147.28.7","path":"/export/home/manasa/priVMw2","port":2049,"type":"NetworkFilesystem"} ,"attachedVmName":"i-2-40-VM","wait":3600} 2014-12-24 15:13:29,422 INFO [c.c.h.v.m.VirtualMachineMO] (DirectAgent-325:ctx-0dbf332a 10.147.40.14, job-760/job-761, cmd: MigrateVolumeCommand) Look for disk devic e info from volume : ROOT-40-000001.vmdk with trimmed base name: ROOT-40 2014-12-24 15:13:29,422 INFO [c.c.h.v.m.VirtualMachineMO] (DirectAgent-325:ctx-0dbf332a 10.147.40.14, job-760/job-761, cmd: MigrateVolumeCommand) Test against disk d evice, controller key: 1000, unit number: 0 2014-11-24 15:13:29,422 INFO [c.c.h.v.m.VirtualMachineMO] (DirectAgent-325:ctx-0dbf332a 10.147.40.14, job-760/job-761, cmd: MigrateVolumeCommand) Test against disk b acking : [f159abf3a6863ff1ad6a24d59100bdb9] vm-test/ROOT-40-000001.vmdk 2014-12-24 15:13:29,422 ERROR [c.c.h.v.r.VmwareResource] (DirectAgent-325:ctx-0dbf332a 10.147.40.14, job-760/job-761, cmd: MigrateVolumeCommand) Catch Exception java. lang.Exception due to java.lang.Exception: No such disk device: ROOT-40-000001.vmdk java.lang.Exception: No such disk device: ROOT-40-000001.vmdk at com.cloud.hypervisor.vmware.resource.VmwareResource.getVirtualDiskInfo(VmwareResource.java:3366) at com.cloud.hypervisor.vmware.resource.VmwareResource.execute(VmwareResource.java:3319) at com.cloud.hypervisor.vmware.resource.VmwareResource.executeRequest(VmwareResource.java:403) at com.cloud.agent.manager.DirectAgentAttache$Task.runInContext(DirectAgentAttache.java:304) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:178) at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:292) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) 2014-11-24 15:13:29,424 DEBUG [c.c.a.m.DirectAgentAttache] (DirectAgent-325:ctx-0dbf332a) Seq 5-6870804181507113730: Response Received: 2014-11-24 15:13:29,425 DEBUG [c.c.a.t.Request] (DirectAgent-325:ctx-0dbf332a) Seq 5-6870804181507113730: Processing: { Ans: , MgmtId: 6662467354709, via: 5, Ver: v1 , Flags: 110, [{"com.cloud.agent.api.storage.MigrateVolumeAnswer":{"result":false,"details":"Catch Exception java.lang.Exception due to java.lang.Exception: No such d isk device: ROOT-40-000001.vmdk","wait":0}}] } 2014-12-24 15:13:29,425 DEBUG [c.c.a.t.Request] (Work-Job-Executor-151:ctx-1199aeb3 job-760/job-761 ctx-63b18747) Seq 5-6870804181507113730: Received: { Ans: , MgmtI d: 6662467354709, via: 5, Ver: v1, Flags: 110, { MigrateVolumeAnswer } } 2014-11-24 15:13:29,425 DEBUG [o.a.c.s.m.AncientDataMotionStrategy] (Work-Job-Executor-151:ctx-1199aeb3 job-760/job-761 ctx-63b18747) copy failed com.cloud.utils.exception.CloudRuntimeException: Failed to migrate volume org.apache.cloudstack.storage.volume.VolumeObject@58ac8d1b to storage pool org.apache.clouds tack.storage.datastore.PrimaryDataStoreImpl@28641518 at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.migrateVolumeToPool(AncientDataMotionStrategy.java:390) at org.apache.cloudstack.storage.motion.AncientDataMotionStrategy.copyAsync(AncientDataMotionStrategy.java:430) at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:68) at org.apache.cloudstack.storage.motion.DataMotionServiceImpl.copyAsync(DataMotionServiceImpl.java:73) at org.apache.cloudstack.storage.volume.VolumeServiceImpl.migrateVolume(VolumeServiceImpl.java:1081) at com.cloud.storage.VolumeApiServiceImpl.liveMigrateVolume(VolumeApiServiceImpl.java:1835) at com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:1821) at com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:2611) at sun.reflect.GeneratedMethodAccessor900.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:2627) at sun.reflect.GeneratedMethodAccessor633.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at $Proxy185.handleVmWorkJob(Unknown Source) at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:103) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:536) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53) at org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:493) at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334) at java.util.concurrent.FutureTask.run(FutureTask.java:166) at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1110) at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:603) at java.lang.Thread.run(Thread.java:722) 2014-11-24 15:13:29,427 DEBUG [c.c.a.m.AgentAttache] (DirectAgent-325:ctx-0dbf332a) Seq 5-6870804181507113730: No more commands found 2014-12-24 15:13:29,480 WARN [o.a.c.s.d.ObjectInDataStoreManagerImpl] (Work-Job-Executor-151:ctx-1199aeb3 job-760/job-761 ctx-63b18747) Unsupported data object (VOLU ME, org.apache.cloudstack.storage.datastore.PrimaryDataStoreImpl@5fadb7eb), no need to delete from object in store ref table 2014-12-24 15:13:29,481 DEBUG [c.c.s.VolumeApiServiceImpl] (Work-Job-Executor-151:ctx-1199aeb3 job-760/job-761 ctx-63b18747) migrate volume failed:com.cloud.utils.exc eption.CloudRuntimeException: Failed to migrate volume org.apache.cloudstack.storage.volume.VolumeObject@58ac8d1b to storage pool org.apache.cloudstack.storage.datast ore.PrimaryDataStoreImpl@28641518 2014-11-24 15:13:29,481 ERROR [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-151:ctx-1199aeb3 job-760/job-761 ctx-63b18747) Invocation exception, caused by: java.la ng.NullPointerException 2014-12-24 15:13:29,481 INFO [c.c.v.VmWorkJobHandlerProxy] (Work-Job-Executor-151:ctx-1199aeb3 job-760/job-761 ctx-63b18747) Rethrow exception java.lang.NullPointerE xception 2014-12-24 15:13:29,481 DEBUG [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-151:ctx-1199aeb3 job-760/job-761) Done with run of VM work job: com.cloud.vm.VmWorkMigrat eVolume for VM 40, job origin: 760 2014-12-24 15:13:29,481 ERROR [c.c.v.VmWorkJobDispatcher] (Work-Job-Executor-151:ctx-1199aeb3 job-760/job-761) Unable to complete AsyncJobVO {id:761, userId: 2, accou ntId: 2, instanceType: null, instanceId: null, cmd: com.cloud.vm.VmWorkMigrateVolume, cmdInfo: rO0ABXNyACBjb20uY2xvdWQudm0uVm1Xb3JrTWlncmF0ZVZvbHVtZfgl82-871PmAgADSgA KZGVzdFBvb2xJZFoAC2xpdmVNaWdyYXRlSgAIdm9sdW1lSWR4cgATY29tLmNsb3VkLnZtLlZtV29ya5-ZtlbwJWdrAgAESgAJYWNjb3VudElkSgAGdXNlcklkSgAEdm1JZEwAC2hhbmRsZXJOYW1ldAASTGphdmEvbGFuZ y9TdHJpbmc7eHAAAAAAAAAAAgAAAAAAAAACAAAAAAAAACh0ABRWb2x1bWVBcGlTZXJ2aWNlSW1wbAAAAAAAAAAIAQAAAAAAAABq, cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result: null, initMsid: 6662467354709, completeMsid: null, lastUpdated: null, lastPolled: null, created: Mon Nov 24 15:13:27 IST 2014} , job origin:760 java.lang.NullPointerException at com.cloud.storage.VolumeApiServiceImpl.orchestrateMigrateVolume(VolumeApiServiceImpl.java:2612) at sun.reflect.GeneratedMethodAccessor900.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107) at com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:2627) at sun.reflect.GeneratedMethodAccessor633.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.lang.reflect.Method.invoke(Method.java:601) at org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:317) at org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:183) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:150) at org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91) at org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:172) at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:204) at $Proxy185.handleVmWorkJob(Unknown Source) at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:103) at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:536) at org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56) at org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103) {noformat} -- This message was sent by Atlassian JIRA (v6.3.4#6332)