Hi Rafael,
Sorry for the delay, we are trying to go deeper into this issue this weekend
but without success.
The process stuck when CS needs to copy the VDI from storage to another
(primary to secondary for instance..).
Here the stack trace asked :
2018-11-04 23:35:45,376 ERROR [c.c.a.ApiAsyncJobDispatcher]
(API-Job-Executor-19:ctx-9f5e7b06 job-6792) (logid:190de247) Unexpected
exception while executing
org.apache.cloudstack.api.command.user.volume.ExtractVolumeCmd
com.cloud.utils.exception.CloudRuntimeException: Failed to copy the volume from
the source primary storage pool to secondary storage.
at
com.cloud.storage.VolumeApiServiceImpl.orchestrateExtractVolume(VolumeApiServiceImpl.java:2529)
at
com.cloud.storage.VolumeApiServiceImpl.orchestrateExtractVolume(VolumeApiServiceImpl.java:3139)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:3178)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy196.handleVmWorkJob(Unknown Source)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-11-04 23:35:45,377 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
(API-Job-Executor-19:ctx-9f5e7b06 job-6792) (logid:190de247) Complete async
job-6792, jobStatus: FAILED, resultCode: 530, result:
org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Failed
to copy the volume from the source primary storage pool to secondary storage."}
2018-11-02 11:16:07,785 DEBUG [o.a.c.s.v.VolumeObject]
(Work-Job-Executor-59:ctx-9c88dddd job-6718/job-6719 ctx-efc91d21)
(logid:ccd69ffe) Failed to update state
com.cloud.utils.exception.CloudRuntimeException: Failed to transit volume: 619,
due to: com.cloud.utils.fsm.NoTransitionException: Unable to transition to a
new state from Copying via CopyRequested
at
org.apache.cloudstack.storage.volume.VolumeObject.stateTransit(VolumeObject.java:188)
at
org.apache.cloudstack.storage.volume.VolumeObject.processEvent(VolumeObject.java:329)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.copyVolumeFromImageToPrimary(VolumeServiceImpl.java:1311)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.copyVolume(VolumeServiceImpl.java:1413)
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.copyVolumeFromSecToPrimary(VolumeOrchestrator.java:492)
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.copyVolume(VolumeOrchestrator.java:797)
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.createVolumeOnPrimaryStorage(VolumeOrchestrator.java:817)
at
com.cloud.storage.VolumeApiServiceImpl.orchestrateAttachVolumeToVM(VolumeApiServiceImpl.java:1428)
at
com.cloud.storage.VolumeApiServiceImpl.orchestrateAttachVolumeToVM(VolumeApiServiceImpl.java:3145)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:3178)
at sun.reflect.GeneratedMethodAccessor559.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy196.handleVmWorkJob(Unknown Source)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-11-02 11:16:07,785 ERROR [o.a.c.s.v.VolumeServiceImpl]
(Work-Job-Executor-59:ctx-9c88dddd job-6718/job-6719 ctx-efc91d21)
(logid:ccd69ffe) failed to copy volume from image store
com.cloud.utils.exception.CloudRuntimeException: Failed to update
state:com.cloud.utils.exception.CloudRuntimeException: Failed to transit
volume: 619, due to: com.cloud.utils.fsm.NoTransitionException: Unable to
transition to a new state from Copying via CopyRequested
at
org.apache.cloudstack.storage.volume.VolumeObject.processEvent(VolumeObject.java:332)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.copyVolumeFromImageToPrimary(VolumeServiceImpl.java:1311)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.copyVolume(VolumeServiceImpl.java:1413)
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.copyVolumeFromSecToPrimary(VolumeOrchestrator.java:492)
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.copyVolume(VolumeOrchestrator.java:797)
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.createVolumeOnPrimaryStorage(VolumeOrchestrator.java:817)
at
com.cloud.storage.VolumeApiServiceImpl.orchestrateAttachVolumeToVM(VolumeApiServiceImpl.java:1428)
at
com.cloud.storage.VolumeApiServiceImpl.orchestrateAttachVolumeToVM(VolumeApiServiceImpl.java:3145)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:3178)
at sun.reflect.GeneratedMethodAccessor559.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy196.handleVmWorkJob(Unknown Source)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-11-02 11:16:07,811 DEBUG [o.a.c.s.v.VolumeObject]
(Work-Job-Executor-59:ctx-9c88dddd job-6718/job-6719 ctx-efc91d21)
(logid:ccd69ffe) Failed to transit volume: 619, due to:
com.cloud.utils.fsm.NoTransitionException: Unable to transition to a new state
from Uploaded via OperationFailed
2018-11-02 11:16:07,811 DEBUG [o.a.c.s.v.VolumeObject]
(Work-Job-Executor-59:ctx-9c88dddd job-6718/job-6719 ctx-efc91d21)
(logid:ccd69ffe) Failed to update state
com.cloud.utils.exception.CloudRuntimeException: Failed to transit volume: 619,
due to: com.cloud.utils.fsm.NoTransitionException: Unable to transition to a
new state from Uploaded via OperationFailed
at
org.apache.cloudstack.storage.volume.VolumeObject.stateTransit(VolumeObject.java:188)
at
org.apache.cloudstack.storage.volume.VolumeObject.processEvent(VolumeObject.java:329)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.copyVolumeFromImageToPrimary(VolumeServiceImpl.java:1325)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.copyVolume(VolumeServiceImpl.java:1413)
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.copyVolumeFromSecToPrimary(VolumeOrchestrator.java:492)
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.copyVolume(VolumeOrchestrator.java:797)
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.createVolumeOnPrimaryStorage(VolumeOrchestrator.java:817)
at
com.cloud.storage.VolumeApiServiceImpl.orchestrateAttachVolumeToVM(VolumeApiServiceImpl.java:1428)
at
com.cloud.storage.VolumeApiServiceImpl.orchestrateAttachVolumeToVM(VolumeApiServiceImpl.java:3145)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:3178)
at sun.reflect.GeneratedMethodAccessor559.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy196.handleVmWorkJob(Unknown Source)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-11-02 11:16:07,811 ERROR [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-59:ctx-9c88dddd job-6718/job-6719 ctx-efc91d21)
(logid:ccd69ffe) Invocation exception, caused by:
com.cloud.utils.exception.CloudRuntimeException: Failed to update
state:com.cloud.utils.exception.CloudRuntimeException: Failed to transit
volume: 619, due to: com.cloud.utils.fsm.NoTransitionException: Unable to
transition to a new state from Uploaded via OperationFailed
2018-11-02 11:16:07,812 INFO [c.c.v.VmWorkJobHandlerProxy]
(Work-Job-Executor-59:ctx-9c88dddd job-6718/job-6719 ctx-efc91d21)
(logid:ccd69ffe) Rethrow exception
com.cloud.utils.exception.CloudRuntimeException: Failed to update
state:com.cloud.utils.exception.CloudRuntimeException: Failed to transit
volume: 619, due to: com.cloud.utils.fsm.NoTransitionException: Unable to
transition to a new state from Uploaded via OperationFailed
2018-11-02 11:16:07,812 DEBUG [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-59:ctx-9c88dddd job-6718/job-6719) (logid:ccd69ffe) Done
with run of VM work job: com.cloud.vm.VmWorkAttachVolume for VM 502, job
origin: 6718
2018-11-02 11:16:07,812 ERROR [c.c.v.VmWorkJobDispatcher]
(Work-Job-Executor-59:ctx-9c88dddd job-6718/job-6719) (logid:ccd69ffe) Unable
to complete AsyncJobVO {id:6719, userId: 2, accountId: 2, instanceType: null,
instanceId: null, cmd: com.cloud.vm.VmWorkAttachVolume, cmdInfo:
rO0ABXNyAB9jb20uY2xvdWQudm0uVm1Xb3JrQXR0YWNoVm9sdW1lB62v-WGH4hwCAAJMAAhkZXZpY2VJZHQAEExqYXZhL2xhbmcvTG9uZztMAAh2b2x1bWVJZHEAfgABeHIAE2NvbS5jbG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZEoABnVzZXJJZEoABHZtSWRMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hwAAAAAAAAAAIAAAAAAAAAAgAAAAAAAAH2dAAUVm9sdW1lQXBpU2VydmljZUltcGxwc3IADmphdmEubGFuZy5Mb25nO4vkkMyPI98CAAFKAAV2YWx1ZXhyABBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAAAAAAAAJr,
cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
null, initMsid: 2886211232, completeMsid: null, lastUpdated: null, lastPolled:
null, created: Fri Nov 02 11:16:06 CET 2018}, job origin:6718
com.cloud.utils.exception.CloudRuntimeException: Failed to update
state:com.cloud.utils.exception.CloudRuntimeException: Failed to transit
volume: 619, due to: com.cloud.utils.fsm.NoTransitionException: Unable to
transition to a new state from Uploaded via OperationFailed
at
org.apache.cloudstack.storage.volume.VolumeObject.processEvent(VolumeObject.java:332)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.copyVolumeFromImageToPrimary(VolumeServiceImpl.java:1325)
at
org.apache.cloudstack.storage.volume.VolumeServiceImpl.copyVolume(VolumeServiceImpl.java:1413)
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.copyVolumeFromSecToPrimary(VolumeOrchestrator.java:492)
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.copyVolume(VolumeOrchestrator.java:797)
at
org.apache.cloudstack.engine.orchestration.VolumeOrchestrator.createVolumeOnPrimaryStorage(VolumeOrchestrator.java:817)
at
com.cloud.storage.VolumeApiServiceImpl.orchestrateAttachVolumeToVM(VolumeApiServiceImpl.java:1428)
at
com.cloud.storage.VolumeApiServiceImpl.orchestrateAttachVolumeToVM(VolumeApiServiceImpl.java:3145)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
at
com.cloud.storage.VolumeApiServiceImpl.handleVmWorkJob(VolumeApiServiceImpl.java:3178)
at sun.reflect.GeneratedMethodAccessor559.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at
org.springframework.aop.support.AopUtils.invokeJoinpointUsingReflection(AopUtils.java:338)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.invokeJoinpoint(ReflectiveMethodInvocation.java:197)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:163)
at
org.springframework.aop.interceptor.ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:92)
at
org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:185)
at
org.springframework.aop.framework.JdkDynamicAopProxy.invoke(JdkDynamicAopProxy.java:212)
at com.sun.proxy.$Proxy196.handleVmWorkJob(Unknown Source)
at com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:581)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
at
org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
at
org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
at
org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:529)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Best regards,
N.B
-----Message d'origine-----
De : Rafael Weingärtner [mailto:rafaelweingart...@gmail.com]
Envoyé : vendredi 2 novembre 2018 19:00
À : users <users@cloudstack.apache.org>
Objet : Re: Upload Data Volume
Do you have a Java stack trace for these errors?
On Fri, Nov 2, 2018 at 2:55 PM Nicolas Bouige <n.bou...@dimsi.fr> wrote:
> I missed those messages from mgmt-server :
>
> 2018-11-02 17:41:53,779 ERROR [o.a.c.s.v.VolumeServiceImpl]
> (Work-Job-Executor-66:ctx-15d61c99 job-6732/job-6733 ctx-b786c44b)
> (logid:e218d7a4) failed to copy volume from image store
> com.cloud.utils.exception.CloudRuntimeException: Failed to update
> state:com.cloud.utils.exception.CloudRuntimeException: Failed to
> transit
> volume: 623, due to: com.cloud.utils.fsm.NoTransitionException: Unable
> to transition to a new state from Copying via CopyRequested
>
> 018-11-02 17:41:53,805 INFO [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-66:ctx-15d61c99 job-6732/job-6733 ctx-b786c44b)
> (logid:e218d7a4) Rethrow exception
> com.cloud.utils.exception.CloudRuntimeException: Failed to update
> state:com.cloud.utils.exception.CloudRuntimeException: Failed to
> transit
> volume: 623, due to: com.cloud.utils.fsm.NoTransitionException: Unable
> to transition to a new state from Uploaded via OperationFailed
>
> 2018-11-02 17:41:53,805 DEBUG [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-66:ctx-15d61c99 job-6732/job-6733) (logid:e218d7a4)
> Done with run of VM work job: com.cloud.vm.VmWorkAttachVolume for VM
> 532, job
> origin: 6732
>
> 2018-11-02 17:41:53,806 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-66:ctx-15d61c99 job-6732/job-6733) (logid:e218d7a4)
> Unable to complete AsyncJobVO {id:6733, userId: 15, accountId: 12,
> instanceType: null, instanceId: null, cmd:
> com.cloud.vm.VmWorkAttachVolume,
> cmdInfo:
> rO0ABXNyAB9jb20uY2xvdWQudm0uVm1Xb3JrQXR0YWNoVm9sdW1lB62v-WGH4hwCAAJMAA
> hkZXZpY2VJZHQAEExqYXZhL2xhbmcvTG9uZztMAAh2b2x1bWVJZHEAfgABeHIAE2NvbS5j
> bG91ZC52bS5WbVdvcmufmbZW8CVnawIABEoACWFjY291bnRJZEoABnVzZXJJZEoABHZtSW
> RMAAtoYW5kbGVyTmFtZXQAEkxqYXZhL2xhbmcvU3RyaW5nO3hwAAAAAAAAAAwAAAAAAAAA
> DwAAAAAAAAIUdAAUVm9sdW1lQXBpU2VydmljZUltcGxwc3IADmphdmEubGFuZy5Mb25nO4
> vkkMyPI98CAAFKAAV2YWx1ZXhyABBqYXZhLmxhbmcuTnVtYmVyhqyVHQuU4IsCAAB4cAAA
> AAAAAAJv,
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0,
> result: null, initMsid: 2886211232, completeMsid: null, lastUpdated:
> null,
> lastPolled: null, created: Fri Nov 02 17:41:52 CET 2018}, job
> origin:6732
> com.cloud.utils.exception.CloudRuntimeException: Failed to update
> state:com.cloud.utils.exception.CloudRuntimeException: Failed to
> transit
> volume: 623, due to: com.cloud.utils.fsm.NoTransitionException: Unable
> to transition to a new state from Uploaded via OperationFailed
>
> Does the attach disk process works on the same way than template/iso ?
>
> De : Nicolas Bouige [mailto:n.bou...@dimsi.fr] Envoyé : vendredi 2
> novembre 2018 17:26 À : users <users@cloudstack.apache.org> Objet :
> RE: Upload Data Volume
>
> Hi Rafael,
>
>
> I know this is not a proper way to work directly on the DB. But
> sometimes that's permit me to "unblock" situations specially with
> VMs...so why not give a try ?
>
>
> I used the link created by Cloudpatform and i uploaded it in
> Cloudstack using the "upload" button in the storage tabs.
>
>
> The .vhd is present into the cloudstack SSVM with his
> volume.properties file and correct informations...(that's why i just
> tried to change the flag.. )
>
>
> I did not see any related error messages, i going to try again and
> check in real time just in case...thanks !
>
>
> Best regards,
> N.B
>
>
>
> ________________________________
> De : Rafael Weingärtner <rafaelweingart...@gmail.com> Envoyé :
> vendredi 2 novembre 2018 16:53:54 À : users Objet : Re: Upload Data
> Volume
>
> There is a joke I always tell people here; database interventions (in
> CloudStack) is forbidden, unless when I execute them. It does not work
> like that, going to the DB and simply changing a flag. You need to
> understand why the status is stuck in “uploaded”, and is not changing
> to ready.
>
> There is something that called my attention now. Are you uploading the
> file (using your browser)? Or, are you configuring the template/data
> disk as link, and then the SSVM is downloading them? These are two
> distinct operations. I was under the impression that you were using
> the latter. So, which one of these methods are you using? If nothing
> is working, I would expect some error messages either in the
> CloudStack management server, system VMs or in the hosts. Did you check them?
>
> On Fri, Nov 2, 2018 at 12:32 PM Nicolas Bouige <n.bou...@dimsi.fr> wrote:
>
> > Hi,
> >
> > SSVM works fine and we dont use any redirection.
> > We are able to Upload the volume-data after some modifications
> > directly on the cloudplatform SSVM.
> >
> > But the problem is still there as we can't attach the volume to a VM.
> > It is in a state "Uploaded", i tried to modify manually the state
> > into the DB (Uploaded --> Ready) but i ran into a new issue.
> > Exception com.xensource.xenapi.Types$UuidInvalid :VDI getByUuid for uuid:
> > null failed due to The uuid you supplied was invalid.
> >
> > -----Message d'origine-----
> > De : Andrija Panic [mailto:andrija.pa...@gmail.com] Envoyé :
> > vendredi
> > 2 novembre 2018 16:03 À : users <users@cloudstack.apache.org> Objet :
> > Re: Upload Data Volume
> >
> > Ok, so SSVM can access remote web server properly, that is good -
> > but please note, any http redirections are not allowed nor httpS
> > (ssl) when downloading volume/template from remote locations via
> > CloudStack, just plain http and http response/status code 200...
> >
> > Can you please also verify health of the SSVM with the script that
> > exists inside SSVM: /usr/local/cloud/system.../ssvm-check or similar
> > named...there should be no errors...
> >
> > Best,
> > Andrija
> >
> >
> >
> > On Fri, Nov 2, 2018, 11:46 Nicolas Bouige <n.bou...@dimsi.fr wrote:
> >
> > > Hi Andrija,
> > >
> > > I can download the .vhd directly on the SSVM.
> > > I tried with the link provided by CP and link from my webserver.
> > >
> > > Both works fine.
> > >
> > > From CP, to make it work, we had to modifiy symbolic link and add
> > > permission on the .vhd.
> > > 49085a35-73bc-4e11-a40b-c7547c798787.vhd ->
> > > /mnt/SecStorage/null/volumes/4/29907/7f9a2e42-04e1-4733-be3e-a00b4
> > > 1c
> > > b2
> > > f08.vhd
> > > ln -sfn
> > > /mnt/SecStorage/1766927c-d83e-3423-aa68-87a5d0c71771/volumes/4/299
> > > 07
> > > /7 f9a2e42-04e1-4733-be3e-a00b41cb2f08.vhd
> > > 49085a35-73bc-4e11-a40b-c7547c798787.vhd
> > >
> > > Now, the DATA volume is staying in state "Uploaded" and we are not
> > > able to make any operations with it , like attach disk to a VM.
> > >
> > > Best regards,
> > > N.B
> > >
> > > -----Message d'origine-----
> > > De : Andrija Panic [mailto:andrija.pa...@gmail.com] Envoyé :
> > > mercredi
> > > 31 octobre 2018 18:45 À : users <users@cloudstack.apache.org> Objet :
> > > Re: Upload Data Volume
> > >
> > > Login inside SSVM in destination CloudStack and try to wget the
> > > volume as suggested in previous email. See if that works first.
> > >
> > > Best
> > > Andrija
> > >
> > > On Wed, Oct 31, 2018, 18:36 Nicolas Bouige <n.bou...@dimsi.fr> wrote:
> > >
> > > > Nope, i exported the .vhd manually from my Xen pool and put it
> > > > in a webserver to be download by Cloudstack.
> > > > This step works fine. So that's not a problem if the download
> > > > volume feature from CloudPlatform didn' work.
> > > >
> > > > i'm in trouble with the second part, to import the .vhd hosted
> > > > on my webserver to cloudstack.
> > > >
> > > >
> > > > -----Message d'origine-----
> > > > De : Rafael Weingärtner [mailto:rafaelweingart...@gmail.com]
> > > > Envoyé : mercredi 31 octobre 2018 18:26 À : users
> > > > <users@cloudstack.apache.org> Objet : Re: Upload Data Volume
> > > >
> > > > Ah, so the cloud platform is not working properly. I mean, it is
> > > > not exporting the VHDs already. That is your problem, right?
> > > >
> > > > On Wed, Oct 31, 2018 at 2:24 PM Nicolas Bouige
> > > > <n.bou...@dimsi.fr>
> > > wrote:
> > > >
> > > > > Hi Rafael,
> > > > >
> > > > > Did not try as i got an access denied via my web
> > > > > browser...that's why i exported the .vhd on my own webserver
> > > > > instead to use this
> > > feature.
> > > > >
> > > > > Best regards,
> > > > > N.B
> > > > >
> > > > > -----Message d'origine-----
> > > > > De : Rafael Weingärtner [mailto:rafaelweingart...@gmail.com]
> > > > > Envoyé : mercredi 31 octobre 2018 17:36 À : users
> > > > > <users@cloudstack.apache.org> Objet : Re: Upload Data Volume
> > > > >
> > > > > Did you log in the SSVM of your ACS 4.11, and tried to wget
> > > > > the template/data disk using the URL provided by the
> > > > > Citrix/Accelerite cloud platform?
> > > > >
> > > > > On Wed, Oct 31, 2018 at 1:33 PM Nicolas Bouige
> > > > > <n.bou...@dimsi.fr>
> > > > wrote:
> > > > >
> > > > > > Hello All,
> > > > > >
> > > > > >
> > > > > >
> > > > > > I’m currently trying to migrate some VMs from Cloudplatform
> > > > > > 4.7
> > > > > > (XEN) to Cloudstack 4.11.1 (XCP)
> > > > > >
> > > > > >
> > > > > >
> > > > > > Im stuck with the DATA Volume, there are exported on a web
> > > > > > server and accessible by the management-server.
> > > > > >
> > > > > > Webserver is working fine as i can upload ROOT disk from it.
> > > > > >
> > > > > >
> > > > > >
> > > > > > The only difference is for upload ROOT DISK as template SSVM
> > > > > > is involved with the public ip and DATA DISK, the
> > > > > > management server is used with private ip.
> > > > > > Both ip (private and public) are « pingable » from
> > > > > > mgmt-server and SSVM and i can download .vhd.
> > > > > >
> > > > > > Unfortunately, each time i try to upload DATA disk, i got
> > > > > > this error
> > > :
> > > > > >
> > > > > >
> > > > > >
> > > > > > 2018-10-31 15:26:09,493 DEBUG [c.c.a.t.Request]
> > > > > > (AgentManager-Handler-15:null) (logid:) Seq
> 34-3493104460979266769:
> > > > > > Processing: { Ans: , MgmtId: 2886211232, via: 34, Ver: v1,
> Flags:
> > > > > > 10,
> > > > > > [{"com.cloud.agent.api.storage.DownloadAnswer":{"jobId":"186
> > > > > > 69
> > > > > > 70
> > > > > > c-
> > > > > > b1
> > > > > > 9f
> > > > > > -427a-b957-3738bbd50d0e","downloadPct":0,"errorString":"Conn
> > > > > > ec
> > > > > > ti
> > > > > > on
> > > > > > refused (Connection
> > > > > > refused)","downloadStatus":"DOWNLOAD_ERROR","downloadPath":"
> > > > > > /m
> > > > > > nt
> > > > > > /S
> > > > > > ec
> > > > > > St
> > > > > > orage/2949d66c-3490-3636-93f7-f2299fa5f4b2/volumes/12/615/dn
> > > > > > ld
> > > > > > 36
> > > > > > 26
> > > > > > 14
> > > > > > 18
> > > > > >
> 85766458800tmp_","installPath":"volumes/12/615","templateSize":0,"
> > > > > > te mp
> > > > > > latePhySicalSize":0,"result":true,"details":"Connection
> > > > > > refused (Connection refused)","wait":0}}] }
> > > > > >
> > > > > > 2018-10-31 15:26:09,563 WARN [o.a.c.alerts]
> > > > > > (RemoteHostEndPoint-15:ctx-28d55a70) (logid:ebfb4bd8) AlertType::
> > > > > > 28
> > > > > > |
> > > > > > dataCenterId:: 0 | podId:: null | clusterId:: null | message::
> > > > > > Failed to upload volume:
> > > > > > ce2747c9-468a-4d1f-9651-8b6e2373079b
> > > > > > with
> > > > error:
> > > > > > Connection refused (Connection refused)
> > > > > >
> > > > > > 2018-10-31 15:26:09,573 ERROR
> > > > > > [o.a.c.s.i.BaseImageStoreDriverImpl]
> > > > > > (RemoteHostEndPoint-15:ctx-28d55a70) (logid:ebfb4bd8) Failed
> > > > > > to upload
> > > > > > volume: ce2747c9-468a-4d1f-9651-8b6e2373079b with error:
> > > > > > Connection refused (Connection refused)
> > > > > >
> > > > > >
> > > > > >
> > > > > > I don’t know where the « connection refused » comes from.
> > > > > >
> > > > > >
> > > > > >
> > > > > > Someone knows the exact process in cloudstack to upload
> > > > > > volumes from an URL ?
> > > > > >
> > > > > > Best regards,
> > > > > >
> > > > > >
> > > > > >
> > > > > > [image: SignatureBouige]
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Rafael Weingärtner
> > > > >
> > > >
> > > >
> > > > --
> > > > Rafael Weingärtner
> > > >
> > >
> >
>
>
> --
> Rafael Weingärtner
>
--
Rafael Weingärtner
