[
https://issues.apache.org/jira/browse/CLOUDSTACK-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Abhinandan Prateek updated CLOUDSTACK-6459:
-------------------------------------------
Assignee: Min Chen
> Unable to enable maintenance mode on a Primary storage that crashed
> -------------------------------------------------------------------
>
> Key: CLOUDSTACK-6459
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-6459
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: 4.4.0
> Reporter: Chandan Purushothama
> Assignee: Min Chen
> Priority: Critical
> Fix For: 4.4.0
>
>
> Primary storage in my setup got powered off. I am not able to enable
> maintenance mode on this primary storage.
> Enabling maintenance mode on the primary storage fails with the following
> error. It eventually timed out after trying many times
> 2014-04-18 16:43:50,020 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-1:ctx-bd92e323 ctx-7d4c2498) ===END=== 10.214.5.40 -- GET
> command=queryAsyncJobResult&jobId=62f6830a-c409-4449-a9c5-6a35b7b9fbed&response=json&sessionkey=WBpwG%2FryPRNNB1GRuHqam1zbtS8%3D&_=1397865006850
> 2014-04-18 16:43:50,495 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentManager-Handler-9:null) SeqA 2-792: Processing Seq 2-792: { Cmd ,
> MgmtId: -1, via: 2, Ver: v1, Flags: 11,
> [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1,"_loadInfo":"{\n
> \"connections\": []\n}","wait":0}}] }
> 2014-04-18 16:43:50,504 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentManager-Handler-9:null) SeqA 2-792: Sending Seq 2-792: { Ans: ,
> MgmtId: 6638073284439, via: 2, Ver: v1, Flags: 100010,
> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
> 2014-04-18 16:43:52,539 WARN [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-143:ctx-16ea61bc) Async 600 seconds timeout for task
> com.xensource.xenapi.Task@8aa497e8
> 2014-04-18 16:43:52,563 DEBUG [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-143:ctx-16ea61bc) unable to destroy
> task(com.xensource.xenapi.Task@8aa497e8) on
> host(0d2ea73b-12c0-433c-b1c3-e1f193e68f6e) due to You gave an invalid object
> reference. The object may have recently been deleted. The class parameter
> gives the type of reference given, and the handle parameter echoes the bad
> value given.
> 2014-04-18 16:43:52,564 DEBUG [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-143:ctx-16ea61bc) Catch exception
> com.cloud.utils.exception.CloudRuntimeException when stop VM:i-3-3-DR due to
> com.cloud.utils.exception.CloudRuntimeException: Shutdown VM catch
> HandleInvalid and VM is not in HALTED state
> 2014-04-18 16:43:52,569 DEBUG [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-143:ctx-16ea61bc) 10. The VM i-3-3-DR is in Running state
> 2014-04-18 16:43:52,572 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgent-143:ctx-16ea61bc) Seq 1-2385781902599520418: Response Received:
> 2014-04-18 16:43:52,573 DEBUG [c.c.a.t.Request]
> (DirectAgent-143:ctx-16ea61bc) Seq 1-2385781902599520418: Processing: { Ans:
> , MgmtId: 6638073284439, via: 1, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.StopAnswer":{"platform":"viridian:true;acpi:1;apic:true;pae:true;nx:true","result":false,"details":"Catch
> exception com.cloud.utils.exception.CloudRuntimeException when stop
> VM:i-3-3-DR due to com.cloud.utils.exception.CloudRuntimeException: Shutdown
> VM catch HandleInvalid and VM is not in HALTED state","wait":0}}] }
> 2014-04-18 16:43:52,576 DEBUG [c.c.a.t.Request]
> (Work-Job-Executor-2:job-30/job-31 ctx-191e1825) Seq 1-2385781902599520418:
> Received: { Ans: , MgmtId: 6638073284439, via: 1, Ver: v1, Flags: 10, {
> StopAnswer } }
> 2014-04-18 16:43:52,591 WARN [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-2:job-30/job-31 ctx-191e1825) Unable to stop vm
> VM[User|i-3-3-DR]
> 2014-04-18 16:43:52,616 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-2:job-30/job-31 ctx-191e1825) VM state transitted from
> :Stopping to Running with event: OperationFailedvm's original host id: 1 new
> host id: 1 host id before state transition: 1
> 2014-04-18 16:43:52,616 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-2:job-30/job-31 ctx-191e1825) Invocation exception, caused
> by: com.cloud.utils.exception.CloudRuntimeException: Unable to stop
> VM[User|i-3-3-DR]
> 2014-04-18 16:43:52,617 INFO [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-2:job-30/job-31 ctx-191e1825) Rethrow exception
> com.cloud.utils.exception.CloudRuntimeException: Unable to stop
> VM[User|i-3-3-DR]
> 2014-04-18 16:43:52,617 DEBUG [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-2:job-30/job-31) Done with run of VM work job:
> com.cloud.vm.VmWorkStop for VM 3, job origin: 30
> 2014-04-18 16:43:52,617 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-2:job-30/job-31) Unable to complete AsyncJobVO {id:31,
> userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd:
> com.cloud.vm.VmWorkStop, cmdInfo:
> rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAA3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwA,
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
> null, initMsid: 6638073284439, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: Fri Apr 18 16:33:50 PDT 2014}, job origin:30
> com.cloud.utils.exception.CloudRuntimeException: Unable to stop
> VM[User|i-3-3-DR]
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:1533)
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStop(VirtualMachineManagerImpl.java:1385)
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStop(VirtualMachineManagerImpl.java:5175)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5309)
> at
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:495)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:452)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> 2014-04-18 16:43:52,642 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-2:job-30/job-31) Complete async job-31, jobStatus: FAILED,
> resultCode: 0, result:
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFuZy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARMAAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9sYW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAPEpvYiBmYWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBVbmFibGUgdG8gc3RvcCBWTVtVc2VyfGktMy0zLURSXXVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJpbmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgALAAAB73QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EAfgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEAfgAXc3EAfgALAAABxHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4ACwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZhcQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsAAALodAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
> 2014-04-18 16:43:52,659 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-2:job-30/job-31) Done executing com.cloud.vm.VmWorkStop
> for job-31
> 2014-04-18 16:43:52,680 ERROR [c.c.s.StoragePoolAutomationImpl]
> (API-Job-Executor-3:job-30 ctx-51f6a0d8) Exception in enabling primary
> storage maintenance:
> java.lang.RuntimeException: Job failed due to exception Unable to stop
> VM[User|i-3-3-DR]
> at
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:495)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:452)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> 2014-04-18 16:43:52,687 DEBUG [o.a.c.f.j.i.SyncQueueManagerImpl]
> (Work-Job-Executor-2:job-30/job-31) Sync queue (3) is currently empty
> 2014-04-18 16:43:52,688 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (Work-Job-Executor-2:job-30/job-31) Remove job-31 from job monitoring
> 2014-04-18 16:43:52,693 ERROR [c.c.a.ApiAsyncJobDispatcher]
> (API-Job-Executor-3:job-30) Unexpected exception while executing
> org.apache.cloudstack.api.command.admin.storage.PreparePrimaryStorageForMaintenanceCmd
> com.cloud.utils.exception.CloudRuntimeException: Job failed due to exception
> Unable to stop VM[User|i-3-3-DR]
> at
> com.cloud.storage.StoragePoolAutomationImpl.maintain(StoragePoolAutomationImpl.java:276)
> at
> org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl.maintain(CloudStackPrimaryDataStoreLifeCycleImpl.java:459)
> at
> com.cloud.storage.StorageManagerImpl.preparePrimaryStorageForMaintenance(StorageManagerImpl.java:1203)
> at
> com.cloud.storage.StorageManagerImpl.preparePrimaryStorageForMaintenance(StorageManagerImpl.java:176)
> at
> org.apache.cloudstack.api.command.admin.storage.PreparePrimaryStorageForMaintenanceCmd.execute(PreparePrimaryStorageForMaintenanceCmd.java:103)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:134)
> at
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:100)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:495)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:452)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> 2014-04-18 16:43:52,695 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-3:job-30) Complete async job-30, jobStatus: FAILED,
> resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Job
> failed due to exception Unable to stop VM[User|i-3-3-DR]"}
> 2014-04-18 16:43:52,703 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-3:job-30) Done executing
> org.apache.cloudstack.api.command.admin.storage.PreparePrimaryStorageForMaintenanceCmd
> for job-30
> 2014-04-18 16:43:52,709 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-3:job-30) Remove job-30 from job monitoring
> 2014-04-18 16:43:52,952 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-7:ctx-ce58b540) ===START=== 10.214.5.40 -- GET
> command=queryAsyncJobResult&jobId=62f6830a-c409-4449-a9c5-6a35b7b9fbed&response=json&sessionkey=WBpwG%2FryPRNNB1GRuHqam1zbtS8%3D&_=1397865009850
> 2014-04-18 16:43:52,994 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-7:ctx-ce58b540 ctx-747fe328) ===END=== 10.214.5.40 -- GET
> command=queryAsyncJobResult&jobId=62f6830a-c409-4449-a9c5-6a35b7b9fbed&response=json&sessionkey=WBpwG%2FryPRNNB1GRuHqam1zbtS8%3D&_=1397865009850
--
This message was sent by Atlassian JIRA
(v6.2#6252)
