[
https://issues.apache.org/jira/browse/CLOUDSTACK-6459?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anthony Xu resolved CLOUDSTACK-6459.
------------------------------------
Resolution: Invalid
primary storage maintenance mode is used to prevent new volume deployment on
this primary storage and migrate the volumes in this primary storage to other
primary storage.
if primary storage is crashed, we can't put this primary storage in maintenance
mode, this is expected behavior
> Unable to enable maintenance mode on a Primary storage that crashed
> -------------------------------------------------------------------
>
> Key: CLOUDSTACK-6459
> URL: https://issues.apache.org/jira/browse/CLOUDSTACK-6459
> Project: CloudStack
> Issue Type: Bug
> Security Level: Public(Anyone can view this level - this is the
> default.)
> Components: Management Server
> Affects Versions: 4.4.0
> Reporter: Chandan Purushothama
> Assignee: Anthony Xu
> Priority: Critical
> Fix For: 4.4.0, 4.5.0
>
> Attachments: kern.zip, management-server.log.2014-04-18.gz,
> mysql_cloudstack_dump.zip
>
>
> Primary storage in my setup got powered off. I am not able to enable
> maintenance mode on this primary storage.
> Enabling maintenance mode on the primary storage fails with the following
> error. It eventually timed out after trying many times
> 2014-04-18 16:43:50,020 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-1:ctx-bd92e323 ctx-7d4c2498) ===END=== 10.214.5.40 -- GET
> command=queryAsyncJobResult&jobId=62f6830a-c409-4449-a9c5-6a35b7b9fbed&response=json&sessionkey=WBpwG%2FryPRNNB1GRuHqam1zbtS8%3D&_=1397865006850
> 2014-04-18 16:43:50,495 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentManager-Handler-9:null) SeqA 2-792: Processing Seq 2-792: { Cmd ,
> MgmtId: -1, via: 2, Ver: v1, Flags: 11,
> [{"com.cloud.agent.api.ConsoleProxyLoadReportCommand":{"_proxyVmId":1,"_loadInfo":"{\n
> \"connections\": []\n}","wait":0}}] }
> 2014-04-18 16:43:50,504 DEBUG [c.c.a.m.AgentManagerImpl]
> (AgentManager-Handler-9:null) SeqA 2-792: Sending Seq 2-792: { Ans: ,
> MgmtId: 6638073284439, via: 2, Ver: v1, Flags: 100010,
> [{"com.cloud.agent.api.AgentControlAnswer":{"result":true,"wait":0}}] }
> 2014-04-18 16:43:52,539 WARN [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-143:ctx-16ea61bc) Async 600 seconds timeout for task
> com.xensource.xenapi.Task@8aa497e8
> 2014-04-18 16:43:52,563 DEBUG [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-143:ctx-16ea61bc) unable to destroy
> task(com.xensource.xenapi.Task@8aa497e8) on
> host(0d2ea73b-12c0-433c-b1c3-e1f193e68f6e) due to You gave an invalid object
> reference. The object may have recently been deleted. The class parameter
> gives the type of reference given, and the handle parameter echoes the bad
> value given.
> 2014-04-18 16:43:52,564 DEBUG [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-143:ctx-16ea61bc) Catch exception
> com.cloud.utils.exception.CloudRuntimeException when stop VM:i-3-3-DR due to
> com.cloud.utils.exception.CloudRuntimeException: Shutdown VM catch
> HandleInvalid and VM is not in HALTED state
> 2014-04-18 16:43:52,569 DEBUG [c.c.h.x.r.CitrixResourceBase]
> (DirectAgent-143:ctx-16ea61bc) 10. The VM i-3-3-DR is in Running state
> 2014-04-18 16:43:52,572 DEBUG [c.c.a.m.DirectAgentAttache]
> (DirectAgent-143:ctx-16ea61bc) Seq 1-2385781902599520418: Response Received:
> 2014-04-18 16:43:52,573 DEBUG [c.c.a.t.Request]
> (DirectAgent-143:ctx-16ea61bc) Seq 1-2385781902599520418: Processing: { Ans:
> , MgmtId: 6638073284439, via: 1, Ver: v1, Flags: 10,
> [{"com.cloud.agent.api.StopAnswer":{"platform":"viridian:true;acpi:1;apic:true;pae:true;nx:true","result":false,"details":"Catch
> exception com.cloud.utils.exception.CloudRuntimeException when stop
> VM:i-3-3-DR due to com.cloud.utils.exception.CloudRuntimeException: Shutdown
> VM catch HandleInvalid and VM is not in HALTED state","wait":0}}] }
> 2014-04-18 16:43:52,576 DEBUG [c.c.a.t.Request]
> (Work-Job-Executor-2:job-30/job-31 ctx-191e1825) Seq 1-2385781902599520418:
> Received: { Ans: , MgmtId: 6638073284439, via: 1, Ver: v1, Flags: 10, {
> StopAnswer } }
> 2014-04-18 16:43:52,591 WARN [c.c.v.VirtualMachineManagerImpl]
> (Work-Job-Executor-2:job-30/job-31 ctx-191e1825) Unable to stop vm
> VM[User|i-3-3-DR]
> 2014-04-18 16:43:52,616 DEBUG [c.c.c.CapacityManagerImpl]
> (Work-Job-Executor-2:job-30/job-31 ctx-191e1825) VM state transitted from
> :Stopping to Running with event: OperationFailedvm's original host id: 1 new
> host id: 1 host id before state transition: 1
> 2014-04-18 16:43:52,616 ERROR [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-2:job-30/job-31 ctx-191e1825) Invocation exception, caused
> by: com.cloud.utils.exception.CloudRuntimeException: Unable to stop
> VM[User|i-3-3-DR]
> 2014-04-18 16:43:52,617 INFO [c.c.v.VmWorkJobHandlerProxy]
> (Work-Job-Executor-2:job-30/job-31 ctx-191e1825) Rethrow exception
> com.cloud.utils.exception.CloudRuntimeException: Unable to stop
> VM[User|i-3-3-DR]
> 2014-04-18 16:43:52,617 DEBUG [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-2:job-30/job-31) Done with run of VM work job:
> com.cloud.vm.VmWorkStop for VM 3, job origin: 30
> 2014-04-18 16:43:52,617 ERROR [c.c.v.VmWorkJobDispatcher]
> (Work-Job-Executor-2:job-30/job-31) Unable to complete AsyncJobVO {id:31,
> userId: 2, accountId: 2, instanceType: null, instanceId: null, cmd:
> com.cloud.vm.VmWorkStop, cmdInfo:
> rO0ABXNyABdjb20uY2xvdWQudm0uVm1Xb3JrU3RvcALQ4GymiWjjAgABWgAHY2xlYW51cHhyABNjb20uY2xvdWQudm0uVm1Xb3Jrn5m2VvAlZ2sCAARKAAlhY2NvdW50SWRKAAZ1c2VySWRKAAR2bUlkTAALaGFuZGxlck5hbWV0ABJMamF2YS9sYW5nL1N0cmluZzt4cAAAAAAAAAACAAAAAAAAAAIAAAAAAAAAA3QAGVZpcnR1YWxNYWNoaW5lTWFuYWdlckltcGwA,
> cmdVersion: 0, status: IN_PROGRESS, processStatus: 0, resultCode: 0, result:
> null, initMsid: 6638073284439, completeMsid: null, lastUpdated: null,
> lastPolled: null, created: Fri Apr 18 16:33:50 PDT 2014}, job origin:30
> com.cloud.utils.exception.CloudRuntimeException: Unable to stop
> VM[User|i-3-3-DR]
> at
> com.cloud.vm.VirtualMachineManagerImpl.advanceStop(VirtualMachineManagerImpl.java:1533)
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStop(VirtualMachineManagerImpl.java:1385)
> at
> com.cloud.vm.VirtualMachineManagerImpl.orchestrateStop(VirtualMachineManagerImpl.java:5175)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> com.cloud.vm.VmWorkJobHandlerProxy.handleVmWorkJob(VmWorkJobHandlerProxy.java:107)
> at
> com.cloud.vm.VirtualMachineManagerImpl.handleVmWorkJob(VirtualMachineManagerImpl.java:5309)
> at
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:102)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:495)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:452)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> 2014-04-18 16:43:52,642 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-2:job-30/job-31) Complete async job-31, jobStatus: FAILED,
> resultCode: 0, result:
> rO0ABXNyABpqYXZhLmxhbmcuUnVudGltZUV4Y2VwdGlvbp5fBkcKNIPlAgAAeHIAE2phdmEubGFuZy5FeGNlcHRpb27Q_R8-GjscxAIAAHhyABNqYXZhLmxhbmcuVGhyb3dhYmxl1cY1Jzl3uMsDAARMAAVjYXVzZXQAFUxqYXZhL2xhbmcvVGhyb3dhYmxlO0wADWRldGFpbE1lc3NhZ2V0ABJMamF2YS9sYW5nL1N0cmluZztbAApzdGFja1RyYWNldAAeW0xqYXZhL2xhbmcvU3RhY2tUcmFjZUVsZW1lbnQ7TAAUc3VwcHJlc3NlZEV4Y2VwdGlvbnN0ABBMamF2YS91dGlsL0xpc3Q7eHBxAH4AB3QAPEpvYiBmYWlsZWQgZHVlIHRvIGV4Y2VwdGlvbiBVbmFibGUgdG8gc3RvcCBWTVtVc2VyfGktMy0zLURSXXVyAB5bTGphdmEubGFuZy5TdGFja1RyYWNlRWxlbWVudDsCRio8PP0iOQIAAHhwAAAADXNyABtqYXZhLmxhbmcuU3RhY2tUcmFjZUVsZW1lbnRhCcWaJjbdhQIABEkACmxpbmVOdW1iZXJMAA5kZWNsYXJpbmdDbGFzc3EAfgAETAAIZmlsZU5hbWVxAH4ABEwACm1ldGhvZE5hbWVxAH4ABHhwAAAAcnQAIGNvbS5jbG91ZC52bS5WbVdvcmtKb2JEaXNwYXRjaGVydAAYVm1Xb3JrSm9iRGlzcGF0Y2hlci5qYXZhdAAGcnVuSm9ic3EAfgALAAAB73QAP29yZy5hcGFjaGUuY2xvdWRzdGFjay5mcmFtZXdvcmsuam9icy5pbXBsLkFzeW5jSm9iTWFuYWdlckltcGwkNXQAGEFzeW5jSm9iTWFuYWdlckltcGwuamF2YXQADHJ1bkluQ29udGV4dHNxAH4ACwAAADF0AD5vcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGUkMXQAG01hbmFnZWRDb250ZXh0UnVubmFibGUuamF2YXQAA3J1bnNxAH4ACwAAADh0AEJvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0LmltcGwuRGVmYXVsdE1hbmFnZWRDb250ZXh0JDF0ABpEZWZhdWx0TWFuYWdlZENvbnRleHQuamF2YXQABGNhbGxzcQB-AAsAAABndABAb3JnLmFwYWNoZS5jbG91ZHN0YWNrLm1hbmFnZWQuY29udGV4dC5pbXBsLkRlZmF1bHRNYW5hZ2VkQ29udGV4dHEAfgAadAAPY2FsbFdpdGhDb250ZXh0c3EAfgALAAAANXEAfgAdcQB-ABp0AA5ydW5XaXRoQ29udGV4dHNxAH4ACwAAAC50ADxvcmcuYXBhY2hlLmNsb3Vkc3RhY2subWFuYWdlZC5jb250ZXh0Lk1hbmFnZWRDb250ZXh0UnVubmFibGVxAH4AFnEAfgAXc3EAfgALAAABxHEAfgARcQB-ABJxAH4AF3NxAH4ACwAAAdd0AC5qYXZhLnV0aWwuY29uY3VycmVudC5FeGVjdXRvcnMkUnVubmFibGVBZGFwdGVydAAORXhlY3V0b3JzLmphdmFxAH4AG3NxAH4ACwAAAQZ0AB9qYXZhLnV0aWwuY29uY3VycmVudC5GdXR1cmVUYXNrdAAPRnV0dXJlVGFzay5qYXZhcQB-ABdzcQB-AAsAAAR5dAAnamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9ydAAXVGhyZWFkUG9vbEV4ZWN1dG9yLmphdmF0AAlydW5Xb3JrZXJzcQB-AAsAAAJndAAuamF2YS51dGlsLmNvbmN1cnJlbnQuVGhyZWFkUG9vbEV4ZWN1dG9yJFdvcmtlcnEAfgAscQB-ABdzcQB-AAsAAALodAAQamF2YS5sYW5nLlRocmVhZHQAC1RocmVhZC5qYXZhcQB-ABdzcgAmamF2YS51dGlsLkNvbGxlY3Rpb25zJFVubW9kaWZpYWJsZUxpc3T8DyUxteyOEAIAAUwABGxpc3RxAH4ABnhyACxqYXZhLnV0aWwuQ29sbGVjdGlvbnMkVW5tb2RpZmlhYmxlQ29sbGVjdGlvbhlCAIDLXvceAgABTAABY3QAFkxqYXZhL3V0aWwvQ29sbGVjdGlvbjt4cHNyABNqYXZhLnV0aWwuQXJyYXlMaXN0eIHSHZnHYZ0DAAFJAARzaXpleHAAAAAAdwQAAAAAeHEAfgA4eA
> 2014-04-18 16:43:52,659 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (Work-Job-Executor-2:job-30/job-31) Done executing com.cloud.vm.VmWorkStop
> for job-31
> 2014-04-18 16:43:52,680 ERROR [c.c.s.StoragePoolAutomationImpl]
> (API-Job-Executor-3:job-30 ctx-51f6a0d8) Exception in enabling primary
> storage maintenance:
> java.lang.RuntimeException: Job failed due to exception Unable to stop
> VM[User|i-3-3-DR]
> at
> com.cloud.vm.VmWorkJobDispatcher.runJob(VmWorkJobDispatcher.java:114)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:495)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:452)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> 2014-04-18 16:43:52,687 DEBUG [o.a.c.f.j.i.SyncQueueManagerImpl]
> (Work-Job-Executor-2:job-30/job-31) Sync queue (3) is currently empty
> 2014-04-18 16:43:52,688 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (Work-Job-Executor-2:job-30/job-31) Remove job-31 from job monitoring
> 2014-04-18 16:43:52,693 ERROR [c.c.a.ApiAsyncJobDispatcher]
> (API-Job-Executor-3:job-30) Unexpected exception while executing
> org.apache.cloudstack.api.command.admin.storage.PreparePrimaryStorageForMaintenanceCmd
> com.cloud.utils.exception.CloudRuntimeException: Job failed due to exception
> Unable to stop VM[User|i-3-3-DR]
> at
> com.cloud.storage.StoragePoolAutomationImpl.maintain(StoragePoolAutomationImpl.java:276)
> at
> org.apache.cloudstack.storage.datastore.lifecycle.CloudStackPrimaryDataStoreLifeCycleImpl.maintain(CloudStackPrimaryDataStoreLifeCycleImpl.java:459)
> at
> com.cloud.storage.StorageManagerImpl.preparePrimaryStorageForMaintenance(StorageManagerImpl.java:1203)
> at
> com.cloud.storage.StorageManagerImpl.preparePrimaryStorageForMaintenance(StorageManagerImpl.java:176)
> at
> org.apache.cloudstack.api.command.admin.storage.PreparePrimaryStorageForMaintenanceCmd.execute(PreparePrimaryStorageForMaintenanceCmd.java:103)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:134)
> at
> com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispatcher.java:100)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.runInContext(AsyncJobManagerImpl.java:495)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext$1.call(DefaultManagedContext.java:56)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
> at
> org.apache.cloudstack.managed.context.impl.DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
> at
> org.apache.cloudstack.managed.context.ManagedContextRunnable.run(ManagedContextRunnable.java:46)
> at
> org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImpl$5.run(AsyncJobManagerImpl.java:452)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:744)
> 2014-04-18 16:43:52,695 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-3:job-30) Complete async job-30, jobStatus: FAILED,
> resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"uuidList":[],"errorcode":530,"errortext":"Job
> failed due to exception Unable to stop VM[User|i-3-3-DR]"}
> 2014-04-18 16:43:52,703 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-3:job-30) Done executing
> org.apache.cloudstack.api.command.admin.storage.PreparePrimaryStorageForMaintenanceCmd
> for job-30
> 2014-04-18 16:43:52,709 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-3:job-30) Remove job-30 from job monitoring
> 2014-04-18 16:43:52,952 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-7:ctx-ce58b540) ===START=== 10.214.5.40 -- GET
> command=queryAsyncJobResult&jobId=62f6830a-c409-4449-a9c5-6a35b7b9fbed&response=json&sessionkey=WBpwG%2FryPRNNB1GRuHqam1zbtS8%3D&_=1397865009850
> 2014-04-18 16:43:52,994 DEBUG [c.c.a.ApiServlet]
> (catalina-exec-7:ctx-ce58b540 ctx-747fe328) ===END=== 10.214.5.40 -- GET
> command=queryAsyncJobResult&jobId=62f6830a-c409-4449-a9c5-6a35b7b9fbed&response=json&sessionkey=WBpwG%2FryPRNNB1GRuHqam1zbtS8%3D&_=1397865009850
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
