Hi Natalia,
This problem is caused at class method
“org.apache.cloudstack.engine.orchestration.NetworkOrchestrator.releaseNic(VirtualMachineProfile,
long)” line 1631 as you can see by the stack trace.
Did you change something else in the database or ACS while these VMs were
stuck in "stopping"?
The code at that line is the following:
updateNic(nic, network.getId(), -1);
For some reason the network object is null. The line that creates the
“network” object is 1609. The code is the following:
final NetworkVO network = _networksDao.findById(nic.getNetworkId());
It seems that the network that was used to assign the IPs for the VMs NIC
was deleted/removed? Could you check that?
The network interface card table is “nics”, you can look for the NICs of
your VM by filtering for “instance_id”.
Also, if you want to check the networks table is “networks”.
On Tue, Feb 14, 2017 at 11:52 AM, Natalia Costas Lago <nata...@cesga.es>
wrote:
> Dear Rafael,
>
> Thre requested information:
>
> ACS version: 4.9.0
>
> hypervisors used: kvm
> storage systems used: gluster as primary
> networking: advanced
>
> And this is the error...
>
>
>
> 2017-02-14 17:48:56,112 DEBUG [c.c.u.d.T.Transaction]
> (API-Job-Executor-2:ctx-4eca9fd0 job-1329 ctx-39cd3fd6) (logid:2eed47c7)
> Rolling back the transaction: Time = 1 Name = API-Job-Executor-2; called by
> -TransactionLegacy.rollback:879-TransactionLegacy.removeUpTo
> :822-TransactionLegacy.close:646-Transaction.execute:43-Tra
> nsaction.execute:47-NetworkOrchestrator.releaseNic:1600-Netw
> orkOrchestrator.release:1588-UserVmManagerImpl.releaseNetwo
> rkResourcesOnExpunge:2102-UserVmManagerImpl.expunge:
> 2050-UserVmManagerImpl.destroyVm:2636-NativeMethodAccessorImpl.invok
> e0:-2-NativeMethodAccessorImpl.invoke:62
> 2017-02-14 17:48:56,156 ERROR [c.c.a.ApiAsyncJobDispatcher]
> (API-Job-Executor-2:ctx-4eca9fd0 job-1329) (logid:2eed47c7) Unexpected
> exception while executing org.apache.cloudstack.api.comm
> and.user.vm.DestroyVMCmd
> java.lang.NullPointerException
> at org.apache.cloudstack.engine.orchestration.NetworkOrchestrat
> or$6.doInTransaction(NetworkOrchestrator.java:1631)
> at org.apache.cloudstack.engine.orchestration.NetworkOrchestrat
> or$6.doInTransaction(NetworkOrchestrator.java:1600)
> at com.cloud.utils.db.Transaction$2.doInTransaction(Transaction
> .java:50)
> at com.cloud.utils.db.Transaction.execute(Transaction.java:40)
> at com.cloud.utils.db.Transaction.execute(Transaction.java:47)
> at org.apache.cloudstack.engine.orchestration.NetworkOrchestrat
> or.releaseNic(NetworkOrchestrator.java:1600)
> at org.apache.cloudstack.engine.orchestration.NetworkOrchestrat
> or.release(NetworkOrchestrator.java:1588)
> at com.cloud.vm.UserVmManagerImpl.releaseNetworkResourcesOnExpu
> nge(UserVmManagerImpl.java:2102)
> at com.cloud.vm.UserVmManagerImpl.expunge(UserVmManagerImpl.
> java:2050)
> at com.cloud.vm.UserVmManagerImpl.destroyVm(UserVmManagerImpl.
> java:2636)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
> ssorImpl.java:62)
> at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
> thodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at org.springframework.aop.support.AopUtils.invokeJoinpointUsin
> gReflection(AopUtils.java:317)
> at org.springframework.aop.framework.ReflectiveMethodInvocation
> .invokeJoinpoint(ReflectiveMethodInvocation.java:183)
> at org.springframework.aop.framework.ReflectiveMethodInvocation
> .proceed(ReflectiveMethodInvocation.java:150)
> at org.apache.cloudstack.network.contrail.management.EventUtils
> $EventInterceptor.invoke(EventUtils.java:106)
> at org.springframework.aop.framework.ReflectiveMethodInvocation
> .proceed(ReflectiveMethodInvocation.java:161)
> at com.cloud.event.ActionEventInterceptor.invoke(ActionEventInt
> erceptor.java:51)
> at org.springframework.aop.framework.ReflectiveMethodInvocation
> .proceed(ReflectiveMethodInvocation.java:161)
> at org.springframework.aop.interceptor.ExposeInvocationIntercep
> tor.invoke(ExposeInvocationInterceptor.java:91)
> at org.springframework.aop.framework.ReflectiveMethodInvocation
> .proceed(ReflectiveMethodInvocation.java:172)
> at org.springframework.aop.framework.JdkDynamicAopProxy.invoke(
> JdkDynamicAopProxy.java:204)
> at com.sun.proxy.$Proxy200.destroyVm(Unknown Source)
> at org.apache.cloudstack.api.command.user.vm.DestroyVMCmd.execu
> te(DestroyVMCmd.java:123)
> at com.cloud.api.ApiDispatcher.dispatch(ApiDispatcher.java:150)
> at com.cloud.api.ApiAsyncJobDispatcher.runJob(ApiAsyncJobDispat
> cher.java:108)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImp
> l$5.runInContext(AsyncJobManagerImpl.java:554)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable
> $1.run(ManagedContextRunnable.java:49)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedCon
> text$1.call(DefaultManagedContext.java:56)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedCon
> text.callWithContext(DefaultManagedContext.java:103)
> at org.apache.cloudstack.managed.context.impl.DefaultManagedCon
> text.runWithContext(DefaultManagedContext.java:53)
> at org.apache.cloudstack.managed.context.ManagedContextRunnable
> .run(ManagedContextRunnable.java:46)
> at org.apache.cloudstack.framework.jobs.impl.AsyncJobManagerImp
> l$5.run(AsyncJobManagerImpl.java:502)
> at java.util.concurrent.Executors$RunnableAdapter.call(
> Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPool
> Executor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoo
> lExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> 2017-02-14 17:48:56,157 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-2:ctx-4eca9fd0 job-1329) (logid:2eed47c7) Complete
> async job-1329, jobStatus: FAILED, resultCode: 530, result:
> org.apache.cloudstack.api.response.ExceptionResponse/null/{"
> uuidList":[],"errorcode":530}
> 2017-02-14 17:48:56,158 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-2:ctx-4eca9fd0 job-1329) (logid:2eed47c7) Publish async
> job-1329 complete on message bus
> 2017-02-14 17:48:56,158 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-2:ctx-4eca9fd0 job-1329) (logid:2eed47c7) Wake up jobs
> related to job-1329
> 2017-02-14 17:48:56,158 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-2:ctx-4eca9fd0 job-1329) (logid:2eed47c7) Update db
> status for job-1329
> 2017-02-14 17:48:56,159 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-2:ctx-4eca9fd0 job-1329) (logid:2eed47c7) Wake up jobs
> joined with job-1329 and disjoin all subjobs created from job- 1329
> 2017-02-14 17:48:56,186 DEBUG [o.a.c.f.j.i.AsyncJobManagerImpl]
> (API-Job-Executor-2:ctx-4eca9fd0 job-1329) (logid:2eed47c7) Done
> executing org.apache.cloudstack.api.command.user.vm.DestroyVMCmd for
> job-1329
> 2017-02-14 17:48:56,186 INFO [o.a.c.f.j.i.AsyncJobMonitor]
> (API-Job-Executor-2:ctx-4eca9fd0 job-1329) (logid:2eed47c7) Remove
> job-1329 from job monitoring
> 2017-02-14 17:48:59,082 DEBUG [c.c.a.ApiServlet]
> (http-nio-6443-exec-14:ctx-dc7db42c) (logid:078a1edc) ===START===
> 193.144.34.125 -- GET command=queryAsyncJobResult&jo
> bId=2eed47c7-4e14-49fb-830e-b24eb785f678&response=json&_=1487090924322
>
>
>
> El 13/02/2017 a las 20:34, Rafael Weingärtner escribió:
>
>> Could you share the error here?
>> Could you also post some more information (stack trace of the erro, ACS
>> version, hypervisors used, storage systems used, networking
>> configuration[advanced or basic])?
>>
>> On Mon, Feb 13, 2017 at 2:31 PM, Natalia Costas Lago <nata...@cesga.es>
>> wrote:
>>
>> I did that. Now I am able to Destroy the VM... but I get an error when I
>>> try to expunge. I can recover the VM.. destroy.. .but not expunge.
>>>
>>>
>>>
>>> El 13/02/2017 a las 20:22, Rafael Weingärtner escribió:
>>>
>>> I have already had this experience of VMs getting stuck in “Stopping”
>>>> state. It is quite trivial the task, you just have to make sure you are
>>>> changing the states of the right VMs. You just have to change the state
>>>> to
>>>> “Stopped” on the database.
>>>>
>>>> On Mon, Feb 13, 2017 at 2:07 PM, Natalia Costas Lago <nata...@cesga.es>
>>>> wrote:
>>>>
>>>> Dear all,
>>>>
>>>>> I stopped two VMs and it seems they where stopped but they appear
>>>>> "Stopping" in the user interface. I have seen workarounds to this issue
>>>>> that update the status of the VM directly in the database.. but this
>>>>> seems
>>>>> risky to me.
>>>>>
>>>>> How should I proceed? Is that the only way?
>>>>>
>>>>> Thanx,
>>>>>
>>>>> Kind regards,
>>>>>
>>>>>
>>>>> --
>>>>> ====================================================
>>>>> Natalia Costas Lago
>>>>> Senior Communications Technician
>>>>> Galicia Supercomputing Centre (CESGA)
>>>>> (CESGA on Twitter <https://twitter.com/#%21/CESGA_> | CESGA on
>>>>> Facebook <
>>>>> https://www.facebook.com/profile.php?id=100000707537407>)
>>>>>
>>>>> Avenida de Vigo, s/n (Campus Vida)
>>>>> 15705 Santiago de Compostela - SPAIN
>>>>>
>>>>> E-mail: nata...@cesga.es
>>>>> Cell: +34 981 56 98 10 (ext. 237)
>>>>> Fax: +34 981 59 46 16
>>>>> Web: https://www.cesga.es/
>>>>> ====================================================
>>>>> [IMPORTANTE] La información contenida en este mensaje y
>>>>> sus posibles documentos adjuntos es privada y confidencial
>>>>> y está dirigida únicamente a su destinatario/a. Si usted no
>>>>> es el/la destinatario/a original de este mensaje, por favor
>>>>> elimínelo. La distribución o copia de este mensaje no está
>>>>> autorizada.
>>>>>
>>>>>
>>>>>
>>>> --
>>> ====================================================
>>> Natalia Costas Lago
>>> Senior Communications Technician
>>> Galicia Supercomputing Centre (CESGA)
>>> (CESGA on Twitter <https://twitter.com/#%21/CESGA_> | CESGA on
>>> Facebook <
>>> https://www.facebook.com/profile.php?id=100000707537407>)
>>>
>>> Avenida de Vigo, s/n (Campus Vida)
>>> 15705 Santiago de Compostela - SPAIN
>>>
>>> E-mail: nata...@cesga.es
>>> Cell: +34 981 56 98 10 (ext. 237)
>>> Fax: +34 981 59 46 16
>>> Web: https://www.cesga.es/
>>> ====================================================
>>> [IMPORTANTE] La información contenida en este mensaje y
>>> sus posibles documentos adjuntos es privada y confidencial
>>> y está dirigida únicamente a su destinatario/a. Si usted no
>>> es el/la destinatario/a original de este mensaje, por favor
>>> elimínelo. La distribución o copia de este mensaje no está
>>> autorizada.
>>>
>>>
>>
>>
>
> --
> ====================================================
> Natalia Costas Lago
> Senior Communications Technician
> Galicia Supercomputing Centre (CESGA)
> (CESGA on Twitter <https://twitter.com/#%21/CESGA_> | CESGA on Facebook <
> https://www.facebook.com/profile.php?id=100000707537407>)
>
> Avenida de Vigo, s/n (Campus Vida)
> 15705 Santiago de Compostela - SPAIN
>
> E-mail: nata...@cesga.es
> Cell: +34 981 56 98 10 (ext. 237)
> Fax: +34 981 59 46 16
> Web: https://www.cesga.es/
> ====================================================
> [IMPORTANTE] La información contenida en este mensaje y
> sus posibles documentos adjuntos es privada y confidencial
> y está dirigida únicamente a su destinatario/a. Si usted no
> es el/la destinatario/a original de este mensaje, por favor
> elimínelo. La distribución o copia de este mensaje no está
> autorizada.
>
--
Rafael Weingärtner
