Re: Cannot start stopped VMs
Hi Everybody, I just fixed the issue :) As per mentioned in the link below, the issue was systemd-machined daemon which tracks locally running virtual machines. I just restarted it as well as restarted libvirtd and cloudstack-agent and it started to work fine. https://www.softaculous.com/board/index.php?tid=14450 Thanks On Thu, Apr 22, 2021 at 9:57 PM Fariborz Navidan wrote: > Hi Nicolas, > > I just updated the SSL certificate for the hostname of the host with a > valid Let's Encrypt cert but the host is still in Unsecure state. I think > it is something to do with the Also VMs still cannot start. > > Thanks > > On Thu, Apr 22, 2021 at 9:39 PM Nicolas Vazquez < > nicolas.vazq...@shapeblue.com> wrote: > >> Hi Fariborz, >> >> I've been taking a look at the thread - you mentioned the hosts are in >> 'Unsecure' state. You should use the provision certificate API to secure >> the hosts (can do it from the UI as well). Can you try if that solves your >> issue? >> >> Regards, >> Nicolas Vazquez >> >> From: Fariborz Navidan >> Sent: Thursday, April 22, 2021 9:12 AM >> To: users@cloudstack.apache.org >> Subject: Re: Cannot start stopped VMs >> >> Anyone can help me? >> >> On Wed, Apr 21, 2021 at 7:46 PM Fariborz Navidan >> wrote: >> >> > Any chance to identify the reason? >> > >> > Thanks >> > >> > On Wed, Apr 21, 2021 at 1:54 AM Fariborz Navidan > > >> > wrote: >> > >> >> Hi. >> >> \Now there is no "Connection Closed" error in agent.log nut it is still >> >> unable to start VMs. I have replaced agent.log with new one at >> >> https://r9host.com/log/agent.log >> >> >> >> Kindly please check it for me >> >> Thanks >> >> >> >> On Tue, Apr 20, 2021 at 8:01 PM Fariborz Navidan < >> mdvlinqu...@gmail.com> >> >> wrote: >> >> >> >>> Hi, >> >>> >> >>> Thank you for your assistance. Libvirtd and cloudstack-agent services >> >>> are running and are active. "netstat -t | grep 8250" shows two >> ESTABLISHED >> >>> connections. Also in the infrastructure section, the host's status is >> shown >> >>> as "Unsecured". >> >>> >> >>> Thanks >> >>> >> >>> On Tue, Apr 20, 2021 at 9:36 AM Harikrishna Patnala < >> >>> harikrishna.patn...@shapeblue.com> wrote: >> >>> >> >>>> Hi, >> >>>> >> >>>> At first glance on the logs, I could see below reasons for the start >> VM >> >>>> failures >> >>>> >> >>>> management server logs: >> >>>> 2021-04-19 18:31:13,539 INFO [c.c.v.VirtualMachineManagerImpl] >> >>>> (Work-Job-Executor-46:ctx-7073f4ff job-28887/job-2 ctx-70a0c6b3) >> >>>> (logid:e613b60b) Unable to start VM on Host[-1-Routing] due to >> Connection >> >>>> is closed >> >>>> >> >>>> agent logs: >> >>>> 2021-04-19 18:31:25,017 WARN [kvm.resource.LibvirtKvmAgentHook] >> >>>> (agentRequest-Handler-1:null) (logid:92f2a447) Groovy scripting >> engine is >> >>>> not initialized. Data transformation skipped. >> >>>> 2021-04-19 18:31:25,686 WARN >> >>>> [resource.wrapper.LibvirtStartCommandWrapper] >> (agentRequest-Handler-1:null) >> >>>> (logid:92f2a447) LibvirtException >> >>>> org.libvirt.LibvirtException: Connection is closed >> >>>> at org.libvirt.ErrorHandler.processError(Unknown Source) >> >>>> at org.libvirt.Connect.processError(Unknown Source) >> >>>> at org.libvirt.Connect.processError(Unknown Source) >> >>>> at org.libvirt.Connect.domainCreateXML(Unknown Source) >> >>>> at >> >>>> >> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1611) >> >>>> at >> >>>> >> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:85) >> >>>> >> >>>> >> >>>> Can you check if libvirtd service running on your KVM host? Also, you >> >>>> can do a quick KVM agent restart and see if it works
Re: Cannot start stopped VMs
Hi Nicolas, I just updated the SSL certificate for the hostname of the host with a valid Let's Encrypt cert but the host is still in Unsecure state. I think it is something to do with the Also VMs still cannot start. Thanks On Thu, Apr 22, 2021 at 9:39 PM Nicolas Vazquez < nicolas.vazq...@shapeblue.com> wrote: > Hi Fariborz, > > I've been taking a look at the thread - you mentioned the hosts are in > 'Unsecure' state. You should use the provision certificate API to secure > the hosts (can do it from the UI as well). Can you try if that solves your > issue? > > Regards, > Nicolas Vazquez > > From: Fariborz Navidan > Sent: Thursday, April 22, 2021 9:12 AM > To: users@cloudstack.apache.org > Subject: Re: Cannot start stopped VMs > > Anyone can help me? > > On Wed, Apr 21, 2021 at 7:46 PM Fariborz Navidan > wrote: > > > Any chance to identify the reason? > > > > Thanks > > > > On Wed, Apr 21, 2021 at 1:54 AM Fariborz Navidan > > wrote: > > > >> Hi. > >> \Now there is no "Connection Closed" error in agent.log nut it is still > >> unable to start VMs. I have replaced agent.log with new one at > >> https://r9host.com/log/agent.log > >> > >> Kindly please check it for me > >> Thanks > >> > >> On Tue, Apr 20, 2021 at 8:01 PM Fariborz Navidan > > >> wrote: > >> > >>> Hi, > >>> > >>> Thank you for your assistance. Libvirtd and cloudstack-agent services > >>> are running and are active. "netstat -t | grep 8250" shows two > ESTABLISHED > >>> connections. Also in the infrastructure section, the host's status is > shown > >>> as "Unsecured". > >>> > >>> Thanks > >>> > >>> On Tue, Apr 20, 2021 at 9:36 AM Harikrishna Patnala < > >>> harikrishna.patn...@shapeblue.com> wrote: > >>> > >>>> Hi, > >>>> > >>>> At first glance on the logs, I could see below reasons for the start > VM > >>>> failures > >>>> > >>>> management server logs: > >>>> 2021-04-19 18:31:13,539 INFO [c.c.v.VirtualMachineManagerImpl] > >>>> (Work-Job-Executor-46:ctx-7073f4ff job-28887/job-2 ctx-70a0c6b3) > >>>> (logid:e613b60b) Unable to start VM on Host[-1-Routing] due to > Connection > >>>> is closed > >>>> > >>>> agent logs: > >>>> 2021-04-19 18:31:25,017 WARN [kvm.resource.LibvirtKvmAgentHook] > >>>> (agentRequest-Handler-1:null) (logid:92f2a447) Groovy scripting > engine is > >>>> not initialized. Data transformation skipped. > >>>> 2021-04-19 18:31:25,686 WARN > >>>> [resource.wrapper.LibvirtStartCommandWrapper] > (agentRequest-Handler-1:null) > >>>> (logid:92f2a447) LibvirtException > >>>> org.libvirt.LibvirtException: Connection is closed > >>>> at org.libvirt.ErrorHandler.processError(Unknown Source) > >>>> at org.libvirt.Connect.processError(Unknown Source) > >>>> at org.libvirt.Connect.processError(Unknown Source) > >>>> at org.libvirt.Connect.domainCreateXML(Unknown Source) > >>>> at > >>>> > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1611) > >>>> at > >>>> > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:85) > >>>> > >>>> > >>>> Can you check if libvirtd service running on your KVM host? Also, you > >>>> can do a quick KVM agent restart and see if it works "systemctl > restart > >>>> cloudstack-agent" > >>>> > >>>> Regards, > >>>> Harikrishna > >>>> > >>>> From: Fariborz Navidan > >>>> Sent: Tuesday, April 20, 2021 12:22 AM > >>>> To: users@cloudstack.apache.org > >>>> Subject: Cannot start stopped VMs > >>>> > >>>> Hello, > >>>> > >>>> Today, I just updated SSL certificate and it failed re-deploying > system > >>>> VMs. Also, when I try to start a VM that is stopped, it fails with > >>>> message: > >>>> Unable to create deployment due to insufficient capacity. I couldn't > >>>> find > >>>> the reason from logs. I have uploaded partial management server and > >>>> agent > >>>> logs. > >>>> > >>>> https://www.r9host.com/log/management-server.log > >>>> https://www.r9host.com/log/agent.log > >>>> > >>>> It is a mission critical issue for us. I would appreciate if you guys > >>>> check > >>>> the logs and let us know what we can do to fix this issue. > >>>> > >>>> Thanks. > >>>> > >>>> harikrishna.patn...@shapeblue.com > >>>> www.shapeblue.com<http://www.shapeblue.com> > >>>> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SG > >>>> @shapeblue > >>>> > >>>> > >>>> > >>>> > > nicolas.vazq...@shapeblue.com > www.shapeblue.com > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK > @shapeblue > > > >
Re: Cannot start stopped VMs
Hi Fariborz, I've been taking a look at the thread - you mentioned the hosts are in 'Unsecure' state. You should use the provision certificate API to secure the hosts (can do it from the UI as well). Can you try if that solves your issue? Regards, Nicolas Vazquez From: Fariborz Navidan Sent: Thursday, April 22, 2021 9:12 AM To: users@cloudstack.apache.org Subject: Re: Cannot start stopped VMs Anyone can help me? On Wed, Apr 21, 2021 at 7:46 PM Fariborz Navidan wrote: > Any chance to identify the reason? > > Thanks > > On Wed, Apr 21, 2021 at 1:54 AM Fariborz Navidan > wrote: > >> Hi. >> \Now there is no "Connection Closed" error in agent.log nut it is still >> unable to start VMs. I have replaced agent.log with new one at >> https://r9host.com/log/agent.log >> >> Kindly please check it for me >> Thanks >> >> On Tue, Apr 20, 2021 at 8:01 PM Fariborz Navidan >> wrote: >> >>> Hi, >>> >>> Thank you for your assistance. Libvirtd and cloudstack-agent services >>> are running and are active. "netstat -t | grep 8250" shows two ESTABLISHED >>> connections. Also in the infrastructure section, the host's status is shown >>> as "Unsecured". >>> >>> Thanks >>> >>> On Tue, Apr 20, 2021 at 9:36 AM Harikrishna Patnala < >>> harikrishna.patn...@shapeblue.com> wrote: >>> >>>> Hi, >>>> >>>> At first glance on the logs, I could see below reasons for the start VM >>>> failures >>>> >>>> management server logs: >>>> 2021-04-19 18:31:13,539 INFO [c.c.v.VirtualMachineManagerImpl] >>>> (Work-Job-Executor-46:ctx-7073f4ff job-28887/job-2 ctx-70a0c6b3) >>>> (logid:e613b60b) Unable to start VM on Host[-1-Routing] due to Connection >>>> is closed >>>> >>>> agent logs: >>>> 2021-04-19 18:31:25,017 WARN [kvm.resource.LibvirtKvmAgentHook] >>>> (agentRequest-Handler-1:null) (logid:92f2a447) Groovy scripting engine is >>>> not initialized. Data transformation skipped. >>>> 2021-04-19 18:31:25,686 WARN >>>> [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-1:null) >>>> (logid:92f2a447) LibvirtException >>>> org.libvirt.LibvirtException: Connection is closed >>>> at org.libvirt.ErrorHandler.processError(Unknown Source) >>>> at org.libvirt.Connect.processError(Unknown Source) >>>> at org.libvirt.Connect.processError(Unknown Source) >>>> at org.libvirt.Connect.domainCreateXML(Unknown Source) >>>> at >>>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1611) >>>> at >>>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:85) >>>> >>>> >>>> Can you check if libvirtd service running on your KVM host? Also, you >>>> can do a quick KVM agent restart and see if it works "systemctl restart >>>> cloudstack-agent" >>>> >>>> Regards, >>>> Harikrishna >>>> >>>> From: Fariborz Navidan >>>> Sent: Tuesday, April 20, 2021 12:22 AM >>>> To: users@cloudstack.apache.org >>>> Subject: Cannot start stopped VMs >>>> >>>> Hello, >>>> >>>> Today, I just updated SSL certificate and it failed re-deploying system >>>> VMs. Also, when I try to start a VM that is stopped, it fails with >>>> message: >>>> Unable to create deployment due to insufficient capacity. I couldn't >>>> find >>>> the reason from logs. I have uploaded partial management server and >>>> agent >>>> logs. >>>> >>>> https://www.r9host.com/log/management-server.log >>>> https://www.r9host.com/log/agent.log >>>> >>>> It is a mission critical issue for us. I would appreciate if you guys >>>> check >>>> the logs and let us know what we can do to fix this issue. >>>> >>>> Thanks. >>>> >>>> harikrishna.patn...@shapeblue.com >>>> www.shapeblue.com<http://www.shapeblue.com> >>>> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SG >>>> @shapeblue >>>> >>>> >>>> >>>> nicolas.vazq...@shapeblue.com www.shapeblue.com 3 London Bridge Street, 3rd floor, News Building, London SE1 9SGUK @shapeblue
Re: Cannot start stopped VMs
Anyone can help me? On Wed, Apr 21, 2021 at 7:46 PM Fariborz Navidan wrote: > Any chance to identify the reason? > > Thanks > > On Wed, Apr 21, 2021 at 1:54 AM Fariborz Navidan > wrote: > >> Hi. >> \Now there is no "Connection Closed" error in agent.log nut it is still >> unable to start VMs. I have replaced agent.log with new one at >> https://r9host.com/log/agent.log >> >> Kindly please check it for me >> Thanks >> >> On Tue, Apr 20, 2021 at 8:01 PM Fariborz Navidan >> wrote: >> >>> Hi, >>> >>> Thank you for your assistance. Libvirtd and cloudstack-agent services >>> are running and are active. "netstat -t | grep 8250" shows two ESTABLISHED >>> connections. Also in the infrastructure section, the host's status is shown >>> as "Unsecured". >>> >>> Thanks >>> >>> On Tue, Apr 20, 2021 at 9:36 AM Harikrishna Patnala < >>> harikrishna.patn...@shapeblue.com> wrote: >>> Hi, At first glance on the logs, I could see below reasons for the start VM failures management server logs: 2021-04-19 18:31:13,539 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-46:ctx-7073f4ff job-28887/job-2 ctx-70a0c6b3) (logid:e613b60b) Unable to start VM on Host[-1-Routing] due to Connection is closed agent logs: 2021-04-19 18:31:25,017 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-1:null) (logid:92f2a447) Groovy scripting engine is not initialized. Data transformation skipped. 2021-04-19 18:31:25,686 WARN [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-1:null) (logid:92f2a447) LibvirtException org.libvirt.LibvirtException: Connection is closed at org.libvirt.ErrorHandler.processError(Unknown Source) at org.libvirt.Connect.processError(Unknown Source) at org.libvirt.Connect.processError(Unknown Source) at org.libvirt.Connect.domainCreateXML(Unknown Source) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1611) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:85) Can you check if libvirtd service running on your KVM host? Also, you can do a quick KVM agent restart and see if it works "systemctl restart cloudstack-agent" Regards, Harikrishna From: Fariborz Navidan Sent: Tuesday, April 20, 2021 12:22 AM To: users@cloudstack.apache.org Subject: Cannot start stopped VMs Hello, Today, I just updated SSL certificate and it failed re-deploying system VMs. Also, when I try to start a VM that is stopped, it fails with message: Unable to create deployment due to insufficient capacity. I couldn't find the reason from logs. I have uploaded partial management server and agent logs. https://www.r9host.com/log/management-server.log https://www.r9host.com/log/agent.log It is a mission critical issue for us. I would appreciate if you guys check the logs and let us know what we can do to fix this issue. Thanks. harikrishna.patn...@shapeblue.com www.shapeblue.com 3 London Bridge Street, 3rd floor, News Building, London SE1 9SG @shapeblue
Re: Cannot start stopped VMs
This is the exception I found in the agent.log: 2021-04-22 12:21:29,045 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-1:null) (logid:4d875f04) Groovy scripting engine is not initialized. Data transformation skipped. 2021-04-22 12:21:47,260 WARN [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-4:null) (logid:1a2bdac6) LibvirtException org.libvirt.LibvirtException: Failed to activate service 'org.freedesktop.machine1': timed out at org.libvirt.ErrorHandler.processError(Unknown Source) at org.libvirt.Connect.processError(Unknown Source) at org.libvirt.Connect.processError(Unknown Source) at org.libvirt.Connect.domainCreateXML(Unknown Source) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1611) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:85) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:45) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtRequestWrapper.execute(LibvirtRequestWrapper.java:78) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.executeRequest(LibvirtComputingResource.java:1643) at com.cloud.agent.Agent.processRequest(Agent.java:661) at com.cloud.agent.Agent$AgentRequestHandler.doTask(Agent.java:1079) at com.cloud.utils.nio.Task.call(Task.java:83) at com.cloud.utils.nio.Task.call(Task.java:29) at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:834) Why is this exception raised? Thanks On Thu, Apr 22, 2021 at 4:42 PM Fariborz Navidan wrote: > Anyone can help me? > > On Wed, Apr 21, 2021 at 7:46 PM Fariborz Navidan > wrote: > >> Any chance to identify the reason? >> >> Thanks >> >> On Wed, Apr 21, 2021 at 1:54 AM Fariborz Navidan >> wrote: >> >>> Hi. >>> \Now there is no "Connection Closed" error in agent.log nut it is still >>> unable to start VMs. I have replaced agent.log with new one at >>> https://r9host.com/log/agent.log >>> >>> Kindly please check it for me >>> Thanks >>> >>> On Tue, Apr 20, 2021 at 8:01 PM Fariborz Navidan >>> wrote: >>> Hi, Thank you for your assistance. Libvirtd and cloudstack-agent services are running and are active. "netstat -t | grep 8250" shows two ESTABLISHED connections. Also in the infrastructure section, the host's status is shown as "Unsecured". Thanks On Tue, Apr 20, 2021 at 9:36 AM Harikrishna Patnala < harikrishna.patn...@shapeblue.com> wrote: > Hi, > > At first glance on the logs, I could see below reasons for the start > VM failures > > management server logs: > 2021-04-19 18:31:13,539 INFO [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-46:ctx-7073f4ff job-28887/job-2 ctx-70a0c6b3) > (logid:e613b60b) Unable to start VM on Host[-1-Routing] due to Connection > is closed > > agent logs: > 2021-04-19 18:31:25,017 WARN [kvm.resource.LibvirtKvmAgentHook] > (agentRequest-Handler-1:null) (logid:92f2a447) Groovy scripting engine is > not initialized. Data transformation skipped. > 2021-04-19 18:31:25,686 WARN > [resource.wrapper.LibvirtStartCommandWrapper] > (agentRequest-Handler-1:null) > (logid:92f2a447) LibvirtException > org.libvirt.LibvirtException: Connection is closed > at org.libvirt.ErrorHandler.processError(Unknown Source) > at org.libvirt.Connect.processError(Unknown Source) > at org.libvirt.Connect.processError(Unknown Source) > at org.libvirt.Connect.domainCreateXML(Unknown Source) > at > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1611) > at > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:85) > > > Can you check if libvirtd service running on your KVM host? Also, you > can do a quick KVM agent restart and see if it works "systemctl restart > cloudstack-agent" > > Regards, > Harikrishna > > From: Fariborz Navidan > Sent: Tuesday, April 20, 2021 12:22 AM > To: users@cloudstack.apache.org > Subject: Cannot start stopped VMs > > Hello, > > Today, I just updated SSL certificate and it failed re-deploying system > VMs. Also, when I try to start a VM that is stopped, it fails with > message: > Unable to create deployment due to insufficient capacity. I couldn't > find > the reason from logs. I have uploaded partial management server and > agent > logs. > >
Re: Cannot start stopped VMs
Any chance to identify the reason? Thanks On Wed, Apr 21, 2021 at 1:54 AM Fariborz Navidan wrote: > Hi. > \Now there is no "Connection Closed" error in agent.log nut it is still > unable to start VMs. I have replaced agent.log with new one at > https://r9host.com/log/agent.log > > Kindly please check it for me > Thanks > > On Tue, Apr 20, 2021 at 8:01 PM Fariborz Navidan > wrote: > >> Hi, >> >> Thank you for your assistance. Libvirtd and cloudstack-agent services >> are running and are active. "netstat -t | grep 8250" shows two ESTABLISHED >> connections. Also in the infrastructure section, the host's status is shown >> as "Unsecured". >> >> Thanks >> >> On Tue, Apr 20, 2021 at 9:36 AM Harikrishna Patnala < >> harikrishna.patn...@shapeblue.com> wrote: >> >>> Hi, >>> >>> At first glance on the logs, I could see below reasons for the start VM >>> failures >>> >>> management server logs: >>> 2021-04-19 18:31:13,539 INFO [c.c.v.VirtualMachineManagerImpl] >>> (Work-Job-Executor-46:ctx-7073f4ff job-28887/job-2 ctx-70a0c6b3) >>> (logid:e613b60b) Unable to start VM on Host[-1-Routing] due to Connection >>> is closed >>> >>> agent logs: >>> 2021-04-19 18:31:25,017 WARN [kvm.resource.LibvirtKvmAgentHook] >>> (agentRequest-Handler-1:null) (logid:92f2a447) Groovy scripting engine is >>> not initialized. Data transformation skipped. >>> 2021-04-19 18:31:25,686 WARN >>> [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-1:null) >>> (logid:92f2a447) LibvirtException >>> org.libvirt.LibvirtException: Connection is closed >>> at org.libvirt.ErrorHandler.processError(Unknown Source) >>> at org.libvirt.Connect.processError(Unknown Source) >>> at org.libvirt.Connect.processError(Unknown Source) >>> at org.libvirt.Connect.domainCreateXML(Unknown Source) >>> at >>> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1611) >>> at >>> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:85) >>> >>> >>> Can you check if libvirtd service running on your KVM host? Also, you >>> can do a quick KVM agent restart and see if it works "systemctl restart >>> cloudstack-agent" >>> >>> Regards, >>> Harikrishna >>> >>> From: Fariborz Navidan >>> Sent: Tuesday, April 20, 2021 12:22 AM >>> To: users@cloudstack.apache.org >>> Subject: Cannot start stopped VMs >>> >>> Hello, >>> >>> Today, I just updated SSL certificate and it failed re-deploying system >>> VMs. Also, when I try to start a VM that is stopped, it fails with >>> message: >>> Unable to create deployment due to insufficient capacity. I couldn't find >>> the reason from logs. I have uploaded partial management server and >>> agent >>> logs. >>> >>> https://www.r9host.com/log/management-server.log >>> https://www.r9host.com/log/agent.log >>> >>> It is a mission critical issue for us. I would appreciate if you guys >>> check >>> the logs and let us know what we can do to fix this issue. >>> >>> Thanks. >>> >>> harikrishna.patn...@shapeblue.com >>> www.shapeblue.com >>> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SG >>> @shapeblue >>> >>> >>> >>>
Re: Cannot start stopped VMs
Hi. \Now there is no "Connection Closed" error in agent.log nut it is still unable to start VMs. I have replaced agent.log with new one at https://r9host.com/log/agent.log Kindly please check it for me Thanks On Tue, Apr 20, 2021 at 8:01 PM Fariborz Navidan wrote: > Hi, > > Thank you for your assistance. Libvirtd and cloudstack-agent services are > running and are active. "netstat -t | grep 8250" shows two ESTABLISHED > connections. Also in the infrastructure section, the host's status is shown > as "Unsecured". > > Thanks > > On Tue, Apr 20, 2021 at 9:36 AM Harikrishna Patnala < > harikrishna.patn...@shapeblue.com> wrote: > >> Hi, >> >> At first glance on the logs, I could see below reasons for the start VM >> failures >> >> management server logs: >> 2021-04-19 18:31:13,539 INFO [c.c.v.VirtualMachineManagerImpl] >> (Work-Job-Executor-46:ctx-7073f4ff job-28887/job-2 ctx-70a0c6b3) >> (logid:e613b60b) Unable to start VM on Host[-1-Routing] due to Connection >> is closed >> >> agent logs: >> 2021-04-19 18:31:25,017 WARN [kvm.resource.LibvirtKvmAgentHook] >> (agentRequest-Handler-1:null) (logid:92f2a447) Groovy scripting engine is >> not initialized. Data transformation skipped. >> 2021-04-19 18:31:25,686 WARN >> [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-1:null) >> (logid:92f2a447) LibvirtException >> org.libvirt.LibvirtException: Connection is closed >> at org.libvirt.ErrorHandler.processError(Unknown Source) >> at org.libvirt.Connect.processError(Unknown Source) >> at org.libvirt.Connect.processError(Unknown Source) >> at org.libvirt.Connect.domainCreateXML(Unknown Source) >> at >> com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1611) >> at >> com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:85) >> >> >> Can you check if libvirtd service running on your KVM host? Also, you can >> do a quick KVM agent restart and see if it works "systemctl restart >> cloudstack-agent" >> >> Regards, >> Harikrishna >> >> From: Fariborz Navidan >> Sent: Tuesday, April 20, 2021 12:22 AM >> To: users@cloudstack.apache.org >> Subject: Cannot start stopped VMs >> >> Hello, >> >> Today, I just updated SSL certificate and it failed re-deploying system >> VMs. Also, when I try to start a VM that is stopped, it fails with >> message: >> Unable to create deployment due to insufficient capacity. I couldn't find >> the reason from logs. I have uploaded partial management server and agent >> logs. >> >> https://www.r9host.com/log/management-server.log >> https://www.r9host.com/log/agent.log >> >> It is a mission critical issue for us. I would appreciate if you guys >> check >> the logs and let us know what we can do to fix this issue. >> >> Thanks. >> >> harikrishna.patn...@shapeblue.com >> www.shapeblue.com >> 3 London Bridge Street, 3rd floor, News Building, London SE1 9SG >> @shapeblue >> >> >> >>
Re: Cannot start stopped VMs
Hi, Thank you for your assistance. Libvirtd and cloudstack-agent services are running and are active. "netstat -t | grep 8250" shows two ESTABLISHED connections. Also in the infrastructure section, the host's status is shown as "Unsecured". Thanks On Tue, Apr 20, 2021 at 9:36 AM Harikrishna Patnala < harikrishna.patn...@shapeblue.com> wrote: > Hi, > > At first glance on the logs, I could see below reasons for the start VM > failures > > management server logs: > 2021-04-19 18:31:13,539 INFO [c.c.v.VirtualMachineManagerImpl] > (Work-Job-Executor-46:ctx-7073f4ff job-28887/job-2 ctx-70a0c6b3) > (logid:e613b60b) Unable to start VM on Host[-1-Routing] due to Connection > is closed > > agent logs: > 2021-04-19 18:31:25,017 WARN [kvm.resource.LibvirtKvmAgentHook] > (agentRequest-Handler-1:null) (logid:92f2a447) Groovy scripting engine is > not initialized. Data transformation skipped. > 2021-04-19 18:31:25,686 WARN > [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-1:null) > (logid:92f2a447) LibvirtException > org.libvirt.LibvirtException: Connection is closed > at org.libvirt.ErrorHandler.processError(Unknown Source) > at org.libvirt.Connect.processError(Unknown Source) > at org.libvirt.Connect.processError(Unknown Source) > at org.libvirt.Connect.domainCreateXML(Unknown Source) > at > com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1611) > at > com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:85) > > > Can you check if libvirtd service running on your KVM host? Also, you can > do a quick KVM agent restart and see if it works "systemctl restart > cloudstack-agent" > > Regards, > Harikrishna > > From: Fariborz Navidan > Sent: Tuesday, April 20, 2021 12:22 AM > To: users@cloudstack.apache.org > Subject: Cannot start stopped VMs > > Hello, > > Today, I just updated SSL certificate and it failed re-deploying system > VMs. Also, when I try to start a VM that is stopped, it fails with message: > Unable to create deployment due to insufficient capacity. I couldn't find > the reason from logs. I have uploaded partial management server and agent > logs. > > https://www.r9host.com/log/management-server.log > https://www.r9host.com/log/agent.log > > It is a mission critical issue for us. I would appreciate if you guys check > the logs and let us know what we can do to fix this issue. > > Thanks. > > harikrishna.patn...@shapeblue.com > www.shapeblue.com > 3 London Bridge Street, 3rd floor, News Building, London SE1 9SG > @shapeblue > > > >
Re: Cannot start stopped VMs
Hi, At first glance on the logs, I could see below reasons for the start VM failures management server logs: 2021-04-19 18:31:13,539 INFO [c.c.v.VirtualMachineManagerImpl] (Work-Job-Executor-46:ctx-7073f4ff job-28887/job-2 ctx-70a0c6b3) (logid:e613b60b) Unable to start VM on Host[-1-Routing] due to Connection is closed agent logs: 2021-04-19 18:31:25,017 WARN [kvm.resource.LibvirtKvmAgentHook] (agentRequest-Handler-1:null) (logid:92f2a447) Groovy scripting engine is not initialized. Data transformation skipped. 2021-04-19 18:31:25,686 WARN [resource.wrapper.LibvirtStartCommandWrapper] (agentRequest-Handler-1:null) (logid:92f2a447) LibvirtException org.libvirt.LibvirtException: Connection is closed at org.libvirt.ErrorHandler.processError(Unknown Source) at org.libvirt.Connect.processError(Unknown Source) at org.libvirt.Connect.processError(Unknown Source) at org.libvirt.Connect.domainCreateXML(Unknown Source) at com.cloud.hypervisor.kvm.resource.LibvirtComputingResource.startVM(LibvirtComputingResource.java:1611) at com.cloud.hypervisor.kvm.resource.wrapper.LibvirtStartCommandWrapper.execute(LibvirtStartCommandWrapper.java:85) Can you check if libvirtd service running on your KVM host? Also, you can do a quick KVM agent restart and see if it works "systemctl restart cloudstack-agent" Regards, Harikrishna From: Fariborz Navidan Sent: Tuesday, April 20, 2021 12:22 AM To: users@cloudstack.apache.org Subject: Cannot start stopped VMs Hello, Today, I just updated SSL certificate and it failed re-deploying system VMs. Also, when I try to start a VM that is stopped, it fails with message: Unable to create deployment due to insufficient capacity. I couldn't find the reason from logs. I have uploaded partial management server and agent logs. https://www.r9host.com/log/management-server.log https://www.r9host.com/log/agent.log It is a mission critical issue for us. I would appreciate if you guys check the logs and let us know what we can do to fix this issue. Thanks. harikrishna.patn...@shapeblue.com www.shapeblue.com 3 London Bridge Street, 3rd floor, News Building, London SE1 9SG @shapeblue
